Infrastructure requirements for running agents in production · TokenGO Blog<br>NEW·GLM 5.2 just onboarded — $1.26/M in · $3.96/M out, cheapest endpoint online.Try it →

─── BLOG<br>Infrastructure requirements for running agents in production<br>Aug 14, 2026 · TokenGO<br>What agents lack in production isn't the model. It's four foundational primitives: timeouts, retries and circuit breaking, persisted state, and tracing.<br>What agents lack in production isn't the model; it's four foundational primitives.

Many teams moving agents from demo to production find that the first hurdle isn't model performance, but infrastructure. Running successfully once in a demo doesn't mean it can run day in and day out. We've helped many teams navigate this pitfall; what's usually missing are a few foundational setups.

Timeouts: don't let a slow API drag down the entire chain

When an agent calls a tool and the other side doesn't respond, it can't wait indefinitely. Every call needs a timeout. When the time is up, it must either switch paths or throw an error. We once saw an agent where a half-dead third-party API stalled the main chain for 40 minutes. The user thought the system had crashed, but it was just stuck on a single call without a timeout.

Retries and circuit breaking: back off on failure, trip the circuit on continuous failure

Occasional tool failures are normal, and a few automatic retries are fine. But if the same API fails continuously, you have to break the circuit. Don't stubbornly hammer a dead service. We've seen an agent exhaust its quota with dozens of retries just because a downstream service blipped for a second. The right approach is retrying with backoff, and tripping the circuit after a certain number of consecutive failures, then probing again later.

State: resuming from breakpoints after a restart

If a machine restarts halfway through a long-running task, can it resume from the breakpoint instead of starting over? This requires the state of each step to be persisted, not just held in memory. If the state isn't persisted to disk, a restart means a complete rerun, and you simply cannot risk putting long-running tasks into production.

Tracing: pulling up the entire chain when one step is slow

When an agent executes over a dozen steps, you need to be able to pull up the entire execution trace to see exactly which step was slow or threw an error. Without tracing, debugging production issues is just guesswork. When onboarding customers to production environments, the very first thing we usually do is implement tracing. Once, a customer complained about a particularly slow response. We pulled up the trace and instantly saw that a retrieval tool was taking eight seconds. It wasn't a model issue, and troubleshooting took all of ten seconds. We are rolling out our free agent observability feature in waves now. TokenGo Agent Tracing.

Architecture: stateful orchestration layer + stateless tool layer

None of these are difficult on their own. The hard part is that they must exist as a cohesive whole, ideally provided by the platform so that every business team doesn't have to reinvent the wheel. We favor a two-layer architecture: a stateful orchestration layer that remembers where the task is and can resume after restarts, and a stateless tool layer that can be spun up or swapped out at any time. If one dies, just swap it. The two layers communicate using the primitives mentioned above. If a tool goes down, the orchestration layer switches to another one and keeps running; the user on the other end might only wait an extra two seconds. We built a reconciliation agent for a financial client where the tool layer went down twice; both times, the orchestration layer switched to backups in seconds, and the business side was completely oblivious.

Taking agents to production is about whether it can pick itself back up after crashing. With a complete set of primitives, it can recover from crashes invisibly to the user.

How this plays out in a real outage

Let's look at a concrete example. In that reconciliation agent, the retrieval service in the tool layer crashed. The orchestration layer had a timeout set before the call; when it didn't get a response in eight seconds, it marked it as a failure and triggered the circuit breaker, stopping further requests to the dead service. The orchestration layer then swapped to a backup retrieval instance via an alternative path, recovering within ten seconds. The reconciliation task wasn't interrupted at all, and the business side noticed nothing. Looking at the trace afterward, the only anomaly in the entire chain was that the retrieval step jumped from its usual two seconds to an eight-second timeout while the rest proceeded normally. With the timeout + circuit breaker together, these two primitives turned an outage into a seamless switch.

When we set up production environments for customers, the first thing we do is enable these four primitives by default. We don't leave it as an...