Our top agent latency optimisations | Lexifina<br>Skip to main contentFeaturesSecurityPricingBlogChangelogSign in
Blog/Research/Agent latency
Blog/Research/Agent latency
A model can emit a complete tool call while it is still streaming later calls or text. Waiting for the entire provider response turns two independently useful clocks into one long serial clock.
How it works<br>Settle each locally executable tool call into a supervised task as soon as its complete event arrives. More tool calls and the remaining provider stream can continue concurrently. A one-permit publication gate protects ordered history while expensive execution remains parallel, and the runner joins outstanding work only after stream settlement.<br>The key separation is between execution readiness and transcript readiness. The former can be eager; the latter must retain the exact order expected by replay and the next model request.
The Improvement<br>If a shell command takes eight seconds and the model needs another five seconds to finish emitting calls, serial execution costs roughly thirteen seconds. Starting the shell immediately exposes only the slower of the two clocks, plus small coordination overhead.<br>The improvement compounds when the response contains several independent calls, especially searches, reads and remote API requests with variable latency.
How to action it<br>Dispatch at the completed tool-call boundary, not at the end of the whole provider message:<br>Parse incrementally: consume stream events as they arrive and maintain one argument accumulator for each tool-call identity.<br>Use an explicit readiness event: a tool becomes executable only when its block closes, not when the first argument fragment appears.<br>Validate before execution: parse the accumulated payload, apply the authoritative schema and reject partial or invalid input rather than guessing.
Read more<br>E2E latency ↓ 10%Microsoft Research’s SUTRADHARA dispatches tool calls incrementally during model decoding instead of waiting for the complete output. Its combined design reduced median first-token-rendered latency by 15% and end-to-end latency by 10%; those figures are not an isolated streaming-dispatch result. Microsoft Research: SUTRADHARA → Anthropic’s implementation guide specifies the safe parser contract: accumulate input deltas, parse when the tool block closes and reject invalid input. Anthropic tool streaming →
Guardrail<br>Do not publish nondeterministic completion order into model history. Define cancellation behavior for tools still running when the stream fails, bound progress-event backpressure, and start only calls whose arguments are complete and validated.
Metrics tool-ready-to-startstream/tool overlap msuncovered tool timeorphaned-tool count
Agent loops repeatedly call the same model with a mostly unchanged conversation. Treating every continuation as a brand-new HTTP request forces the surrounding API stack to rediscover and reprocess state that was valid milliseconds ago.
continuation cost = new input processing + incremental inference, not full connection + full history processing
How it works<br>Keep a session-scoped streaming connection available across model steps. When non-input request properties are unchanged and the next input strictly extends the prior exchange, the client sends only the new tail with the provider-issued previous-response identity. Server-side state can then reuse prior items, tool definitions, routing decisions and rendered-token artifacts.<br>Retain idle connections for a bounded window and avoid head-of-line blocking: if the cached connection is busy, another request can use a temporary connection.
The Improvement<br>The gain repeats on every tool continuation. It removes connection establishment and prevents conversation-length-dependent request processing from growing throughout the run. This matters increasingly as model inference becomes faster and API/service overhead becomes a larger share of end-to-end time.<br>The optimisation is structural: it preserves the familiar request shape while making the underlying stack incremental.
How to action it<br>Treat continuation state as a leased, fingerprinted session resource:<br>Keep one hot transport per active session: reuse a streaming connection across the model–tool loop instead of reconnecting for every continuation.<br>Store the continuation identity: retain the provider’s previous-response identity alongside a fingerprint of the model, instructions, tools, sampling settings and routing properties.<br>Send only a strict tail: use incremental continuation only when the new input extends the prior exchange and the non-input fingerprint is unchanged.
Read more<br>Up to 40% fasterOpenAI reports up to 40% faster agentic workflows after moving Responses continuations onto WebSockets with connection-scoped state. It attributes the result to incremental validation, token rendering, routing reuse and overlapping non-blocking post-inference work. OpenAI WebSocket engineering →
Guardrail<br>Delta...