Prompt Caching In Agents | EARENDIL
Prompt Caching In Agents
Date:Wed, 22 Jul 2026
From:Earendil Engineering rfc@earendil.com>
To:You
Subject:Prompt Caching In Agents
Large language models are often thought of like functions: send in some text,<br>receive some text. That is a useful abstraction, but it ignores one of the most<br>important parts of running a coding agent: most of the input is the same as last<br>time. In other words we mostly append to it.
A coding agent sends the model its system prompt, tool definitions, project<br>instructions, conversation history, tool calls, and tool results. On the next<br>turn it sends almost all of that again, plus a small amount of new material.<br>Once a session has grown to tens or hundreds of thousands of tokens, recomputing<br>the whole prompt for every turn is slow and expensive.
Prompt caching is what makes this somewhat economic, but it is also quite<br>fragile. A changed tool definition, a model switch or a provider routing<br>decision can turn what one would expect to be a cheap incremental request into a<br>full replay of the context.
For coding agents, cache behavior is therefore not just an implementation detail or<br>optimization. It affects latency, cost, tool design, session design, and even<br>which product features should be made available.
What a KV Cache Contains
A transformer processes a prompt in two broad phases. During prefill , it<br>reads the input tokens and computes attention state for them. During decode ,<br>it produces new tokens one at a time.
At each attention layer, every processed token produces a key and a value. These<br>are not quite like key–value lookups in a hash table: both are arrays of<br>numbers, usually floats or lower-precision quantized values. When processing a<br>new token, the model compares that token's query with the earlier keys<br>to determine how relevant each earlier token is. It then uses those relevance<br>scores to form a weighted mixture of the corresponding values . In that<br>sense, a key is what the model matches against, while a value is the information<br>it retrieves (but the lookup is fuzzy rather than "returning a single exact<br>match" like a dictionary lookup.)
Those keys and values are retained so that the next generated token can attend<br>to everything that came before without recomputing the earlier tokens. This<br>retained state is the KV cache .
Conceptually, a request looks like this:
request 1:
[system][tools][user][assistant][tool result][user]
K and V tensors per token and layer
request 2:
[system][tools][user][assistant][tool result][user][new]
new work
The real representations are more complicated, model-specific, and "quite"<br>large. The important property is that they correspond to a particular token<br>prefix. Two prompts that mean the same thing but tokenize differently do not<br>share a KV cache. If a token changes in the middle, everything after that token<br>is a different continuation.
Prompt caching extends the lifetime of this state beyond one generation. When<br>the next API request from the coding agent begins with the same tokens, the<br>inference system can reuse the stored work for the matching prefix and prefill<br>only the new suffix. So far, the theory.
Where the Cache Lives
In order for a cache to work it needs to be stored somewhere, and it needs to be<br>addressable. There are two broad ways inference systems make KV caches<br>available to a later request.
The simpler approach is session affinity . It works by keeping the KV cache<br>on or near the GPU that computed it, and routing the next request back to the same<br>worker. A session ID or prompt-cache key becomes a trivial routing hint and so<br>you can potentially even deal with this problem on the HTTP load balancer level<br>without having to look into the payload.
request(session-42) --> router --> worker 7 --> GPU 7 KV cache<br>next(session-42) --> router --> worker 7 --> GPU 7 KV cache
This avoids moving a very large cache over the network. It is fast when it<br>works, but it constrains scheduling. The selected worker can become overloaded,<br>restart, or evict the entry. A router may also decide that balancing the fleet<br>is more important than preserving one session's cache. It is however a very<br>attractive solution because it works with little extra deployed infrastructure<br>and hardware.
The other approach is to distribute the cache . KV blocks can be stored in<br>another memory tier or made available across workers, so a request is not tied<br>as tightly to one GPU.
+--------------------+<br>request --> scheduler -->| worker 3 / GPU 3 |<br>| +--------------------+<br>+----------> distributed KV blocks<br>+----------> worker 9 / GPU 9
That improves scheduling flexibility and recovery, but moving, indexing, and<br>retaining KV blocks is itself a systems problem. Implementations mix GPU memory,<br>host memory, local storage, remote storage, prefix-aware routing, and eviction<br>policies in different ways.
To put KV caches into perspective: they can be large but they are in some ways<br>smaller than one would assume. With...