Cache hit rate dropping by 20% doubles your agent's bills - Dirac Posts

Features

Docs

Posts

Token Market Share by<br>Lab

Evals

Discord

GitHub

Install Dirac

On this page

Back to Posts

Prefix Caching

TL;DR

We posit that in the typical long agentic loops, Cache hit rate and Cached input pricing are the most<br>dominant yet ignored cost factors.

The agent loop simulation below lets you estimate costs by tweaking different parameters. This affects<br>both API ($$) and subscriptions (usage limit).

The headline claim is using the standard 1x/10x ratio between uncached read vs cached read cost. This<br>gets even more extreme as cached reads get cheaper (e.g. Deepseek or Xiaomi pricing)

Most providers on OpenRouter cache well below 50%.

Why is prefix caching necessary

Note: While this is an extensive and interesting topic, we will only touch<br>the relevant aspect of prefix caching. Worth looking it up to understand technical details.

At any given point, the LLM you're talking to does one thing: given all of the tokens that have been seen so far and<br>in the exact sequence they appeared, what is the next token I should generate? Once the new token is<br>generated, it gets appended to the context and the model repeatedly asks the same question until generation<br>finishes. This process is known as the Generation Loop and every single token generation is one<br>Forward Pass . LLMs being sensitive to the order of input is a well recognized phenomenon (reference).

In a typical agentic loop, you have a system prompt, tool call definitions, a user prompt, tool calls themselves,<br>tool call outputs and model reasoning tokens. Something like:

SystemPrompt<br>Tool Defs<br>User Prompt<br>Model Reasoning<br>Model Response<br>Tool Call 1 (model decided to emit this tool call based on the exact sequence<br>in the context window up until this point)<br>...<br>Tool Call 10 (same)<br>...<br>More reasoning<br>Tool Call 75 (same)<br>...<br>...<br>Task Completion

The process above takes many API calls and typically grows the context window into tens of thousands or even low<br>hundreds of thousands of tokens. At every turn, the agent sends the complete history of the conversation to<br>the LLM.

And on the LLM side, it is both expensive and unnecessary to do a full forward pass at every turn, so the infra<br>provider typically caches what is computed so far and only does the generation loop on the remaining content. This<br>cache is known as a Prefix Cache . In the most naive form, prefix caching is simply doing a look up<br>of the incoming full conversation against the server-side KV cache, starting from the very first token (hence<br>"prefix" caching), to see how far it can match an available cache.

How much Caching affects your costs/usage limits?

The chart below simulates the actual costs. Each turn appends 200–3000 random input tokens, the model responds with<br>~200 output + ~100 thinking tokens, and a cache hit is rolled probabilistically. Results are averaged over multiple<br>runs. Adjust the parameters and watch the chart respond. 100% caching is obviously impossible, we use 96% as gold<br>standard.

Parameters

Cache hit rate: 96%

Turns

Input ($/M)

Output ($/M)

Cache read ($/M)

Tuning

Min input/turn

Max input/turn

Output/turn

Thinking/turn

Runs (avg)

How do I know my live caching stats?

You can look into the responses from your provider to check what is the cached reads vs full reads on each response.<br>Alternatively, Dirac continuously displays the weighted averaged cache hit rates throughout the agentic loop.

Dirac now displays live cache hit rates on all sessions.

Why do providers not fix their caching?

Honestly, I don't know. It is expensive for both the user, the provider, the environment and a straight downgrade on<br>literally everything to<br>have poor caching. As I noted in a previous post, a majority of the providers on OpenRouter<br>have well below 50% caching.

The most generous explanation is probably memory pressure. Caches take up too much memory. Less generous is bad<br>configuration/infra. Even less generous and more Machiavellian is the reported numbers being different from actual<br>numbers and providers pocketing the difference (since a user has no way to tell and must rely on the token<br>consumption stats in the response).

A few tips if you are writing an agent

Keep the prefix stable: in your agent's "rebuild conversation" step,<br>make sure nothing is sneaking in that would break the cache.

No random or variable elements such as timestamps early in the system<br>prompt - this benefits your next loop by keeping system prompt and tool defs cached across<br>conversations.

Premature compression of the conversation, while tempting, will<br>certainly break the cache and you might end up paying effectively more.

All Posts