Coolhand Labs | Prompt caching cheatsheet: building cost-effective AI agents

Skip to main content

← All Posts

August 17, 2026


# Prompt caching cheatsheet: building cost-effective AI agents

*Cache write, cache read, refresh, breakpoint, TTL, storage fee. Six words that mean different things at three providers — and one of them charges you while nothing is happening.*

Prompt caching is the closest thing to free money in the inference stack, and it is described almost identically by every provider that sells it: send the same context twice, pay less the second time. That description is true. It is also thin enough that teams routinely turn caching on, watch their bill go *up*, and can't say why.

The reason is vocabulary. Every provider uses the same six words for six meaningfully different mechanisms. This piece is the glossary — what each term means, where the providers diverge, and which ones have a price attached. It's the foundation for the four pieces that follow: [the pricing rules](/updates/llm-prompt-caching-pricing-guide-and-cost-breakdown), and then a deep dive each on [Anthropic](/updates/llm-prompt-caching-anthropic-how-to-and-pro-tips), [OpenAI](/updates/llm-prompt-caching-openai-how-to-and-pro-tips), and [Gemini](/updates/llm-prompt-caching-gemini-how-to-and-pro-tips).

## What caching actually does

> **In one line:** the model stores the computed state of a prompt prefix so it doesn't have to recompute it next time.

That "recompute" step isn't free, and it isn't small. Input tokens are billed on every single request, and for most production AI workloads — agents especially — the prompt dwarfs the response: system instructions, tool definitions, and accumulated conversation or tool-call history can run into the tens of thousands of tokens, resent in full on every turn. An agent looping through a multi-step task might resend the same 50,000-token prefix a dozen times in a single run. Without caching, you pay full input price for that prefix every single time. With it, everything after the first request drops to roughly a tenth of the cost. At agent scale, that's rarely a rounding error — it's often the single largest lever available for cutting inference spend.

When you send a request, the model processes your entire prompt from the first token. Caching lets the provider keep the intermediate state — the key-value tensors — for some leading portion of that prompt, so a later request that starts with the identical bytes can resume from that point instead of starting over.

## What can be cached? Two key things to remember

Two consequences fall straight out of that, and both matter more than most of the guidance you'll read.

**Caching only ever discounts input.** No provider's caching touches output token cost. If your workload is output-heavy — long generations from short prompts — caching cannot help you much no matter how well you implement it. Any savings projection that applies the cache discount to your whole bill is wrong by whatever fraction of your spend is output. Anthropic's docs are explicit that [prompt caching has no effect on output token generation](https://platform.claude.com/docs/en/build-with-claude/prompt-caching); OpenAI says the same, adding that [the model still computes a fresh response](https://developers.openai.com/api/docs/guides/prompt-caching) from the cached prefix, so identical requests still aren't guaranteed identical outputs.

**It's a prefix cache, not a substring cache.** The match runs from the very start of the prompt forward. Change one token near the beginning and everything after it is a miss, regardless of how much of the rest is identical. This is why "put static content first, variable content last" is the one piece of advice every provider gives.

That second point is why your **system prompt** is usually the single best thing to cache. It's typically the largest block of genuinely static content in the request — instructions, persona, output format, tool definitions — and it's identical across every call, which is exactly what a prefix cache rewards. Anthropic builds its cache ordering around this directly: prefixes are assembled as `tools`, then `system`, then `messages`, and its own guidance is to ["cache stable, reusable content like system instructions, background information, large contexts, or frequent tool definitions"](https://platform.claude.com/docs/en/build-with-claude/prompt-caching) — in practice, that means the `cache_control` breakpoint usually lands on the last block of the system prompt. OpenAI's advice points the same direction: ["place static content like instructions and examples at the beginning of your prompt, and put variable content, such as user-specific information, at the...