Cache-Control for LLMs

Sign in

Subscribe

Claude Sonnet 5 charges \$2.00 per million fresh input tokens. Writing them into a five-minute cache costs \$2.50, and every read after that costs \$0.20 (see here). So two identical calls cost \$2.70 with caching instead of \$4.00. One reuse pays for the write. Every hit after that is 90% off.<br>For a resource priced like that, the controls are thin. Anthropic offers just two: where the reusable part of a prompt ends, marked at up to four places, one of which automatic caching may spend on its own, and how long the entry lasts, either five minutes or an hour. OpenAI offers the same four places and thirty minutes. That is the whole vocabulary.<br>So developers improvise. The standard trick is a keepalive: replay the prefix on a timer, because every read resets the expiry clock. It works. But it also scales with how many prefixes you keep warm.<br>The trouble isn't a lack of cache controls. They exist, except a layer down. LMCache pins, compresses, clears, and moves KV state across GPU, CPU, and NVMe. SGLang stores and matches reusable prefixes in a radix tree. vLLM lets a caller salt the cache key. What is missing is a portable contract above all of it.<br>HTTP had the same gap in 1996. Caching worked, but what a server could say about it was thin: an expiry date, a conditional request, and Pragma: no-cache. RFC 2068 fixed it by introducing additional vocabulary: Cache-Control for policy, ETag for identity, Vary for what makes two requests the same. That is what LLM caching does not have.<br>Who knows what<br>Every serving system is already solving an optimization problem: maximize hit rate within a fixed memory budget, by choosing what to keep, for how long, and who may reuse it. What the provider lacks information about what each part of the prompt is worth.<br>Take one request. Part of it is a system prompt that has not changed in a month and will be sent again in ten seconds. Part of it is a user turn nobody will ever send again. One is worth keeping and the other is worth dropping the moment it is read, but both arrive as the same thing: tokens, in order, with no note attached.<br>So the provider applies its default to both, and the user turn holds memory until eviction gets to it, taking room something reusable could have had.<br>Such inputs come in two kinds. Some are constraints on the solution: this content may not be retained, this reuse has to be exact, this must never cross a tenant boundary. The rest are parameters the solver can weigh or discard: this branch resumes in about thirty seconds, a miss here is expensive. Everything else, meaning placement, eviction, compression, prefetch, scheduling, and the model- and adapter-specific parts of cache identity, stays with the solver.<br>Supply them and the solver gets better for both sides: the same memory serves more hits, so the application pays less and the provider gets more out of the hardware it already bought.<br>Constraints<br>Constraints are non-negotiable, so no amount of memory pressure makes violating one an option. There are four important types of constraints:<br>Retention. Whether this state may be kept at all, and if so, for how long at most. A user's medical intake form can be cached for the length of the request and no longer.<br>Reuse. Two settings. exact matches only on an identical prefix: two requests share state until they diverge, and everything past that is recomputed. approximate matches each span wherever it was cached, then repairs it. How many tokens get repaired is a choice: repair more and the result lands closer to exact but takes longer. CacheBlend, for instance, repairs 15% of tokens for roughly a 0.02 quality drop, for its benchmark. But another user may choose a different trade-off.<br>Scope. How widely this state may be shared: request, user, tenant, a named group, or public. Since system prompts and tool schemas repeat across customers, sharing cuts cost and latency for everyone. SafeKV estimated (on three datasets) that partitioning per user raised time to first token by 2.3 to 8.9% on Llama-2-13B and 8.3 to 38.9% on Llama-2-70B.

There are safety implications to sharing. Send a candidate prefix, time the reply, and a fast one means somebody sent it before you. Because the cache is a tree, partial matches confirm partial guesses, and they compose: a 2025 audit found cross-user sharing at seven providers, and later work reconstructed whole prompts token by token. Providers now isolate by workspace or organization but the user ought to be able to specify finer or coarser boundaries.<br>Identity. What makes two requests the same, so one can reuse the other's state. The server builds most of the key, chaining each block's hash to its parent alongside model configuration, adapters, and multimodal inputs. But that key tracks exactly one thing: the tokens sent. So identical tokens always match, even when they have stopped meaning what they meant, as when a document is re-approved under a new policy while its text stays...