Tinfoil - Architecting secure prompt caching

← Back to Posts<br>Architecting secure prompt caching<br>Jul 14, 2026•10 min read<br>Tanya Verma and Sacha Servan-Schreiber

We're excited to announce cached prompt pricing in the Tinfoil Inference API. This has been one of the most frequently requested features because it makes eligible inference requests require less compute, which is great for us because we can serve more customers and great for customers because they pay less. It works by caching inputs that the model has recently processed, which has an outsized benefit on long-context tasks such as coding agents. We've taken our time implementing prompt caching because caching is privacy's natural enemy, and a hasty implementation could weaken the privacy guarantees that Tinfoil provides. In this post, we explain these pitfalls, and how we avoid it.

The problem is that caching inherently requires keeping state around, and that state introduces timing side channels that attackers can exploit to learn information they shouldn't. Specifically, an attacker can observe if something is processed faster or slower and infer whether it was cached. While this may seem innocuous at first, timing side channels can leak secret keys and other private information, and have underpinned many high-profile attacks, including Spectre and Meltdown. Unfortunately, caching is also extremely necessary as it enables efficiency across nearly all layers of modern computing infrastructure.

Prefix caching makes inference go brrrr

Inference consists of two phases:

Prefill : reading your prompt. The model ingests all tokens from the entire prompt at once, in parallel. This work scales with the length of the prompt, and it has to finish before the model produces the first token of the response.

Decode : generating the full response. The model then generates the answer one token at a time, and each token depends on the previous one, so output generation cannot be parallelized.

The idea is that the prefill computation can be performed once, then saved, and then reused if part of a future prompt is identical. Inference libraries/frameworks like vLLM and SGLang use different approaches to achieve this central idea.

Or in other words, prefix caching stores the preprocessed conversation history up to the latest user message (i.e., the "prefix") for a given session. Functionally, this means that for each additional user message, the full context up to that message is already preprocessed and cached, which speeds up inference and reduces API usage costs for end users. As a result, all model providers take great steps to ensure that they can optimize the prompt structure to maximize caching. This is especially important when the input context is large, for example with many agentic workloads such as coding. See for instance these write-ups from the Codex team and the Claude Code team on how they architected the agent loop. Figure 1 illustrates how prefix caching reuses prefill computation across requests.

Figure 1: Prefix caching: the prefill computation for a shared prompt prefix is performed once and reused for subsequent requests (conversation turns), reducing latency and cost.

Cache timing attacks and how salting stops them

A cache shared across users introduces a timing side channel that can be exploited by any user of the inference service.

To see this, consider a user, Alice, who makes a request for hello, which gets processed and stored in the prefix cache. Another user, Bob, can make a request for hello and observe that the inference server responded a few milliseconds faster than usual, leading Bob to conclude that Alice's prompt also had the prefix hello.

The solution to prevent this type of cross-user attack is to salt the cache. Recall that a request hits the cache only if its tokens match a cached prefix exactly, starting from the very first token. A cache salt is a per-user value that acts like an invisible first word on every prompt the user sends: Alice's request is effectively ⟨alice's salt⟩ hello and Bob's is ⟨bob's salt⟩ hello. Because they differ from the first token onward, so to the cache the two prompts look nothing alike, and Bob can never hit (or time) an entry that Alice created.

In short, salting partitions the cache to ensure every user only has access to their own cache. Partitioning a cache key by security or privacy context is a well-established mitigation for cache timing attacks, for instance, HTTP caching guidance refers to the analogous technique as "double keying". The same pattern is used in production inference systems: vLLM and NVIDIA Dynamo support per-request cache_salt and hosted providers including OpenAI and Anthropic isolate prompt caches at organization or workspace boundaries. Figure 2 illustrates how standard cache salting partitions the cache per user.

Figure 2: Standard cache salting: the cache is partitioned using a per-user salt, ensuring each user only accesses their own cache and...