KV Cache vs. Prompt Cache: What’s the Difference, and How Are They Related? - Jake blog - Articles From JakeBlog

KV Cache vs. Prompt Cache: What’s the Difference, and How Are They Related?

🇨🇳🇺🇸

1 hour ago<br>Articles From JakeBlog •

159 Views

No comments

Table of Contents

Every time a large language model generates a token, it draws on the content that came before it. If it had to compute everything from scratch at every step, responses would be much slower. When building an agent, the same set of system prompts, tool definitions, and conversation history is used over and over again. If these were reprocessed each time, latency and computational costs would continually increase.

These two types of redundant computations correspond to two concepts that are often confused: KV Cache and Prompt Cache . A model’s processing of a single request is usually split into two stages: prefill and decode . The KV Cache stops the system from re-doing the work on historical token K/V pairs during decoding, and the Prompt Cache lets later requests reuse the same prefix.

In short, the KV Cache is the underlying state and inference mechanism. The Prompt Cache is the strategy or product capability that reuses these preprocessing results across requests. A lot of Prompt Cache implementations rely on reusing pre-computed K/V states.

Tip for reading: This text is going to talk about Q, K, V, prefill, decode, prefix matching, and cache breakpoints (also called cache boundaries). You don’t need to know anything about math or APIs to understand this article. When you’re reading, first think of Q, K, and V as "intermediate vectors" in attention calculations. Then, follow along with the two examples: "Beijing weather" and "product manual."

KV Cache: "Intermediate Results" During Model Generation

Large models generate content token by token. Whenever a new token is generated, the model has to consider the tokens that have already appeared.

For example, in a standard Transformer, each token makes three sets of vectors—Q, K, and V—at every layer. You can think of Q as "what I’m looking for," K as "what I have here," and V as "what information I should extract if I’m selected."

In autoregressive decoding, the Q values of historical tokens aren’t reused in subsequent steps. However, their K and V values are repeatedly queried by tokens generated later. So, the model stores these K and V values—this is the KV Cache .

For example, if you were to ask, "What’s the weather like in Beijing?" During the prefill phase, the model processes the whole question and stores the K and V values for each token at every layer. Once the decoding phase starts, new Q, K, and V values are calculated only for the token just added to the sequence at each step. The model puts the current K and V together with the cached history, then does attention calculations using the current Q on both the historical and current K and V to predict the next token. This way, you won’t have to keep recalculating the K and V of historical tokens.

But that doesn’t mean long context is free of cost. For standard full-attention models, the longer the context, the more video memory the KV Cache uses, and the more historical K/V pairs usually need to be read at each step. So, long conversations might still feel slow. Attention structures like sliding windows limit the history that can be seen.

The KV Cache is usually managed by the inference engine, and application developers rarely interact with it directly. It’s mostly used for incremental decoding within a single generation, but the cached K/V state can also be used by the inference framework for cross-request prefix reuse. The latter is often called a Prompt Cache or Prefix Cache .

Prompt Cache: Eliminating Redundant Processing of Identical Prefixes

When people hear the term "cache," many immediately think of an "output cache," where a previously answered question is simply returned. But the Prompt Cache isn’t the same kind of output cache. Even if there’s a cache hit, the model will still regenerate the response.

The Prompt Cache reuses intermediate results from the prefill phase for prompt prefixes, such as K/V states or other similar preprocessing results. A cache hit reduces redundant prefill computations and shortens the delay for the first token. If the API provider charges for cached inputs, it can also lower the cost of repeated inputs.

For example, let’s say you give the model a 50-page product manual and ask:

Product manual → What's the warranty period?<br>A bit later, you ask another question based on the same manual:

Product manual → What are the requirements for returning an item?<br>The product manual used in both requests is exactly the same, except for the last question. If this common prefix is cached, the second request can reuse the preprocessed results associated with the manual and process only the new question that follows. On the other hand, if you only use this manual once, Prompt Cache...