Understanding KV Cache: The Hidden Memory Cost of Serving LLMs

Understanding KV Cache: The Hidden Memory Cost of Serving LLMs - Melchi Contents

Understanding KV Cache: The Hidden Memory Cost of Serving LLMs Melchi included in GenAI 2026-05-19 3006 words 15 minutes

Contents

How attention architectures evolved to keep KV cache from eating your GPU, and what that means if you self-host. Already comfortable with KV cache and attention? Skip the theory and jump straight to the interactive KV Cache Calculator to size VRAM for your model, batch size, and target GPU.

If you’re planning to self-host a large language model, you’ve probably sized VRAM based on parameters alone. A 70B model in BF16 needs roughly 140 GB just for weights. That’s the easy part: 70 billion parameters × 2 bytes. What’s less obvious is the second memory consumer that grows while the model is actually serving requests: the Key-Value (KV) cache . KV cache scales with every cached token in an active request: prompt tokens, generated tokens, and any prefix-cache entries the engine keeps resident. It also scales with the number of concurrent sequences. At 32K–128K context, KV cache can easily become the largest single thing on the GPU. If you don’t budget for it, you’ll serve one long-context user when you wanted to serve many. This post walks from the basics of attention through the architectural and runtime tricks people use to shrink KV cache. By the end you should be able to look at a model card and roughly predict its memory profile. Part 1: Attention, a quick refresher The original transformer attention The 2017 paper “Attention Is All You Need” introduced Multi-Head Attention (MHA) , the mechanism that lets a model look back at previous tokens when generating the next one. Three steps: Project. For each token, build three vectors (a Query (Q), a Key (K), and a Value (V)) by multiplying the token representation by learned weight matrices. Score. Take the dot product of Q with K. This answers “how relevant is each previous token to what I’m generating now?” Aggregate. Softmax the scores, then take a weighted sum of V. The result is a context-aware representation of the current token. The formula: $$ \text{Attention}(Q, K, V) = \text{softmax}\left(\frac{QK^T}{\sqrt{d_k}}\right)V $$ Why we need a cache During training, the model usually processes the whole training sequence in parallel. There’s no decode-time KV cache because every token is available at once. Training is still memory-hungry (activations, gradients, optimizer state) but that’s a different problem. Inference is the autoregressive case, where the model generates tokens one at a time. Each new token needs to attend to every token that came before it. Without a cache, generating token #1000 would require recomputing K and V for all 999 previous tokens at every step. The KV cache is the obvious fix: store the K and V projections once and reuse them on every subsequent step. Hugging Face’s documentation explains it the same way (Hugging Face cache explanation ). Key insight: We cache K and V, not Q. Q is the “question” being asked by the current token and only matters for that one step. K and V are the “memory” the question gets asked against, and that memory has to stick around.

Part 2: How big is the KV cache? The formula The serving-time number you actually want is: Total KV cache bytes = 2 × num_layers × num_key_value_heads × head_dim × cached_tokens × active_sequences × bytes_per_element

Where: 2 because we store both K and V. num_layers because every attention layer has its own KV cache. num_key_value_heads is the number of KV heads, which is not always the same as the number of query heads. head_dim is the per-head vector size, often 64, 128, or 256. cached_tokens is prompt tokens plus generated tokens still resident in the cache. active_sequences is your active batch / concurrent sequences. bytes_per_element is 2 for BF16/FP16, 1 for FP8/INT8, 0.5 for INT4-style packed storage. For standard Multi-Head Attention (MHA) : num_key_value_heads = num_attention_heads

For Grouped-Query Attention (GQA) or Multi-Query Attention (MQA) : num_key_value_heads That distinction is the whole game. GQA and MQA shrink KV cache by reducing how many K/V heads are stored, while keeping more Q heads for model capacity. A concrete example: a 70B-scale MHA baseline The example below is intentionally a worst-case MHA baseline . It is not a claim that every 70B-class model uses this exact configuration; many of them use GQA, MLA, sliding windows, or hybrid attention. ParameterValueLayers80Query heads64KV heads64Head dimension128PrecisionBF16 (2 bytes)Per token: 2 × 80 × 64 × 128 × 2 = 2,621,440 bytes ≈ 2.5 MiB

Now scale it up: ScenarioCached TokensActive SequencesKV Cache SizeSingle user, short chat2,04815 GiB Single user, long context32,768180 GiB 8 users, moderate context8,1928160 GiB 16 users, long context32,768161.25 TiB Reality check: A 70B...

Understanding KV Cache: The Hidden Memory Cost of Serving LLMs

Related Articles

Elevated error rates on requests to multiple models

Donald Trump and sons to be 'forever' exempt from tax audits

PopuLoRA: Co-Evolving LLM Populations for Reasoning Self- Play

Old Reddit Is Down

The ultimate female fantasy – A feminist critique of Beauty and the Beast