A primer: how an LLM actually serves a request | The Inference Wall

A primer for “The Inference Wall”. Read this before<br>Part 1 if the words KV cache, prefill, decode,<br>or batch are fuzzy. It explains the machine the five posts go on to break; it deliberately<br>stops before any of their findings.

Manas Pathak · August 21, 2026

The five posts in this series each take a working LLM server, turn one knob until something<br>breaks, and read why. To follow why each break happens, you need a mechanical picture of<br>what the server is doing between the moment a request arrives and the moment its answer<br>finishes streaming. That picture is small, it is not math-heavy, and once you have it every<br>post is a variation on it. This primer builds it once. No benchmarks here, no surprises, just<br>the machine.

The model is a pile of weight matrices in memory

A language model is, physically, a large collection of weight matrices , fixed numbers<br>learned during training. For the model this series uses (Qwen3.5-4B), that collection is<br>8.6 GB . When you start the server, those 8.6 GB are loaded once into the GPU’s memory and<br>they stay there, unchanged, for the life of the server.

The GPU has two relevant parts. There is its memory (called HBM), which is large, holds<br>those 8.6 GB comfortably, but is relatively slow to read from. And there are its compute<br>cores , which do the actual multiplying, are extremely fast, but have almost no storage of<br>their own. This split is the single most important fact in the whole series, so hold onto it:<br>the weights live in the slow, roomy memory; the fast cores that use them cannot keep the<br>weights parked next to themselves.

Producing one token = streaming all the weights through the cores

“Running the model on a token” means taking that token, represented as a vector of numbers,<br>and multiplying it through every weight matrix in turn, layer by layer (this model has 32<br>layers), until numbers come out the other end that tell you the next token. That single sweep<br>through all the matrices is called a forward pass .

Because the cores cannot hold 8.6 GB, doing a forward pass means streaming all 8.6 GB of<br>weights out of HBM and through the cores. The multiplying itself is quick; the moving of<br>those bytes is the slow part. You will see the series lean on this again and again: the cost<br>of producing a token is dominated by how many bytes of weights have to be streamed to produce<br>it, not by the arithmetic done with them.

Two phases: prefill reads the prompt, decode writes the answer

Every request runs in two distinct phases, and they behave very differently.

Prefill is the first phase: the model reads your whole prompt. Crucially, all the prompt<br>tokens already exist (you typed them), so they can all be pushed through the forward pass<br>together, in one sweep. A 100-token prompt is one forward pass over 100 tokens. Prefill is<br>where the model does a lot of arithmetic at once, because every prompt token interacts with<br>every other (a prompt of length N does roughly N-by-N work as each token looks at all the<br>others).

Decode is the second phase: generating the answer, one token at a time. Here is the<br>constraint that shapes everything downstream: to produce output token 2, the model needs<br>output token 1 as input, because a language model predicts each token from the ones before<br>it. So the tokens of an answer cannot be produced together, the way a prompt’s tokens can.<br>Each output token is its own forward pass, over just one new token, and each such pass<br>streams all 8.6 GB of weights again. Prefill amortizes one weight-stream over the whole<br>prompt; decode is stuck paying one weight-stream per output token. That asymmetry is why<br>decode, not prefill, is the phase this series spends most of its time on.

The KV cache: why decode does not reread the whole conversation

If each output token needs “the tokens before it,” you might think every decode step reprocesses<br>the entire conversation so far. It does not, and the thing that saves it is the KV cache .

When the model processes a token, part of its work produces two vectors for that token, a<br>key and a value (K and V), which together are how later tokens will “look back” at this<br>one. The KV cache simply stores those K and V vectors for every token the model has already<br>seen. So when the model generates the next token, it does not recompute the past, it looks up<br>the cached K and V of every earlier token and attends to them.

Two things to keep straight, because they trip people up:

The KV cache stores per-token data (K and V vectors) , not weight matrices. The 8.6 GB of<br>weights are one thing; the KV cache is a separate, much smaller pile that grows as the<br>conversation grows. On this model a cached token is around 130 KB, so a hundred cached<br>tokens is barely ten megabytes, tiny next to 8.6 GB.

Because of the cache, a decode step feeds the model only each request’s single most recent<br>token , not its whole history. The history is already in the cache; only the newest...