How LLMs Work, Part 4: Using the Trained Model | Shubham Raizada’s Blog

Table of Contents

How LLMs Work, Part 4: Using the Trained Model

A while back I wrote a post on TurboQuant, about compressing the KV cache to make inference cheaper. At that point I was reasoning about the size of the KV cache without knowing how it gets filled in the first place, or where the Keys and Values come from. I knew it stored Key and Value vectors, but I could not have explained what actually happens between hitting enter and seeing the first word appear, or why the response streams out one word at a time instead of all at once.

I have been writing up how LLMs work for software engineers who don’t have an ML background. This is the last part of the series. Parts 1, 2, and 3 covered how text is processed, how models learn, and how training scales. All of that produces a model sitting in memory as billions of trained parameters. This post covers what happens when you type a prompt and hit enter. How the model generates one token at a time, why that is slow, what the KV cache does about it and how decoding strategies shape the response.

Inference: Autoregressive Generation

During training, the model saw entire sequences and predicted the next token at every position in parallel. But during inference, there is no response to look at. The model has to build it one token at a time.

In Part 1 I covered the forward pass. Token IDs flow through embeddings and transformer layers, and the final layer outputs a score (logit) for every token in the vocabulary. During inference the same forward pass runs, but the model can only predict one new token at a time. It does not know what comes after that because the rest of the response has not been generated yet.

Given a prompt, the model generates a response in the following steps.

User types a prompt. The tokenizer converts it to token IDs.

Run one forward pass with all the prompt tokens as input. The model produces logits for the token that would come after the entire prompt.

Sample one token from those logits.

Append it to the sequence.

Run another forward pass with the original prompt plus the one new token.

Sample the next token.

Repeat until you reach the desired length or hit a stop token.

This is autoregressive generation . Each step feeds into the next, so the model has to go one token at a time.

Let me walk through an example. The user sends “What is gravity?”, which the tokenizer converts to [What, is, gravity, ?] (4 tokens).

Step 1 : Input is [What, is, gravity, ?] (4 tokens from the prompt).

Forward pass produces logits for 128,256 possible next tokens. After SFT and alignment (Part 3), the model responds to questions rather than just completing text. “Gravity” gets the highest logit.

Sample “Gravity” (pick it from the probability distribution). Append to sequence.

Step 2 : Input is now [What, is, gravity, ?, Gravity] (5 tokens).

Forward pass produces logits. The token for “is” gets the highest logit.

Sample “is”. Append to sequence.

Step 3 : Input is now [What, is, gravity, ?, Gravity, is] (6 tokens).

Forward pass produces logits. The token for “the” gets the highest logit.

Sample “the”. Append to sequence.

Step 4 : Input is now [What, is, gravity, ?, Gravity, is, the] (7 tokens).

Forward pass produces logits. The token for “force” gets the highest logit.

Sample “force”. Append to sequence.

The model keeps going like this, one token per step, building “Gravity is the force…” until it hits a stop token or a length limit. The mechanics are the same as a base model completing text. The difference is that SFT and alignment (Part 3) shaped the parameters so the model produces answers instead of continuations.

At each step, the model processes the entire sequence so far. Step 1 reads 4 tokens. Step 2 reads 5. Step 3 reads 6. At step 100, the model is re-reading tokens 1 to 99, which were already processed in previous steps. Most of that work is redundant.

Unlike training, where the entire sequence is fed at once and all tokens are processed in parallel (causal mask prevents looking ahead), inference is inherently sequential . You cannot generate step 2 until step 1 is done, because step 1 produces the token that step 2 needs as input. A 1000-token response means 1000 forward passes. Training processes all 1000 tokens in one.

Prefill and Decode

In practice, inference splits into two phases.

Prefill phase , which processes the entire prompt at once, in parallel. A 50-token prompt runs in one forward pass. The GPU processes all 50 tokens simultaneously through every layer. Each token only attends to the tokens before it (the causal mask from Part 1 prevents looking ahead), but the GPU computes all these attention patterns in one batched operation. This is what GPUs are good at, large matrix multiplications with high parallelism. The forward pass works the same way as during training. The model produces a next-token prediction at every position. But...