Prefill vs. decode in LLM inference<br>Skip to main content Contact sales Sign in<br>About usModelsPricingBlogCareersDocs<br>Guides Prefill vs. decode in LLM inference<br>Meghana Madhyastha<br>August 7, 2026 · 7 min read

Author<br>Meghana Madhyastha

Last updated<br>August 7, 2026 · 7 min read

Table of contents<br>How prefill, decode, and the key-value (KV) cache work<br>How the two phases show up in latency and throughput<br>Start with the workload shape<br>Serving patterns manage the trade-off<br>Test the request path your users actually take<br>Measure the phase before optimizing it<br>FAQ: What’s the difference between prefill and decode?

LLM inference has two phases: prefill, when the model processes the input context before it can produce output, and decode, when it generates that output one token at a time. The distinction becomes visible in product behavior. A response may take too long to start, leaving a blank pause after a user sends a request, or it may begin promptly and arrive in slow, uneven bursts.<br>Prefill and decode put pressure on different parts of inference. A first-token delay and a slow stream can call for different measurements and serving choices; aggregate tokens per second can't show whether an individual request starts promptly, streams smoothly, or finishes within the time your product needs.<br>How prefill, decode, and the key-value (KV) cache work<br>Every autoregressive LLM request has a request path. Before generation can begin, the serving system assembles system instructions, retrieved context, conversation history, and the user’s prompt into the input token sequence sent to the model. During prefill , the model processes that input and builds the key-value (KV) cache: request state that helps it generate the response. During decode , the model produces the response one token at a time using that state.<br>The KV cache isn't the model’s “memory of the conversation.” It's serving state for the request context. It's created as the input is processed, then read as output tokens are generated. As the response grows, that state grows too.<br>The phases have different access patterns. Prefill can process prompt tokens in parallel, so it's typically compute-bound: the system has substantial model work to perform before it can emit the first token. Decode is sequential. The next token cannot be generated until the previous one is available, and each step repeatedly reads model and KV-cache state from memory. In common autoregressive serving, decode is therefore typically memory-bandwidth-bound as well as sequential. Redis’s overview of prefill and decode and WEKA’s technical guide describe the same broad distinction.<br>That doesn't mean every LLM, runtime, or request behaves identically. Model architecture, prompt shape, output length, queueing, and concurrency all affect what a user experiences. It does mean that a long wait before output and a slow stream deserve separate investigation.<br>How the two phases show up in latency and throughput<br>The right metrics answer different questions:<br>Time to first token (TTFT) measures when a streamed response starts.<br>Inter-token latency (ITL) measures the gap between output tokens and whether the stream feels smooth or stuttered.<br>End-to-end latency captures total completion time.<br>Throughput measures total work completed by the system over time.<br>TTFT is a useful front-of-request signal. Long prompts, large retrieval payloads, and extensive conversation history can all increase prefill work. But high TTFT doesn't prove prefill is the sole cause. Queueing, network overhead, and admission control can also delay the first token.<br>ITL is the corresponding streaming signal. A response can have a good TTFT and still feel poor once generation starts. Long outputs, code generation, and multi-step agent responses make token cadence and end-to-end perceived time especially important.<br>Throughput complements the latency measures; it answers a different, system-level question. A change that improves aggregate throughput can still hurt an individual user’s end-to-end perceived time. That trade-off shows up whenever a serving system packs more work onto shared resources.<br>The symptom is a starting point, not a final diagnosis. A long wait before streaming can involve prefill, queueing, or the network path. Uneven streaming can involve decode, cache pressure, or contention with other work.<br>Start with the user-visible latency symptom, then measure time to first token, inter-token latency, or both phases under representative load.For a broader framework on matching latency requirements and traffic shape to an inference mode, see How to choose the right managed inference architecture.<br>Start with the workload shape<br>Workload categories are useful starting points for a measurement plan. The same application can move between prefill and decode pressure as traffic, model choice, prompt construction, and concurrency change.<br>Long-context and retrieval-heavy requests should send you first to the input-token distribution, TTFT,...