Prefill and Decode Want Different Computers | Aditya Kumar Light Dark System

Post<br>Cancel<br>Prefill and Decode Want Different Computers<br>Contents Prefill and Decode Want Different Computers

Three announcements inside a year, all making the same architectural bet.<br>AWS is pairing Trainium with Cerebras: Trainium runs prefill, a Cerebras CS-3 runs decode, and the result ships as a premium tier on Bedrock.1 AMD is pairing Helios rack-scale systems with the Cerebras Wafer-Scale Engine on the same split, claiming 5x higher tokens per second per watt, with Helios going into Cerebras datacenters before the end of 2026.2 And NVIDIA, after licensing Groq’s technology in a reported $20 billion deal, is shipping Groq 3 LPX as a dedicated decode co-processor inside the Vera Rubin platform: Rubin GPUs run prefill, LPX racks run the latency-sensitive part of the decode loop.3<br>None of this is fashion. It falls out of a property of the transformer decode loop that has been true since the first autoregressive model and gets worse with every hardware generation. This post is about why the split is inevitable. The four that follow are about why programming the result is so much harder than the press releases suggest.<br>The two phases are not the same computation<br>Serving one request has two phases with almost nothing in common.<br>Prefill processes the whole prompt at once. For a prompt of $N$ tokens and hidden dimension $d$, each projection in each layer is a matrix multiply of shape $(N \times d) \times (d \times d)$. That is $2Nd^2$ floating-point operations against $d^2$ weights read from memory. At two bytes per parameter, arithmetic intensity is<br>\[I_{\text{prefill}} = \frac{2Nd^2}{2d^2} = N \ \text{FLOP/byte}\]For a prompt of a few thousand tokens, that is a few thousand FLOP per byte. Every accelerator built in the last decade is compute-bound in that regime.<br>Decode emits one token per sequence per step. The same projection becomes $(1 \times d) \times (d \times d)$ — a matrix-vector product. You read the entire weight matrix to produce a single token:<br>\[I_{\text{decode}} = \frac{2d^2}{2d^2} = 1 \ \text{FLOP/byte}\]That is the whole problem in one line. Prefill and decode differ in arithmetic intensity by three orders of magnitude, on identical weights, in the same model, for the same request.<br>Batching is the standard answer and it is a partial one. Running $B$ sequences together turns the GEMV back into a GEMM of shape $(B \times d) \times (d \times d)$, so intensity rises to roughly $B$. To approach the compute-bound regime on a modern GPU you need $B$ in the hundreds. Two things push back. Larger batches raise per-token latency for every sequence in the batch, which is the metric users feel most directly. And the KV cache scales with $B$ times context length, so the batch you want for arithmetic intensity is often the batch you cannot fit.<br>Attention makes it worse. At decode step $t$, attention reads the entire KV cache accumulated so far and does $O(t)$ work on $O(t)$ bytes. There is no reuse to find. That component is memory-bound at every batch size, and it grows linearly with context, which is precisely the direction the industry is moving.<br>So decode is bandwidth-starved by construction, and long context starves it further.<br>The ceiling, in one division<br>The abstraction becomes concrete fast. At batch 1, generating a token requires reading every weight in the model exactly once. So the upper bound on single-stream decode throughput is a division:<br>\[\text{tokens/sec} \le \frac{\text{memory bandwidth}}{\text{bytes of weights}}\]Nothing about the kernel, the framework, or the compiler enters that expression. An 8B model at fp16 is 16 GB. On an accelerator with 3 TB/s of bandwidth, batch-1 decode cannot exceed roughly 190 tokens per second no matter what you do to the software. A 70B model at fp16 is 140 GB, which puts the same ceiling near 21 tokens per second and does not fit in one H100’s memory in the first place.<br>That division is why decode latency is a hardware property rather than an optimization target. You can improve the constant factor with quantization, which shrinks the numerator’s denominator, or with speculative decoding, which amortizes one weight sweep across several accepted tokens. Both are real and both are bounded. What you cannot do is make a batch-1 GEMV compute-bound.<br>It is also why 21 PB/s is the number Cerebras leads with. Put the weights in SRAM and the division comes out somewhere else entirely.<br>The design has a cost that belongs in the same paragraph. Those 44 GB are distributed across 900,000 cores as small local memories rather than pooled the way HBM is, and a model whose weights exceed 44 GB needs more than one wafer. For frontier-scale models the decode side is a multi-wafer system with its own partitioning problem, which is a real constraint rather than a footnote.<br>One chip cannot be right for both<br>Put the two phases on the same accelerator and you buy hardware that is wrong for one of...