Speculative Decoding, Explained: The Free Speed Toggle Your Local LLM Is Probably Not Using
Skip to content
Subscribe
Dark
There is a setting in LM Studio and llama.cpp that can make a local model generate 1.5 to 2.5 times faster without changing a single word of its output. Not a lower quant, not a smaller model, the same model producing mathematically identical answers, faster. Most people leave it off, and the people who turn it on often pair it wrong and make things slower. It is called speculative decoding, and 2026 is the year it went from a power-user trick to something model makers ship in the box. Here is how it works, what it really delivers, and the cases where it backfires.<br>The idle-GPU problem it solves<br>Start from a fact we keep coming back to on this site: token generation is limited by memory bandwidth, not compute. Each new token forces the GPU to read the model's active weights from VRAM, and as we covered in Bandwidth, Not TFLOPS, the arithmetic units spend most of that time waiting. The same hardware that crawls through generation at 20 tokens per second can chew through a prompt at 800+ tokens per second, because prompt processing evaluates many tokens in one parallel pass while generation does one at a time.<br>Speculative decoding is the trick that converts generation into something that looks like prompt processing. A small, fast draft model guesses the next several tokens. The big model then checks all of those guesses in a single parallel pass, the kind of work your GPU is good at. Every guess the big model agrees with is accepted instantly. The first guess it disagrees with gets thrown away, and the big model's own choice is used instead.<br>The part that surprises people: this is lossless . It is not an approximation and there is no quality tradeoff to weigh. The verification step uses a rejection-sampling rule that provably reproduces the big model's output distribution, so you get exactly what the big model alone would have produced. The original Google paper puts it plainly: the speedup comes "without changing the distribution," with "identical outputs." Your only costs are VRAM for the draft model and some extra computation, which is why the technique shines precisely where compute sits idle: single-user local inference.<br>What the research says it is worth
Method (paper)ApproachReported speedup
Speculative decoding (Leviathan et al., Google, 2022)Separate small draft model2 to 3x on T5X<br>Speculative sampling (Chen et al., DeepMind, 2023)Draft model + modified rejection sampling2 to 2.5x on Chinchilla 70B<br>Medusa (Cai et al., 2024)Extra decoding heads on the model itself, no separate draft2.2 to 3.6x<br>EAGLE (Li et al., 2024)Tiny drafter that reads the big model's hidden states2.7 to 3.5x on Llama 2 70B
Paper-reported figures on datacenter hardware, linked in the sources below. Local speedups are usually smaller; treat 1.5 to 2x as a good outcome on a home box.<br>The whole game is a single ratio: how often the draft's guesses are accepted, versus what the drafting costs. A draft model that agrees with the big model 70 to 80% of the time roughly doubles your speed. One that agrees half the time barely breaks even, and the drafting overhead can push you underwater. That is why the acceptance rate, which llama.cpp and LM Studio both report, is the first number to check when you try this.<br>Two practical rules fall out of that ratio. First, the draft must share the big model's vocabulary, which in practice means same family: Qwen drafts for Qwen, Llama drafts for Llama. Second, the draft should be roughly a tenth the size of the main model or smaller. LM Studio's own pairing guidance is a 1B draft for an 8B model, up to about a 1.5B draft for a 32B model. Bigger drafts guess slightly better but cost too much per guess.<br>The 2026 shift: the draft model now comes in the box<br>The reason this technique stopped being niche is called multi-token prediction (MTP) . Instead of you hunting for a compatible draft model, the model maker trains a small drafting head into (or alongside) the model itself, so speculation works out of the box with a perfectly matched drafter. DeepSeek-V3's technical report mainstreamed the idea in 2024, training with an MTP objective and noting the extra head "can also be reused for speculative decoding." In 2026 it went fully consumer:<br>Qwen 3.6 ships MTP layers, and llama.cpp gained support for them this spring (--spec-type mtp). One owner running the 27B on an M2 Max 96GB reported: "the results are amazing: 2.5x speed increase, bringing it to 28 tok/s". That thread's worked examples (quant choice, KV cache settings, context limits by RAM) are worth reading in full.<br>Gemma 4 shipped official MTP drafter checkpoints for every size in May 2026, down to a 78-million-parameter drafter for the phone-sized E2B. Google claims up to 3x "while guaranteeing the exact same quality as standard generation," and support landed across Ollama, vLLM, SGLang, and MLX.<br>Ollama , which for...