Nothing Is Easy When You're an LLM: The Flat Latency Problem

SubscribeSign in

Nothing Is Easy When You're an LLM: The Flat Latency Problem<br>Dose #9 — Production Agentic AI Under Pressure

Dr. Ryan Rad<br>Jun 05, 2026

Share

When you walk to the kitchen, your brain barely registers it. When you solve a differential equation, it burns. Humans spend energy proportional to the difficulty of the task.<br>LLMs don’t.<br>Ask a model where the capital of France is. Ask it to explain recent advances in quantum computing. To the model, both answers cost exactly the same — one full forward pass per token, every time, no exceptions. Predicting the word “Paris” and the word “Qubit” are computationally identical events.<br>That is not an inefficiency. That is the architecture. And it’s costing you more than you think as it’s integral into every single interaction you have with LLMs.

The Uniform Compute Problem<br>Every token your model generates receives identical compute — one full forward pass through every layer, every attention head, every weight. An obvious continuation produces a sharply peaked output distribution, one candidate dominates, the answer is never in doubt. A critical reasoning token, that determines the entire downstream branch of your pipeline, produces a near-uniform one, several candidates compete,. Same forward pass. Same cost. The architecture cannot tell the difference.

Consider a model generating a Python function. After for i in range(len(my_list , the next token is near-deterministic, any model trained on a week of GitHub data predicts the closing parenthesis ) with near-certainty. Then comes the second closing bracket ) , the colon : , the newline, and the indent. Each one burning full frontier compute to predict syntax a rule-based system could have handled.<br>Then the logic lands. The one token that actually matters. Same price as every boilerplate token that preceded it.<br>That compounds in multi-agent systems. Every agent pays full model cost for every token, including the ones that never needed a frontier model in the first place.<br>This is the flat latency problem. You cannot fix it by scaling hardware.

Speculative Decoding: Run Ahead, Verify Fast<br>Not every token deserves a frontier model. So we should stop sending them all to one and use a hybrid strategy. Speculative decoding works like this.

A small, cheap Draft Model runs ahead of the main model. It generates K tokens speculatively — fast, low cost, imperfect. The large Verifier Model then checks all K drafted tokens in a single parallel forward pass. Not K passes. One.<br>If the draft tokens match what the verifier would have generated, you have produced K tokens for the cost of roughly one verification pass. You have paid once and received K. That is the efficiency gain.<br>If they diverge at token N , you accept tokens 1 through N-1 , grab the verifier’s corrected token for position N , discard the rest of the draft, and let the drafter resume. You have lost nothing in output quality. The final sequence is always what the large model would have produced. The drafter never contaminates the output. It only accelerates the path to it.<br>If the draft model is accurate on 7 out of every 10 tokens, your effective generation speed increases dramatically without compromising model quality, model size, or hardware. The verifier is still the brain. The drafter is just the leg work.

One important caveat. The drafter has to be fast enough and accurate enough to pay for its own overhead. A drafter that’s wrong on 8 out of 10 tokens means you are running a second model for almost no gain. The draft acceptance rate is the number to watch. If it falls below roughly 60%, you are adding complexity without recovering the latency.

Why Nobody Shipped This in Production — Until Now

Speculative decoding has been supported in HuggingFace for years — a single assistant_model argument in generate() is all it takes technically. The problem was never the API. It was finding a drafter that actually works with your model.<br>The draft model and verifier have to share the same tokenizer and output distribution. In practice that means same-family models, or a distilled or quantized version of the verifier itself. Pair the wrong two models and your acceptance rate collapses — you are running two models and gaining nothing. Most combinations don’t work. So most teams didn’t bother.<br>The operational question was always: which smaller model is actually compatible with mine, and who maintains that pairing as the verifier gets updated? There was no clean answer.<br>Gemma 4 changed everything!

Everything above assumes you have a drafter worth trusting. Gemma 4 ships with one.<br>Its built-in MTP (Multi-Token Prediction) drafter is not a separate model bolted on — it is prediction heads trained directly into the architecture. That distinction matters: the drafter shares the target model’s KV cache and activations, so it never recalculates context the larger model already has. This is not just...