DFlash Changes What Tokens per Second Means | Michał Piszczek

Skip to content

AI INFRASTRUCTURE

DFlash Changes What Tokens per Second Means

August 11, 2026·14 min read·Michał Piszczek

#DFlash#speculative decoding#llama.cpp#local LLM#AI infrastructure

I spent a night trying to fit a dense 30B model, 256K context, vision, and speculative decoding onto one 24 GB GPU. The fastest quant lost. The quant with the lowest perplexity lost too. What won was the configuration that made the whole system useful, not any single number impressive.

The final setup runs Meta Muse Glimmer 30B on an NVIDIA RTX PRO 4000 Blackwell SFF capped at 70 watts. It holds the target model, a separate vision projector, a five-layer DFlash drafter, and a 262,144-token slot on one card. On a coding task it reaches 84.64 tokens per second. On mixed code, prose, reasoning, and infrastructure work it falls to 38.34. With the KV cache actually filled to 262,116 input tokens, decode falls again to 21.56.

Same weights. Same GPU. Same drafter. Three very different machines, depending on what they are asked to produce and where the cursor sits in memory.

With speculative decoding, tokens per second is no longer purely a hardware benchmark. It is also a predictability benchmark.

That is the useful result. DFlash did not just make this model faster. It changed what throughput measures.

DFlash is not a faster Glimmer

Muse Glimmer 30B is a dense model with roughly 29.6 billion parameters, 52 layers, and a native 131,072-token context that can be extended to 262,144. Dense matters here. Every generated target token activates the whole model. There is no MoE shortcut where only a small subset of parameters runs.

DFlash sits beside it. The drafter has five layers and prepares a block of candidate tokens in parallel. In this llama.cpp configuration, n_max=15 means up to 15 draft candidates around a 16-token block. Glimmer then verifies that block with the full target model. Candidates survive only while they match the target's accepted continuation. The first mismatch ends the accepted prefix.

The drafter is not blind autocomplete. It receives target-model features from layers 1, 13, 25, 37, and 49, injecting information about Glimmer's internal state into its own cache before proposing the next block. That is the key idea in the DFlash paper: a lightweight block-diffusion model drafts in parallel, while the expensive autoregressive model verifies.

DFlash does not make one Glimmer forward pass cheaper. It tries to buy several output tokens with that pass. When it guesses well, the cost of verification is amortized across an accepted prefix. When it guesses badly, the drafter and verification work become overhead.

This is also why speculative decoding can be lossless with respect to the target model under the same sampler. The drafter proposes. The target remains the authority. A weak drafter should reduce speed, not intelligence. If output quality changes materially, suspect quantization, sampling differences, or an implementation bug before blaming the speculative idea itself.

Code and planning are different workloads

On the same Q5_K_M target, regular decoding produced 17.98 tok/s. DFlash on the long coding task produced 80.87 tok/s, a 4.50 times speedup. The best tuned 256K run reached 84.64 tok/s with 38.67% token acceptance.

Then I ran a mixed workload: code, prose, reasoning, and infrastructure. Throughput dropped to 38.34 tok/s and global acceptance to 14.41%.

WorkloadDecodeAcceptanceWhat it measures

Regular decoding, code17.98 tok/sn/aTarget model alone<br>DFlash, code80.87 tok/s36.18%Predictable structured output<br>DFlash, best tuned code84.64 tok/s38.67%256K slot, vision loaded<br>DFlash, mixed agent work38.34 tok/s14.41%Code, prose, planning, infra<br>DFlash, full KV cache21.56 tok/sworkload-specific262,116 input tokens

Code is unusually friendly to speculative decoding. Syntax, indentation, APIs, repeated identifiers, boilerplate, and local patterns constrain the next tokens. After for (int i = 0; i , there are relatively few sensible continuations. A drafter can often travel several tokens before it diverges.

Planning is different. After “the safest migration strategy is”, several continuations can be equally correct. The drafter chooses one. Glimmer chooses another. Neither continuation has to be bad, but an exact-token verifier sees a mismatch, rejects the remaining speculative prefix, and starts another cycle.

This means an agent has at least two throughput regimes. During implementation, repetitive edits, code completion, JSON, command lines, and schema-constrained tool calls can fly. During architecture, ambiguous reasoning, or conversational explanation, DFlash may spend much more time asking the target, “did you mean this exact path?”

A single average hides that difference. If an agent benchmark contains 80% code emission, DFlash looks extraordinary. If it contains long planning traces and divergent prose, the same setup...