The DGX Spark isn't slow. It's bandwidth-starved — and speculative decoding helps.📉"> Notes

01 — The problem

Bandwidth, not compute

The DGX Spark pairs a GB10 superchip with 128 GB of unified LPDDR5X. Enough room for a 70B model on a desk, but only 273 GB/s to feed it. Running Qwen3.8-27B at Q4_K_XL (17.5 GB of weights), I measured 11.5 tokens per second . Across eight tasks (Chinese and English prose, code generation and editing, JSON, reasoning, translation, a 2,177-token summary) every median landed between 11.47 and 11.66 tok/s. A 1.7% total spread. The workload doesn’t matter; that’s what a machine looks like when it spends its time waiting on memory instead of computing.

Every generated token reads all 17.5 GB of weights. At the published 273 GB/s that caps generation near 15.6 tok/s; measured 11.5 is 74% of the ceiling. No configuration flag fixes arithmetic.

Speculative decoding fits this imbalance precisely: verifying k drafted tokens costs one forward pass (a single 17.5 GB read), so accepted guesses are paid for with idle compute instead of scarce bandwidth. The thing people complain about, the wasted FLOPS, is exactly what makes the fix work. The same reasoning applies to Strix Halo, Apple Silicon, and any unified-memory design.

02 — The lie

Where 208 tok/s came from

llama.cpp’s --spec-type ngram-mod drafts from repeated n-grams, no draft model needed. Benchmarked the usual way (same prompt, five runs, median) it reported up to 208 tok/s. But the first request in a fresh process is always ~11.5, and every later one climbs: the n-gram cache persists across requests inside the server process , so by run two it contains the complete answer from run one. The benchmark was measuring the cache replaying its own output. Setting "cache_prompt": false disables the KV prompt cache. It does not touch the n-gram draft cache, and no request-level flag does.

If you benchmark llama.cpp's n-gram speculation by repeating a prompt, your numbers are wrong. Restart the process or vary the prompt. The honest cold-start figure here is 11.5–12.4 tok/s, no measurable benefit at all.

03 — The results

Five strategies on one binary

Same model, same quantization, same server binary, idle machine; the only variable is the --spec-type flag and its draft model. Five runs per cell, median reported, spread ±0.1–2.8%.

Token generation · tok/s<br>Qwen3.8-27B UD-Q4_K_XL · median of 5 · higher is better

TaskBaseline<br>Draft 2BDSparkDFlash2DFlash2 ×

zh-prose11.6614.4218.4322.15<br>1.90×<br>en-prose11.5413.2417.1921.09<br>1.83×<br>code-gen11.5415.5423.8928.06<br>2.43×<br>code-edit11.5412.9618.7123.09<br>2.00×<br>json-out11.5419.5820.8323.33<br>2.02×<br>reasoning11.5423.3526.6128.33<br>2.45×<br>translate11.5324.1124.6829.48<br>2.56×<br>long-ctx11.4714.0519.6322.02<br>1.92×

ngram-mod (cold, honest) is omitted: 11.5–12.4 tok/s, indistinguishable from baseline.

DFlash2 wins every task , at 1.83× to 2.56× the baseline. It is a block-diffusion drafter: it emits a whole block of guesses in one pass and traces a coherent path through the candidates, and it’s a 2 GB file sitting next to a 17.5 GB model. Note that its numbers finally vary by task (21–29 tok/s) while the baseline was flat: the bandwidth ceiling is no longer the binding constraint.

04 — Findings

Three things the numbers taught me

Acceptance rate is not comparable across drafter architectures

On code generation the sequential 2B drafter accepts 88% of its guesses and delivers 15.5 tok/s; DFlash2 accepts 75% and delivers 28.1. Ranking by acceptance rate, the metric every paper reports, picks the slower system by 1.8×. A rejected sequential guess is a whole wasted forward pass; a rejected block position costs almost nothing. When comparing sequential against block drafters, measure wall-clock throughput or nothing.

Lossless in distribution, not reproducible in practice

At temperature 0, DSpark and DFlash2 matched a no-speculation control byte-for-byte on only 6 of 8 tasks (the control itself was fully deterministic). The divergence I traced landed exactly on the sequence’s second-narrowest top-2 logprob margin, 0.022, where a different verification batch shape changed floating-point reduction order and flipped the argmax. The distribution is preserved, but bit-for-bit reproduction on a GPU is not happening. If a regression suite pins exact model output, speculative decoding will break it, and that is not a bug you can fix.

The speedup belongs to the workload, not the setup

Deleting four words from one prompt (an instruction not to answer with an outline) changed what the model wrote, moved draft acceptance from 68% to 55%, and moved throughput 21%. Any single-prompt benchmark of this technique is one sample from a wide distribution. That’s why everything above uses eight tasks, and eight is still too few.

05 — The other path

What about just running a MoE model?

The standard advice for bandwidth-limited hardware. A 90 GB DeepSeek-V4-Flash quant generates at 19.5–20.0 tok/s on the same tasks, 1.7× the...