DFlash 2: Keep Drafting Parallel — Inco AI<br>Skip to contentInference is the bottleneck of the agent era. Agents read, plan, and call<br>tools, often for hours or days. They consume tokens at a rate chat never<br>approached. Every one of those tokens takes a full forward pass over the<br>model. At Inco AI, we are building the inference stack scaled to the token<br>economics of tomorrow. This post is a sneak peek.
Our team released DFlash in January; it<br>now runs in SGLang, vLLM, TensorRT-LLM, and llama.cpp. NVIDIA measured<br>up to 15× throughput<br>with it on Blackwell GPUs; Google reported<br>3× more tokens per second<br>on TPUs; CoreWeave's production Kimi K2.7 Code endpoint, the fastest for<br>that model on Artificial Analysis,<br>runs DFlash by default. The ecosystem now builds on it:<br>NVIDIA,<br>Red Hat, and<br>Modal have all published<br>DFlash drafters; Meta<br>(Muse Glimmer),<br>Poolside (Laguna),<br>Xiaomi<br>(MiMo-V2.5-Pro),<br>and NVIDIA<br>(Nemotron 3.5 Lightning)<br>ship official drafters with their own models. On Hugging Face, DFlash models have<br>been downloaded more than 3.5 million times (as of August 2026).
Speculative decoding is a core piece of the modern inference<br>stack.1 A small draft model guesses a block of tokens,<br>and the target model verifies the whole block in one forward pass. Good guesses<br>turn one pass into several tokens; bad ones just get thrown away. For years, though, the draft itself stayed autoregressive :<br>one token at a time. DFlash made it one-pass too: the entire block, every<br>position, predicted in parallel .
DFlash 2 drafting for Qwen3.8-27B on an Apple M5 Max with oMLX, side by side with autoregressive decoding.<br>DFlash 2 pushes parallel drafting one step further: over 20% more output from<br>every verification pass, for around 1% added cycle latency , with the output<br>provably unchanged. Across benchmarks the gain runs 16–25%. With the<br>Qwen3.8-27B drafter released today, SGLang serves at 2.7–3.4× the<br>throughput of autoregressive decoding at batch size 1. Predicting every<br>position independently leaves headroom in two places: choosing the right<br>tokens and holding accuracy to the end of the block. DFlash 2 recovers<br>both without giving up the one-pass design.
Run It Now
DFlash 2 already runs in the mainstream inference engines:
SGLangvLLMllama.cppoMLX<br>pip install "sglang[all] @ git+https://github.com/sgl-project/sglang.git#subdirectory=python"
python -m sglang.launch_server \<br>--model-path Qwen/Qwen3.8-27B \<br>--speculative-algorithm DFLASH \<br>--speculative-draft-model-path incoai/Qwen3.8-27B-DFlash2 \<br>--speculative-num-draft-tokens 8
pip install -U "vllm @ git+https://github.com/vllm-project/vllm.git@refs/pull/52816/head"
vllm serve Qwen/Qwen3.8-27B \<br>--speculative-config '{<br>"method": "dflash",<br>"model": "incoai/Qwen3.8-27B-DFlash2",<br>"num_speculative_tokens": 7<br>}'
git clone https://github.com/ggml-org/llama.cpp.git<br>cd llama.cpp<br>git fetch origin pull/27342/head:pr-27342<br>git switch pr-27342
# NVIDIA CUDA<br>cmake -B build -DCMAKE_BUILD_TYPE=Release -DGGML_CUDA=ON<br>cmake --build build -j
# Apple Silicon<br>cmake -B build -DCMAKE_BUILD_TYPE=Release -DGGML_METAL=ON<br>cmake --build build -j
./build/bin/llama-server \<br>-hf ggml-org/Qwen3.8-27B-GGUF:Q4_K_M \<br>-hfd incoai/Qwen3.8-27B-DFlash2-GGUF:Q4_K_M \<br>--spec-type draft-dflash \<br>--spec-draft-n-max 7
:first-child]:mt-0 [&>:last-child]:mb-0 [&_li:last-child>p:last-child]:mb-0">Download and install the prebuilt oMLX with DFlash 2 support.<br>To run Qwen3.8-27B with DFlash 2:
Open the oMLX Model Downloader and download:
mlx-community/Qwen3.8-27B-4bit
incoai/Qwen3.8-27B-DFlash2
Open the Model Manager and edit mlx-community/Qwen3.8-27B-4bit. Configure DFlash with the following settings:
DFlash : enabled
Draft model : incoai/Qwen3.8-27B-DFlash2
Draft quantization : enabled
Runtime block size : 5
Verify mode : dflash
Save the settings and load the target model.
The Right Tokens Are Already There
DFlash predicts every position independently, in parallel. Each pick is<br>plausible on its own. Yet nothing makes them fit together, and an<br>incoherent block is cut short at verification.<br>Recent methods such as Domino and<br>DSpark buy coherence with sequential<br>heads that rewrite each position's full-vocabulary distribution.<br>But is that costly autoregressive correction really necessary?
No. The evidence is already in DFlash's own candidate lists. Take the<br>first position: DFlash's top pick is right 85.4% of the time, but the<br>right token is in its top 16 candidates 99.5% of the time. Even when the top pick<br>is wrong, the right token is usually on the list.
[role=checkbox]]:translate-y-[2px] whitespace-nowrap text-[0.7rem] font-medium tracking-wider uppercase text-left">Metric[role=checkbox]]:translate-y-[2px] whitespace-nowrap text-[0.7rem] font-medium tracking-wider uppercase text-right">0[role=checkbox]]:translate-y-[2px] whitespace-nowrap text-[0.7rem] font-medium tracking-wider uppercase text-right">1[role=checkbox]]:translate-y-[2px] whitespace-nowrap text-[0.7rem] font-medium tracking-wider uppercase...