Laguna XS 2.1 33B on a RTX 3090: 296 tok/s peak, 152 tok/s at 256K context | lucebox
July 2026<br>By Davide Ciffa<br>Laguna XS 2.1 33B on a RTX 3090: 296 tok/s peak, 152 tok/s at 256K context<br>poolside’s Laguna XS 2.1 ships with a DFlash speculative-decoding drafter, and lucebox runs the pair on a single RTX 3090 : 296 tok/s peak at short context, and a flat 152 tok/s at 256K tokens where the full KV cache would not even fit in 24 GB. Prefill runs at ~3,500 tok/s (256K tokens in 67 s). Three optimizations got the same GPU from 22 to 152 tok/s at 256K in one pass: a drafter KV ring cache, sliding-window ring caches, and KVFlash paging. And the speculative decode is lossless : every committed token is one the model itself would produce.<br>Both scenes on the same RTX 3090: a short prompt streaming at the measured 296 tok/s peak, then a 256K-token prompt (prefill as a labeled timelapse, decode at the measured 152 tok/s). TL;DR<br>Decode speed stops depending on context length. Laguna XS 2.1 Q4_K_M on a 3090 holds 155 tok/s at 30K and 152 tok/s at 256K of real-code context. At the start of this work the same 256K request decoded at 22 tok/s.<br>Prefill: 256K tokens in 67 seconds (~3,500 tok/s), down from 411 s. Using batched pooled prefill + ring attention; the remaining cost is the MoE expert GEMMs.<br>Short context: 242 tok/s mean, 296 peak , HumanEval-10 at 230 tok/s, GSM8K answers 10/10. Speculative decoding is verify-exact, so quality is the model’s own at any speed.<br>Two new engine optimizations, both model-agnostic. The DFlash drafter now keeps a context-KV ring cache (draft step 9.9 → 2.4 ms, flat at any context), and sliding-window layers attend a 2K position ring instead of the whole KVFlash pool (verify 19.4 → 13.7 ms, ~1 GB VRAM freed).<br>Built with poolside’s drafter. Acceptance stays ~56% on long real-code prompts;<br>The model, fast MoE with a drafter in the box<br>Laguna XS 2.1 is poolside’s 33B MoE: 40 layers, fine-grained experts, 2048-token hidden size, and a 3-in-4 pattern of 512-token sliding-window attention layers. It ships with an official DFlash drafter : a 5-layer block-diffusion head that reads the target’s hidden states and proposes 16 tokens per step. lucebox verifies those proposals in one batched target forward and commits only tokens the target itself would emit: speculative decoding with exact greedy equivalence, which is why every number below comes with unchanged output quality (GSM8K 10/10).<br>Short context was already fast: 242 tok/s mean, 296 peak on a 3090 after the earlier rounds of work (quantized q4 drafter on HuggingFace, fused decode loop, CUDA-graph replay everywhere). Long context was a bigger problem: at 256K tokens the same server decoded at 22 tok/s and took 20+ minutes to prefill. This post is about closing these gaps:
Three optimizations<br>The 22 tok/s at 256K was not one bottleneck but three, and each fix is now a default in the lucebox engine.<br>1. KVFlash paging for the full-attention layers (background). The pool holds 8K of the 256K context on the GPU; cold 64-token chunks page to host RAM bit-exact, and a drafter scores which chunks deserve residency. Prefill runs through the pool in 1024-token batches with CUDA-graph replay. This is what makes 256K fit at all: the full KV cache would need ~20 GB next to 19 GB of weights.<br>2. A context-KV ring cache for the drafter. DFlash drafters re-encode their feature window (up to 4K tokens of target hidden states) every step. That cost ~10 ms per step once the window filled, on every DFlash model family, hidden until now behind slower targets. The drafter’s K/V now live in a position-indexed ring that only encodes newly committed tokens: draft step 9.9 → 2.4 ms, constant at any context .<br>per-step time at 256K context, RTX 3090
before draft 9.9 │ verify 19.4 │ heads 0.8 ≈ 22 tok/s*<br>+ draft-KV draft 2.4 │ verify 19.5 │ heads 0.8 → 109 tok/s<br>+ SWA rings draft 2.4 │ verify 13.3 │ heads 0.8 → 152 tok/s
* session start also paid thermal taxes 3. Ring caches for the sliding-window layers. 30 of Laguna’s 40 layers attend a 512-token window, but under paging they flash-attended the whole 8–16K pool span with 97% of it masked out, so most of the attention FLOPs in both prefill and decode were masked-out work. Window layers now keep a small position ring (slot = pos mod 2048) and skip the pager entirely: verify 19.4 → 13.7 ms, prefill −27%, ~1 GB of VRAM freed , acceptance unchanged.<br>Speed and quality<br>All runs: Laguna XS 2.1 Q4_K_M with the Q4_K_M DFlash drafter, single RTX 3090, real-code prompts, 256-token timed generation. “Start of this work” is the first working KVFlash port of laguna (32K pool, no drafter cache, no rings); “after” is the shipping default.<br>ContextConfigPrefillDecode tok/sAcceptance shortkvflash off—241.5 mean / 295.7 peak ~81% 30Kstart of this work19.1 s84.8~56% 30Kafter, 8K pool13.1 s 154.6 58.6% 256Kstart of this work26.4 min22.1~55%...