Qwen3.8 27B at 256K: 50 TPS on a 24 GB GPU | Michał Piszczek

Skip to content

AI INFRASTRUCTURE

Qwen3.8 27B at 256K: 50 TPS on a 24 GB GPU

August 17, 2026·17 min read·Michał Piszczek

#Qwen3.8#MTP#NVFP4#llama.cpp#local LLM#GPU inference

I gave Qwen3.8's MTP drafter another 69.2 MiB of precision. Throughput fell from 50.44 to 37.02 tokens per second. That result sums up the whole experiment: the best local inference setup is rarely made from the individually "best" parts.

I wanted a dense 27B model, its full 262,144-token context, multimodal input, maximum useful quality, and speculative decoding on an NVIDIA RTX PRO 4000 Blackwell SFF with 24 GB of VRAM. The server also had to survive real agent work after printing model loaded. The experiment followed a hunch I had written about earlier: careful operation may matter as much as moving to a larger model.

The finished system averages 50.44 tok/s in the current ten-run production series. On a strict runtime A/B, the custom llama.cpp build reaches 55.40 tok/s versus 45.42 for clean master, a 21.97% gain. Against target-only greedy decoding, embedded MTP moves 21.19 to 59.46 tok/s, or 2.81 times the throughput. At the far end of a genuinely occupied 256K cache, it still produces 12.61 tok/s without an out-of-memory failure.

Those numbers came from different gates and should stay separate. Combining them into one heroic speedup would make a better headline and a worse benchmark.

The winning setup came from the fit between the quant, drafter, CUDA kernels, memory layout, and workload. No component won on its own.

The target was deliberately unreasonable

Qwen3.8 27B is a 64-layer dense model. Its repeating pattern contains three Gated DeltaNet layers followed by one full-attention layer, giving 48 recurrent layers and 16 conventional attention layers. It has a native 262,144-token context, a one-layer MTP head, and a separate 27-layer vision encoder.

The hardware is lopsided in a useful way:

GPU0: RTX PRO 4000 Blackwell SFF, 24 GB GDDR7 with ECC, a 192-bit memory interface, 432 GB/s peak memory bandwidth, 24,467 MiB reported capacity, and sm120a. It holds the target, embedded MTP, recurrent state, graphs, and the 256K KV cache.

GPU1: RTX 2000 Ada, 15,996 MiB, sm89. It holds the F16 multimodal projector and other auxiliary services.

Runtime: Debian 13, CUDA 12.9.86, GCC 14.2, dual-architecture CUDA build.

Only the 16 full-attention layers grow a conventional KV cache with sequence length, which makes 256K less absurd than it first appears. With Q4 K and V, that cache costs roughly 4.25 GiB before allocator overhead. DeltaNet adds recurrent state and checkpoints instead. Four checkpoints were the useful minimum; the default 32 spent memory I needed elsewhere.

NVIDIA quotes 432 GB/s of peak bandwidth. That is a hardware ceiling rather than an application metric from llama.cpp, but it matters here. Autoregressive decode repeatedly streams quantized weights, and the 16 attention layers add increasingly expensive KV reads as context fills. This is why the same profile averages about 50 tok/s on the production task and 12.61 tok/s at the far end of a 261.5K-token cache.

The original plan was simple: estimate the capacity, select a quant, then benchmark it. The machine immediately taught me that capacity estimates are just admission tickets. The real test begins after loading.

The first winner was Q4_0, and it was the wrong winner

I began with public GGUFs at 40K context. Q4_0 was surprisingly strong. Target-only decoding reached 22.40 tok/s, and MTP with n_max=3 reached 44.95. It beat smaller Q3_K_M and nominally smarter Q4_K_M variants because file size and quant label do not describe the CUDA kernel that actually runs.

QuantTarget onlyMTP n=3Acceptance

Q3_K_M17.00 tok/s31.34 tok/s83.98%<br>IQ4_XS, iMatrix20.63 tok/s34.40 tok/s64.87%<br>Q4_022.40 tok/s44.95 tok/s80.40%<br>Q4_K_M17.57 tok/s26.15 tok/s66.86%

Then quality testing spoiled the easy answer. On a short, identical WikiText-2 control, IQ4_XS scored 6.1175 perplexity while Q4_0 scored 6.3798. Q4_0 led the speed table. Hermes needed a main model, though, and that quality trade felt too expensive for a few hundred milliseconds. I would have been using a 27B model as oversized autocomplete.

The opposite extreme failed too. Q4_1 reached 6.1127 PPL, marginally ahead of IQ4_XS, but its memory footprint made 256K plus F16 vision uncomfortable. The useful point was somewhere between a fast blunt quant and a precise file that left no room for the rest of the system.

Loading 256K proves almost nothing

Early capacity tests looked excellent. Q4_0, MTP, Q4 KV, four recurrent checkpoints, and the F16 projector all allocated at 262,144 context. That still did not answer the question I cared about.

I filled the slot with 261,500 input tokens, generated another 256, and then reused the hot cache. No truncation. No OOM. The first Q4_0 profile decoded at 12.06 tok/s near the end of the cache, compared with 44.95...