The Attention Rebuild: How 2026's Open Models Made 1M Context Fit on Machines You Own
Skip to content
Subscribe
Dark
Every headline model of 2026 advertises a million-token context window: Kimi K3, DeepSeek V4, Inkling, Nemotron 3. Not long ago that number would have been a joke for local hardware, and our own KV cache explainer spelled out why: on a classic dense transformer, context is a memory tax that grows with every token. Run the math on a Llama 3.3 70B and it is brutal. Its cache costs about 0.66MB per token (80 layers, 8 KV heads, 128-dim heads, 16-bit), so 128k tokens of context is roughly 84GB , and a million tokens would be around 655GB, more memory than the model itself .<br>Yet here is a measured, real-world 2026 result: a 48GB Mac runs Qwen 3.6 27B at its full 262k context , model and cache together, with room left for macOS. Nothing about that machine changed. The models changed. While everyone was watching the MoE takeover, the same labs quietly made a second, matching bet: they rebuilt attention itself. This piece is about that second convergence, because it decides how much context your machine can really hold.<br>The problem: full attention pays rent on every token<br>In a standard transformer, every layer stores a key and value vector for every token you have ever fed it, and every new token looks back at all of them. That store is the KV cache. It grows linearly with context length, multiplied across every layer, and it lives in your fast memory alongside the weights. Double your context, double the tax. This design has perfect recall, and a price that made long context a datacenter luxury.<br>The 2026 releases attack that price from three directions, usually in combination:<br>1. Sliding windows: only remember what's nearby. Most layers watch only the last few thousand tokens, so their cache stops growing at the window size. Mistral 7B (Jiang et al., 2023) mainstreamed this in open models, using sliding window attention "to effectively handle sequences of arbitrary length with a reduced inference cost."<br>2. Linear attention: replace the cache with a fixed-size state. Instead of storing everything, a recurrent-style layer compresses history into a state whose size never grows, the lineage of Mamba (Gu & Dao, 2023), which showed "linear scaling in sequence length" with a constant-size state and roughly 5x inference throughput over comparable transformers. The modern refinement is Moonshot's Kimi Delta Attention (the Kimi Linear paper), a gated linear-attention module that cuts KV cache usage "by up to 75%" in long-sequence generation.<br>3. Keep a few full-attention layers as the safety net. Pure linear attention historically loses precise long-range recall, so the winning recipe is a hybrid: mostly cheap layers, with periodic full-attention layers preserving global lookback. Kimi Linear interleaves them 3:1; Inkling runs 55 sliding-window layers against 11 global ones.<br>Who is running what
2026 modelAttention designWhat it does to context cost
Kimi K3 (2.8T)Kimi Delta Attention (hybrid linear)Cache/state stays in the tens of GB even at the 1M window<br>Qwen 3.6 27BHybrid: only 16 of 65 layers keep KV, rest are linear~4x less KV memory than a standard dense design, plus a fixed ~0.9GB recurrent state (owner-measured)<br>Inkling (975B)55 sliding-window + 11 global layers, banded position biasMost layers' cache capped at the window, 1M supported<br>MiniMax M3 (428B)MSA sparse attentionVendor-reported 28.4x cut in per-token attention compute at 1M context, with up to 14.2x faster prefill on an H800<br>DeepSeek V4 familyCompressed sparse attention (vendor description)1M context on both the 1.6T Pro and the 284B Flash<br>Kimi Linear 48B (the research vehicle)KDA + full attention, 3:1Up to 75% KV reduction (paper-reported)
Sources linked in each row and below; "owner-measured" and "vendor-reported" labels matter, and we keep them.<br>Set that table against the dense baseline from the intro and the shift is stark. The old question was "how much context can you afford?" The new question is "does your runtime understand the model's layout?", because on these architectures the cache stopped being the thing that runs out first.<br>What owners measure<br>The best public numbers come from the community member who converted Qwen 3.6 27B for llama.cpp, froggeric on r/LocalLLaMA, whose fit tables we referenced above: "only 16 of 65 layers use KV cache (verified). The other 48 are linear attention (fixed 898 MiB recurrent state). KV memory is ~4x less than a standard dense model."<br>His follow-on warning is the practical gotcha of this whole transition: "Runtimes that don't handle this (e.g. vllm) allocate KV for all 65 layers and show much higher memory usage." The architecture only saves you memory if your runtime knows about it. Support for each hybrid design lands in llama.cpp, vLLM, and MLX at different speeds, which is exactly why the Inkling and K3 GGUFs are still waiting on merged support (as we covered in the K3 hardware piece),...