The GPU Is the New Storage Controller | Murat Karslioglu<br>Home<br>About<br>Blog<br>Books<br>Tools<br>PM
For 40 years, the CPU has been the I/O initiator for storage. It decides what gets read, when, and where it lands in memory. Every protocol in the stack assumes this: NVMe command queues, the Linux block layer, O_DIRECT, scatter-gather lists, interrupt-driven completions. All designed for a CPU host. AI inference is breaking that model. The GPU now knows what data it needs next (the next KV cache tensor, the next attention head), and routing that request through the CPU adds microseconds of latency that the GPU can’t afford. GPU-initiated I/O, direct NVMe reads over P2P DMA, is the architectural response. But the software stack wasn’t built for it, and the standards bodies haven’t caught up.
The CPU Was Always the I/O Initiator
Every storage architecture since the IBM PC/AT has followed the same flow: the CPU decides what data to fetch, builds a command descriptor, submits it to a device queue, and handles the completion interrupt. NVMe refined this model (65,535 queues, polling instead of interrupts, multi-core submission), but it didn’t change the fundamental assumption. The CPU is the host. The storage device is the target. Data flows from device to host memory, and if a GPU needs that data, the CPU copies it again.
This worked for 40 years because the CPU was the one doing the computation. It knew what data it needed because it was the one processing it. Even when GPUs became the dominant compute engines for training workloads, the data pipeline still made sense: the CPU prefetches training batches from storage, stages them in host DRAM, and the GPU pulls them over PCIe when ready. Training is sequential and predictable. You know which batch comes next because you designed the data loader.
Inference is different.
Why Inference Breaks the Model
Training reads data in large, predictable sequential sweeps. A DataLoader shuffles the dataset once per epoch, then streams batches in order. The CPU can prefetch effectively because the access pattern is known.
Inference generates its access pattern dynamically, one token at a time. Each forward pass through the model produces a new token, which changes what the next forward pass needs. And the dominant memory consumer in inference isn’t model weights (those are static, loaded once). It’s the KV cache.
The KV Cache Problem
Every transformer-based model maintains a key-value cache: the accumulated attention context from all previous tokens in the sequence. For each new token generated, the model reads the entire KV cache to compute attention, then appends the new token’s key-value pair. The cache grows linearly with sequence length.
The numbers are large and getting larger:
ModelContext LengthKV Cache Size (FP16)NotesLlama 3.1 8B32K tokens~4 GBFits in single GPU HBMLlama 3.1 8B128K tokens~16 GBStill fits, barelyLlama 3.1 70B128K tokens~40 GB (with GQA)Half of an H100’s HBMLlama 3.1 70B128K tokens~320 GB (without GQA)Exceeds any single GPUAny 70B+ model1M tokens~150+ GBMulti-GPU required<br>GQA (Grouped Query Attention) and MLA (Multi-head Latent Attention) compress the KV cache by 4-8x, but the scaling problem remains. At 128K context, 4 concurrent requests on a Llama 70B model need roughly 160 GB of KV cache. That exceeds a single H200’s 141 GB of HBM. At 1M token context (which Claude, Gemini, and GPT-4 all support), the KV cache for a single session can exceed 15 GB with modern optimizations.
GPU HBM is precious. An H100 has 80 GB, an H200 has 141 GB, and a Blackwell B200 has 192 GB. Model weights for a 70B parameter model in FP16 consume ~140 GB alone. There is not enough HBM for both the model and all active KV caches. Something has to spill.
The Tiering Imperative
The industry’s answer is tiered KV cache storage. Hot cache stays in HBM. Warm cache spills to host DRAM. Cold cache goes to NVMe. The memory hierarchy that CXL is reshaping for storage metadata applies equally to inference state:
┌─────────────────────────────────────────────────────┐<br>│ GPU HBM │ ~ns access │ 80-192 GB │<br>│ (active KV) │ TB/s BW │ $$$$$ │<br>├─────────────────────────────────────────────────────┤<br>│ Host DRAM │ ~μs access │ 512 GB - 2 TB │<br>│ (warm KV) │ ~26 GB/s │ $$$ │<br>│ │ (PCIe Gen4) │ │<br>├─────────────────────────────────────────────────────┤<br>│ Local NVMe │ ~100 μs │ 4-60 TB │<br>│ (cold KV) │ 7-14 GB/s │ $$ │<br>├─────────────────────────────────────────────────────┤<br>│ Network Storage │ ~ms access │ Petabytes │<br>│ (shared KV/ckpt) │ variable │ $ │<br>└─────────────────────────────────────────────────────┘<br>The latency differences are brutal. vLLM’s KV offloading connector (v0.11.0+) measures 83.4 GB/s bidirectional transfer between GPU and CPU memory at 2 MB block sizes, yielding a 2-22x reduction in time-to-first-token for single requests. But that’s the best case, the DRAM tier. Moving KV cache from NVMe to GPU crosses both the PCIe bus and the NVMe latency floor, adding 100+ microseconds per...