Forget the GPU Shortage: The Real AI Bottleneck Was Diagnosed in 2007

SubscribeSign in

Forget the GPU Shortage: The Real AI Bottleneck Was Diagnosed in 2007<br>Why the constraint isn’t compute - it’s memory, and a 2007 paper already mapped it

Salar<br>Jul 05, 2026

Share

This past week, AMD and Qualcomm both announced new memory-packaging approaches within days of each other - AMD bolting LPDDR5X onto its Versal Premium Gen 2 SoCs, Qualcomm previewing “High Bandwidth Compute” for future inference accelerators. A few weeks earlier, Nvidia locked down SK Hynix HBM supply through 2030, which is less a procurement deal than a hedge against the physics problem that actually limits AI. H100 lead times are sitting at 36–52 weeks, and data centers are on track to consume 70% of all memory chips produced in 2026. Everyone is panicking about a GPU shortage. The GPUs are fine. The memory is the bottleneck - and that was diagnosed in 2007.<br>The 2007 paper

I read Ulrich Drepper’s “What Every Programmer Should Know About Memory” the way most engineers do - late, and a little embarrassed I’d waited. It came out in 2007, the year the iPhone launched, when “AI” meant chess engines and the idea of training a trillion-parameter model sat somewhere between science fiction and heresy. Drepper wasn’t thinking about any of that. He was thinking about cache lines.<br>His thesis was straightforward: CPUs were getting exponentially faster, and memory was not. The gap between what the processor could do and what it could be fed was widening every year. He walked through the whole hierarchy - hard drive, RAM, L1/L2 cache - and pointed out that the distance between those layers, measured in nanoseconds, was the only number that actually mattered. A program that kept forcing the CPU to fetch from main RAM instead of its on-die cache would leave a fast processor idle most of the time, waiting for bytes to arrive over the bus. He called it the Memory Wall, and his message to programmers was that they could no longer treat memory as a flat, uniform bucket. Where your data lived determined how fast your code ran.<br>The same wall, one stack up

The names have changed but the shape is identical. In 2007 the hierarchy was hard drive → RAM → L1/L2 cache. For AI today it’s HBM → GPU on-chip SRAM. HBM is the new main RAM. The GPU’s SRAM is the new L1 cache. Every token a large model generates requires moving gigabytes of weights across that gap, over and over, millions of times a second.<br>This is why the news this week reads the way it does. The leaps in raw compute - TFLOPS - are outpacing the leaps in memory bandwidth by an order of magnitude on chips like Blackwell. The matrix multiplies everyone obsesses over are cheap; moving the data to do them is what’s expensive, both in time and in power. Nvidia isn’t locking up HBM through 2030 because it’s greedy - it’s doing it because without that memory, Blackwell’s compute cores sit idle. AMD and Qualcomm aren’t reinventing packaging for fun; they’re trying to buy back bandwidth the logic side already outpaced. The “compute shortage” that dominated earnings calls this spring was, almost entirely, a memory-bandwidth shortage wearing a GPU label.<br>The fix was already in the paper

The people who actually solved this solved it the way Drepper would have predicted. FlashAttention, one of the biggest breakthroughs in making large models viable, didn’t shrink the math or simplify the architecture. It rewrote the attention computation so that data stays inside the GPU’s on-chip SRAM as long as possible and minimizes round-trips back to HBM. That’s locality of reference - the exact concept Drepper was begging programmers to understand in 2007 - ported to a transformer nineteen years later.<br>The small-language-model wave is the same concession from the other direction. Llama, Phi, Mistral, the open-weight ecosystem running on whatever commodity hardware people already have - if you can’t move a 70-billion-parameter model across the bus fast enough, shrink the model until it fits inside the cache hierarchy. As a working software engineer, the reason Sonnet and Opus get the job done for me isn’t that they’re smarter than the frontier. It’s that the frontier is too memory-hungry to deploy at the scale the market actually wants.<br>The physics of it

Drepper ended the essay with a line that aged well: the physics of hardware dictate the limits of software. We treat code as abstract, unshackled from copper and silicon. This past week’s news is what it looks like when that pretense runs out of road. The companies that win the next decade of AI won’t be the ones with the most FLOPS — they’ll be the ones who finally internalize what a Linux kernel hacker tried to tell everyone in 2007. The chips changed. The wall didn’t.

Subscribe

Share

Discussion about this post<br>CommentsRestacks

TopLatest

No posts

Ready for more?

Subscribe

© 2026 Salar · Privacy ∙ Terms ∙ Collection notice<br>Start your SubstackGet the app<br>Substack is...