Running Gemma 4 26B at 5 tokens/sec on a 13-year-old Xeon with no GPU | Neomind

There’s a server in my basement that has no business running a modern language model. It’s a repurposed HP StoreVirtual storage box, roughly thirteen years old, two Ivy Bridge Xeons, no GPU. It was built to hold disks, not do math. As of this week it runs Google’s Gemma 4, a 26-billion-parameter open-weights mixture-of-experts model, at about five tokens per second. Reading speed.

HardwareRepurposed HP StoreVirtual: dual Xeon E5-2690 v2 (Ivy Bridge, 2013), DDR3, no GPU<br>Instruction setsAVX1 only — no AVX2, no FMA3<br>ModelGemma 4 26B-A4B (MoE), Q8_0<br>Decode~5.2 tokens/sec<br>Prompt eval~16 tokens/sec<br>Cost of the boxunder $300

Anybody can rent a GPU. It’s harder to take a modern MoE model and a dead enterprise box and make them meet in the middle, and that gap is the whole reason I’m writing this up. “Good with AI” has quietly come to mean “pays for a subscription.” I think the real skill is different: knowing a model well enough to point it at a problem nobody packaged for you, and telling whether the answer it hands back is actually correct. So rather than claim we’re good at this, here’s a worked example, on hardware that had no business cooperating.

The post that started it

A couple of weeks ago a piece called “A 10 year old Xeon is all you need” made the rounds on Hacker News. The author runs Gemma 4 on a single 2016 Xeon with no GPU and 128 GB of slow DDR3, using ik_llama.cpp and about 25 carefully chosen flags. It’s a great read, and it leans on every trick in the modern inference playbook: speculative decoding, CPU-aware mixture-of-experts routing, flash attention ported to the CPU, run-time weight repacking. Real engineering.

“I have a Xeon too,” I thought. Several, in fact. So I tried it. It didn’t run.

What an AI agent is actually good for

The build died on startup. I handed the failure to Claude and asked what was wrong. The answer came back fast and specific. The author’s 2016 chip is a Broadwell part. Mine are Ivy Bridge, the generation Intel calls “v2.” The fast kernels in that fork assume AVX2 and FMA3, instruction sets that didn’t ship until Haswell, the “v3” generation, in 2014. My CPUs are older than the instructions the code was written against. The optimized paths weren’t there to execute.

So I asked the obvious follow-up: can we make it run anyway? I’d already taken a first swing with a free model that got close but couldn’t land it. Claude picked up that half-finished approach, agreed it was the right one, and finished it off, reworking the hot paths so they fall back cleanly on a pre-AVX2 chip instead of reaching for instructions that aren’t there.

This is the part I care about. This didn’t come from typing “fix it” once and getting a working patch back. Somebody had to read another person’s performance-critical C++, work out why a kernel wasn’t valid on this particular microarchitecture, and route around it without throwing away the optimizations that made the fork worth using. Claude did that work. My job was narrower: run the right experiments and recognize when the output was finally correct. I came away impressed.

The result

Gemma 4’s 26B mixture-of-experts model now generates text at reading speed on hardware that was retired before the model’s architecture existed. The original write-up never published a tokens-per-second figure, just “reading speed,” so here’s the concrete one: about five tokens a second on thirteen-year-old silicon, for borderline free.

Proof it runs: Gemma 4 26B answering on the basement box, CPU-only.

The patch is up as ikawrakow/ik_llama.cpp#2138 if you want the exact diff — still open and awaiting maintainer review as I write this, so run it from the branch for now. The hope is that anyone else sitting on ancient enterprise iron can keep a local model around: a fallback for when the paid APIs are down, or a cheap way to grind through slow batch jobs when paying per token doesn’t make sense.

For the people who want the actual bug

Full disclosure before I go further. I’m not a C++ programmer. I can read a stack trace and I know my way around a build system, but I did not hand-write kernel fallbacks for a quantized matmul engine, and I won’t pretend I did. What I did was drive. I ran the experiments, read the output, asked the next question, and knew what “correct” had to look like. The diagnosis and the patch came from the Claude instance running on the server itself. I asked it to write up what it fixed, and the rest of this section is that summary, lightly edited. If you came here from Hacker News for the real teardown, this part’s for you.

What was actually broken

The engine we needed was ik_llama.cpp, ikawrakow’s fork of llama.cpp that adds the optimizations Gemma 4’s MoE inference depends on. It assumes AVX2 as its floor. The Xeon E5-2690 v2 in this box has AVX1 but not AVX2. Turn GGML_USE_IQK_MULMAT off at build time and most of the codebase respects it: the...