Your LLM re-reads the same text a thousand times — Reame

Your LLM re-reads the same text a thousand times

I spent two weeks trying to make CPU inference faster. The thing<br>that finally worked was not in the engine.

Measured on a free Oracle ARM box — 4 cores, €0/month. Everything<br>below is reproducible; the scripts are linked at the end.

I maintain a CPU-first inference server built on llama.cpp. Not as a fallback<br>for a missing GPU — as the target. The hardware I care about is the free tier:<br>two to four ARM cores, no accelerator, no budget.

The bottleneck there is prefill. Before a model writes a single token it has<br>to read your entire prompt, and on a document of any size that read dominates<br>everything else. So I went looking for speed in the obvious places.

Three things that didn't work

I bumped llama.cpp forward by a year of upstream kernel work, expecting the<br>ARM optimizations to show up. On this quantization and this CPU: 0% .

I tried a coarser quantization. Q4_0 is 37% faster at prefill and<br>19% faster at decode — and it dropped 5 facts out of 20 on my extraction test.<br>Rejected.

I halved the number of active experts during prefill on a mixture-of-experts<br>model. 44% faster, and it silently corrupts the cache: a KV cache built with 4<br>experts and read back with 8 scores 11/20 against a 14/20 control. The damage<br>is baked into the cached representation, not just the output. That one was<br>worth knowing about.

What worked was the text

The same page. The same model. The same question. Rewritten as<br>label: value, one fact per line, with conditions kept attached to<br>what they qualify:

Input formTokensTime to first tokenFact exam

Original prose213740.5 s19/20<br>Compressed118521.0 s20/20<br>Fact sheet 405 6.2 s 20/20

Six and a half times faster, and more accurate. That second part is<br>what I didn't expect, and it's the reason this is worth writing down. Fewer<br>tokens should mean less information and worse answers. It didn't.

The model saw the right number and answered wrong

To understand why accuracy went up, I dumped the post-softmax attention while<br>the model answered a question about a page listing four similar prices — one per<br>service, all in the same block.

Document formAttention on the right priceOn the wrong oneAnswer

Prose, dense 3B7.0%0.6%correct<br>Prose, small MoE1.9%1.7%wrong<br>label: value, small MoE2.9%0.4%correct

1.9% against 1.7% is a coin flip. The model was never blind to the number —<br>it just couldn't bind it to the question. Prose puts the price and the<br>service in the same sentence as everything else; label: value makes<br>the binding structural instead of semantic, and the ratio goes from 1.1:1 to<br>7:1.

The practical consequence: when a small model gets a fact wrong, "use a<br>bigger model" is one answer, but "write the document differently" is often<br>cheaper and works better.

Then I went looking for where the bytes go

Having stopped guessing, I counted what a model actually reads per generated<br>token. Two things came out that I have not seen written down anywhere, both<br>invisible on a GPU.

1. The output head is a quarter of your bandwidth

In many small models the embedding matrix is tied: it doubles as the<br>output projection and is read in full for every token generated, to score all<br>151,936 vocabulary entries.

Read per generated tokenMBShare

Output head (tied embedding)127.629.4%<br>Active experts (8 of 232)176.240.6%<br>Attention, dense layers, norms129.729.9%

Nearly a third of the bandwidth per token goes to scoring words in languages<br>this deployment will never emit. My domain — Italian, my own code, technical<br>English — uses 9,314 of those 151,936 entries.

Trimming the vocabulary to 32k turns out to be safe by construction, for two<br>reasons worth stating precisely. The tokenizer is byte-level and all 256<br>byte-characters are kept, so no text becomes unrepresentable — the worst case is<br>that a trimmed word costs an extra token. And the embedding is Q6_K with rows<br>spanning a whole number of quantization blocks, so whole rows drop out without<br>ever splitting a block: the surviving weights are bit-for-bit<br>identical . No requantization, no numerical loss.

Measured on the free box, three independent runs, under 2% spread:<br>+17.8% decode , from 46.7 to 55.0 tokens/second. The 20-question<br>fact exam scores the same before and after — 13/13 critical facts both times.<br>The cost, measured on held-out text, is 1.9% more tokens for the same document,<br>so this pays off most when the input is already short.

The theoretical ceiling from bandwidth alone was +30%. I got +17.8%. The gap<br>is time that isn't weight reading, and I'd rather publish the measured number<br>than the calculated one.

2. Weight repacking costs 4.2 GB nobody mentions

My server showed 9,825 MB resident for a 5.37 GB model. I assumed a leak.<br>It isn't:

load_tensors: CPU_Mapped model buffer size = 5068.51 MiB

load_tensors: CPU_REPACK model buffer size = 4254.45 MiB

llama.cpp repacks quantized weights at load time into a layout its NEON<br>kernels read...