Caveman Promises 65% Fewer Tokens. My Bill Didn't Move. — Ernie

Let's talk

← Back to articlesEN/RU

Caveman is a Claude Code plugin with 97,000 GitHub stars — 86,000 when I ran these benchmarks.<br>It saves you tokens by making the AI drop the filler and grunt. Fewer words out, smaller bill.<br>Obvious, right?

I tried it. The AI really did get terser — on five of the seven jobs I ran; on the other two it got<br>wordier. My bill didn't budge. Here's why.

Not all tokens cost the same

You pay for three kinds of token, at very different prices. On Sonnet, per million:

output — the words the model writes back — $15

input — the text you send — $3

cache-reads — context it has already seen — $0.30

So one word the model writes costs as much as fifty words it re-reads.

And it re-reads a lot. Every turn, the model goes back over the whole conversation — your files, the<br>test output, everything said so far — then adds a few words at the end. Those few words are about<br>0.6% of the tokens in a session. The other 99% is re-reading.

Token economics<br>Best case: 0.6% of tokens

If the model wrote nothing at all<br>In tokensIn dollars

0.6%<br>The most a skill can save by making answers shorter

OutputEverything else (mostly cache-read)<br>Now count it in dollars →

Few words, but each one fifty times dearer — so in money they come to about 20% of your bill<br>(the arithmetic is in the appendix).

That 20% is a ceiling , not a result. It's the whole slice a skill can touch by making the<br>model's words shorter — the other 80% is re-reading, and no amount of terseness makes that<br>smaller.

65% of a fifth is 13%

Take that ceiling and multiply it by how much shorter the answers actually get:

Answers get 100% shorter — the model says nothing at all → bill drops 20% . The ceiling,<br>and impossible: you'd get no answers.

Answers get 65% shorter , which is what the<br>README advertises<br>(pinned)<br>→ 65% of 20% = 13% off the bill . The best you could hope for if the claim were true.

Answers get 6% shorter , which is what I measured on average → 6% of 20% = 1% off the<br>bill . What you actually get, and it's inside the noise.

20% is the ceiling. 13% and 1% are what you'd actually save. Two different things, and the<br>label quotes neither.

Thirteen percent isn't nothing — I'd install a tool that reliably did that. But nothing I tested<br>got near it. (On a subscription your "bill" is a quota, but Anthropic says<br>limits are cost-based —<br>same math.)

What 140 runs showed

Seven coding jobs — a one-line helper up to a multi-file refactor — each done ten times: five<br>with the skill, five without — 70 runs, and 140 counting the second skill I checked in the<br>appendix. Ten times each, because the same prompt costs a different amount every time you run it. Each run's real dollar cost was metered with<br>vigiles , an open harness I built, and every run had to<br>pass a correctness check — no "savings" from broken answers. You can<br>run the whole thing yourself, on<br>your own subscription.

The skill was unmistakably live — its SessionStart hook fires from the first message, the answers<br>telegraphic. Yet output shrank by about 6% on average — not 65% — and that average hides the<br>real result: it cut output hard on two jobs (−31% and −28%), and made it longer on two others<br>(+54%, +21%). The bill came out 1% cheaper, less than the same task swings between two runs<br>with no skill at all. The honest reading of both numbers is no change I could see, not a 1%<br>saving off a 6% trim.

Could seven jobs miss a real saving? A big one, no; a small one, sure — but the argument never<br>rested on the measurement. The ceiling is arithmetic — prices times shares — and you can redo it<br>without my benchmark: 65% of a fifth is 13%, whatever my runs did. The runs answer the one thing<br>arithmetic can't: does the skill beat its own ceiling in practice? It doesn't.

I'm not the first to check the label. Max Taylor<br>benchmarked caveman against the two words "be brief"<br>and found it didn't beat that boring default on quality or on tokens. JetBrains<br>ran one study over 86 real coding tasks<br>with the skill forced on — their words: an upper bound, not a typical result. Output fell 8.5%<br>against the advertised 65% — about 1.7% of a bill, same multiplication. Cost: about 10% cheaper<br>on average, and in their words "routinely erased by single-trial variance" — run it once<br>and the noise beats the effect. That study also shows how badly single runs lie: their ten-task<br>smoke test, run once, said −29.5% ; the same ten tasks at three runs each said −6.7% .<br>Nothing changed but the repeats. Theirs ran three trials a condition; mine ran five.

Where the 65% came from

Not a lie — a measurement of the wrong thing. The headline 65% was self-measured on 10 one-shot<br>prompts (their own range: 22–87%). One prompt, one reply, no files, no tools, no re-reading — the<br>one setting where the reply basically is the whole session, with no re-reading for it to get<br>lost in. Trim it there and you trim almost the whole token count.

Real coding is the...