Speaking to AI Agents like Cavemen Saves 65% of Tokens. We Test.

JetBrains AI

Supercharge your tools with AI-powered features inside many JetBrains products

Follow

Follow:

RSS RSS

Explore More

Agentic AI<br>AI<br>AI Assistant<br>Does Speaking to Agents Like Cavemen Really Save 65% of Tokens? We Test

Denis Shiryaev

A paired A/B benchmark of the token-compression skill Caveman on Claude Code, run on SkillsBench: does it actually save tokens, and does it degrade AI agent output quality?

Advertised saving: 65%. Measured saving: 8.5%.

Output-token saving on real agentic tasks, with the skill forcibly activated. This is the ceiling, not the usual-case result.

Why we ran this

We at JetBrains are investing more and more into proper testing of the tooling around coding agents, and one skill got our attention: “Caveman” . Its pitch is best described in its own dialect:

Skill make agent talk like caveman. Why use many token when few do trick. Filler die. Code, commands stay byte-exact. 65% output token saved. Every reply. Forever. Work with 30+ agents. Many GitHub star.

We think:

Claim cheap to make. Verify expensive. Agent not chat window. Agent output mostly tool call, file edit, code: skill promise not touch those. So we measure two things README not measure: real saving on multi-step agent work, and whether squeezing agent think-out-loud hurt task outcome.

Setup

Harness Harbor 0.17: Docker-sandboxed trials, task-level verifiers, paired runs.Agent Claude Code 2.1.200, headless, bypassPermissions.Model claude-sonnet-5, reasoning effort low (--effort low).Benchmark SkillsBench (benchflow/skillsbench): 86 of 87 tasks. Each task is auto-graded by its own tests on a 0-1 scale, where 1 means solved and partial credit is possible.Arm A no-skill: stock Claude Code.Arm B with-skill-forced: Caveman installed via Harbor --skill plus one instruction line forcing activation: “Use caveman mode…”Pairing Same tasks, same model, same settings, same budget per arm; excluded tasks excluded from both arms.Volume 3 runs, about 240 billed trials, about USD 106 total.

Why “forced” matters: Caveman is user-activated. It triggers on phrases like “caveman mode” or “be brief”. We forced it on in every reply, which means every number below is the skill’s best case. In normal use, where the agent must decide to activate it on its own, the realized saving can only be equal or lower than the roughly 10% ceiling measured here.

Finding 1: the saving is about 8.5%, not 65%

Advertised savings come from chat-style prose answers. Agentic output is different: code, diffs, tool invocations, and exact error strings dominate the token stream, and Caveman correctly leaves all of it verbatim. Only the narration between tool calls gets compressed, and there is not much of it.

Output-token saving vs. baseline

smoke: 10 tasks, k=1<br>re-run: 10 tasks, k=3<br>full: 86 tasks, k=1

-29.5%<br>-6.7%<br>-8.5%<br>small-sample noise<br>headline result, 82 clean pairs

advertised -65%

Output-token saving of the forced-Caveman arm across the three runs. The eye-catching -29.5% from the first small run did not replicate; at scale the saving converges to -8.5% (592k to 542k output tokens over 82 paired tasks). The advertised -65% is off-chart.

Finding 2: no detectable quality degradation

The question we actually cared about: does making the agent terse make it worse? Across 82 paired tasks in the full run, the answer is no: the arms are statistically indistinguishable.

Per-task paired outcomes

64 tied<br>10<br>skill scored higher<br>identical score in both arms<br>skill scored lower

Per-task paired outcomes, full run. Sign test over the 18 non-ties: p = 0.82, far from any significant difference. Average task score was 0.326 for baseline vs. 0.311 for the skill arm, a -0.015 gap on a 0-1 scale.

Average task score per run

no-skill

with-skill-forced

0.5<br>0.25

0.38<br>0.25<br>0.45<br>0.39<br>0.33<br>0.31<br>smoke: k=1<br>10 tasks: k=3<br>86 tasks: k=1<br>looked like a regression<br>gap shrinks<br>statistically flat

Average task score per arm. The scary early gap shrinks as sample size grows: the pattern of noise, not of a real effect. Individual tasks flip freely between passing and failing on repeat attempts in both arms.

Style transfer itself works exactly as designed: forced-arm transcripts are unmistakably caveman, while code artifacts stay untouched and normal.

Finding 3: the cost saving is real but fragile

Cost tracks the roughly 8.5% token saving, so the skill arm should come out roughly 10% cheaper, and per task, it does. But the raw arm totals in our full run showed the skill arm 11.6% more expensive: USD 40.60 vs. USD 36.39. The entire inversion came from a single trial: one dependency-audit task ballooned past the 200k long-context pricing tier in the skill arm and billed USD 8.29 vs. USD 0.33 . In an earlier run the same task threw a USD 3.25 outlier in the baseline arm. It is a property of the task, not the skill.

Outcome

Safe, honest about style, oversold on savings. Forced on, Caveman...