rtk Claude Code Token Savings: A Skill Trial Benchmark

JetBrains AI

Supercharge your tools with AI-powered features inside many JetBrains products

Follow

Follow:

RSS RSS

Explore More

AI<br>AI Assistant<br>Does "rtk" skill really cut agent tokens by 60–90%? We tested it

Denis Shiryaev

Does "rtk" reduce Claude Code token usage?

Part 2 of a series where we take public “token saving” add-ons for coding agents and run the same paired A/B benchmark against each of them. Part 1 was the caveman skill (advertised −65%, measured −8.5%).

TL;DR: rtk advertised saving: 60–90%. Measured on real agent work: +7.6% more expensive at low reasoning effort (p=0.004), ±0% at high effort. Setup: Claude Code 2.1.201 · claude-sonnet-5 low and high efforts · SkillsBench. Task quality: unchanged in both arms, at both effort levels.

Why we ran this

rtk (“Rust Token Killer”) is a CLI proxy with a simple, appealing pitch: your agent runs git status, rtk intercepts it, runs the real command, and hands the model a compressed version of the output — * master / M a.txt / ?? b.txt instead of eleven lines of porcelain. A Claude Code PreToolUse hook rewrites eligible shell commands transparently, so the model doesn’t even have to know rtk exists. The README promises 60–90% less token consumption and walks through a 30-minute session where 118k tokens of command output become 24k.

The compression itself is real and often tasteful. Here is rtk on a live repo, captured from our test container:

# git status # rtk git status<br>On branch master * master<br>Changes not staged for commit: M a.txt<br>(use "git add ..." to update…) ?? b.txt<br>modified: a.txt<br>Untracked files:<br>(use "git add ..." to include…)<br>b.txt<br>no changes added to commit …

# python -m pytest (19 lines) # rtk pytest<br>…full pytest output… Pytest: 2 passed, 1 failed<br>Failures:<br>1. [FAIL] test_fail<br>test_demo.py:3: in test_fail<br>E AssertionError: one is not two

We liked the idea enough to test it properly. Two questions the README doesn’t answer:

First, how much of a real agent session is Bash output at all? The savings table assumes the agent shells out for everything. But Claude Code reads files with its built-in Read tool, searches with Grep, and both bypass the Bash hook completely (rtk’s docs acknowledge this). Whatever those tools carry, rtk can never touch.

Second, does compression cost correctness? A filter that summarizes test output is making an editorial judgment about what the model needs. If it drops the one line that mattered, the agent re-runs commands, reads files raw, or, worse, declares victory on a failing build. Token savings that come with a quality tax are not savings.

Setup

Harness Harbor 0.18 – Docker sandboxes, task verifiers, paired runsAgent Claude Code 2.1.201, headless, bypassPermissions, pinned in both armsModel claude-sonnet-5 – full run twice: at low and at high reasoning effortBenchmark SkillsBench, 86 of 87 tasks, auto-graded 0–1 with partial creditArm A stock Claude CodeArm B rtk v0.43.0 exactly as rtk init -g ships it: binary + PreToolUse hook + RTK.mdVolume 4 paired runs (10-task smoke, same 10 at k=3, full 86 at low effort, full 86 at high effort) total of 425 billed trials, ≈USD 320 (Harbor-recorded USD 317 plus reconstructed subagent spend)

Because the hook rewrites every eligible Bash call mechanically, arm B measures rtk’s as-shipped ceiling: no “did the model remember to use it” gap to argue about. Every with-rtk trial also persists rtk’s own audit log and analytics database, as proof the treatment actually fired.

Finding 1: Most agent bytes never touch the hook

Before spending anything we replayed 83 existing baseline transcripts (same model, same benchmark) and asked: if rtk had been installed, what could it even have touched?

Two structural reasons. First, Claude Code reads files with its built-in Read/Grep tools, which bypass the Bash hook entirely; rtk’s own README admits this in a footnote. Second, half of what agents run in a shell is python3 … and other uncovered commands, and a sixth uses pipes-to-files, heredocs and substitutions that rtk deliberately refuses to rewrite. What’s left, 33% of Bash calls, carries just under 20% of tool-result chars; and tool results are themselves only a slice of what a session bills as input, because the same context is re-read on every turn. Squeeze rtk’s whole share by 70% and the cap works out to ≈3% of input tokens . This number cost nothing to compute, and it predicted the outcome.

Finding 2: No token savings; but a small, significant cost increase

We ran the ladder the caveman eval taught us to run. The k=1 smoke on ten deliberately Bash-heavy tasks (rtk’s best case) showed the rtk arm a median +35% more expensive. Alarming, until you know that identical attempts of the same task in the same arm differ by a median 22% in cost anyway. At k=3 most of the scare evaporated into noise (Wilcoxon p≈0.65), exactly as a k=1 mirage should.

Then the full 86 tasks gave the noise-resistant answer, and...