I Tested Six Ways to Save $$$ on 5.6 Sol. The only one that worked was using Terra — Stet

I Tested Six Ways to Save $$$ on 5.6 Sol. The only one that worked was using Terra<br>July 20, 2026 · Updated July 22, 2026<br>TL;DR

I compared six approaches meant to save tokens (Caveman, Ponytail, RTK, Context Mode, Mandarin prompts, and 5.6 Terra xhigh) with a baseline across ten real tasks. I repeated all seven arms twice for a total of 140 agent runs.

On the recorded workload of ten tasks, none of the six reduced total tokens in both runs.

Averaged per task, Caveman and the cheaper Terra model looked mildly cheaper both times, but the differences were small enough that run-to-run randomness could explain them.

Context Mode and Mandarin used far more tokens no matter how I counted.

The trajectories suggest changes in agent behavior: search, delegation, validation, and patch scope. Several local savings coincided with code review losses, patches that were not equivalent, and lower robustness.

The only repeated drop in recorded cost came from switching to the cheaper 5.6 Terra xhigh model: −49% in dollars while the recorded workload used slightly more tokens.

An agent is not a file you can compress

Tools for saving tokens make the same pitch: lower usage without significant tradeoffs. This treats the agent like a file. Shrink the dialogue, filter the shell output, index the context, and the total gets smaller.

But an agent is not a file. It is a system that decides what work to do next, and every one of these interventions changes those decisions. An agent that receives compressed command output can respond by running more commands. An agent told to write less code gets to decide which requirements count as optional. Nothing about a smaller local surface guarantees a smaller trajectory, and the trajectory is what you pay for.

So that is the question this study asks: what does the agent do next, after its context is compressed?

What "saving tokens" means

"Did it save?" hides two separate choices: what you count (tokens or dollars) and how you summarize the ten tasks. I report every result both ways:

The recorded workload : add up everything the ten tasks actually consumed and compare totals. This is closest to the real bill for this exact batch, but one or two enormous tasks can dominate the number.

The typical task : pair the treatment with the baseline on each task, take the percentage change on each pair, and average those changes (geometrically). This stops the biggest tasks from deciding the answer, but it no longer adds up to the batch total.

Both views exist for tokens and for cost, so the tables below carry all four columns. Cost adds one wrinkle of its own: dollars follow the price structure, including cache discounts, so cost can move differently from tokens when the mix of cached input, fresh input, output, model, or reasoning effort changes.

Caveman shows why the distinction matters. Its workload totals moved +7% and −11%, while its paired point estimates were −5% and −9%. Terra's workload totals rose 6% and 5%, while its paired point estimates were −23% and −12%. Those four are descriptive point estimates, and all four sit in the noise band: each paired bootstrap interval includes zero. Caveman's −9% estimate carries a 95% interval from −24% to +6%, and Terra's −23% carries one from −62% to +38%. The scale of the noise is visible in the baseline itself, which cost $6.93 per task in run 1 and $9.00 in run 2, a 30% swing with the same tasks and the same configuration.

Methodology

The tasks are ten real merged changes from the production repo of Stet, the local eval harness I'm building. Stet replayed these tasks and graded each attempt on more than whether its tests passed or failed. I ran a baseline (5.6 Sol at medium effort) and six treatments, all with the Codex CLI:

Caveman : a terse interaction prompt pack that strips conversational filler and compresses the agent's dialogue (your agent is now a Neanderthal).

Ponytail : an output minimization instruction set that guides the agent to act like a lazy senior engineer. It pairs "The best code is the code never written" with a YAGNI (you aren't gonna need it) ladder to climb before writing anything and a cap on validation work.

RTK : a wrapper that filters and compresses shell command output before it reaches the model.

Context Mode : batches commands, indexes large results, and returns only the sections judged relevant.

Mandarin mode : the claim that Chinese's denser tokenization saves tokens (中文最好的吗？).

5.6 Terra at xhigh : the simplest way to save money is to use a cheaper model.

Seven arms ran twice: 14 arm runs × 10 tasks = 140 agent runs. The repetitions used the same tasks, so they measure consistency rather than giving us 20 independent tasks.

Quality included tests, semantic equivalence, code review, deterministic footprint risk, and eight LLM graders scoring on a 0–4 scale across several rubrics meant to capture software...