Half the tokens for the same 36 tool calls: optimizing a coding agent | app.nz Blog

BlogListen to this article<br>On-device voice<br>Uses the voice built into your browser; no article text leaves this page.

Audio narration is not supported by this browser.

app agent is the coding agent in the app.nz CLI: the tool loop runs in your terminal, against the app.nz model gateway. We benchmarked it against Codex a few days ago and it came out cheaper. Then we went looking for the rest.<br>This is what that pass found, including the change that made things worse and the measurement that caught it.<br>The one fact that matters<br>A chat completion API is stateless. Every turn re-sends the entire conversation: system prompt, tool schemas, every file the agent has read, every command's output. So the cost of a run is roughly<br>transcript size x turns<br>Not "transcript size". Not "turns". The product. A file read on turn 3 of a 40-turn run is paid for 37 more times. That single fact is where every real optimization in an agent comes from, and it is also why intuitions from ordinary programming mislead: the expensive thing is not the work, it is the residue the work leaves behind in the transcript.<br>Why the benchmark could not guide this<br>Our existing harness runs two fixture repos with failing test suites through the agent and scores only whether the suite goes green without the tests being edited. It is the right way to measure outcomes, and it is nearly useless for attributing cost:<br>The model chooses how many turns to take. Same agent, same task, same<br>prompt: 19k tokens one run, 78k the next.<br>Those fixtures are solved in three or four turns, so nothing about long-run<br>context management is even exercised.<br>Every run costs real money and real minutes, so you cannot iterate on a design<br>by re-running it.<br>So we built the missing half: a harness with no model in it at all .<br>Weighing the wire<br>TestLongRunWireCost scripts a fixed sequence of 36 tool calls — the shape of a real exploration-and-repair session, including the parts nobody designs for:<br>Copycycle := []model.ToolCall{<br>{Name: "search", Arguments: `{"pattern":"line 1","path":"handler.go"}`},<br>{Name: "read_file", Arguments: `{"path":"handler.go"}`},<br>{Name: "read_file", Arguments: `{"path":"store.go"}`},<br>{Name: "shell", Arguments: `{"command":"cat test.log"}`},<br>{Name: "read_file", Arguments: `{"path":"handler.go"}`}, // re-read<br>{Name: "search", Arguments: `{"pattern":"line 1","path":"handler.go"}`}, // re-search<br>{Name: "shell", Arguments: `{"command":"cat bundle.min.js"}`},<br>{Name: "read_file", Arguments: `{"path":"router.go"}`},<br>{Name: "shell", Arguments: `{"command":"cat test.log"}`}, // re-run<br>A fake gateway answers each request with the next scripted call and weighs the request body . Four cycles, 37 turns, one number out the other end: how many bytes the harness put on the wire. Every tool call is deterministic, so the number is identical run to run — which turns out to matter more than we expected.<br>Copygo test ./internal/agent -run TestLongRunWireCost -v<br>Baseline, before any of this work:<br>wire cost: 4157949 bytes (~1039487 tokens) over 37 turns + 1 compaction call, 112377 bytes/turn<br>A million tokens for 36 tool calls against four small files. That is the real target, and no benchmark of three-turn tasks was ever going to show it to us.<br>Round one: the obvious things<br>Terser tool schemas. The whole tool block ships with every request, so a sentence in a tool description is paid once per turn, not once per session — it is the most expensive prose in the program. Most of it was restating parameter names: "path": "File path.", "new_string": "Replacement text.". Deleting a description entirely rather than serialising an empty one, and cutting the rest to what actually changes model behaviour, took the block from 5,193 to 4,556 bytes — about 160 tokens a turn. A test now fails the build if it grows back:<br>Copytool schemas are 5104 bytes (~1276 tokens per turn), over the 5000 byte budget;<br>shorten a description or drop one that restates a parameter name<br>Workspace-relative paths. ripgrep was being run with an absolute target, so every one of up to 80 match lines carried the same 60-character prefix. Eighty copies of a useless string, re-sent every later turn. Same for the wrote /abs/path/… confirmations. This one also made the terminal output legible, which is how we noticed it.<br>Per-line truncation. Output was capped in total bytes, so a single minified bundle or base64 blob could consume the whole budget and push out the lines that mattered. Now over-long lines are truncated first — every line survives, only the pathological ones get shortened.<br>Collapse repeated tool results. Agents re-run git status, re-grep the same pattern, and re-run the same test command constantly. When a result is byte-identical to one already in the transcript, we replace it with a pointer. That is not hiding information: "this has not changed since you looked" is information.<br>Count the fixed cost in the budget. The...