We improved 15 LLMs at coding in one afternoon. Only the harness changed. — Stencil
In fact only the edit tool changed. That's it.<br>+15ptsavg over patch, 16 models<br>10×Grok Code Fast 1<br>−61%output tokens, best case<br>$0training compute
SORTΔ PATCHΔ REPLACEHASHLINE % PATCH REPLACE HASHLINE V2<br>MODEL0255075100Δ PATCHΔ REPLTOK<br>01Grok Code Fast 1+64.6+4.6−49%<br>02MiniMax M2.1+41.7+10−42%<br>03Devstral Medium+40.5+3.8—<br>04GLM-4.5 Air+27+0.4−17%<br>05GLM-4.7+23.3+8.3−32%<br>06GPT-5.1 Codex Mini+20.3+4.2—<br>07Grok-4.1 Fast+19.2+2.5−20%<br>08Grok 4 Fast+17.4−0.4−61%<br>09Qwen Turbo+16.6−1.7—<br>10Claude Sonnet 4.5+14.4+3.3−24%<br>11Claude Haiku 4.5+13+11.3−22%<br>12Gemini 2.5 Flash Lite+10±0—<br>13Kimi K2.5+10+5−26%<br>14Gemini 3 Flash+8+11.3−21%<br>15GPT-5.2 Codex+4.6−0.4+26%<br>16DeepSeek V3.2−5−8.3+20%
Pass rate per model per edit format · 3 runs × 180 tasks, fresh session each ·<br>hashline beats patch in 14/16 models; the v2 revision improves further in 12/16 —<br>largest gain GPT-5.1 Codex Mini, 60.0% → 77.5%<br>The Wrong Question<br>The conversation right now is almost entirely about which model is best at coding, GPT-5.3 or Opus. Gemini vs whatever dropped this week. This framing is increasingly misleading because it treats the model as the only variable that matters, when in reality one of the bottlenecks is something much more mundane: the harness.<br>Not only is it where you capture the first impression of the user (is it uncontrollably scrolling, or smooth as butter?), it is also the source of every input token, and the interface between their output and every change made to your workspace.<br>Why bother, you ask? Opus may be a great model, but Claude Code to this day leaks raw JSONL from sub-agent outputs, wasting hundreds of thousands of tokens. In an open harness, we get to just fix that: subagents output structured data now.<br>Tool schemas, error messages, state management, everything between "the model knows what to change" and "the issue is resolved." This is where most failures happen in practice.<br>Being model agnostic, it is a great testing ground, as the model is but a parameter. The real variable is the harness, which you have unimaginable control over.<br>Anyhow — about that one variable we changed yesterday.<br>Edit Tool!<br>Before we explain what we built, it's worth understanding the state of the art.<br>Codex uses apply_patch : It takes a string as input, which is essentially an OpenAI-flavored diff, and instead of relying on a structured schema, the harness just expects this blob to follow a strict set of rules. Since OpenAI folks are without a doubt smart, the token selection process is almost certainly biased to fit this structure at the LLM gateway for the Codex variants of GPT, similar to how other constraints like JSON schemas or required tool calls work.<br>But give this to any other model, completely unaware of it? Patch failures go through the roof. Grok 4's patch failure rate in our benchmark was 50.7% , GLM-4.7's was 46.2% . These aren't bad models — they just don't speak the language.<br>Claude Code (and most others) use str_replace : find the exact old text, swap in the new text. Very simple to think about. But the model must reproduce every character perfectly, including whitespace and indentation. Multiple matches? Rejected. The "String to replace not found in file" error is so common it has its own GitHub issues megathread (+27 other issues). Not exactly optimal. Gemini does essentially the same thing plus some fuzzy whitespace matching.<br>Cursor trained a separate neural network : a fine-tuned 70B model whose entire job is to take a draft edit and merge it into the file correctly. The harness problem is so hard that one of the most well-funded AI companies decided to throw another model at it, and even then they mention in their own blog post that "fully rewriting the full file outperforms aider-like diffs for files under 400 lines."<br>Aider's own benchmarks show that format choice alone swung GPT-4 Turbo from 26% to 59%, but GPT-3.5 scored only 19% with the same format because it couldn't reliably produce valid diffs. The format matters as much as the model.<br>The Diff-XYZ benchmark from JetBrains confirmed it systematically: no single edit format dominates across models and use cases. EDIT-Bench found that only one model achieves over 60% pass@1 on realistic editing tasks.<br>As you can see, there is no real consensus on the "best solution" to the simple "how do you change things" problem. Our take: none of these tools give the model a stable, verifiable identifier for the lines it wants to change without wasting tremendous amounts of context and depending on perfect recall. They all rely on the model reproducing content it already saw. When it can't — and it often can't — the user blames the model.<br>Hashline!<br>Now bear with us here. What if, when the model reads a file, or greps for something, every line comes back tagged with a 2-3 character content hash:<br>hello.js — read<br>1:a3|function hello() {<br>2:f1| return "world";<br>3:0e|}<br>When the model edits, it references...