Sol loves to cheat — jumploops

tl;dr

What started as an exploration into building a custom harness, unintentionally led to hacking benchmarks with a model that wants to win at all costs.

Background

I’ve been running a “spec-driven” development flow for the past ~year.

It’s pretty simple.

Before asking an LLM to do something, I first ask it to draft a doc for what it needs to do.

I use this strategy for feature development, greenfield projects, debugging, you name it.

The pattern works for me, but it’s a bit repetitive.

So I decided to automate it.

chum-codex

The idea was straightforward: I’d create a supervisor agent , that would run a “spec-driven process” by delegating to worker subagents who would actually write the docs, do the work, etc.

Note: when trying to do this with vanilla Codex or Claude Code, it would somewhat work, but the default prompts are catered to a user much more so than a “supervisor”

I hypothesized that the supervisor agent need only have the ability to read files and call workers, because that’s what I do.

Rather than rebuild a coding harness for the workers, I looked at Pi, OpenCode, and Codex’s App Server.

I’d been using Codex for quite awhile, so I decided to give app-server a spin. The other options are cool, you should check them out.

Anyhow, the first version worked well enough: the supervisor would size the task, call a worker with e.g. a design request, the worker would spit out a doc, the supervisor would then ask the worker to turn that doc into an implementation spec (split by phase, as appropriate), and then finally ask the worker to actually implement the thing.

Note: this simplified diagram omits the user feedback portions e.g. design doc review

config:<br>sequence:<br>mirrorActors: false<br>sequenceDiagram<br>participant S as Supervisor<br>participant W as Worker

S->>S: Size task<br>S->>W: Design request<br>W-->>S: Design doc

S->>W: Create implementation spec<br>W-->>S: Phased implementation spec

S->>W: Implement<br>W-->>S: Result<br>Woot! I’d saved some time in my development process.

(or did I?)

The Rabbit Hole

Great, it worked; hacky, but working.

Note: this is where I should have stopped

Sitting on my high horse, I surveyed the landscape and thought “wow, everyone should see this!”

What’s the best way to do that? Benchmarks!

What’s the best benchmark to use? Not Terminal Bench!

What benchmark did I dive too deep on? Terminal Bench 2.1!

Terminal Bench

If you’re not familiar with agentic benchmarks, Terminal Bench’s name is telling. It’s a set of tasks that can be accomplished from the terminal, covering a range of one-off tasks from chess to DNA assembly.

Because it’s so simple, it’s probably one of the worst benchmarks to test a spec-driven development flow.

Due to its simple nature, however, it was easy to test against.

I started with a few of the tasks that vanilla Codex w/GPT-5.5 failed at, such as DNA assembly/insert, video extraction/processing, ELF extraction, and protein assembly.

It worked.

These tasks benefited from a “design pass” before implementation, as the doc helped avoid narrowing and circular validation.

The horse I was riding just got a lot taller.

Note: Terminal Bench 1.x/2.x is likely saturated , more on that later.

GPT-5.6?

The published GPT-5.5 benchmark is 83.8% (~74/89 tasks, 5 runs).

chum-codex was hitting 89.9% or ~80/89 tasks.

Excited to share the news of beating Codex, I ran a couple of vanilla Codex benchmarks just to make sure.

For context: this was on June 25th, 2026 and rumors were spreading that GPT-5.6 was imminent.

I ran three vanilla Codex benchmarks… and my heart sank: 88.8%

My harness was just one task ahead of vanilla Codex.

Some tasks were clearly improved, others had regressed.

The next day, GPT-5.6 Sol was announced.

Interestingly, Terminal Bench 2.1 was the only coding-related benchmark they initially shared, showing 88.8% on GPT-5.6 Sol and 91.9% on Sol Ultra.

The new model (sans Ultra) is hitting the same score as the old model (on the latest version of Codex)? Is the harness what matters here?

I reached out to OpenAI, and they confirmed all my request IDs hit GPT-5.5, not GPT-5.6

Sol Ultra spawns parallel subagents to do work, though in my testing it’s quite a bit more token-heavy than most people want/need for the majority of their tasks.

In either case, I was excited to see the new frontier!

Steering

GPT-5.6 is much harder to steer.

Switching from 5.5 to 5.6 made my harness drop in effectiveness. Things that were easy to do before, were now much more difficult.

I traced part of this delta to a change in the base Codex prompt. For GPT-5.5, the prompt is coding-focused and spends a lot of time on “engineering judgment” including frontend guidance, editing constraints, and having “sympathy with the codebase already in front of you.”

Excerpt from GPT-5.5 prompt

The Codex prompt for GPT-5.6 is much different, spending almost zero energy on engineering related specifics. Instead it focuses on...