The Benchmark Behind the Benchmark

← All Posts

A benchmark is two things : the tasks, and the judge that decides whether you succeeded. Everyone treats the judge as an afterthought, but it decides every number you publish, and a weak one makes the whole thing reward-hackable .

So we spent a long time building ours. An aligned judge has to be consistent in four ways:

The same trace twice. Judge one finished run again and get the same score.

Across tasks. Task 90 is held to the same bar as task 3.

Across models and harnesses. Browser Use and Codex get graded the same way.

At the run level. If model A beats model B, that shouldn't flip when you judge it again.

None of that is free. Start with how much of the score is even real.

What moves the score

Two things. Only one of them is the model.

Var(score)=can’t fixσagent2​​​+oursσjudge2​​​<br>Agent noise is the web and the model. A site blocks you. A wrong turn on step 12. You measure it. You can't remove it.

Judge noise is the same finished run, graded twice, coming back with two different verdicts. That one we can kill.

Agent noise

We ran GPT-5.5 five times on the same 106 tasks. Same model. Same harness. Same config.

Each run solved about 89. Never the same 89.

Every column is one task. Every row is one run. Green means it worked.

64 tasks work every time. 2 never work. The 40 in the middle are the whole problem. Same model, same task, passing and failing at random. A misread filter. A wrong click. Giving up one step early.

Run once and you score 85%. Give it five tries and 104 of the 106 tasks work at least once.

So a single run doesn't measure what a model can do. It measures how often it does it.

Judge noise

Now freeze the agent. We took 104 finished runs from that set, the ones we have human labels for, and changed only the model doing the grading.

Same idea: every column a task, every row a judge, all reading identical work.

45% of the benchmark depends on which judge you asked. The most lenient reports 83.7%, the strictest 62.5%. Twenty-one points, wider than the gap between most models people compare.

If the judge moves the score more than the model does, you can't tell whether a change helped.

Building the judge

One prompt with the whole trace

The first judge was a single LLM call. Whole trace in, verdict out.

Fine with one harness. Then we added more. Traces look nothing alike, run hundreds of steps, and compaction eats the part you need.

The evidence is one thing, buried

Task: extract listings under $2,000. On step 34 the agent sets the filter to $20,000. Everything after that is clean, confident, and wrong.

You can't fit that in a prompt. And one pass gives one shot at finding it.

So the judge became an agent. It gets the trace and the workspace and goes hunting, as Codex or Claude Code with read access. It can open the CSV, scroll back to step 34, check the number against the page it came from.

What the judge can see

An agent judge can only verify what it can open. Easy to break, though. Screenshots are heavy. Page text is heavy. Something truncates to fit a budget.

Cut the wrong thing and a true claim becomes unverifiable, which to the judge looks identical to an invented one. It starts calling correct answers hallucinations. Get the evidence contract right before you touch the prompt.

Why we stopped asking for pass/fail

The judge has to output something, and pass or fail is the obvious choice. It's also the wrong one, because most tasks aren't binary.

Real prompts are loose. Check one of the top contributors. Open the trending puzzle. Nobody can verify which one was right, including the judge. And agents fail partially: ask for JSON, get a CSV with all the right data in it. After 400 correct steps that isn't a zero.

One from our set:

Go to spaces and navigate to one of the recommended spaces to view. Check the profile of one of the top contributors in this space and return how many followers they have.

The agent verified 227 followers . But the recommended-spaces page was login-gated, so it used a publicly reachable Space instead.

Five of our eleven judges passed it. Six failed it. Same screenshots, same follower count. They split on whether a substituted Space counts.

So it isn't 0 or 1. Call it a 60. You can argue for 45, and that argument is the point: the score doesn't remove the judgment call, it puts it somewhere you can see.

So we ask for a number instead: what percentage of the requested outcome was delivered, correct and backed by evidence. Here is the same judge on the same traces, run four times, changing only a setting that shouldn't move the answer at all.

20 points versus half a point. The binary judge isn't more wrong, it just amplifies. Every borderline task is a coin flip, and every coin flip becomes a whole task in the final score.

Ensembles

The other lever: run the judge several times and combine the answers.

It works because judge noise is random. Every run is the real score plus...