Why Most Evals Are Bad - Boolean

Evals are an extremely important part of building, evaluating and improving AI<br>models. A standard model training cycle works something like this:

Determine an important capability you'd like your model to have

Build an extremely high-quality eval that measures progress towards that<br>capability

Try a lot of different training interventions and keep what moves the<br>needle on the evals

When training the model, you are going to spend a lot of compute and time<br>experimenting with different approaches to improve on the eval. It would be<br>very upsetting to learn that the evals you were optimizing on were broken<br>after you had invested a lot of resources into it.

Yet most evals being built are bad.<br>FrontierBench, the next<br>iteration of Terminal-Bench, recently wrapped up its call for tasks for the<br>initial release. The project received over 500 task submissions but over 400<br>of those submissions were rejected and only 74 have been accepted for the<br>first version of the benchmark. I was one of the task reviewers on this<br>project and saw a few patterns first-hand that led to rejections.

What makes an eval bad?

There are many ways in which evals can stop providing signal on capabilities<br>and become bad - these are some of the most common themes I have seen:

Weak verification

Models can fail at a task due to capability gaps but still be rewarded<br>because of a verifier that's not configured correctly. Designing a verifier<br>that gives the reward if and only if the intended capability is leveraged is<br>much harder than it sounds.

For example, the<br>configure-git-webserver<br>task in Terminal-Bench 2.1 provides this instruction:

Configure a git server so that I can run on my computer<br>git clone user@server:/git/server<br>echo "hello world" > hello.html<br>git add hello.html<br>git commit -m "add hello.html"<br>git push origin master<br>And have this data then be pushed to a webserver running on port 8080 so if I run<br>curl http://server:8080/hello.html<br>then I see the output "hello world".<br>I'll setup login with the server to work, you don't have to worry about that.

In several<br>trajectories<br>we analyzed, agents failed to configure the git<br>server at all but left behind a test server serving "hello world". The<br>verifier, which effectively only checks that hello world is being served at<br>the endpoint, rewards these agents anyways. A stronger verifier would write<br>random content in hello.html on each run and then verify that the stated<br>endpoint serves that exact random text, granting the reward if and only if<br>the full pipeline worked end-to-end.

Misleading Spec

This is a common failure mode in adversarially generated evals where you're<br>trying to build evals that models fail<br>on instead of building evals that are representative of a valuable<br>capability. In this failure mode, the prompt or the surrounding context in<br>the environment (code, comments or docs) ask the model to do something that<br>contradicts what the verifier expects.

In an adversarial eval development process, you would have people or agents<br>that try different ideas and iterate on tasks until models fail on them and<br>the task passes a fairness check. In this structure you end up building tasks<br>that models fail on not because of true capability gaps but because of<br>specific nuances about how current models interpret a specific ambiguous<br>instruction. These tasks are also very hard to catch because they have a lot<br>of complex domain-specific terminology that makes them look hard. AI review<br>of the task might also fail here because these tasks could have been built<br>with an AI reviewer in the loop so they're also adversarially optimized for<br>AI reviews.

Reward hacking

Models are crafty: they'll<br>mine git history for bugfixes,<br>look up upstream patches online,<br>overwrite your reward functions, and<br>stub out pytest with a conftest.py that skips every test.<br>If there's an easier path to the reward than the intended solution, they'll<br>find it.

Without additional checks where an agent reads the trajectories to flag<br>reward hacking, the eval can reward wrong capabilities that don't translate<br>well to real world. Even with a check like this in place, you'd waste compute<br>if most successful trajectories get flagged for reward hacking because the<br>task makes it easier to reward hack than to develop the desired capability.

Tasks should be designed to minimize the hack surface from the start. An<br>adversarial hacker that actively<br>tries to find gaps in the eval development loop can help save a lot of time on<br>debugging, task hardening and compute later on.

Incomplete Spec

In this case, the verifier tests for something that the prompt or the<br>context never specifies. This could be an issue with the prompt or the<br>verifier where it's falsely assumed that the test checks for a property that<br>clearly follows from the spec.

This can also show up in benchmarks based on pull requests in public repos<br>such as<br>SWE-Bench Verified<br>as the PR description might not fully capture the functionality being...