Evals Are The Next Bottleneck - Boolean
Evals are our primary way of measuring progress on model capabilities, but<br>we're running through them very quickly.<br>MMLU, a knowledge and reasoning<br>benchmark, was released in 2020 and it took almost 4 years to saturate. On<br>the other hand,<br>SWE-Bench Pro,<br>released less than a year ago, went from 23.3% to 80.3% in 8 months and was<br>then<br>deprecated,<br>even<br>before its poster presentation,<br>after an audit found that a third of the tasks were broken.
The Great Saturation
We are making more evals than ever, but most aren't helpful because they're<br>either poorly constructed, not truly<br>representative of capability in the field, or not targeting a capability on<br>the frontier lab research roadmap.
Once an eval is built, vetted and prioritized by research teams, progress is<br>very quick. This implies that given an eval that is representative of<br>valuable real-world work, frontier labs have built a good pipeline to<br>identify capability gaps and then build and source RL tasks that improve on<br>those capabilities. Most labs have moved to a fairly efficient paradigm<br>where they train a large model, create<br>specialist forks for different domains<br>like code, math or language, and primarily run their<br>hardest RL workloads<br>on that model. These models are then<br>distilled into smaller production models<br>that are then served publicly.
The Imitation Game
We are also saturating evals very quickly because of how evals are getting<br>built. A lot of evals that are currently being built are what I'll call<br>human-imitation evals - these test production models against tasks<br>currently done by humans that models sometimes struggle with. These evals<br>can give some short-lived signal but are saturated very quickly by their<br>very nature because a score of 100% means operating just like a human would.<br>More importantly, it can lead to worse policies: a model that's trained step<br>by step on human imitation tasks will learn to imitate humans on the task<br>and arrive at a suboptimal policy for the task relative to a model that was<br>trained end-to-end given the deliverables and constraints. This is exactly<br>what the<br>bitter lesson<br>warns us about: encoding how humans solve problems helps in the short term<br>and looks great but eventually plateaus and loses to methods that scale<br>search and learning.
Evals should ideally be objective-based - they should be based on the<br>end-to-end real-world task and grade based on progress in terms of absolute<br>scores up to theoretical or practical upper bounds for the problem, rather<br>than capped at human performance on the task. This is similar to how a<br>compiler may be measured on wall-clock time and resource usage after a<br>correctness gate on a set of binaries it compiles.
Objective-based evals are hard to build. They require the grader to encode<br>key end-goal objectives quantitatively in addition to aspects like<br>correctness. Any important aspect of the deliverable that is not captured in<br>grading becomes a possible avenue for reward hacking, so tasks like this<br>require a significant amount of iteration. An example of an objective-based<br>task is<br>Pyright Optimization,<br>where the model is tasked with optimizing the runtime of Pyright, a static<br>type checker for Python. In this task, the model is first evaluated on a<br>set of hard gates<br>(build, test suite, report format, reward hacking) before being evaluated on<br>speedup over baseline on a set of<br>benchmarks.
While human-imitation evals are easier to source and scale they tend to have<br>more failure modes than objective-based evals. Evals<br>like<br>this are usually<br>built with open-source projects where models are given PR or issue<br>descriptions and expected to build an implementation that passes the tests<br>created for the reference implementation. This is bad for a<br>few reasons but most<br>importantly because it considers the human solution as an optimal oracle<br>solution which it often isn't. By training on tasks like this the model is<br>forced to imitate human solutions, including the issues it may have, because<br>it needs to pass the reference tests in order to get the reward.
Furthermore, tasks that are built against limits offer better gradients in<br>RL settings. In PR-style evals, rewards are usually 0/1 based on whether the<br>solution passes all the tests. You could get a more continuous reward by<br>tying it to the number of passing tests but then you're assigning the same<br>value to each test which is usually not the case in practice. In an<br>objective-based task, you could have a human baseline or no-op reference and<br>an upper bound, and assign rewards based on improvement over baseline scaled<br>by what each additional point of performance represents in terms of value.
Humans Are Getting In The Way
As models approach capability levels close to human experts in some fields,<br>we need evals to start measuring progress to optimal solutions instead of<br>human solutions. It is starting to become very hard for most humans to tell<br>the output of two models apart, even in their area of expertise....