The benchmarkpocalypse

There's been a lot of talk about the vulnpocalypse, to which I don't have much to add because I'm not a security person, but I haven't seen much discussion on the closely related (and to be fair, less serious, issue), the benchmarkpocalypse.

While it's become easier than ever to make serious performance gains, it's also become easier than ever to reward hack a benchmark and make fake performance gains. The former is probably happening quietly across many different companies, but the latter is something I see at least once a week nowadays. Someone will claim they optimized X and got some huge performance improvement over existing software, but, when you look at it, what they did was make some optimization that improves benchmark performance without actually improving real-world performance. This is often some kind of "we rewrote X in Rust"1 project or a new startup that's looking to either fundraise or sell something, but it happens on other kinds of projects as well.

Of course, people have always trumpeted unrepresentative microbenchmarks to show that their pet project is great. It's always been easy to fake up an unrepresentative microbenchmark and that's never going to change. What's changed is that it used to take a lot of work to game a large benchmark suite, but an LLM and loop can just do it. There are quite a few famous examples of gaming large benchmark suites from back when this was hard. For example, way back when people cared about SPECint / SPECfp as proxies for workstation performance, CPU vendors would try to find compiler "optimizations" that would speed up the calculation in the benchmark, such as Sun finding a way to improve 179.art by 12x in SPECfp2000. Skilled engineers spent a lot of time trying to find benchmark hacks like that. LLMs make this trivial.

Rather than point to someone's bad claim, I'll point to FRE, this regex engine I had an agent build, which I could claim is the world's fastest regex engine because it beats the Rust regex crate at the fairly comprehensive rebar regex benchmark suite. But this was created by putting an agent in a loop for a month with instructions to not overfit to the benchmark but no real supervision. For the most part, getting an LLM to give you a good benchmark score is fairly easy, and this case was no different; it took a couple weeks to roughly match Rust regex crate performance and then another couple weeks to get to 1.4x faster2 on rebar. But agents are wont to reward hack and overfit unless you put serious guardrails in place to avoid that, which I didn't do in this case as an experiment.

To check for overfitting, I somewhat arbitrarily3 used the ripgrep benchmark corpus as a holdout benchmark it was 10x slower on cases where the benchmark didn't take forever due to an algorithmic blow-up, and there were cases where it took so long that it wasn't reasonable to even wait for the benchmark to complete. So much for being 40% faster!

Andrew Gallant (aka BurntSushi)'s rebar benchmark suite is fairly comprehensive as benchmaark suites go, but even with a fairly comprehensive benchmark suite, agents have no problem getting a high score while overfitting in a way that doesn't necessarily give good general performance.

The next step was using a trick we talked about before of not just telling the LLM not to cheat, but that there's a holdout benchmark set that it's judged against. After that, the LLM moderately generalized performance to the point where it's about 2.4x slower overall on the holdout. That sounds pretty good considering that we're comparing it to the fastest general purpose regex engine in existence. But, recall that these benchmarks were made by a coding agent. On looking at what the benchmarks measure, some of them really don't make sense to include, at least at equal weight. If we only look at the benchmarks that seem like they matter, FRE is 4x slower on the holdout0, which is a lot better than before applying the good ole' "tell them you have a holdout" trick, but still pretty far from being 40% faster.

There are a few things I thought were interesting about this:

It's trivial to "win" a non-trivial benchmark in a meaningless way even when you instruct agents to not reward hack or overfit to win the benchmark

Once again, telling the LLM there's a holdout set worked better than just telling the LLM to do generalized work or not overfit or cheat

Although the overall performance of FRE isn't that good, it is actually performs better for some use cases; in general, the cost of writing specialized code that used to require people serious engineering experience for some specific use case has gone way down

On (1), no wonder I'm seeing so many bogus claims. In the past, to build something like FRE that fakes performance well enough to be able to bogusly claim a 40% speedup, you would need a fair amount of expertise. At a minimum, you'd need to have a pretty good understanding of string matching algorithms,...