Everyone's Watching the Wrong Benchmark - by Zach
SubscribeSign in
Everyone's Watching the Wrong Benchmark<br>Why the gap between open and closed models is about to widen, and what it means for the data you thought was yours.
Zach<br>Jul 09, 2026
Share
Picture this: it’s 11pm and a former Goldman analyst is sitting at her kitchen table, laptop open, working through a stack of forms. She’s not building a pitch or a model of a leveraged buyout. She’s writing out, in careful detail, how she’d actually reason through one: the judgment calls, the shortcuts, the things you only learn after a decade on a desk. She’s getting paid $200 an hour for it. On the other end is a frontier lab, and the thing she is patiently teaching will, if it works, be very good at her old job.<br>Multiply that scene by tens of thousands of experts (doctors annotating clinical reasoning, litigators marking up briefs, senior engineers reviewing architecture decisions) and paying out more than $1.5 million a day at a single vendor, and you have the most under-reported story in AI. A brand-new industry has materialized in about 24 months, almost entirely to sell one thing to a dozen buyers. And it tells you something the leaderboards don’t.<br>Is the open-source gap really closing?
The consensus right now is comforting if you like open source: the gap is closing, maybe closed. And on the public benchmarks, that’s largely true. Epoch AI’s aggregate capability index puts open-weight models only about four months behind the closed frontier. Stanford’s 2026 AI Index tells the same story from another angle: the top U.S. model’s lead over China’s fast-rising — and largely open-weight — competitors has narrowed to 2.7% as of March 2026, after DeepSeek-R1 briefly matched the best American model a year earlier. Four months. Measured against valuations that price in a durable moat, that’s a rounding error.
Credit: Stanford’s 2026 AI Index Report<br>If that were the whole picture, this post would be an argument for open source, and I’d be writing it.<br>But look closer at what these models can and can’t actually do. The same 2026 AI Index that shows convergence also shows frontier models now matching or beating human experts on PhD-level science and competition mathematics — while still failing at, among other things, conducting financial analysis. Sit with that. The benchmarks that are saturated are the public, gameable ones. The capabilities that would actually let a model replace an incumbent (the messy, high-context expert work) aren’t there yet. Epoch’s own footnotes say the quiet part: open-weight models do worse on private benchmarks than public ones, the polite way of saying they’re trained to the test, and the leading labs don’t always release their most capable models. The frontier you can measure isn’t the frontier that exists.<br>That’s the whole thesis in two sentences from the people who build the charts. The benchmarks are saturated, so everyone can climb them. What they can’t do is climb the things nobody publishes.<br>Where is the future gap being created?
For most of the deep-learning era, everyone trained on more or less the same raw material: the public internet. That’s why open source kept pace — the ingredients were a commons, and the rest was compute and technique, both of which diffuse fast. Compute gets cheaper. Papers get published. Weights leak or get released.<br>The internet is now largely used up as a source of new signal, and the labs have quietly moved the fight to ground open source can’t stand on: data they pay to create. Not the old crowd-labeling of stop signs, but expert reasoning, the tacit, hard-won knowledge that lives in people’s heads and in the workflows of companies that would never hand it over. The vendors selling this are growing at rates that don’t look real. One crossed $1.2 billion in revenue while bootstrapped. Another went from a $1 million to a $500 million run rate in 17 months. A third went from $7 million to $100 million in a single year. Meta paid nearly $15 billion for a stake in one of them.<br>Add it up and the labs are spending somewhere around $10–15 billion a year creating training data today, and at least one operator close to it thinks that number passes $100 billion within two years. To put that in context: the entire global market for enterprise application software — every CRM, ERP, and HR system on earth — is ~$300 billion a year. We are watching a data-creation industry grow, from a standing start, toward the scale of the software industry it aims to automate in a few years (relative to 20+ years of SaaS development to reach this scale).<br>Almost all of this is private, undisclosed opex. Unlike the compute capex that gets guided and dissected every quarter, nobody has to report what they spend on data. When a category is this hard to see and growing this fast, the reported figures are a floor, not a ceiling. Our view at Gradient is that the true size of the data-provision business is materially...