AI Is Superhuman Wherever the Scoreboard Is Cheap · Ground Truth

▚">

ContentsThe internet ran out<br>The mechanism, which is almost insultingly simple<br>Which makes the whole map predictable in advance<br>"But models can make their own training data"<br>The genuinely bleak part<br>Three ways to misread the table<br>Can you cheat your way off the slow side?<br>What would blow this up

Here is a pattern that looks completely arbitrary until you find the single rule underneath it, at which point it stops looking arbitrary forever.

AI is now superhuman at competitive programming. Superhuman at formal mathematics. Better than most professionals at finding software vulnerabilities. It plays board games at a level no human being will ever reach again, and nobody even finds that remarkable anymore.

AI is also, after all of that, roughly fine at telling you whether to take the job offer. Fine at sensing when a negotiation has quietly turned against you. Fine at taste.

The obvious explanation is that the first list is easy and the second list is hard. That explanation feels correct and is completely wrong. Competitive programming is not easier than having good judgment about people. Ask literally any programmer.

The real rule is stranger, and once you see it you cannot stop seeing it:

AI becomes superhuman at anything where checking the answer is cheap, and stays mediocre at everything where checking the answer is expensive.

Not doing. Checking.

To see why, we have to go back to the moment the industry ran out of internet.

The internet ran out#

For several years the recipe for a better model was gloriously simple. Get more text. Use more compute. Get a better model. It worked so reliably and for so long that people started treating it as a law of nature.

Then the text ran out.

The rough numbers: somewhere around 10^14 useful tokens of high-quality human text exist in the world. Models had already been trained on about 10^13 of them. Which leaves one order of magnitude of headroom, and it is the low-quality order. The good stuff was eaten first, the way it always is.

The remaining headroom is one order of magnitude, and it is the low quality order.<br>That is the well. The robot is scraping the bottom of it, and what is coming up in the bucket is mud.

This did not cause a crisis and it did not slow anything down, which is the part people consistently get wrong. It caused a redirection. Denied more text, the frontier pivoted to synthetic data, to reinforcement learning on verifiable outcomes, and to self-play in domains where you can cheaply check whether an answer was right.

That pivot could not possibly have been neutral across domains. It made the entire enterprise dependent on something that is distributed across human activity with spectacular unevenness: a cheap way to find out whether the answer was any good.

The mechanism, which is almost insultingly simple#

A training loop needs a signal. Something that says yes, more of that, or no, less of that. Millions of times over.

So consider two loops.

Loop one is code with a unit test. The model writes code. The test runs. It passes or it fails. That is your signal. It costs approximately nothing, it takes milliseconds, it needs no human at all, and it can run millions of times a day , forever, all night, while everyone is asleep. Call it the Millisecond Loop .

Loop two is career advice. The model gives advice. And now, to discover whether that advice was any good, you wait six months and observe how someone's life went. Then you attempt to disentangle the effect of the advice from eleven other things that also happened. (Good luck. Nobody has ever done this convincingly, including the person whose life it was.)

Call this one the Six-Month Loop . It runs approximately never.

Same model, same budget. The only thing that changed is what it costs to find out whether the answer was any good.<br>Same robot. Same compute. Same everything. The one on the left has done five million reps since breakfast. The one on the right is sitting on the floor next to a clock, waiting, and will still be waiting next quarter.

There is no clever architecture that fixes a loop you cannot run.

Which makes the whole map predictable in advance#

Once you accept that capability flows toward cheap verification, you can write down where AI will be strong and weak without knowing anything at all about the models. Just order the world by how much the ground truth costs:

FastMediumSlowFormal math, competitive programmingComputational chemistry, materialsExperimental biologyCode with testsNarrow imaging diagnosticsClinical judgmentStructured retrievalLegal document analysisNegotiation, tasteGames with defined win conditionsForecasting with resolutionSocial science, policyCyber exploits (self-verifying)Grid dispatch and marketsSpecialty harvest, last-metre delivery

Now picture that table as a hill.

Same robots, opposite outcomes, decided entirely by which side of the hill the work happens to sit...