The Loss Function Is the Product · Ryan Merlin

writing about now connect<br>On this page The governor<br>Error correction is intelligence<br>Three costs<br>The self-correction paradox<br>Composed loss functions<br>Retreat, patch, redesign<br>The governor for AI fleets<br>What are we training people to do?

Jensen Huang has a rule of thumb: he would be, in his words, “deeply alarmed” if a $500,000 engineer didn’t burn through at least a quarter-million dollars a year in AI tokens. He’s right that the spend is coming. He’s just measuring the wrong side of it.

That money buys tokens, the small chunks of text an AI reads and writes. It’s tempting to treat them as raw fuel: more tokens means more output, and more output means more value. But every token is really an allocation decision. Some of it goes to generation: the forward motion, the new code, the new draft. Much of the rest goes to making that generation usable, the retrieval, coordination, verification, retries, and correction that turn a draft into something you can trust. The ratio between forward motion and everything it takes to make that motion trustworthy is the number almost nobody reports. It’s the one that decides whether all that spend buys anything real.

The available evidence and field reports suggest the industry is measuring its units of intelligence at the wrong boundary.

A 2025 CodeRabbit analysis of 470 open-source pull requests found that AI-co-authored code carried roughly 1.7 times as many issues as human-written code, with some security categories running as much as 2.74 times higher. The study leaned on authorship signals rather than confirmed provenance, so treat the exact multipliers as directional rather than precise. In one vivid weekend experiment, Saiprapul Thotapally burned 202 million tokens pushing agentic workflows across a few thousand requests. The cause was spawn loops, malformed retries, and context drift. A LeanOps analysis found that an agent working on a multi-step task burns ten to a hundred times more than a simple chat. The reason: at every step it re-reads everything it knows before it acts. And the errors compound. Assume for a moment that step failures are independent: a ten-step chain where each step is 85 percent reliable comes out right end to end only about a fifth of the time. Picture a factory line where four of every five products fall off the belt before the end.

The industry is optimizing generation. But the part that actually decides whether the output is any good is something else. Engineers call it the loss function. Strictly, the loss function is just the score, the number that says how far off you are. The real machine is the loop around it: the instrumentation that senses the drift, the trace that attributes it, and the mechanism that corrects. I will use loss function as shorthand for that whole correction system, because the score is where it starts. That system is the real product.

You could object that this is just control theory with new nouns, or MLOps, or the error budgets that site-reliability teams have run for years. Fair, and half the point: the same shape keeps getting rediscovered because it holds weight. What’s actually new is where it now has to live. Not around one model or one service, but across a fleet of agents handing work to each other, where the failure hides in the handoff and nothing ever throws an error. That is the part the old playbooks don’t cover, and it’s where this is headed.

The governor

The history of channeling fire is the history of this problem.

Open flame is pure generation. Raw power, no regulation. It cooks your dinner or it burns down the village. For most of human history, that was the tradeoff.

The Newcomen steam engine arrived in 1712. It offered containment. Operators watched the engine and manually adjusted valve timing based on observation. That was better than open flame. But the correction loop was human, slow, and did not scale. This is where most AI guardrails sit today: permissions, approval gates, human-in-the-loop review. Containment with slow, manual feedback.

James Watt’s flyball governor changed the equation in 1788. Spinning arms sensed rotational speed and mechanically throttled steam intake. No human in the loop. Continuous, automatic correction. James Clerk Maxwell analyzed the mathematics in his 1868 paper On Governors. In doing so he effectively founded control theory.

The breakthrough was not a bigger fire. It was not a stronger box around the fire. It was a feedback loop that could sense error and correct without waiting for a human to notice.

We are at the Newcomen stage of AI systems. We have containment. We do not yet have the governor. The teams that build it will define the next era of this industry.

Error correction is intelligence

Is that a metaphor, or is the same structure genuinely repeating? Four domains reached it independently, without ever comparing notes. Read them and decide for yourself.

Biology. DNA polymerase has a raw...