Scaling Laws: The Law Behind Every LLM — Kaplan vs. Chinchilla vs. Inference-Optimal (2020-2026)

SubscribeSign in

Scaling Laws: The Law Behind Every LLM — Kaplan vs. Chinchilla vs. Inference-Optimal (2020-2026)<br>Dose #10 — Production Agentic AI Under Pressure

Dr. Ryan Rad<br>Jul 29, 2026

Share

For decades, this curve was blueprint of deep learning. If your model is too simple (relative to the complexity of the task), it underfits, it can’t capture the pattern. Increasing the complexity of the model usually led to better results, but if you keep pushing the complexity up on a fixed dataset, then something worse happens: error stops falling and starts climbing again. The model isn’t learning anymore. It’s memorizing noise (overfitting).

That second half of the curve is the one that mattered. It’s why “just make the model bigger” used to be bad advice, not good advice. And it wasn’t really a modeling problem. It was a supply problem.<br>Data was always limited. Every gain in model capacity ran straight into the cost of labeling more of it. Human annotation is slow, expensive, and does not scale linearly with your ambition. For decades, that constraint (not compute nor the architecture) was the actual bottleneck on how good a model could get.<br>The wall didn’t move until someone stopped asking humans to label the data at all.

Self-supervision is the trick: use the text itself as the label. Predict the next word. “The cat sat on the ___” and the answer, “mat,” was sitting right there in the training corpus the whole time. No annotator required. Every book, every codebase, every web page instantly became labeled data, for free.<br>That single move deleted the data bottleneck. Suddenly the constraint on how good a model could get wasn’t how much labeled data existed. It was how much compute you were willing to spend processing it. The bias-variance wall didn’t get solved. It got sidestepped, by making the “not enough labeled data” side of the curve functionally irrelevant.<br>Which immediately created a new problem: if data is no longer scarce, and compute is the only real constraint, how do you decide where a bigger compute budget actually goes?

The Equation Behind Every Training Run<br>Every training run has a cost, measured in FLOPs, and it comes down to a deceptively simple formula:<br>C ≈ 6 × N × D<br>N is parameter count. D is dataset size in tokens. The 6 is fixed, we need 2 FLOPs for the forward pass, and 4 for the backward pass, per parameter, per token. You don’t get to negotiate this. The only lever you control is how you split your budget between N and D.<br>That fundamental allocation question, bigger model or more data, has been answered three different ways in five years, and each answer produced a visibly different generation of models.

1. The Kaplan Era: Go Big or Go Home

In 2020, OpenAI’s Kaplan et al. modeled loss as a function of N and D independently and found something that looked decisive: the exponent on parameters was small enough that scaling the model mattered far more than scaling the data. Their math said that for every unit of new compute, roughly 73% should go into making the model bigger, and only 27% into feeding it more text.

Look at the left panel: every parameter count eventually flattens out, hitting a data ceiling regardless of how much more text you feed it. That flattening is exactly what told Kaplan et al. that parameters, not tokens, were the lever worth pulling.<br>Labs took that seriously. GPT-3 landed at 175B parameters trained on 300B tokens. PaLM went to 540B parameters on 780B tokens. That’s a token-per-parameter ratio under 2x on both — models with enormous capacity and comparatively little to actually learn from.<br>The cost is that “bigger” and “smarter” quietly stopped meaning the same thing. A model can have hundreds of billions of parameters and still be undertrained, plenty of room to memorize, not enough signal to fill it. Kaplan’s numbers weren’t wrong about the trend. They were wrong about the destination.<br>Note: the gap wasn’t a math error so much as a methodology one. Kaplan’s fits excluded embedding parameters, were built mostly on smaller models, and used fixed learning-rate schedules that penalized runs with more data, all of which quietly tilted the conclusion toward “make it bigger.” Get your fitting methodology wrong at this level and the entire first generation of frontier models inherits the bias.

2. The Chinchilla Correction: Balance Beats Brute Force

Two years later, DeepMind’s Hoffmann et al. re-ran the experiment properly, over 400 models, tuned learning-rate schedules, IsoFLOP contours instead of single-variable fits. Their model of loss added a parameter-bound term and a data-bound term together instead of treating them as separate stories, and the two exponents came out almost identical.

That efficient frontier line running through the contours on the left is the whole paper in one stroke. For any fixed compute budget, there is one model size that sits at...