Researchers pinpoint why larger language models pick up skills that small ones miss

Ad

Skip to content

The Decoder

AI, Menschen, Wirtschaft

-->

Sign In

Register

Subscribe Now

The Decoder

Opens discord in a new tab<br>Opens LinkedIn in a new tab

Researchers pinpoint why larger language models pick up skills that small ones miss

Jonathan Kemper

View the LinkedIn Profile of Jonathan Kemper

Jun 7, 2026

Nano Banana Pro prompted by THE DECODER

A new study suggests that instead of endlessly inflating models, it may be more efficient to increase the frequency of specific tasks in training data to anchor rare skills in smaller models.

A new study from researchers at Anthropic, Stanford, and other institutions explains why larger language models learn certain tasks that smaller ones fail at. The finding goes beyond the conventional wisdom that big models simply learn faster.

In some cases, small models can't reliably learn rare tasks even with extremely long training runs. Even well-known scaling laws show that a small model never reaches the loss of a large one, no matter how much data you throw at it.

Only the larger OLMo models learn the rarely interspersed tasks reliably, as can be seen from the orange-colored fields at the bottom right of both tasks. | Image: Huang et al.<br>Common tasks crowd out rare ones

To isolate the mechanism, the researchers tested a mix of tasks with varying frequency and complexity. A model with N neurons gets assigned the N "most useful" features, where usefulness is based on how often a task appears and how important it is. Frequent, simple tasks get priority. Rare, complex ones get dropped. In the experiments, only models that were large enough learned tasks that made up just 0.25 percent of the training data.

A model with N neurons assigns the N most useful features, while larger models also pick up rarer tasks further down the list. | Image: Huang et al.<br>The core of the paper is its explanation of why size helps. As long as frequent tasks aren't well-learned yet, they pull the model strongly in their direction at every training step, overwriting much of what the model picked up about rare tasks. Once a large model has mostly mastered the frequent tasks, that pull fades. The freed-up capacity goes to rare tasks, and learned signals are more likely to stick.

Small models rarely reach that point, according to the study. They fall into an "update-and-forget" loop. A rare example gets briefly learned, then largely erased by the next training steps on frequent tasks. When the next rare example shows up, the model starts over from scratch.

One experiment was designed to cleanly separate this effect. The total frequency of a rare task stays constant, but the gap between individual observations varies. The larger the gap, the more the signal decays in narrow models. Wide models hold onto it better between observations and build on it.

Real language models show the same pattern

To test the theory during pre-training, the team trained OLMo models ranging from 4 million to 4 billion parameters on up to 210 billion tokens from the Dolma corpus. They mixed two artificial tasks into the data, a number comparison and a modular addition, with frequencies ranging from about 1,000 instances per batch down to one instance every ten batches.

In the middle row, all models get the same clear signal (the peaks) where the task appears. In the bottom row, the difference is stark: in the small 20M model (purple), the rest of the language training constantly pushes in random directions and disrupts the signal. In the larger models (300M and 1B), the line stays near zero, leaving the signal intact. | Image: Huang et al.<br>Only the larger OLMo models picked up the rare tasks by learning the rule behind them and applying it to new cases, rather than just memorizing individual examples.

This was especially clear with modular addition, where the researchers observed what's known as grokking. A model memorizes a task first, then suddenly clicks on the actual principle after more training. Only the bigger models hit that moment, and only when the task showed up often enough in the data.

A look inside the models tells the same story. In the one-billion-parameter model, every training step that included the rare task pushed clearly toward the right answer. In the 20-million-parameter model, that signal drowned in noise from everything else. Almost no real learning took place.

Memorization turns out to be a stepping stone

The study treats memorization as a prerequisite for generalization, rather than an unwanted side effect. A model needs to hold onto individual observations long enough for a broader pattern to take shape across many batches.

This offers a practical alternative to just making models bigger. Instead of scaling up the model, the frequency of a target task in the training data can be increased to anchor a specific skill, the research suggests.

There's more than one theory for why...