How can LLM RL Work Despite Information-Theoretic Inefficiency

Epistemic Status : Obviously speculative and maybe obvious.

The success of RL in LLMs has been puzzling me for a while. People have developed various information-theoretic style arguments by which they argue that RL is extremely informationally inefficient compared to pretraining, that it can only impart a tiny amount of bits, and that it can only bring out behaviours that are already in the base models etc. The basic intuition here is extremely obvious and essentially falls immediately out of the formulation of the two methods. Pretraining (and SFT etc) compute a loss on every token so every token gives them information. RL, of the policy gradient style that works for LLMs, computes a single scalar reward, often binary, denoting success or failure at the end of a long rollout which can often be hundreds of thousands of tokens long. In a naive sense, this means that RL accumulates 1/rollout_length bits per sample compared to pretraining. What is worse is that computing these rollouts requires decoding the actual model which is much more expensive than prefill due to the memory-bandwidth-limited nature of transformer inference at long sequence lengths. This makes RL doubly inefficient – first we waste substantial compute doing rollouts vs prefilling and then for each rollout instead of computing the loss on all tokens we only get a single scalar. Finally, even the reward functions themselves are often crude. People see great results with merely binary reward indicating task failure or success, where a lot of the time the rollout fails for some silly reason such as incorrectly calling a single tool or forgetting to put \boxed around their answer.

The arguments for the profound inefficiency of RL seem compelling. However, empirically, reality works essentially the opposite way. Far from using information inefficiently, RL is vastly more sample efficient than pretraining. RL can achieve extremely rapid gains in performance, often taking a model from being barely able to do a task to completing it near-perfectly, in only a few hundreds to thousands of steps. Moreover, strong gains can be seen with RL in often as few as ten steps. Achieving anything in ten steps with pretraining is basically impossible. However, from the information-theoretic perspective this is exactly backward. What possible information could RL have derived in these ten * num rollout reward scalars that is so fundamental?

Moreover, LLMs are additionally surprising in that they do not use many of the core RL techniques developed in the last decade of independent RL research. From a pure-RL researcher’s perspective, the methods used in LLMs today are very primitive. They are just slight variants of policy gradients and are extremely sensitive about being on-policy, while old-school RL has a complex network of value functions, MCTS-style policy search, replay buffers with intelligent routing and selection of prior memories, model-based rollouts inside the latent space, and so on. How is LLM RL so ridiculously effective when it uses such relatively primitive methods, and secondly why do the methods that we know work well in other RL scenarios not work for LLMs. It is not like nobody has thought of giving an LLM a value function or a replay buffer during RL; these methods just perform poorly and are outperformed by regular GRPO-style policy gradients.

We also know it is not pure on-policyness that is vital to the performance of RL. RL achieves performance gains that regular iterative SFT on on-policy traces cannot accomplish, despite SFT being essentially pretraining-equivalent in the level of information density the loss provides. The standard bits-per-token view above would predict that iterative-SFT on the models’ own successful traces would utterly destroy policy gradient RL. Of course, the opposite occurs and in practice iterative self-SFT is not as effective.

Another interesting phenomenon is that LLM RL is incredibly sensitive to staleness and any numerical differences between trainer and rollouter. Tiny numerical differences that appear to make no difference to output quality, and were routinely ignored in inference stacks previously, suddenly become decisive, causing RL training to destabilize and collapse. Unlike old-school RL, even moderate degrees of off-policyness appear highly damaging and also causes performance plateaus or instabilities. By contrast, old-school RL was basically the wild west. Atari-style agents do not care about on-policyness at all. You can have a replay buffer with traces from whenever and the agent just chugs through them and learns. People did distributed RL back in the day and nobody cared to match their inference vs trainer implementations down to numerical precisions; the training just went on regardless1.

This is all profoundly mystifying. Somehow all of the intuitions and understanding we have of pretraining, the previous paradigm of...