AVB on X: "https://t.co/FJ1Ai5BKvY" / X<br>Post

Log inSign up

Post

AVB

@neural_avb

What are Looped Transformers? Explained clearly<br>So there are basically two ways people make a LLM "smarter":<br>1. give it more parameters (a bigger brain),<br>2. or, give it more compute/data to train on (a longer education).<br>Both work, but both are expensive!<br>The ultimate motive of all types of research is to maximize objectives under resource constraints. In that light, we would rather be chasing a tempting third option: reuse the parameters you already have, instead of buying new ones.<br>The idea has led the world to discover/invent a Looped Transformer.<br>A looped transformer takes a section of the model and run the same input through it multiple times in a loop, letting it "think" a bit more each pass, almost like re-reading a paragraph a few times to understand it better. If this works, you could get the intelligence benefits of a much bigger model while only storing the weights of a small one.<br>Instead of training 100 transformer layers, you might train 25 layers, but loop over them 4 times.<br>The forward pass latency and the number of FLOPs remain similar, but the number of weights decrease 4x.<br>Effectively, you have trained your weights to be polymorphic - the same weights are capable of reflecting and iterating over past versions of its own output.<br>Looped models have done surprisingly well on reasoning-heavy tasks, in-context learning, and abstract puzzle-solving benchmarks like ARC-AGI and Sudoku.<br>The earliest Looped Transformer<br>One of the most cited papers is this little known 2018 ICLR paper on "Universal Transformers"<br>In Universal Transformers we take the standard Transformer and replace its fixed stack of N distinct layers with a single shared transition function applied recurrently across time steps.<br>Instead of processing depth as "layer 1 → layer 2 → ... → layer N," UT iteratively refines the representation of every position in parallel, applying the same weights at each recurrent step.<br>Some of these themes are present in classic RNNs (recurrent neural networks) as well, where a single module is reused multiple times to process data sequentially. Whereas RNNs looped over the entries in a sequence across time, UT (and Looped Transformers as well) processes each input token parallelly across time. The loop is only applied at the layer level.<br>UT got a lot of cool results but its lasting legacy is for introducing this crazy idea. It is now cited as the direct architectural ancestor of essentially every modern looped/recurrent-depth language model.<br>Why UT "failed" (in 2018)<br>There are several reasons why UT wasnt perceived as a bigger deal back then.<br>1. The compute-vs-parameters trade-off wasn't understood yet<br>Given an N× increase in compute, simply making the model N× bigger tends to beat looping it N times. In 2018-19, scaling laws (Kaplan et al., 2020) hadn't even been published, so the field had no formal framework for asking "is recurrent compute or parameter compute more valuable per FLOP?" UT was evaluated at matched parameter count, not matched training compute, which flatters recurrence (recurrent steps are "free" parameters) while hiding its true compute cost.<br>2. The timing was wrong<br>UT (2018) landed right as the field was entering the "scale is all you need" era. Think of all the work that came back then: BERT (2018), GPT-2 (2019), and soon GPT-3 (2020) demonstrated that brute-force parameter and data scaling of vanilla, easily-parallelizable Transformers delivered massive, predictable gains. Recurrent depth is fundamentally sequential - you cannot compute step t+1 until step t finishes. 3. Infrastructure wasn't ready<br>3. The gains were real but not dramatic enough<br>UT was more of a research paper than a full blown millions of dollars model. The results were cool but they did not feel huge enough to shift the paradigm back then, because the scale they operated in was small.<br>4. Advances in tech<br>UT was tested only as a dense architecture. Modern LTs work with knowledge of 7 more years of dedicated research and funds. We got Mixture of Experts models now, more advanced sparse attention methods (like DSA, and more recently CSA).<br>UT was also an encoder-decoder architecture. In current times, all LMs are decoder only architectures.<br>5. Compute money<br>GPUs in 2026 are way more capable than in 2019. Investors are also more likely to invest in cutting edge AI research right now than back in 2019 (which is kinda surprising given most of the backbone of modern AI was arguably built from 2017-2020).<br>The Modern Looped Transformer (Latent Thinking)<br>Universal Transformer's core idea was: apply the same self-attention + transition block repeatedly, refining the representation of every token in parallel, with per-token halting deciding when each symbol is "done."<br>The modern Looped Transformer keeps the same weight-tying thing alive but reframes almost everything else around what a decoder-only, billion-to-trillion-parameter causal...