The 1991 Unnormalized Linear Transformer
-->.
Jürgen Schmidhuber (2025, based on [FWP])
Pronounce: You_again Shmidhoobuh
AI Blog
Twitter: @SchmidhuberAI
The 1991 Unnormalized Linear Transformer (ULTRA)
The T in ChatGPT[GPT3] stands for an artificial neural network (NN) called Transformer. In March 1991, when compute was millions of times more expensive than today, even before the LSTM, Schmidhuber published a first Transformer variant, which is now called the unnormalised linear Transformer.[ULTRA][FWP0]<br>It had to be more efficient than Google's 2017 quadratic Transformer:[TR1] ULTRA's computational costs scale linearly in input size, rather than quadratically (in 1991, no journal would have accepted an NN that scales quadratically).<br>In 1991, Schmidhuber also introduced self-supervised pre-training to enable deep learning in sequence-processing NNs (see the P in ChatGPT).[UN][UN0-2]<br>His 1993 recurrent ULTRA extension[FWP2] talked about learning "internal spotlights of attention”—compare the recent attention terminology, e.g., "attention is all you need,"[TR1] and tweets of<br>2022 &<br>2023.<br>Like modern quadratic Transformers, the 1991 ULTRA is highly parallelizable.<br>It was a by-product of more general research on<br>NNs that learn to program the fast weights of other NNs.[FWP]<br>The 1991 experiments were similar to today's: predict some effect, given a sequence of inputs.[FWP0]
How does the unnormalized linear Transformer work?<br>There are two feedforward NNs (FNNs) called the slow net and the fast net.<br>The slow net has a special unit for each fast net<br>unit from which at least one fast connection is originating. In 1991, the vector of real-valued activations across these units was called<br>FROM (blue in the image); in today's Transformer terminology it's called KEY.<br>The slow net also has a special unit for each fast net unit to which at least one fast connection is leading. In 1991, the vector of activations across these units was called TO (red in the image); today it's called VALUE. At every time step of sequence processing, each fast weight may rapidly change in proportion to the product of the current activations of the corresponding units in KEY and VALUE generated by the slow net. This product is simply added<br>to the fast weight (which then may be normalized by a squashing function[FWP0]). The<br>additive part by itself essentially<br>overcomes the vanishing gradient problem.[FWP]
The current INPUT to which the fast net is applied is called the QUERY.<br>Essentially, the QUERY is processed by the fast weight matrix, which is<br>a sum of outer products of previously generated KEYs<br>and VALUEs (ignoring normalizations and projections).<br>The KEYs/VALUEs/QUERIES implement READ/WRITE operations on the separate storage represented by the fast network.<br>Since all operations of both networks are differentiable, we obtain end-to-end differentiable<br>active control of fast weight changes through additive outer products.[FWP0-3a]<br>Hence the slow net can learn by gradient descent in some given error function to rapidly modify the fast net during sequence processing, by inventing good context-dependent KEYs and VALUEs at the right times.
This is mathematically equivalent to what was later called an unnormalised "linear Transformer" with "linearized self-attention."[FWP6][TR5-6a][DLH][ULTRA]
The "quadratic" Transformers of 2017[TR1-2] are a combination of Schmidhuber's 1991<br>additive outer product fast weight principle[FWP0-2]<br>and softmax:<br>attention (QUERY, KEY, VALUE) ~ softmax (QUERY KEY) VALUE.<br>The attention weights in Transformers<br>can be viewed as context-dependent weight vectors or<br>NN-programmed fast weights.[FWP]
In the interest of efficiency,<br>linear Transformers of 2020-21[TR5-6]<br>abandoned the softmax, essentially resurrecting the original 1991 system,[ULTRA][FWP0-1][FWP] whose costs scale linearly in input size, rather than quadratically.[TR1]
Of course, plain outer products in NNs<br>go back at least to Konorski's informal 1948 rule[HEB48] (later often called the "Hebb rule"[HEB49])<br>and<br>concrete formal implementations through<br>Steinbuch's Learning Matrix around 1960.[ST61-63][AMH1-2][KOH72][LIT74][PAL80]<br>See also Kosko's bidirectional associative memories.[KOS88]<br>However, these authors described pre-wired rules to<br>associate user-given patterns with each other. Their<br>systems did not learn by gradient descent to use such rules for<br>associating self-invented KEY/VALUE patterns, like the ULTRAs and other Transformers<br>since 1991.[ULTRA]<br>(Neither did early NNs with fast weights by Malsburg (1981) and others.[FAST][FASTa,b][DLP])
The 1991 ULTRAs are essentially NN-programming NNs<br>whose elementary programming instructions<br>are additive outer product rules. What was the key novelty? Errors are backpropagated through these differentiable rules such that ULTRA can learn to minimise its objective function by invoking and using the rules wisely, generating appropriate KEYs/VALUEs at the right times to create useful changes of fast...