🧠 MLPs are Hebbian Memories: A Simple Recipe for Fact-Storing Transformers · Hazy ResearchJul 22, 2026 · 10 min read<br>🧠 MLPs are Hebbian Memories: A Simple Recipe for Fact-Storing Transformers<br>Roberto Garcia*, Jerry Liu*, Ronny Junkins*, and Chris Ré

⚡ TL;DR

It sounds like sci-fi, but it's real: we can instantly build knowledge into a Transformer block, no training required!

Previously, we showed that MLPs can be constructed to store facts. Since then, we found a much simpler and more powerful explanation: a Transformer MLP is naturally a Hebbian memory. This view lets us develop the first closed-form fact-storing MLP construction , requiring no gradient descent 🤯, that packs facts at the information-theoretically optimal rate and is usable within Transformer blocks for factual recall !

Full paper team: Roberto Garcia*, Jerry Liu*, Ronny Junkins*, Sabri Eyuboglu, Atri Rudra, Chris Ré

📄 Paper

💻 GitHub

The Hazy Research Bot Constructing a Fact-Storing MLP.

Beyond efficient fact-storing MLPs (click to expand!)Excitingly, our construction's impact goes beyond efficiently building MLPs! This post focuses on the MLP construction and its storage capacity, but in our paper we discuss how it also:

Gives the first theoretical account of why LLMs can store facts at an information-theoretically optimal rate, as observed in prior work.

Offers a constructive, rather than mechanistic, interpretability of fact storage inside Transformers. Since we can construct fact storing MLPs in closed-form, we can exactly understand how facts live in them.

Enables seamless fact editing in Transformer blocks without retraining.

🔑 Fact Storage

Prior work (e.g., Allen-Zhu and Li 2024, Morris et al. 2025) has empirically found that LLMs store facts near the information-theoretically optimal rate of Θ(Flog⁡F)\Theta(F\log F)Θ(FlogF) parameters for FFF facts. Our goal is to match this rate by construction in our MLPs, and to show that Transformers can actually use them.

What do we mean by facts? We study facts through the lens of a fact set, which is simply a mapping from keys to values. Concretely, a fact set is defined by the key embeddings K={k1,…,kF}K=\{k_1,\ldots,k_F\}K={k1​,…,kF​}, value embeddings V={v1,…,vF}V=\{v_1,\ldots,v_F\}V={v1​,…,vF​}, and a mapping between them f:[F]→[F]f:[F]\to[F]f:[F]→[F]. For example, the key embedding for "France" might map to the value embedding for "Paris" in the capitals fact set.

What do we mean by a fact-storing MLP? We say an MLP\mathrm{MLP}MLP stores the fact set if, when queried on key kik_iki​, its output has the highest dot product with the correct value vf(i)v_{f(i)}vf(i)​:

⟨MLP(ki),vf(i)⟩>⟨MLP(ki),vj⟩for all j≠f(i).\langle \mathrm{MLP}(k_i), v_{f(i)} \rangle > \langle \mathrm{MLP}(k_i), v_j \rangle \quad \text{for all } j \neq f(i).⟨MLP(ki​),vf(i)​⟩>⟨MLP(ki​),vj​⟩for all j=f(i).

This matches how language models decode tokens: the model produces a hidden vector, then scores possible next tokens by dot product with output embeddings.

🔧 A simple MLP construction

How do we build an MLP that stores a fact set? Surprisingly, a very simple closed-form recipe works.

The construction. Given keys kik_iki​ and values vf(i)v_{f(i)}vf(i)​, sample random matrices A,G∈Rm×dA,G \in \mathbb{R}^{m\times d}A,G∈Rm×d from a Gaussian distribution and construct the gated MLP:

MLP(x):=B ϕ(x),ϕ(x):=(Ax)⊙(Gx),B:=1F∑ivf(i) ϕ(ki)⊤\mathrm{MLP}(x):=B\,\phi(x), \qquad \phi(x) := (Ax)\odot(Gx), \qquad B := \frac{1}{F}\sum_i v_{f(i)}\,\phi(k_i)^\topMLP(x):=Bϕ(x),ϕ(x):=(Ax)⊙(Gx),B:=F1​i∑​vf(i)​ϕ(ki​)⊤

That's the whole construction, no gradient descent!

Our Hebbian memory view of MLPs (click to expand!)A classical Hebbian memory stores key-value pairs as a sum of outer products, W=∑iviki⊤W = \sum_i v_i k_i^\topW=∑i​vi​ki⊤​, and retrieves values with Wq=∑ivi⟨ki,q⟩Wq = \sum_i v_i \langle k_i, q\rangleWq=∑i​vi​⟨ki​,q⟩. This is a weighted sum of stored values, where each value's weight is its key's similarity to the query. Our construction is exactly a Hebbian memory, but with the similarity measured in the MLP's feature space ϕ\phiϕ instead:<br>MLP(q)=B ϕ(q)⏟MLP=1F∑ivf(i) ⟨ϕ(ki),ϕ(q)⟩=1F∑ivf(i) K(ki,q)⏟Hebbian Memory,\underbrace{\mathrm{MLP}(q) = B\,\phi(q)}_{\text{MLP}} = \frac{1}{F}\sum_i v_{f(i)}\,\langle \phi(k_i), \phi(q)\rangle = \underbrace{\frac{1}{F}\sum_i v_{f(i)}\,K(k_i, q)}_{\text{Hebbian Memory}},MLPMLP(q)=Bϕ(q)​​=F1​i∑​vf(i)​⟨ϕ(ki​),ϕ(q)⟩=Hebbian MemoryF1​i∑​vf(i)​K(ki​,q)​​,<br>Intuitively, the kernel similarity K(ki,q)K(k_i, q)K(ki​,q) acts as a soft lookup. When we query with a stored key q=kjq = k_jq=kj​, the matching term K(kj,kj)K(k_j, k_j)K(kj​,kj​) is large while the mismatched terms K(ki,kj)K(k_i, k_j)K(ki​,kj​) for i≠ji \neq ji=j are small, so the weighted sum collapses onto the correct value vf(j)v_{f(j)}vf(j)​.<br>MLPs are Hebbian memories. An MLP compares a query to stored keys in feature space, then returns a similarity-weighted sum of stored values.<br>📈 MLP storage capacity. How many...