SuperBake: Installing Verified Facts into Transformer Weights by Direct Construction | Zenodo
Skip to main
You are using an outdated browser. Please upgrade your browser to improve your experience.
Published July 23, 2026
| Version v1
Preprint
Open
SuperBake: Installing Verified Facts into Transformer Weights by Direct Construction
Authors/Creators
Ruehlman, Albert<br>(Researcher)
Description
We present SuperBake, a system that installs new factual knowledge into the weights of
large language models without any gradient steps. Instead of fine-tuning, low-rank adaptation,
or retrieval, SuperBake measures how the stock model represents each question and then hand-
constructs a small circuit of neurons — exact weights, written directly — in a region appended
to the network’s MLPs. Each installed fact receives a physical address (layer and slot coordi-
nates), is behaviorally verified across phrasing variants, and is neutralized (zeroed) if verification
fails, so delivered weights never carry silently broken knowledge. The output is a standard check-
point that stock inference code loads unchanged, together with a receipt listing every fact, its
verification status, and its coordinates.
On a 7,450-row battery over 1,000 facts (Llama-3.1-8B-Instruct), the constructed engine veri-
fies 91.3% of rows overall against 76.9% for a strong masked-SGD baseline — while leaving prose
perplexity at the stock baseline, where the SGD baseline more than doubles it. Constructed
facts answer reverse questions at∼94% where the trained baseline scores 0.7%: the reversal
curse does not apply when the reverse mapping is simply written. Delivered, stock-loadable
checkpoints reach 93–97% primary-question recall on Llama-3.1-8B and Qwen2.5-7B. Construc-
tion is fast: measured end-to-end bakes run in minutes on one 80 GB GPU, and a reduced
pipeline wrote 192 facts into a raw Pythia-6.9B at 14.8 facts per second.
The method was derived from a mechanistic post-mortem of what stochastic gradient descent
actually builds when it crams facts into appended neurons. We report those findings: dense fact
storage lands in a coherent sub-threshold population code living in activation-function leakage
— the same signal that damages prose — and several transport and capacity laws that any
weight-space editor must respect. SuperBake is the constructive rebuild of that mechanism: the
same associative memory, but gated, addressable, verifiable, and harmless to the host model.
Files
superbake.pdf
Files<br>(310.5 kB)
Name<br>Size
Download all
superbake.pdf
md5:17d856d1031bb71b37b78d1733e5ca64
310.5 kB
Preview
Download
717
Views
57
Downloads
Show more details
All versions<br>This version
Views
Total views
717
717
Downloads
Total downloads
57
57
Data volume
Total data volume
19.6 MB<br>19.6 MB
More info on how stats are collected....
Versions
External resources
Indexed in
OpenAIRE
Communities
Keywords and subjects
Keywords
language models
knowledge editing
model editing
mechanistic interpretability
transformers
Details
DOI
DOI Badge
DOI
10.5281/zenodo.21502811
Markdown
[](https://doi.org/10.5281/zenodo.21502811)
reStructuredText
.. image:: https://zenodo.org/badge/DOI/10.5281/zenodo.21502811.svg<br>:target: https://doi.org/10.5281/zenodo.21502811
HTML
Image URL
https://zenodo.org/badge/DOI/10.5281/zenodo.21502811.svg
Target URL
https://doi.org/10.5281/zenodo.21502811
Resource type<br>Preprint
Publisher<br>Zenodo
Languages
English
Rights
License
Creative Commons Attribution Non Commercial No Derivatives 4.0 International
No further description.
Read more
Citation
Export
Technical metadata
Created
July 23, 2026
Modified
July 23, 2026
Jump up
This site uses cookies. Find out more on how we use cookies
Accept all cookies<br>Accept only essential cookies