The Hidden Geometry of Transformer Weights: A Canonical Basis for Interpreting Transformer Language Models | Zenodo
Skip to main
You are using an outdated browser. Please upgrade your browser to improve your experience.
Published August 14, 2026
| Version v2
Preprint
Open
The Hidden Geometry of Transformer Weights: A Canonical Basis for Interpreting Transformer Language Models
Authors/Creators
Gernone, Gianluca
Description
Version 2 of the paper "The Hidden Geometry of Transformer Weights".
Rotating a Transformer language model into a coordinate system aligned<br>with its own weight matrices — the canonical basis — reveals a hidden<br>internal structure. Version 1 established the descriptive phenomena.<br>Version 2 establishes that they are functional and general:
- Causal evidence: zeroing a single canonical axis (0.11% of the model)<br>collapses MMLU from 47.50% to 21.25% and destroys output coherence;<br>five control axes show no effect.<br>- Six per-layer spectral indices (cohesive, torsional, informational,<br>dimensional, rhythmic, vorticity) with effective dimensionality 4.77/6,<br>exposing a 41x isotropic collapse between weight and activation spectra.<br>- Lossless realignment verified on RMSNorm and LayerNorm architectures<br>(Qwen 2.5 0.5B, SmolLM2 1.7B, Pythia 1.4B), plus native cross-layer<br>alignment measurements across eight architectures and a MoE model.<br>- Corrected cross-layer alignment numbers vs. version 1.
Every measurement is reproducible: scripts and pre-computed data<br>accompany the paper (https://github.com/todotge/canonical-basis),<br>including an interactive per-axis control chat<br>(demo video: https://youtu.be/WOJwkjj9VT0).
Files
A_Canonical_Basis_for_Interpreting_Transformer_Language_Models.pdf
Files<br>(463.7 kB)
Name<br>Size
Download all
A_Canonical_Basis_for_Interpreting_Transformer_Language_Models.pdf
md5:3c7758bd1010d5eff37bc3110a6388c1
463.7 kB
Preview
Download
Additional details
Related works
Is supplemented by
Working paper:
https://github.com/todotge/canonical-basis
(Other)
Dates
Updated
2026-08-14
Software
Repository URL
https://github.com/todotge/canonical-basis
Programming language
Python
156
Views
93
Downloads
Show more details
All versions<br>This version
Views
Total views
156
Downloads
Total downloads
93
Data volume
Total data volume
66.3 MB<br>0 Bytes
More info on how stats are collected....
Versions
External resources
Indexed in
OpenAIRE
Communities
Keywords and subjects
Keywords
transformer interpretability
canonical basis
singular value decomposition
model compression
sparse autoencoders
homeostasis
neural network geometry
layer normalization
mechanistic interpretability
ablation
MoE
Details
DOI
DOI Badge
DOI
10.5281/zenodo.21935673
Markdown
[](https://doi.org/10.5281/zenodo.21935673)
reStructuredText
.. image:: https://zenodo.org/badge/DOI/10.5281/zenodo.21935673.svg<br>:target: https://doi.org/10.5281/zenodo.21935673
HTML
Image URL
https://zenodo.org/badge/DOI/10.5281/zenodo.21935673.svg
Target URL
https://doi.org/10.5281/zenodo.21935673
Resource type<br>Preprint
Publisher<br>Zenodo
Languages
English
Rights
License
Creative Commons Attribution 4.0 International
The Creative Commons Attribution license allows re-distribution and re-use of a licensed work on the condition that the creator is appropriately credited.
Read more
Copyright
Copyright (C) 2026 Gianluca Gernone
Citation
Export
Technical metadata
Created
August 14, 2026
Modified
August 14, 2026
Jump up
This site uses cookies. Find out more on how we use cookies
Accept all cookies<br>Accept only essential cookies