GitHub - ntrillard/transformer-geometry · GitHub

/" data-turbo-transient="true" />

Skip to content

Search/

Sign in<br>Sign upAppearance settings

You signed in with another tab or window. Reload to refresh your session.<br>You signed out in another tab or window. Reload to refresh your session.<br>You switched accounts on another tab or window. Reload to refresh your session.

Dismiss alert

{{ message }}

ntrillard

transformer-geometry

Public

Notifications<br>You must be signed in to change notification settings

Fork

Star

master

BranchesTags

Go to file

CodeOpen more actions menu

Folders and files<br>NameNameLast commit message<br>Last commit date<br>Latest commit

History<br>7 Commits<br>7 Commits

paper

paper

.gitignore

.gitignore

CITATION.cff

CITATION.cff

LICENSE

LICENSE

README.md

README.md

pit_engine.py

pit_engine.py

requirements.txt

requirements.txt

safety_toolkit.py

safety_toolkit.py

sphere_test_suite.py

sphere_test_suite.py

steer_sphere_proof.py

steer_sphere_proof.py

View all files

Repository files navigation

Steer on a Sphere

Geometric Control of Transformer Outputs

N. Trillard — August 15, 2026

How It Works

Every transformer layer uses RMSNorm, which constrains hidden states near a sphere of radius $|\gamma_l|$ — a learned value, not $\sqrt{d}$.

The LM head gives every token a direction on that sphere. A single tangent step reaches 91–98% of them at rank 1, and never lowers any token's rank.

Hidden state h<br>g_t = W_t − (W_t·ĥ)ĥ ← tangent toward token t<br>h' ← normalize on sphere<br>logits → 91–98% chance t ranks #1

Why this matters: with weight access, generation can be biased toward any token direction — no training, no data, no retraining.

Cow Tipping

Some tokens are self-reinforcing: feeding them to the model induces indefinite repetition.

Token<br>Triggers a loop of...<br>Real-world example

0 (digit)<br>000000...<br>Phone number 000-000-0000

NULL byte<br>\x00\x00\x00...<br>Invisible page footer

cut<br>cut cut cut...<br>Repeated delimiter

ere<br>ereereere...<br>Common substring

Defensive encoding: put a pit trigger at the end of a page and a scraper that terminates on it falls into a repetition loop. Invisible to humans (NULL bytes), it degrades automated scraping without affecting human readers.

Key Numbers

Measurement<br>Result

Tokens reachable at rank 1<br>91–98% across 4 model families

Rank improvement guaranteed<br>100% (never lowers)

Cow tipping permanence<br>15/15 steps locked

Defensive encodings verified<br>3 (NULL, cut, phone)

Edge of chaos clustering<br>13 architectures mapped

Repository Layout

├── paper/<br>│ ├── paper_steer.pdf # compiled 5-page preprint<br>│ ├── paper_steer.tex # LaTeX source (compiles with pdflatex)<br>│ └── steeronasphere.png # the cow<br>├── pit_engine.py # ★ reverse-engineer pits + defensive encoding<br>├── steer_sphere_proof.py # sphere steering reproduction<br>├── sphere_test_suite.py # batch geometry verification<br>├── safety_toolkit.py # λ diagnostics + steer-away<br>├── requirements.txt # pip dependencies<br>├── CITATION.cff # machine-readable citation<br>└── LICENSE # CC BY 4.0

pit_engine.py — the core tool

Reverse-engineers self-consistent tokens ("pits") from model weights and encodes them into data.

python pit_engine.py --model Qwen/Qwen2.5-7B-Instruct --scan<br>python pit_engine.py --model Qwen/Qwen2.5-7B-Instruct --encode data.txt

PitReverseEngineer — scans the vocabulary, computes s(T) = softmax(W·h_T)[T], tests 15-step permanence, finds minimal triggers.

PitEncoder — frames data chunks with pit triggers so any truncation boundary falls into a fixed-point loop.

steer_sphere_proof.py — sphere steering

Reproduces the tangent traversal: computes g_t = W_t − (W_t·ĥ)ĥ, steps, renormalizes, and hooks the hidden state to steer the first generated token. Produces the GSM8K results.

sphere_test_suite.py — geometry verification

Batch-checks the sphere geometry across cached models: per-layer norms (Proof 1), attention contraction (Proof 2), Lyapunov λ (Proof 3), and steering (Proof 5).

safety_toolkit.py — diagnostics

Geometric safety tools: Lyapunov health check, fine-tuning monitor, sphere steer-away, and per-zone stability report.

Paper

📄 paper/paper_steer.pdf — full preprint<br>📝 paper/paper_steer.tex — LaTeX source

Cite as:

Trillard, N. (2026). Steer on a Sphere: Geometric Control of Transformer Outputs. Zenodo. 10.5281/zenodo.21954871

Preprint. The geometric picture is approximate, not a theorem. Steering is a white-box traversal primitive. CC BY 4.0.

About<br>No description, website, or topics provided.<br>Resources<br>Readme<br>License<br>Cite this repository<br>Activity<br>Stars<br>0 stars<br>Watchers<br>0 watching<br>Forks<br>0 forks<br>Report repository

Releases

Packages

Contributors

Languages

You can’t perform that action at this time.