GitHub - ibackstrom/QuartAI: Reproducible real versus quaternion Transformer crossover study on TinyStories · GitHub

/" data-turbo-transient="true" />

Skip to content

Search or jump to...

Search code, repositories, users, issues, pull requests...

-->

Search

Clear

Search syntax tips

Provide feedback

--><br>We read every piece of feedback, and take your input very seriously.

Include my email address so I can be contacted

Cancel

Submit feedback

Saved searches

Use saved searches to filter your results more quickly

-->

Name

Query

To see all available qualifiers, see our documentation.

Cancel

Create saved search

Sign in

/;ref_cta:Sign up;ref_loc:header logged out"}"<br>Sign up

Appearance settings

Resetting focus

You signed in with another tab or window. Reload to refresh your session.<br>You signed out in another tab or window. Reload to refresh your session.<br>You switched accounts on another tab or window. Reload to refresh your session.

Dismiss alert

{{ message }}

ibackstrom

QuartAI

Public

Notifications<br>You must be signed in to change notification settings

Fork

Star

main

BranchesTags

Go to file

CodeOpen more actions menu

Folders and files<br>NameNameLast commit message<br>Last commit date<br>Latest commit

History<br>1 Commit<br>1 Commit

articles

articles

qbench

qbench

qllm

qllm

tests

tests

.gitignore

.gitignore

README.md

README.md

quaternion_llm_instructions.md

quaternion_llm_instructions.md

quaternion_phase2_protocol.md

quaternion_phase2_protocol.md

requirements.txt

requirements.txt

run_overnight.sh

run_overnight.sh

View all files

Repository files navigation

Quaternion small language-model comparison

This is a compact Karpathy-style decoder experiment where only the Transformer projections change. The real,<br>complex, and quaternion models otherwise use the same data, batches, optimizer, schedule, attention, and head.

Phase 2 measured a data-dependent crossover. Quaternion projections had lower validation loss through 2M<br>training tokens, 5M was inconclusive, and real projections won from 10M through 50M.

Articles

Why I ran this test

Quaternion Transformers win early, then lose

Short LinkedIn post

source .venv/bin/activate<br>python -m unittest qllm.tests.test_layers<br>python -m qllm.compare # quick pipeline test<br>python -m qllm.compare --token-budget 50000000 # minimum evidence run (slow)

The quick command compares a normal model with quaternion models at (1) equal width and (2) equal total<br>parameter count, then writes an honest table and plots to qllm/report/. It is deliberately labeled a smoke<br>test because a few thousand tokens and one seed cannot establish the hypothesis.

Full-size YAML configurations are in qllm/experiments/configs/; run one with:

python -m qllm.data.prepare --vocab-size 8192 --train-mb 64 --val-mb 8<br>python -m qllm.train qllm/experiments/configs/real-base.yaml

Run the fixed-budget, three-seed equal-parameter follow-up overnight with:

./run_overnight.sh

It trains only the two arms needed to answer the main hypothesis (300M total tokens), skips completed runs when<br>relaunched with the same output directory, and writes manifest.json, per-run results, summary.json, and<br>summary.md under qllm/runs/overnight-50m/.

Phase 2 (qbench)

Phase 1 commands and caches above remain unchanged. Phase 2 byte-copies the pinned Phase-1 tokenizer,<br>training binary, and training text, then encodes a separate 8 MiB validation prefix with that tokenizer:

python -m qbench.data<br>python -m unittest tests.test_models<br>python -m qbench.verify --device auto # strict 2x200-step Stage A<br>python -m qbench.benchmark # 50 warmups, 200 iterations<br>python -m qbench.run --model real --tokens 500000 --seed 1<br>python -m qbench.analysis.analyze<br>python -m qbench.generate # after final 50M checkpoints exist<br>python -m qbench.run_crossover # full 635M-token sweep; long<br>python -m qbench.run_ablations

qbench/all.sh prepares data, runs Phase-1/layer/model tests and strict real-data verification, benchmarks, runs<br>the crossover and ablations, generates samples, then builds the report. Do not invoke it merely as a quick test.<br>Runs use source/data/config digests and atomic resumable 5%-boundary checkpoints. Every final metric uses exactly<br>2M fixed sequential validation tokens at batch 64; every curve point uses the same preregistered 131,072-token<br>prefix. Generated caches, results, checkpoints, and plots are ignored, while<br>qbench/benchmarks/throughput.csv and qbench/REPORT.md are intended deliverables. The Stage-D width-100 real<br>control is deliberately omitted because introducing low-rank projections would not be a clean algebra-only control.

About

Reproducible real versus quaternion Transformer crossover study on TinyStories

Topics

machine-learning

reproducible-research

transformers

pytorch

quaternion

Resources

Readme

Uh oh!

There was an error while loading. Please reload this page.

Activity

Stars

stars

Watchers

watching

Forks

forks

Report repository

Releases

No releases...