GitHub - vixhal-baraiya/microgpt-c: The most atomic way to train and inference a GPT in pure, dependency-free C · GitHub
/" data-turbo-transient="true" />
Skip to content
Search/
Sign in<br>Sign upAppearance settings
You signed in with another tab or window. Reload to refresh your session.<br>You signed out in another tab or window. Reload to refresh your session.<br>You switched accounts on another tab or window. Reload to refresh your session.
Dismiss alert
{{ message }}
vixhal-baraiya
microgpt-c
Public
Notifications<br>You must be signed in to change notification settings
Fork<br>91
Star<br>617
main
BranchesTags
Go to file
CodeOpen more actions menu
Latest commit
History<br>6 Commits<br>6 Commits
Folders and files<br>NameNameLast commit message<br>Last commit date<br>data
data
docs
docs
src
src
.gitignore
.gitignore
LICENSE
LICENSE
Makefile
Makefile
README.md
README.md
View all files
Repository files navigation
microGPT-C
The most atomic way to train and inference a GPT in pure, dependency-free C.
A character-level transformer with forward pass, backprop, Adam and<br>sampling, in one C file with nothing beyond libc. It trains on ~32k names<br>in a couple of seconds and generates new ones.
Build and run
make run
Or run it directly, on any corpus with one item per line:
./microgpt data/names.txt
Builds on macOS, Linux and Windows (MSYS2), on ARM64 with NEON and x86-64<br>with AVX2. The Makefile picks the flags for the host.
step 5000 / 20000 | loss 2.6036 (avg 2.2940)<br>step 10000 / 20000 | loss 1.9639 (avg 2.2564)<br>step 15000 / 20000 | loss 2.7007 (avg 2.2151)<br>step 20000 / 20000 | loss 2.3463 (avg 2.2201)
inference<br>sample 1: kayley<br>sample 2: maria<br>sample 3: arana<br>sample 4: shayan<br>sample 5: jayden<br>sample 6: saria<br>sample 7: kaylen<br>sample 8: amari<br>sample 9: alina<br>sample 10: mailyn<br>c fp32+NEON 10168430 tok/sec
Notes
The model has 4192 parameters and generalises rather than memorises.<br>Trained on 20000 of the 32033 names, it scores 2.2054 nats per character<br>on those and 2.2039 on the 12033 it never saw, beating an interpolated<br>trigram that has nearly five times as many parameters.
Training and inference use separate forward passes. gpt_forward stores<br>activations for backprop; gpt_forward_infer is a specialised<br>single-token path whose logits match it to within fp32 rounding.<br>docs/PERFORMANCE.md covers how that path works and<br>what limits it.
machine<br>backend<br>tok/sec
Apple M5 Pro<br>NEON<br>10,168,430
AMD Ryzen 5 5600H<br>AVX2<br>6,927,775
About<br>The most atomic way to train and inference a GPT in pure, dependency-free C<br>Topics<br>artificial-intelligencecc-programmingdeep-learninggptllmmachine-learningmicrogpt<br>Resources<br>Readme<br>MIT license<br>Activity<br>Stars<br>617 stars<br>Watchers<br>6 watching<br>Forks<br>91 forks<br>Report repository
Releases
Packages
Contributors
Languages
You can’t perform that action at this time.