GitHub - slvDev/esp32-ai 路 GitHub
/" data-turbo-transient="true" />
Skip to content
Search or jump to...
Search code, repositories, users, issues, pull requests...
-->
Search
Clear
Search syntax tips
Provide feedback
--><br>We read every piece of feedback, and take your input very seriously.
Include my email address so I can be contacted
Cancel
Submit feedback
Saved searches
Use saved searches to filter your results more quickly
-->
Name
Query
To see all available qualifiers, see our documentation.
Cancel
Create saved search
Sign in
/;ref_cta:Sign up;ref_loc:header logged out"}"<br>Sign up
Appearance settings
Resetting focus
You signed in with another tab or window. Reload to refresh your session.<br>You signed out in another tab or window. Reload to refresh your session.<br>You switched accounts on another tab or window. Reload to refresh your session.
Dismiss alert
{{ message }}
slvDev
esp32-ai
Public
Notifications<br>You must be signed in to change notification settings
Fork<br>84
Star<br>708
main
BranchesTags
Go to file
CodeOpen more actions menu
Folders and files<br>NameNameLast commit message<br>Last commit date<br>Latest commit
History<br>30 Commits<br>30 Commits
data
data
experiments
experiments
firmware
firmware
media
media
src
src
.gitignore
.gitignore
.python-version
.python-version
README.md
README.md
RESULTS.md
RESULTS.md
pyproject.toml
pyproject.toml
uv.lock
uv.lock
View all files
Repository files navigation
Running a 28.9M parameter LLM on an $8 microcontroller
Open to Work 聽路<br>饾晱 slvDev 聽路<br>LinkedIn
This is a 28.9 million parameter language model that generates text on an ESP32-S3,<br>a microcontroller that costs about $8. It runs on the chip itself, with nothing<br>sent to a server, and it writes each word to a small screen wired to the chip at<br>roughly 9 tokens per second. The last language model people ran on a chip like this had 260<br>thousand parameters, so this one holds about a hundred times more. It fits because<br>most of the model lives in flash instead of RAM, using an idea from Google's Gemma<br>models called Per-Layer Embeddings.
The numbers
Parameters<br>28.9M stored (25M of them in a flash lookup table)
Chip<br>ESP32-S3, about $8, with 512KB SRAM, 8MB PSRAM and 16MB flash
Speed<br>about 9.5 tok/s end to end (9.7 tok/s of pure compute)
Connectivity<br>none, everything runs on the device
Model size<br>14.9MB at 4-bit
Why it is hard, and how it fits anyway
A microcontroller has very little fast memory. The ESP32-S3 gives you 512KB of SRAM.<br>Normally the whole model has to be reachable from there, which keeps you stuck with<br>tiny models, and that is why the previous model on a chip like this had only 260<br>thousand parameters.
The way around it is to stop putting the model in fast memory at all. Most of a<br>language model's parameters sit in an embedding table, which the model reads from<br>rather than computes on. So you can leave that 25 million row table in slow flash<br>and pull only the few rows each token needs, about 450 bytes, while the small part<br>that does the actual work stays in fast memory. The large model then costs almost<br>nothing to run, because you never load most of it. It just sits in flash and gets<br>sampled a little at a time.
That idea is Google's Per-Layer Embeddings, from Gemma 3n and Gemma 4. Here it runs<br>on the memory layout of a microcontroller instead of a phone or a GPU. As far as I<br>can tell, nobody had tried it on a chip this small.
SRAM (fast, tiny) the "thinking" core, used on every token<br>PSRAM (medium) the output head and working memory<br>FLASH (huge, slow) the 25M-param table, about 6 rows read per token (~450 B)
What it does, and what it does not
The model was trained on TinyStories, so it writes short, simple stories and mostly<br>keeps them coherent. It will not answer questions, follow instructions, write code,<br>or know facts. That limit comes from the small part of the model that does the<br>reasoning, and the memory trick does not change it. What is interesting here is the<br>architecture, fitting a large model onto a tiny chip, rather than what a 28.9 million<br>parameter model can say.
Running it yourself
The firmware, the wiring, and the flashing steps live in<br>firmware/esp32_llm/README.md. The training,<br>ablation, and quantization code is in src/ and experiments/. The full method,<br>the ablations, and the on-chip measurements are written up in<br>RESULTS.md.
Credit
TinyStories is the dataset this trains on: short synthetic stories simple enough<br>that a small model can still learn to write coherently (Ronen Eldan and Yuanzhi Li,<br>Microsoft Research, arXiv:2305.07759). The other<br>half is Per-Layer Embeddings, Google's design from the Gemma models, which is what<br>lets a big model fit on a small chip.
Andrej Karpathy's llama2.c is why a lot of<br>people, me included, believe you can train a tiny language model and run it in plain<br>C at all. This grew out of that.
How this actually went
I left the messy history in the repo on purpose. That includes a bug I found in...