Dev proves LLMs will run on anything – even a $10 microcontroller
Jump to main content
Search
REG AD
EDGE AND IOT
Dev proves LLMs will run on anything – even a $10 microcontroller
Tobias Mann
Tobias<br>Mann
SYSTEMS EDITOR
Published<br>tue 4 Aug 2026 // 22:20 UTC
Getting a small local language model running on a notebook or even smartphone in 2026 is trivial. But what about something even smaller and lower-power. Say, like an ESP32 microcontroller that costs less than $10?<br>It might sound impossible — the device is primarily designed for things like remote sensors, IoT, and other embedded applications, not running generative AI models — yet, that's exactly what a developer who goes by the handle SlvDev has managed to do.<br>In a process detailed on GitHub, and recently showcased on the Better Stack YouTube channel, SlvDev documented how he managed to get a small language model running at nearly 10 tokens a second locally on a microcontroller that costs about the same as a fancy cup of coffee.
REG AD
Tiny stories on a tiny microcontroller
REG AD
Cramming a large language model (LLM) onto something as small as a ESP32 microcontroller isn't a trivial task.<br>There's a reason that these models are trained and run on GPUs. LLMs are memory-hungry beasts that typically require between one and four bytes per parameter just to hold their weights in memory.<br>With just 520 KB of SRAM and 8 MB of pseudo SRAM (PSRAM) on the ESP32-S3, you aren't going to be running a model like DeepSeek V4 Flash .<br>To make it work, the dev had to drop the "large" from the language model and settle for something nearly 10,000 times smaller: TinyStories, a 28.9 million-parameter model originally developed by Microsoft Research.<br>However, even this model is asking a lot of an ESP32-S3 module. At 16-bit precision, the model requires about 60 MB of memory that the ESP32 simply doesn't have. So, the dev employed several techniques, some of which we've previously explored, to shrink the model’s footprint.<br>The first is quantization, a process by which weights are compressed by reducing their precision from something like 16-bits of precision to eight, or even four.<br>This enabled SlvDev to trade a bit of accuracy for a 75 percent reduction in memory required. Instead of about 60 MB of memory to hold the weights, they now require just 14.9 MB.<br>But that still wasn’t enough. Thankfully, in addition to the 8.5 MB of working memory, the ESP32-S3 can also be bought with up to 16MB of flash storage.
REG AD
By borrowing a technique called per-layer-embedding (PLE) from Google's Gemma family of models, the dev was able to offload the majority of the model's weights, about 25 million parameters or about 12 MB worth, to flash with minimal performance degradation.<br>Offloading model weights to NVMe storage is an old trick for getting massive frontier-class models like DeepSeek V3 running on hardware that wouldn't have the necessary memory and GPU capacity to serve it otherwise. The downside of this approach, historically, is that it murders performance. Instead of tokens a second, you're usually looking at seconds, or in some cases minutes, per token. It works but it's not remotely practical.<br>PLE manages quite a bit better because these weights are accessed rather sparingly, which keeps the flash's glacially slow bandwidth relative to DRAM or SRAM from nerfing performance.<br>The result is that rather than trying to cram 14.9 MB into the ESP32's memory, the model now only needs about 2 MB. Specifically the output head, embeddings, and KV cache are kept in the chip's PSRAM, while activations are handled in the chip's 520 KB of SRAM.<br>Using this approach, the dev says they were able to get 9.88 tokens a second out of the microcontroller, which is faster than the average person can read.
MORE CONTEXT
Bypassing AI guardrails is so easy a script kiddie can do it
China turns up the heat with open model blitz as US model makers panic
MediaTek lines up $5B war chest for AI datacenter push
Google dev kit spurs first-ever agent-on-agent violence
So, what can I do with it?<br>While you may be able to get a small generative AI model running on a microcontroller like an ESP32, you won't get much from the practice beyond dumb simple pride.<br>Tiny Stories is a great proof-of-concept, but aside from generating short, reasonably coherent stories on demand, it can't do much. You aren't going to build a chatbot with it, generate code, or power an agent. There is another model, called Barista, that can answer questions at roughly twice the performance, but only on topics pertaining to espresso.
REG AD
TinyStories and Barista are, well, just too tiny to do much else. Yet the fact these models run on an ESP32 at all is impressive in itself<br>That said, if you've got a device with just a bit more memory and compute, say a Raspberry Pi or a smartphone, there are far more capable models out there.<br>Google's Gemma 4-E2B-it, launched back in April, employs the same quantization and PLE...