Summary
-
ESP32-AI incorporates an LLM of 28.9M parameters into an ESP32-S3 that can tell coherent stories.
-
It uses memory-mapped flash for a 25M parameter embedding table, and only needs a ~560K dense core in fast RAM.
-
It runs entirely on-chip at ~9 tokens/sec, which is impressive given the small size and RAM limits of the ESP32.
Ever since LLMs came into public view, people have tried to group them into the smallest things they can find. We’ve seen people use Raspberry Pis and ESP32 to run an AI locally, with varying results. Now, someone managed to squeeze a 28.9 million LLM into ESP32-S3 and can tell a pretty good story.
ESP32-AI adds a 28.9M LLM to its microcontroller
It’s very impressive for the size of the chip.
in it ESP32 Subreddituser slvDev_ managed to run an LLM on an ESP32-S3. It can store 28.9 million parameters; For context, the first version of ChatGPT had 117 million parameters. As such, this ESP32 project only has about a quarter of the intelligence of OpenAI’s first model, but considering how it was integrated into an ESP32 device, it’s a much more impressive feat than it might at first seem.
So how did they manage to get all those parameters in ESP32-S3? Well, this is how they explain it:
The trick (Gemma’s per-layer embeddings): Most parameters in a language model are a big embedding table that is read from, not computed.
So I keep that 25 million row table in flash (XIP memory mapped) and read only ~6 rows per token (~450 bytes). Only a ~560K dense core needs fast memory.
The 25M parameter table is basically free to run, just sitting in flash and sampling a few rows at a time.
The end result is an AI that can tell a story and keep track of variables, which is pretty good for such a small device. If you want to download it and run it on your own hardware, head over to ESP32-AI GitHub Page for all the details.
