Taalas-Style On-Chip Weights on a $250 FPGA: a Language Model at 60k tok/s | Michael Ayles
I was so impressed by the chatjimmy.ai demo by Taalas, I wanted to see what I could squeeze inside the fabricThe FPGA's reconfigurable logic. Same thing as 'PL'. Where the whole model runs here. of a $250 FPGAField-Programmable Gate Array: a chip full of reconfigurable logic you wire into a custom digital circuit, instead of running software on a fixed CPU.. By not using the (4 GB available) DDRThe off-chip DRAM (the board's main memory), ~20 GB/s, shared by both the CPU and the fabric. This shared controller is 'the bandwidth wall'., and keeping everything in URAMUltraRAM: the big, wide on-chip SRAM (~18 Mb, 64 blocks). Holds the resident INT4 weight image. Crucially, it is true dual-ported, which enables the two-cohort 'split-brain'./BRAMBlock RAM: small, flexible on-chip SRAM blocks (~5 Mb total). Holds activations, scratch, and the KV cache here., I set myself a target of 100,000 tok/s.
The headline, bit-exact and measured on silicon: 59,965 tokens per second on the fabric. The same model on this board’s own Arm cores manages 11 tok/s. My laptop’s RTX 3050 Ti manages 719.
Taalas managed to bake Llama 3.1 8B into their chip, and have a usable chat. I had ~3 MB of memory to work with, which means the most powerful model we can fit is a TinyStoriesA synthetic corpus of about 2.1 million very simple children's stories, designed so that even tiny models can learn coherent English from it. The training data here.-class one: 3.16M parameters, ~1.5 MB at INT4. Initially, I wanted to shrink it further by lemmatisingReduce each word to its dictionary root ('saying' to 'say', 'words' to 'word'). The Keviniser lemmatises the corpus down to content words, about 30% fewer characters for the same story. the training corpus and making it sound like Kevin Malone, ‘why use many word when few do trick’, but since this didn’t actually make the model any smaller, it just speeds it up by ~30%, I went back on myself and just allowed the hardware to flip-flop between both models.
First, talk to it.
The live demo
This widget is a real WebSocketA persistent two-way connection between browser and server, unlike ordinary HTTP's request-then-hang-up. What lets the chat stream characters live. connection to the board. Your words go through a Cloudflare tunnel, to a serving box, to the Kria, into the fabric, and back. If the status dot is green, you are talking to a single FPGA in Wales. It’s a story generator, not an assistant. It doesn’t understand questions. Give it “once upon a time” and it finishes it.
kevin on kria connecting...
few word do trick. type something, kevin reply from inside the chip.
once upon timethere be little girlthe dog run fastkevin like eat<br>send<br>A real conversation with a 3.16M-parameter model living entirely inside the on-chip memory of a $250 Kria KV260 FPGA. No GPU, no DDR in the token loop. Output is deliberately telegraphic: the compression is the speed. When two models are in rotation the badge shows which is live and counts down to the next swap; at zero, the whole chip is reprogrammed with the other model (~25 s). If the dot is red the board is asleep or under load. Open the full demo.<br>I hope you didn’t get your hopes up, there’s only so much you can do with ~1.5 MB of weights.
Why I even had this board
This is all running on my AMD KV260 dev board, featuring the K26 SOMSystem-on-Module: the plug-in compute module that the KV260 board is built around.. This is one of the latest SoCs in the Zynq line (specifically the Zynq UltraScale+ ZU5CG/ZU5EV, catchy eh?), the successor to the OG 7000s. God, how I wanted a dev board for one of those 10 years ago.
I bought the KV260 for a different side project, a deterministic vision pipeline. The board is sold as a “vision AI starter kit,” but the VitisAMD/Xilinx's application development stack for their boards. The KV260's 'vision AI' demos run on it, mostly on the Arm cores rather than the fabric. object detection runs on the quad core A53The quad-core Arm Cortex-A53 CPU on the KV260 (~1.33 GHz). The baseline the fabric is compared against, and the orchestrator when the CPU is in the loop.s, and the A53 is a weak core with no hardware matmul. The Vitis libraries turned out to be basically OpenCV on Linux rather than anything that pushes the fabric. If I wanted probabilistic AI running on a CPU, I sure as heck wouldn’t choose a quad core A53.
I did manage to get my deterministic depth engine running in Verilog on the fabric, but since it’s a robotics project, the hardware has a lot of downtime for safety reasons. So I put a fresh Linux image on a spare SD card and set up SSH so I can work on this from anywhere in what little free time I have (it very quickly spiralled out of control: another remote machine now has JTAG access to it, and the PSU is on a Zigbee relay so I can force reboots remotely).
The wall
One fact drives everything. Generating one token at...