A new inference engine to run Kimi K3 2.78T parameter with 29GB of RAM

The WASTE inference engine - Marco Bambini

Marco Bambini

SubscribeSign in

The WASTE inference engine Running Kimi K3 2.78T parameter with 29GB of RAM

Marco Bambini Jul 30, 2026

Today we are releasing WASTE, an open-source inference engine designed to run models whose weights are substantially larger than the memory available on the host machine. The first model fully supported by WASTE is Kimi K3, a 2.78 trillion parameter mixture-of-experts model. We converted the original 1.42 TB checkpoint into a 982 GB container and ran the complete model on a MacBook Pro with 64 GB of unified memory. This is not a distilled, pruned, or reduced variant . A typical execution currently looks like this: $ waste run ~/models/k3.waste "What is the capital of Italy?"

waste: no --budget, using 46.24 GB of 64.00 GB (expert cache 17.56 GB)

The capital of Italy is Rome.

[16 tokens, 49.31 s, 0.32 tok/s] The current performance, approximately one third of a token per second, is not yet suitable for most interactive applications. However, the result demonstrates that available RAM does not necessarily have to impose a hard limit on the total size of a model that can be executed locally. WASTE is an initial step in a broader effort to make increasingly capable models available on hardware controlled by the people and organizations using them. The goal is to provide greater control over infrastructure costs, data privacy, availability and deployment. Moving the memory boundary

Kimi K3 is a mixture-of-experts model. Although it contains 896 routed experts, only 16 experts are selected in each layer for a given token. Most of the model weights are therefore inactive at any particular point in the computation. WASTE takes advantage of this property by separating the model into two main parts. The model trunk remains resident in memory. The expert weights remain on NVMe storage and are read when selected by the router. Any remaining RAM is used as a bounded expert cache. This changes the primary constraint. Instead of requiring the entire model to fit in memory, the system needs to retrieve the active portion of the model within an acceptable amount of time. For K3, each token accesses approximately 17 GB of expert data. This makes storage bandwidth and cache behavior central to performance. WASTE stores experts in a layout designed around this access pattern. The matrices required for one expert are adjacent and can be loaded with a single positional read. The engine bypasses the operating system page cache and manages expert caching directly, so its behavior remains representative even when the complete model is much larger than memory. The runtime is written in C and has no external inference dependencies. Python is used for conversion and validation, but not during inference. The project currently includes: an embeddable C API;

a command-line client;

an OpenAI-compatible HTTP server;

CPU implementations for ARM and x86 systems;

support for macOS, Linux and Windows;

text and image input for Kimi K3.

The same engine also runs Kimi-Linear 48B from a 19 GB container at approximately 8.9 tokens per second on the same machine. Developing through measurements

One of the most useful parts of this project has been the development process itself. The repository includes a document called What we know, and how we know it. It records not only successful optimizations, but also assumptions and proposed designs that were later disproved by measurements. We adopted a simple working rule: Before starting a long or expensive operation, run a smaller real-world test capable of invalidating the underlying assumption.

This was particularly important because many operations in this project are expensive. Downloading the source model involves 1.42 TB of data. A complete conversion produces almost one terabyte of output. Changes to the expert format can require converting the model again. Testing an assumption first is considerably less expensive than discovering a problem at the end of that process. Storage placement

We initially planned to run the converted model from an external NVMe drive. The nominal performance of the drive appeared sufficient, but conventional storage benchmarks did not represent the pattern used by WASTE. The engine reads expert-sized records from many different positions rather than processing one continuous file sequentially. We therefore wrote a benchmark that reproduced the actual access pattern. On the target machine, the external SSD reached approximately 0.94 GB/s. The internal SSD reached 12.78 GB/s. The main limitation was the USB bridge in the external enclosure rather than the NVMe device itself. As a result, the external drive is used for downloading and staging the source checkpoint, while the converted WASTE container is placed on the internal SSD. This measurement changed an important deployment decision before the full model was downloaded and converted. Working with the released...

A new inference engine to run Kimi K3 2.78T parameter with 29GB of RAM

Related Articles

(no title)

Scientists reverse brain aging, with a nasal spray

AI has torched the market for junior programmers

Al Vigier: Canada's AI strategy shouldn't include secret Palantir bills

Google will expand age checks on Android worldwide till the end of the year