qMLX: Dogfood Tales #1 - Multiplexing & Cache Stability · Andryo Marzuki - Net Zero Productivity by 2050↓Skip to main content<br>Table of Contents
I recently wrote about my custom fork of rapid-mlx, qMLX: Maximising my AI psychosis by minmaxing my Mac Studio, in a post last week. It honestly received way more traction than I was expecting so I’ve decided to keep a technical diary going forward which I’m going to call “Dogfood Tales”.<br>With only two weeks out of my ten weeks of parental leave remaining I am now on borrowed time to push this project as far as it can go. Rather than winging it like I have up until this point I’ve decided to keep notes every time a noteworthy (read: annoying bug) happens as I use qMLX. My hope is that if I’m unable to spend much time on this project as work creeps back into the picture, some other person can continue the dream.<br>This post will cover off the three primary areas of concern I’ve identified while dogfooding qMLX over the last week:<br>Leveraging qMLX’s deliberate design principles (serial processing) and only using SSD KV cache restores to maximise multiplexing capability<br>Experimenting with 3.7bit quant and pushing Qwen3.6 to the largest context on a 96GB Mac Studio that is stable<br>Screaming about my cache being invalidated because the harness has reordered system prompt blocks (harness complaints)<br>tl;dr: I have completely removed the hot cache path, and made qMLX a-restore-from-SSD inference engine that limits requests in a serial manner. qMLX can now achieve concurrency without parallelism, which finally lets me run sub-agents off a local model.<br>Checkpoint Splitting & Enabling Multiplexing #<br>As of PR#27 I’ve made the rather opinionated decision to rip out hot caching from qMLX. With only 96GB unified on my Mac Studio, it just seems entirely wasteful to use scarce RAM on maintaining a hot cache. For my particular use case which is agentic workflows, moving to a cold cache has basically meant I can expand the number of sessions that I concurrently run by simply expanding my disk space. Unlike RAM, disk space is essentially dirt cheap.<br>Until recently, I have been limiting my interactions with qMLX to a single session at a time to make it easier for me to isolate issues - with the majority of the core bugs resolved, I wanted to set my sights on “concurrency”, this is mostly due to the fact that the bandwidth limitations of Apple Silicon mean things like continuous batching don’t generally help [citation needed]. So the approach I decided to implement is the opposite of continuous batching, multiplexing.<br>As I dogfooded qMLX with multiple concurrent sessions, I continued to get wedged requests that basically grind everything to a halt. On the rare occasion, this was due to things like thought loops, but by far the biggest culprit was having to do an entire cold prefill on 100K+ tokens.<br>The culprit of this was my rather aggressive LRU evictor, it would reduce the qMLX’s cache folder to 80% by basically removing the oldest/least used checkpoints. This was a problem born as a consequence of my own stupidity and short sightedness, as part of the initial POC I didn’t bother optimising the checkpointing - I instead bruteforced the situation by using a TB5 2TB NVME enclosure (lol).<br>But the reason why any of this was an issue at all was the fact that Qwen3.5 122b uses Gated DeltaNet for its KV caching, this means that it can’t be split cleanly like SoftMax attention (for example) could. That means that rather than creating incremental checkpoint slices, qMLX was checkpointing the entire state every time it hit the default threshold of 256.<br>Luckily for me, I’ve already opened the pandora’s box of maintaining my own fork of an inference engine, so this problem was entirely solveable. The recurrent portion of Qwen3.5 122b’s KV cache is not splittable, but the remaining full attention layers can be split like any other KV cache. I’m also happy to say that it worked fantastic and has been implemented as of PR#37.<br>The Results #<br>If I could summarise the results in a few words, they’d be “pretty sick”. The changes have basically meant I am now able to host practically an infinite amount of concurrent sessions and use them all through multiplexing. This is particularly huge for me as I can now leverage sub-agents and sub-sessions, with a cheap cloud frontier model like MiMO V2.5 Pro I can make it do all the heavy lifting with my local Qwen3.6 122b.<br>To illustrate this in action, I ran three concurrent sessions and measured the outcome. Across the 20-minute test, 789,351 prompt tokens were offered to the model. Only 6.2% were recomputed on the GPU, the rest were restored from the on-disk KV cache.
Interval by interval, disk-cache reuse dominates recompute, the multiplexer keeps every conversation’s prefix warm, so new turns extend cached KV instead of...