qMLX: Maximising my AI psychosis by minmaxing my Mac Studio · Andryo Marzuki - Net Zero Productivity by 2050&darr;Skip to main content<br>Table of Contents

A follow-up question on a 50,000 token conversation took three to five minutes before the first token appeared. Not the full answer. The first token. That is not a chatbot, it is a batch job, and you go and make a cup of coffee while it thinks.<br>To the dozen other geeks obsessed with maximising your Mac Studio, I come with good tidings. By good tidings I mean I spent three weeks debugging a cache leak so you don&rsquo;t have to.<br>This started with antirez/ds4, a brilliant experiment in running large models on consumer hardware. Two months ago I impulse-bought an M3 Mac Studio Ultra. I feared RAM prices would make the 96GB configs unattainable soon (spoiler: they did), and I really wanted to dive into local inference stacks whilst I am on parental leave. The goal was simple: run a frontier model on a single machine, keep it warm, and have a conversational AI that actually understands context.<br>It turned out to be more complicated than that. The model fits in unified memory and answers coherently, But after weeks of daily use I made a hard pivot: I dropped DS4 Flash and switched to Qwen 3.5 122B. Two separate things then happened, and it is worth keeping them apart. The model swap was a speed-and-fit decision. Making the new model actually usable meant fixing three bugs in my own serving stack that had nothing to do with which model I ran.<br>Why I switched models #<br>DS4 Flash is a genuinely good model and antirez&rsquo;s stack is a brilliant piece of work. It just was not the right model for what I do. This is a fit story, not a fault story.<br>My use case is long-context agentic coding: pair programming where the model holds thousands of tokens of conversation, code, and tool output, and I need near-instant turns to stay in the flow. For that specific workflow the prefill latency was the dealbreaker. Past 50k tokens a simple follow-up took three to five minutes before the first token appeared, and you cannot pair program with a model that makes you wait for a cup of tea. By the time it caught up I had already moved on to the next problem. That is not DS4 being slow in some absolute sense, it is a mismatch between how that setup handles long-context prefill on my hardware and what my workflow needs.<br>Qwen 3.5 122B looked like a better fit for the M3 Ultra on every axis I cared about:<br>Near frontier, fully local. No API calls, no rate limits, no data leaving the machine. A 122B model that rivals proprietary systems, running entirely offline.<br>The right active-param size for the Ultra&rsquo;s bandwidth. The M3 Ultra&rsquo;s memory bandwidth is large. A 122B MoE with roughly 10B active params sits in the sweet spot where the GPU can feed the compute units without stalling.<br>A good balance of tool calling and reasoning. Enough reasoning depth for complex code logic, robust enough for reliable function calling and agent orchestration.<br>The right size for a unified KV cache. With 96GB of unified memory, a 122B model at low bit-width leaves just enough headroom for a deep SSD-backed KV cache, which is the whole game for long-context retention without swapping.<br>The closest serving stack I could find was rapid-mlx. But it went a different direction on hybrid attention. Given that divergence, I forked it instead. The result is qMLX, available on GitHub. The fork is specialized for hybrid attention (more on that at the end).<br>The real work: killing three bugs #<br>The model fits, but it was unusable out of the box for a different set of reasons that had nothing to do with the model itself: every follow-up message reprocessed the entire conversation from scratch. On a 130k-token context that is a multi-minute wait before the model emits a single word.<br>The cause is the hybrid attention architecture, a mix of GatedDeltaNet (SSM) layers and dense attention. The recurrent state in the SSM layers cannot be rewound or trimmed to an earlier position, so to avoid a memory leak the in-memory cache drops any entry containing those layers. For this model the in-memory prefix cache misses every single time. In a normal window I measured zero in-memory hits against 109 disk hits. The only thing keeping the model warm is the disk cache: checkpoint the attention KV to SSD, restore it on the next turn. Disk restore is not a fallback here, it is the entire cache. And it kept breaking, in three separate ways, each hiding behind the last.<br>Bug one: a timestamp in the system prompt #<br>KV reuse is byte-exact. If the prompt changes even slightly, the match fails at the first difference and everything after it recomputes.<br>The agent framework was stamping a unique message ID into the system prompt on every turn. That unique value, near the top of a 130k-token prompt, meant the prompt was never byte-stable. Turn two differed from turn one within the first few hundred tokens. The cached agent got thrown out, the...