@adlrocha - Forget “Fat” Models. The Future is Narrow
@adlrocha Beyond The Code
SubscribeSign in
@adlrocha - Forget “Fat” Models. The Future is Narrow<br>How to REAP your experts to fight the upcoming open-weight crunch<br>adlrocha<br>Jul 12, 2026
Share
A quick confession before we start. I told you across the last two posts that I was working towards building my own local inference rig, and I’ve decided to hold off on the hardware for now. After my analysis, it was clear that the market is moving too fast to freeze a build , we may be at an all-time-high of hardware prices, and I’d rather keep my money in my pocket a bit longer. But I haven’t dropped the thread (I can be pretty obsessive). I’m still deep in the design space, still trying to work out what the right box actually looks like before I spend a penny on it.<br>And then this week a piece of news dragged me right back in. China’s Ministry of Commerce has reportedly spent the past month in meetings with Alibaba, ByteDance and Z.ai about restricting overseas access to their best AI models. And not only the closed ones. The report says it covers open-weight models too , the freely downloadable kind, with Qwen and GLM-5.2 named directly. The idea being floated is a tiered system: basic models wave through, high-performance ones get a security review, and the most sensitive frontier ones either stay inside China or don’t get released at all. Nothing is law yet, so take it with the appropriate salt, but the direction of travel is hard to miss.<br>A big chunk of the open-weight models the local community runs are Chinese. DeepSeek, Qwen, GLM, they’re the reason a person can run something close to the frontier at home at all. Even more, they are the reason we can get lots of serverless LLM providers offering cheap and performant tokens able to compete with the big labs demand. A ban or freeze on the flow of open-weights models from China into the West could be catastrophic for our access to intelligence.<br>Which brings to the original inspiration of this post. I’ve been following for a while how the local inference community is trying to compress models so they fit in the kind of hardware that you can afford at home. Some people have been able to run DeepSeek on a single DGX, and I’ve already talked about in this newsletter about how Antirez managed to get DeepSeek to run on an M5 with DS4. These are all models that in theory would only fit in expensive compute infrastructures, but can be adapted to replace many use cases at home.<br>To no one’s surprise, this is part 3 of the topic that has been obsessing me for a couple of months, i.e. how to run LLMs at home. Part 1 was the hardware , and why memory bandwidth is the number that actually decides what you can run. Part 2 was the software, the flags that decide how much of that hardware you actually get to use. This one is about the model itself, and something the China news sharpened for me. Every big lab is building fat models, huge generalists good at a hundred benchmarks with nothing to do with each other, and then almost nobody uses them like that. You use one for coding. Or for pulling fields out of documents. Or for one industry’s very specific paperwork. The model hauls around a mountain of capability you’ll never touch, and that really adds up when you are trying to run these models in constrained hardware.<br>So this post is about the techniques that can allow you to throw what you don’t need away, so you compress the model to your needs. If the supply of new open weights really is about to get choppier, doing that well matters more than ever if we want to be able to run these models at home. And my take is that this will not only affect individuals, but also organisations that increasingly rely on LLMs for their day-to-day.
Throwing away the experts that you don’t need
Quick recap, because it matters here. As we’ve discussed before in this newsletter, most of the interesting open-weight models you have the chance to run at home are Mixture-of-Experts (MoE). I went into this in part 2, but the short version: instead of one big dense network where every parameter fires on every token, a MoE model is chopped into a lot of smaller “expert” sub-networks , and a little router picks a handful to switch on for each token. DeepSeek-V4-Flash activates 6 experts plus a shared one per token, out of a pool of hundreds.
The reason everyone builds them this way is efficiency: you get the knowledge of a giant model but only pay to run a slice at a time. The key thing here is that the slices you never use don’t go anywhere. They still sit in memory, still take up disk, still set the floor on how much hardware you have to buy. Your coding model carries experts for languages you don’t speak and domains you’ll never ask about, but as you don’t know the specific experts and network layers that activate to access that knowledge, you are stuck with the full model.<br>In order to compress models, the obvious technique is to use...