I Couldn't Find a Local LLM Tool Fast Enough, So I Built My Own Called Squish - Squish
Skip to content
Initializing search
konjoai/squish
Guides
Architecture
Integrations
Reference
Contributing
Blog
Categories
I Couldn't Find a Local LLM Tool Fast Enough, So I Built My Own Called Squish¶
This requires Apple Silicon (M-series) and macOS 13 (Ventura) or later. If you're on Intel, Linux, or Windows, the numbers in this article will not apply to your hardware.
TL;DR: Squish is a local LLM inference server for Apple Silicon, 1.15 to 14.7× faster than Ollama depending on how much your prompts repeat. Skip the story and install it →
Since February, many of my commits have been written by a local AI in under two seconds. No API keys, no rate limits, no internet connection, and no data leaving my machine. Getting it working took considerable effort, but once it did, it's been very reliable across dozens of repositories. The local AI software I'm describing didn't exist, not without heavy modifications to source code you don't control, or building your own from scratch. So I built it, and it solved my problem. It may not solve yours, but you can clone it, fork it, take it apart, and have fun with it.
It started with Gemini. I wired the API to a script I wrote that automates git commits and pushes. It was fast and intelligent, until I hit the hard free tier rate limits. In my case, I hit them every day. Sometimes on the second commit, sometimes on the first. Gemini returned a response message saying I was rate limited. I used it once or used it twice and was cut off. This was for simple stuff, maybe 500 to 1,000 tokens for a commit, a little more for a large diff. I put up with it for a while, but the annoying limits drove me to drop it. I searched around but couldn't find anything that could solve my problem.
So I went local, off the cloud entirely, with no rate-limiting. I pointed the script at Ollama running Mistral and configured and tested it for weeks: built a custom Modelfile, iterated prompts, fine-tuned output until the commit messages described the diff accurately. The descriptions came out great. They also took too long. End to end, a response landed anywhere from seven to ten seconds on a normal commit, and north of a minute for a large diff. So I pulled every lever and turned every knob, but the adjustments failed to reduce the response time, and my problem was still unsolved. The slow and unpredictable responses were the hard wall. When the software is written by someone else, their speed limit is your speed limit. I wanted a coherent commit message in under five seconds, ideally under three. Ollama and the models it serves could not deliver. I thought about it for over a month and came to the conclusion that I had to build something myself. Something lean and elegant that doesn't just work, it beats the baseline outright. That something is called Squish, a local AI inference server.
What Squish Is¶
To be clear: I did not build a model. I built the server that runs models, a framework that quantizes and compresses them, and a local format that reduces how much memory they need to run. Squish is an MLX-based local inference server, and five architectural components set it apart from existing tools:
KONJO AI / SQUISH INFERENCE SERVER<br>How Squish runs on Apple Silicon<br>One shared pool of memory: the weights, the cache, and the GPU all live in it.
Your prompt
APPLE SILICON · UNIFIED MEMORY
GPU<br>The engine<br>math on the<br>compressed weights
Unified memory (RAM)
Quantized weights<br>small (INT3), kept warm
KV cache<br>reuses seen work
Scratch<br>working buffers
GPU (static arrow + traveling pulse) -->
the full weight set,<br>moved every token
RAM entry -->
tokens (static arrow + traveling pulse) -->
Tokens stream back
RUNS IN THE BACKGROUND<br>Memory governor<br>Watches memory in tiers:<br>shrinks caches gently, then<br>more aggressively under<br>pressure. Rejects new work<br>only if it gets severe.<br>The weights stay resident.
watches
evicts
evicts
PLATFORM Apple Silicon · MEMORY unified · DEFAULT INT3, KV fp16
1. A persistent daemon that keeps the model awake. Ollama loads its model lazily, on the first request, and that first request pays for it: twenty to thirty seconds of cold start while the weights load, before a single token comes back. Worse, the model gets unloaded once it sits idle, so an intermittent workflow pays that toll over and over. Squish loads the model once, when the daemon starts, and keeps it resident for as long as it's running. The cost is paid a single time; every request after is warm. Load once, never load again.
2. A two-tier KV cache that remembers. Before a model can answer a prompt, it runs prefill. It reads the entire prompt and builds the internal attention state required in memory before producing a single token. That memory state is called the KV cache. Normally the KV cache is discarded once a response is completed, and on the next request it gets rebuilt from...