The State of Open-Source LLM Inference
Shwetha Krishnamurthy
SubscribeSign in
The State of Open-Source LLM Inference<br>What happens after you leave the frontier API and decide to build.
Shwetha Krishnamurthy<br>Jul 15, 2026
Share
Disclaimer: Written in a personal capacity; opinions are mine, not my employer’s. Everything here comes from public docs, published papers, and benchmarks I ran on rented hardware with my own money. Used Claude in drafting the blog. All ideas are mine but Claude helped me in communicating them better.<br>Scope: this is a post about serving LLMs. Many of the patterns transfer to serving other modalities, at different capacities and with different breaking points, but text serving is the spine of the story.<br>Thanks for reading! Subscribe for free to receive new posts and support my work.
Subscribe
You didn’t choose inference. It chose you.
You run an enterprise business. Few months ago one of your teams shipped a feature on a frontier model API, and it was the right call: nothing to deploy, nothing to operate, state-of-the-art quality from the first request. It worked. Users came.<br>Then one of three emails arrives.<br>The first is from finance. The API bill has a slope, and the slope is steeper than the revenue attached to it.<br>The second is from legal. A customer contract wants to know who, besides you, can see the data: which vendors touch the tokens, what they are allowed to do with them, and how long they keep them. You would like the honest answer to be a shorter list than it currently is.<br>The third is from your own engineers. They need a latency guarantee the API won’t sign and a knob the API doesn’t expose, and they have started saying words like “p99” in meetings with a haunted look.<br>Any one of these emails is enough to send you looking for alternatives to the frontier API. This post is a map of what you’ll find when you do, told in the order you’ll find it. Because here is the thing about the open-source inference stack in 2026: you don’t adopt it as a stack. You adopt it one component at a time, and every component arrives because something just broke.<br>The fork
Before anything breaks, you have one decision to make, and it splits the world in two.
The fork. Left: managed inference, someone else’s pager. Right: your GPUs, your stack, your pager. This post is about the right-hand box.<br>Buy. Managed inference providers (Fireworks, Together, Baseten, Modal, and their peers) will run the open model of your choice on dedicated hardware and hand you back the thing you already like: an API. You keep the frontier-API experience, you gain model choice and data control, and someone else carries the pager. Most teams that leave the frontier APIs land here, and for many of them it is the right landing.<br>Build. Open weights, your own GPUs (rented or owned), and the open-source serving stack.<br>I’m not going to referee this fork. The trade-offs are real on both sides, they depend on your team and your traffic, and vendor-neutral is a load-bearing adjective in this post. Both roads have stories worth telling; this post covers the second one. If you decide to build, here is what happens next.<br>Chapter 1: The monolith (one replica, and it runs fine)
What just broke: nothing yet. You begin the way everyone begins: with one replica. One serving engine process holding one copy of the model. A small model fits on a single GPU; a bigger one spans two or four or eight, with the serving engine managing the split behind a flag. Either way, the unit you deploy, scale, and reason about is the replica.<br>You picked your open model (that decision is someone else’s blog post). You rented the GPUs for your first replica. You typed vllm serve with no flags, braced for a week of tuning, and instead got a competent, batching, cache-managing server that just... works. For a while, it is genuinely fine.
The whole system, chapter 1 edition: users, requests, and one box. Weights and KV cache live inside the box, and so, for now, does your entire inference strategy. Each chapter of this post adds to this diagram; orange marks what’s new.<br>What you now own, whether you know it or not: batching, KV cache management, and scheduling within a replica. Continuous batching plus paged, block-based KV management (PagedAttention in vLLM’s vocabulary, RadixAttention in SGLang’s) is the canonical pair, and the serving engine handles both. The serving engines have also quietly absorbed a lot of the folk wisdom. I benchmarked Gemma 4 12B on a rented H100 across three configurations, and the defaults already shipped chunked prefill, prefix caching, and CUDA graphs; nearly every tuning tip the internet offered me was either already on or did nothing outside run-to-run noise. The one change that beat noise was quantizing the weights to FP8: +14% saturated throughput, and p99 time-to-first-token at high load dropped from about 24.5 seconds to about 15.8. I wrote that whole investigation up separately, with a pizza kitchen standing in for...