Why Open-Source Models Haven't Killed the Big Dogs
The AI Company Builder Memo
SubscribeSign in
Why Open-Source Models Haven't Killed the Big Dogs<br>Spoiler alert: Inference is Hard
Karthik Ravi<br>Aug 06, 2026
Share
Over the last few weeks, I’ve spent a ton of time talking with investors.<br>Who knew trying to build a neocloud with weird hardware would get this much attention?!<br>Obviously, I’m flattered. Nothing validates my existence as a founder quite like a bunch of people with bags of cash who might actually give some of it to me.<br>I’m joking. Mostly. 😂<br>Putting aside all the questions about whether CueCloud can win in an increasingly crowded neocloud market, one much bigger question keeps coming up:<br>If open-source models are getting this good, why isn’t everyone serving them? And why haven’t they killed OpenAI and Anthropic?
Okay, technically that’s two questions. But they both get at the same thing: if model quality is becoming commoditized, why are OpenAI and Anthropic still making ungodly amounts of money while charging more for their best models?<br>The short answer is that open weights aren’t the same thing as usable inference. A model can be free to download and still be expensive, annoying and surprisingly difficult to serve reliably.<br>OpenAI and Anthropic aren’t just selling intelligence. They’re selling the convenience of accessing that intelligence through an API.<br>And as we’re about to see, that convenience covers up an absolutely disgusting amount of infrastructure work.
One API call. About 47 moving parts.<br>Part 1: “Just Self-Host It” Is Doing a Lot of Work
1.1 The Fantasy
The pitch for open-source AI sounds almost offensively simple.<br>Go to Hugging Face, find a model that performs nearly as well as the best closed models and download it for free. Rent some hardware, get the model running and put an API in front of it.<br>Congratulations. You’ve replaced OpenAI.<br>Somebody should probably tell Sam Altman that it’s over.<br>Honestly, I understand why people think about it this way. I had basically the same reaction when we first started looking seriously at open models. If the model is free and the hardware costs less than the API bill, what exactly are we still paying OpenAI and Anthropic for?<br>And if you’re only trying to get a model running for yourself, this argument isn’t completely wrong. The tooling has gotten good enough that a technical person can rent a machine, follow some documentation and start generating tokens without building everything from scratch.<br>That’s usually the moment where people stop doing the math.<br>And if all you look at is the sticker price, the math can look pretty compelling.<br>Runpod currently lists an H100 SXM for $2.99 an hour. Meanwhile, GPT-5.5 costs $30 per million output tokens.<br>Let’s do some extremely irresponsible napkin math.<br>Suppose your open model generates 50 tokens per second on that H100. That number is intentionally hypothetical. Actual speed will depend on the model, quantization, runtime, context length and how many requests you’re processing at once.<br>But 50 tokens per second gives us a nice, round example.<br>If you kept the GPU generating at that speed for a full hour, it would produce 180,000 output tokens. Buying the same number of output tokens from GPT-5.5 would cost $5.40.<br>The H100 costs $2.99.<br>Holy shit. Fire Sam Altman. We’ve solved inference.
Time to tell Sam to pack up… or not??<br>Except that comparison quietly assumes your GPU is generating tokens every second of every hour. Real usage generally doesn’t arrive in one perfectly smooth line. People go to lunch. Developers stop working. Traffic spikes, disappears and then returns at exactly the moment you’ve decided to change something.<br>At 25% utilization, the same GPU would produce only 45,000 output tokens during that hour. Those output tokens would cost $1.35 from GPT-5.5, while your H100 still costs $2.99.<br>Using these admittedly crude assumptions, the GPU needs to average roughly 28 output tokens per second across the entire hour just to break even against GPT-5.5’s output pricing. If it tops out at 50 tokens per second, you need to keep more than half of its capacity busy all the time.<br>And this is the generous comparison.<br>We’re treating our hypothetical open model and GPT-5.5 as if they provide the same quality, consume the same number of reasoning tokens and complete the same amount of useful work. Obviously, that isn’t always going to be true. A cheaper model that needs more attempts, writes worse code or requires more human correction may not actually be cheaper at all.<br>The math gets even less exciting when you compare self-hosting with an API that already serves open models. Together AI currently charges $1.04 per million tokens for Llama 3.3 70B. At that point, you’re not choosing between an expensive closed model and a cheap open model. You’re choosing between operating the open model yourself and paying someone else a relatively small amount to make it their problem.<br>That doesn’t mean...