What Is a Container, Really? Five Years of GPU Infrastructure | Beam← All postsEngineering

What Is a Container, Really? Five Years of GPU Infrastructure<br>Luke Lombardi<br>July 7, 202610 min read

We've been doing this for a while. We started the company with the idea that it should be dead simple to deploy models – an AWS Lambda for GPUs. Since then, we've faced almost every infrastructure problem there is. Let me start at the beginning.<br>Starting naively on ECS<br>When we started building, we were building a cloud IDE, sort of like Replit for ML. For the first year or so, we were focused on making it easy to fork and launch machine learning models from your browser, so we invested heavily in the front end. The backend left a lot to be desired. I'd used ECS before for data pipelines and was familiar with it, so we started there.<br>Our ECS architecture was naive. We'd generate a Dockerfile for each user, bundle their code into it, and spawn an ECS container that embedded a small webserver alongside their code. It was simple, and it kind of worked. This is the demo we raised our pre-seed round with.<br>As soon as we started running real models, it stopped working. We kept ECS workers running, so it wasn't truly serverless. We just dynamically loaded and unloaded user code on demand to fake it. As we began onboarding our first users into production, we came across the core problem: cold start. That was the thing we'd be fighting (in one form or another) for years.<br>Knative scaling woes<br>The cold-start problem pushed us toward a second version of the system, and toward Kubernetes. I tend to reach for the easiest available thing, so I went looking for something off-the-shelf, truly scale-to-zero, and Kubernetes-native. Knative fit, so we built V2 on top of it: a custom operator, CRDs for our deployment resources, and internally roughly the same server as before: dynamically loaded user code, and a separate image tag per deployment version.<br>Knative plus Kubernetes was a real improvement. It let us lean on the K8s ecosystem for things like proactive image caching, just warming the disk on nodes with a tool called kube-fledged. And around this time we did something we should have done much earlier: we instrumented the system end to end. We'd been so focused on getting something that worked that we knew when things were slow but not why. So we started shipping metrics.<br>The data confirmed our suspicions and ranked them. Three things were slow: starting GPU nodes, loading large images stuffed with CUDA dependencies, and loading model weights into RAM and then into VRAM. By far the biggest was starting the GPU node itself. We kept an overprovisioned warm pool to hide it, but that was expensive, so we spent real effort optimizing node bootstrap. That turned out to be wasted work. We were optimizing at the wrong layer. (In today's market it makes even less sense to optimize there: GPUs are scarce enough that you cling to whatever capacity you can get. But that's another story.)<br>I remember the moment when this became a hair-on-fire problem for us. One of our first production users was running a large inference workload, and at that stage we didn't even have a queuing system for inference. The way we "scaled" their workload was to add containers via Knative, with load balancing at the Kubernetes service level, in front of what was (at that point) just a FastAPI server. It was trash, and it failed in the most visible way possible. We'd set up a shared Slack channel with the customer, and they'd wired a hook to post a message every single time their pipeline threw an error. So we sat there getting bombarded with error messages, over and over.<br>We tuned every Knative setting there was. We forked it. We changed the autoscaling parameters, rewrote the proxy behavior, tried a month of small tweaks. None of it worked, and watching exactly where it broke is what produced the diagnosis the whole rest of this story follows from.<br>The problem was twofold.<br>First, Knative's autoscaling and request buffering were built for web traffic – they can't handle APIs where each request carries this much latency. You cannot scale an inference pipeline the way you scale a CRUD service; you're operating under false pretenses, and it will not work.<br>Second, even with perfect request buffering, container start and model load time were high enough that it wouldn't matter — you can't buffer your way out of not being able to start the container fast enough.<br>So there were two problems, not one. How fast can you start a container, and how do you autoscale and proxy requests for workloads that don't behave like web requests. We had to solve both.<br>(For what it's worth, the Knative fork was a short blip, about a month. Nobody really pushed back on the decision to leave it, mostly because nobody outside the company knew what we were doing internally.)<br>What is a container, really?<br>The second problem (request routing, queuing, proxying) we solved by building our own control...