Roboflow Serverless Inference: 1,000 Models on a Shared GPU Fleet
Solutions
Resources
Pricing<br>Docs<br>Blog
Search
Sign In<br>Book a demo<br>Get Started
Search
Sign in<br>Book a demo<br>Get Started
Search
Blog<br>Roboflow Serverless Inference: A Thousand Models on a Shared GPU Fleet
Sachin Agarwal
Published<br>Jul 16, 2026<br>18 min read
The Challenge: Heavy Payloads, Unpredictable Compute, and Thousands of Models
To an external client, our serverless inference API looks like a standard web service: you POST an image and get back a clean JSON block of predictions, usually in under 100 milliseconds. Behind that simple promise sits some genuinely gnarly engineering. If your background is mostly in typical CRUD services, machine vision inference will break a few of your production mental models. Three traits do the breaking:<br>Ingress inversion. Most web services ingest kilobytes and emit the heavy bytes. Vision pipelines flip that: we ingest multi-megabyte, high-resolution frames and compute them down to a few kilobytes of detections. The expensive direction is inbound, and the whole pipeline has to be shaped to absorb that weight.<br>A 6,000× latency chasm. Latency here isn't a bell curve; it's aggressively bimodal. A model already warm in GPU memory answers in about 5ms. A cold one has to be fetched from the model registry, deserialized, and streamed into VRAM — routinely 30 seconds, and a minute or more for the largest foundation models. In production that means a median well under 100ms with 1–2% of requests landing past 10 seconds. There is no "typical" latency to design around: there's a fast lane and a slow lane, and you don't know which one a request is in until you're already committed.<br>VRAM multi-tenancy. We don't serve a static set of models; we host a spiky, long-tailed catalog of thousands of customer-trained models alongside heavyweight foundation ones. Only a small fraction of that catalog fits in GPU memory at once, so which models happen to be warm — and where — decides who gets the fast lane. Hold that thought; it shapes this design more than anything else on the list.<br>Why synchronous request-response fails
Serve this workload with a plain synchronous request-response API and four things go wrong, reliably — and the last one is the killer:<br>Timeouts fire at the worst moments. Client, load balancer, and server each keep their own timeout, and none of them knows whether the request in flight is a 5ms warm hit or a 30-second cold load. The shortest timer wins and the connection dies — but the GPU keeps grinding to finish work nobody will receive. Then the client retries, re-sending its multi-megabyte frame into an already-struggling system: a self-inflicted denial of service.<br>Slow requests starve fast ones. A synchronous server holds a connection, buffers, and a handler for as long as a request runs. One burst of cold starts for long-tail models saturates the connection pool and chokes out the warm-path traffic that should be flying through.<br>Cold models form convoys. The inference server runs requests in parallel, so a cold load doesn't freeze the whole node — but every request for the model being loaded convoys behind it for the full 30 seconds, arriving faster than they resolve. With a long-tailed catalog of thousands of models, somewhere in the fleet that convoy is always forming. Head-of-line blocking isn't an unlucky edge case here. It's Tuesday.<br>The load balancer can't see your VRAM. A synchronous front end routes with round-robin or least-connections — both completely blind to which GPU happens to have the requested model warm. So a request for a model sitting warm on node three gets routed to node seven, which pays a thirty-second cold load to serve it — and evicts a model some other request was counting on. At catalog scale this doesn't just miss the cache occasionally; it shreds it. The same model gets loaded redundantly across the fleet while requests keep landing cold, and every misroute triggers an eviction somewhere else — feeding the first three failures: more cold loads, more timeouts firing, more starved connections, more convoys. The fast lane exists — blind routing just never steers anyone into it.<br>Underneath all four failures is a mismatch in resource economics. Accepting requests is lightweight I/O that runs happily on cheap commodity CPUs; running inference is heavy compute on scarce, expensive GPUs. Couple the two — so that accepting more traffic requires more GPUs, or a GPU stall backs up into your web tier — and you waste the expensive resource and destabilize the cheap one. The mismatch isn't only accept-versus-execute, either: plenty of real work is CPU-bound — visualization overlays, long-lived LLM calls waiting on tokens — and running it on a GPU node means the fleet's scarcest resource idles while a CPU spins. Decouple the tiers and CPU-shaped work can route to CPU-shaped workers.<br>So here's the design goal in one sentence: accept work at wire speed on cheap machines, execute it...