How Photoroom Cut Inference Cold Start to Seconds - leads with the entity + outcome | PhotoroomSkip to content

How we cut inference cold start to seconds with NVIDIA Dynamo<br>July 23, 2026<br>th]:bg-background-subdued [&_tr:first-child>th]:align-middle [&_tr:first-child>*]:border-t [&_tr:first-child>:first-child]:rounded-ss-200 [&_tr:first-child>:last-child]:rounded-se-200 [&_tr:last-child>:first-child]:rounded-es-200 [&_tr:last-child>:last-child]:rounded-ee-200 [&_table_p]:my-0 [&_:where(li_p):not(:where([class~='not-prose'],[class~='not-prose']_*))]:my-0 [&>*:first-child]:mt-0! [&>span>*:first-child]:mt-0! [&>*:last-child]:mb-0! [&>span>*:last-child]:mb-0! [&_.yt-embed]:my-10 sm:[&_.yt-embed]:my-12 sm:prose-h1:heading-900 prose-h1:mb-6 prose-h1:mt-12 sm:prose-h1:mt-14 [&_h1_strong]:font-700 blogpost group mt-10">div>*:first-child]:mt-0">Photoroom cut GPU inference cold starts from ~220s to 35–45s using NVIDIA Dynamo Snapshot. The approach snapshots a fully-loaded process with NVIDIA's cuda-checkpoint and CRIU, then restores it on any GPU in a Kubernetes fleet. Here's how we wired it into LitServe and ArgoCD.<br>At Photoroom, we run dozens of computer vision and generative AI models in production. Every background removal, studio scene, retouch, or expand is powered by its own model, each with its own latency budget and traffic pattern.<br>Some models run under continuous load; others are bursty or rarely touched. For the quiet ones, we scale to zero, because there's no sense holding an expensive GPU idle for a model that gets a handful of requests an hour.<br>Scale-to-zero has one well-known downside: the cold start. The first request after a scale-up pays the full cost of bringing a replica online, and for a large diffusion or transformer model that can mean several minutes. This post covers how we cut that time to seconds using GPU memory checkpointing.<br>What actually happens when a container serving with LitServe starts<br>LitServe is an open-source, PyTorch-native serving framework from LightningAI: you just fill in setup() to load the model and predict() to run inference, and it handles the HTTP server, batching, and workers. We reach for it whenever we want to get a model served quickly. The trade-off: it loads models sequentially, one setup() run start to finish before the server accepts a request, whereas frameworks like Triton load them in parallel. That serial cold start is why we focus on LitServe here, and why a fast restore helps so much.<br>When a LitServe container starts, it will not serve a single request until the model is fully loaded. It launches a worker process, runs your setup() inside it, and only brings up the HTTP server once that worker reports READY. setup() is where all the heavy lifting happens, roughly in this order:<br>Import the inference libraries (PyTorch, TensorRT, diffusers, …).

Download the model weights.

Move the weights from host memory into GPU memory.

(Optional) Warm up the model with JIT / torch.compile or CUDA graph capture.

Only then does the HTTP server come up and start serving.<br>Where does the time actually go? On our setup: imports take about 15 s (the ML libraries are surprisingly heavy to load); downloading a ~55 GB model at ~1 GB/s (~8 Gbps) is about a minute, or zero if the weights ship in the image or on a mounted volume; moving them host-to-GPU is only a second or two, because the link is PCIe Gen5 x16 (~50 GB/s); and warm-up depends entirely on the model, from near-zero for an AOT-compiled engine to ~40 s for our regionally-compiled (compile_repeated_blocks) image-edit pipeline.<br>Where a cold start's time goes: the setup() phases and their rough durations<br>For a typical large diffusion or transformer setup, expect tens of seconds of imports, seconds to a few minutes of download depending on the bandwidth to the storage that contains the weights, a second or two to load the weights onto the GPU, and anywhere from typically several minutes of warm-up. Put together, that is a few minutes on every cold start, and most of it (the download and warm-up) is repeated every single time.<br>What is GPU memory checkpointing?<br>What if we could freeze the process after all of this is done, and thaw it back on the next cold start? That's memory checkpointing. In short, GPU memory checkpointing freezes a fully-loaded inference process, including its GPU memory, to disk, and restores it on a later cold start instead of reloading and recompiling the model. Serverless platforms have used CPU-memory checkpointing for years, but GPU state was out of reach until NVIDIA shipped its cuda-checkpoint utility. It's been around for a little while, but only from driver r580 onward can a checkpoint be restored onto a different physical GPU than the one it was captured on, which is exactly what you need in a dynamic, autoscaled fleet.<br>Memory checkpointing in one picture: producer and consumer<br>How to add GPU memory checkpointing to a LitServe app<br>NVIDIA Dynamo Snapshot<br>NVIDIA Dynamo is a framework for...