Where gVisor cold-boot time goes - and why a bigger instance won't fix it — Apoxy Blog<br>All systems operationalApoxy:// Field notes/Engineering<br>EngineeringJun 6, 202612 min read<br>Where gVisor cold-boot time goes - and why a bigger instance won't fix it<br>We measured runsc cold boot on a number of hosts across three clouds and one Mac. The fast number isn't the cloud, the slow part isn't the kernel, and most of the obvious fixes do nothing.
Dmitry Ilyevsky<br>Co-founder & CTO
For our CLRK runtime we run agents inside gVisor sandboxes<br>because we believe its isolation properties are the right ones<br>for untrusted code that touches LLM tools and the customer's network. So we wanted to know exactly what that isolation costs at boot, and which of<br>the available knobs can reduce that number.
Across eight hosts ranging from an M1 Mac to an AWS bare-metal instance, the same workload took<br>anywhere from 81 ms to 280 ms of runsc create plus runsc start. The fastest host was the M1<br>laptop, which beat every cloud instance - including the biggest-core-count boxes, which landed dead<br>last. Most of the obvious tuning knobs turned out to be either neutral or slightly negative. The one<br>trick that worked the best recovered a 4.5x reduction, but we had to cheat a little bit.
Why we care about gVisor cold start
CLRK agent sandboxes are launched on a worker pod with one of two purposes. For a long-lived<br>DaemonAgent, boot latency is amortized over the agent's lifetime and barely matters. For a<br>short-lived TaskAgent, boot latency is added to user-perceived response time on every dispatch.<br>That second case is the one we wanted to bound.
Existing gVisor benchmarks<br>tend to measure either steady-state syscall throughput (workload runs inside the sandbox) or<br>whole-container boot including image pull. Neither is what we cared about. We wanted the wall time<br>from "controller decides to dispatch this task" to "the init process inside the sandbox has been<br>scheduled," holding the image pull and worker overhead aside as separate concerns.
So we built a measurement spike that ran the exact same runsc argv as the CLRK worker, with the<br>same OCI spec, the same sentrystack plugin network, and the same --platform=systrap flag. We<br>measured each phase end to end. Then we replicated the measurement on eight different hosts to see<br>which axes of variance matter.
How we ran the benchmark
We tried to reuse existing CLRK components as much as possible - the image store and rootfs cache,<br>the plugin network the sandbox attaches to, and the runsc (the worker binary doubles as runsc,<br>so the spike forks and measures the actual thing rather than a reimplementation that could drift<br>from it). Same goes for other bits - capabilities, mounts, namespades, and cgroup - Claude was<br>entrusted to rip out/import all of that out byte-for-byte to make the bench as close to the real<br>thing as possible. A human ensured no funny business was going on with the benchmark setup.
Per-phase boundaries are split into two complementary streams:
Outer wall times measured externally around runsc create, start, wait, and delete.
Internal phase markers parsed from the per-sandbox --debug-log by matching known Sentry log<br>strings (Create container, Gofer started, PID, Installing seccomp filters, loader.go:519] CPUs:, Process should have started, and others).
Each run was 5 warmup iterations (discarded) plus 50 measured cold iterations using alpine:3.20 as<br>the rootfs and /bin/true as the init process - the cheapest possible boot. Image extraction is<br>timed separately, since the worker's ImageStore caches the extracted rootfs across sandboxes and<br>the per-sandbox cost is a near-zero digest cache hit.
We re-ran the same spike across:
Apple M1 Pro inside an OrbStack arm64 Ubuntu VM (10 vCPU)
AWS c7g.xlarge (Graviton 3, arm64)
AWS c7g.metal (Graviton 3, arm64, 64 vCPU)
AWS c7i.xlarge (Sapphire Rapids, amd64)
AWS c7i.metal-24xl (Sapphire Rapids, amd64, 96 vCPU)
GCP c4a-standard-4 (Google Axion, arm64)
GCP c3-standard-4 (Sapphire Rapids, amd64)
Each host ran Ubuntu 24.04. The cloud guests ran the stock Canonical 6.17.0-aws / 6.17.0-gcp<br>kernels; the OrbStack VM ran the custom 6.19.13-orbstack kernel. The spike was compiled fresh on<br>each host with go-1.23.
Hard numbers
CORE is the sum of runsc create + runsc start. OVERALL adds runsc wait (/bin/true<br>exit + reap) and runsc delete. All values are over 50 measured iterations, systrap, /bin/true<br>init.
gVisor cold boot<br>FIG. 01
Hover a row for p50 / p95 / max
Apple M1 ProOrbStack
78 103<br>Apple M1 ProOrbStackCORE78 p5083 p9584 maxOVERALL103 p50110 p95111 max
GCP c4a-standard-4Google Axion
129 174<br>GCP c4a-standard-4Google AxionCORE129 p50139 p95144 maxOVERALL174 p50186 p95195 max
AWS c7i.xlargeIntel
134 183<br>AWS c7i.xlargeIntelCORE134 p50145 p95154 maxOVERALL183 p50195 p95201 max
AWS c7a.xlargeAMD EPYC 9R14
137 185<br>AWS c7a.xlargeAMD EPYC 9R14CORE137 p50146 p95148 maxOVERALL185 p50195 p95199 max
GCP c3-standard-4Sapphire Rapids
143 188<br>GCP...