Scaling to 1 million concurrent sandboxes in seconds
Inkling by Thinking Machines is now available on Modal Learn more
All posts<br>Back Engineering<br>July 16, 2026•10 minute read
Scaling to 1 million concurrent sandboxes in seconds
Colin Weld Member of Technical Staff
Connor Adams Member of Technical Staff
At Modal, we build sandboxes, among other things. Agents run in sandboxes, and agents are eating software. Today, Modal runs millions of sandboxes per day, supports up to fifty thousand concurrent sandboxes per customer, and supports a variety of use cases at scale, from reinforcement learning to background agents.<br>Increasingly, our users require more and more sandboxes, created at higher and higher rates. Reinforcement learning can require running millions of sandboxes concurrently, and creating bursts of hundreds of thousands of sandboxes at the beginning of rollouts. Similarly, agents increasingly require massive scale and high concurrent creation rates to deal with traffic bursts.<br>Our existing sandbox platform is really good, but it wasn’t designed for these scales; nor is any other existing solution. We’re obsessed with scale and performance, and we want our infrastructure to accelerate the growth of agents, not add friction. So we went back to the drawing board.<br>Over the last few months, we’ve rebuilt our core sandbox platform from the ground up for both scale and reliability. On our new system, users can run millions of sandboxes concurrently and create tens of thousands of sandboxes per second. We’ve removed all central bottlenecks from our control plane so there are no practical scaling limits, and we’ve optimized every part of container scheduling and startup, simplifying the scheduling path to a layer of load balancers which create containers directly on our worker fleet.<br>As a demonstration of what our platform is capable of, we’ve run a million sandboxes concurrently, creating all 1 million in under a minute.<br>Evidence that we can run a lot of sandboxes.Why most solutions don’t scale<br>Running 1 million sandboxes pushes the limits of any container platform, both because of the sheer number of containers, but also because running this many sandboxes requires many tens of thousands of compute nodes. There will be many operations which are either O(containers), O(nodes), or both, which will cause traditional container platforms to hit scaling limits.<br>For Kubernetes, as an example:<br>The scheduling algorithm is O(n x p) for n nodes and p pods in the worst case, and scheduling is serialized by default.<br>Each pod causes multiple writes to etcd (the central Kubernetes durable store) over its lifetime, which can create serious issues under high pod creation rates or high pod churn, and etcd is not natively shardable within a keyspace<br>Each node must write to etcd at least once per heartbeat interval to signal liveness, so baseline etcd write load is O(nodes) completely independent of pod creation<br>Approximation of Kubernetes scheduling flow. New pods are written to etcd (a strongly consistent durable store) by the API server. The Kubernetes scheduler watches for new unassigned pods, and assigns them to nodes via a call to the API server, which again writes to etcd; after this write goes through, then a node can start the pod.Kubernetes can be scaled, but it requires serious work. To run large numbers of nodes, etcd generally must be rewritten or replaced. Supporting high scheduling throughput requires building a complex scatter-gather system to parallelize the scheduling algorithm while still maintaining a single source of truth for pod state. Sharding and parallelization is not easy by default, because Kubernetes relies on strong consistency as a backbone of its design.<br>Modal’s original sandbox architecture has similar issues. Like Kubernetes, we rely on strong consistency throughout our backend, so creating and scheduling sandboxes requires global coordination, and O(sandboxes) writes to Postgres, which we cannot trivially shard.<br>Modal’s original sandbox control plane architecture. When sandboxes are created, they are placed on a queue and written to Postgres. Scheduling is optimistic and run in parallel, with central coordination required to avoid conflicts. Assigning a sandbox to a worker (compute node) requires an additional write to Postgres.Because we don’t build on Kubernetes, we’ve been able to scale out many parts of this system. For example, scheduling is parallelized by default, which allows us to achieve very high burst sandbox creation rates. But as we scaled to larger and larger numbers of nodes and sandboxes, we continually encountered new bottlenecks arising from operations which were either O(sandboxes) or O(nodes) but were not easy to scale out.<br>For example, we run a durable workflow for each sandbox that finishes, so high sandbox churn rates would create massive event backlogs. We’d repeatedly run into RPCs called at a rate of O(sandboxes) which caused unexpected load issues throughout...