SandboxGym, How Well Your Sandboxes and Virtual Machines Really Work | Nebula Security

Risk map

X = finding count · Y = escapes×100 + cross-tenant×10 + bypass

Critical<br>High<br>Moderate

SandboxGym

Last updated July 15, 2026<br>Next update August 15, 2026

AI agents now browse, execute code, install dependencies, and reach private systems from inside<br>"isolated" execution environments. We audited eight sandbox and virtual-machine products, from<br>major labs to small startups, to measure how much of the advertised boundary actually holds.

Nearly every sandbox or virtual-machine implementation we examined contained a critical vulnerability<br>capable of collapsing its intended isolation boundary, turning systems designed to safely contain<br>untrusted code into potential entry points to the host, its data, and adjacent workloads.

See Results<br>Nominate a sandbox/VM

Goal

SandboxGym exists so users can see the real security posture of the sandbox or VM they rely on<br>for untrusted code — not only whether isolation fails, but how seriously vendors treat security<br>and how responsibly they fix what we report.

Disclosure policy<br>Each month we report findings to vendors and refresh this benchmark for the previous month.<br>The public update ships on the 15th of every month .

Experiment setup

We selected eight open-source sandbox and virtual-machine projects for this audit.<br>If an open-source sandbox belongs on this list and isn’t here yet,<br>nominate it here.

Method

Each product was audited by Nebula Security’s<br>code scanning agent<br>against a fixed open-source<br>snapshot. Scans covered the entire codebase at deep depth, with findings retained only when they<br>were exploitable from inside the sandbox and could collapse the intended isolation boundary.

Auditor Nebula Security

Scope Entire codebase, deep scan depth

Snapshot Fixed open-source commits, see appendix

Products surveyed

36<br>High-impact isolation failures, all exploitable from inside the sandbox

Full escapes / host code execution & memory corruption

100%<br>Of scored products had at least one isolation failure

01Finding count ranking

Ranked by total finding count , highest first. Every product we scored had at least one<br>isolation failure; the breakdown distinguishes full escapes,<br>policy bypasses, and<br>cross-tenant access.

More findings do not necessarily mean higher risk; failure class and impact still matter.

Full escape / host memory corruption<br>Policy bypass<br>Data exposure / cross-tenant

Finding composition by product

Stacked by failure class and ordered by total finding count.

Product<br>Isolation model<br>Escape<br>Bypass<br>Cross-tenant<br>Total<br>Disclosure

Counts include only issues with direct host memory corruption, host file overwrite, network-policy<br>bypass, or impact on other sandboxes on the host. Unauthorized out-of-sandbox file reads and<br>GPU-dependent findings are excluded from these totals.

Commit activity

Each point is a commit from Feb 9 through July 15, 2026; denser bands mean higher update<br>frequency. Red markers are introducing commits for<br>attributed findings (probable-cause analysis, not raw HEAD blame).

Commit activity shows how often developers maintain and update a product — how quickly they<br>ship fixes, and how often they introduce new issues. That cadence helps assess overall<br>security posture and engineering maturity.

smolvm tends to introduce issues in a short window — dense red markers during<br>intensive feature work, when security review does not keep pace with shipping. They did fix<br>vulnerabilities once we reported them, which shows they take disclosed issues seriously, but<br>lack an effective way to catch them during development.

runtm has the lowest update frequency for both bug fixes and new features. The<br>vulnerabilities we reported were never fixed.

beta9 began intensive feature work in late May. Development after that is steady,<br>and relatively few vulnerabilities were introduced in that burst compared with the volume of<br>commits.

cua often lands large changes as monolithic PRs. Every counted vulnerability was<br>introduced in a single commit<br>(7d1fa31:<br>~30k lines added across 197 files) — again pointing to missing security gates during development<br>rather than a slow drip of regressions.

sandbox-runtime’s published criticals are a late-June Windows ACL cluster: the<br>separate-user / deny-ACE foundation land in<br>4785dcc<br>(2026-06-26), and the follow-on refactor that drops group/ACL-stamp modes in<br>daef9ed<br>(2026-06-30) introduces the remaining criticals. Red markers on the sandbox-runtime lane are<br>snapshot-blame introducing commits for those findings.

nono is the fastest on remediation: they acknowledged and fixed reported issues in<br>about two days — the quickest response in this set. Their developers also noted they are still<br>in fast development, so vulnerabilities are expected for now; we expect security focus to<br>increase as the product stabilizes.

amika provides a Docker-based runtime. Relying on mature Docker isolation means<br>severe runtime escapes are rare,...