Firecracker disk snapshots in O(changed bytes), not O(disk size) — Tensorlake

HomeBlogPricingCareersDocsGitHubSlack communityTalk to FounderDashboard →

◆ ON THIS PAGE 10 MIN<br>Sandboxes used to be a place where an agent ran a Python script, opened up a browser to read web pages, and went away. The coding agents our users run today inside sandboxes scaffold a full-stack application, start Postgres next to it, load a few gigabytes of test data, and run an integration suite. The sandbox needs to be close to production for the tests and benchmarks to provide meaningful signals.

We build our sandbox infrastructure on Firecracker, and we built it for performance from the start. This post is about an overlay-based storage engine we built for Firecracker: the design and performance measurements.

Spoiler for the impatient: the copy-on-write subsystem we added made the disk faster, and it turned snapshots into something a running database barely notices.

Firecracker's storage story ends at a raw disk image

Stock Firecracker gives you a raw disk image over virtio-block. It's genuinely fast but is also the entire storage feature set. No copy-on-write, no shared base images, and the only way to snapshot a disk is to keep the whole file.

Duplicating a 100 GB disk image on the NVMe we benchmark below takes 101 seconds, and it produces 100 GB of snapshot. It does not matter if the guest changed one file or rewrote everything, the cost of a snapshot is the whole disk, and the disk has to stay quiesced while you copy it. Snapshotting a Tensorlake sandbox with the same one-file change will cost 167 milliseconds and 105 MB.

Stock Firecracker keeps no record of which blocks the guest wrote. The disk is one opaque file, so the only safe snapshot is to copy every block, changed or not. Our storage engine sits in the write path and marks each 4 KiB block as the guest dirties it, so at snapshot time it already knows the 105 MB that changed and skips the other 99.9 GB.

The native block overlay

Our fork adds an overlay engine to Firecracker's block device. The guest cannot tell it exists. From inside the VM you see one ordinary virtio-block disk with ext4 on it. No special drivers, no FUSE mounts, no overlayfs. If you ran mount in the guest looking for the trick, you would not find it.

Under the hood, each drive is backed by a read-only base image (the container image, materialized as a flat ext4 block image), a sparse per-VM live overlay file, and optionally a stack of frozen overlay layers from earlier snapshots. The bookkeeping is two bitmaps at 4 KiB granularity: a dirty bitmap that records which blocks live in the overlay, and a zero bitmap that records blocks known to be all zeroes. That is the whole data structure. A read checks the bitmaps and issues exactly one read against the right file. A read of a known-zero block does no I/O at all. Writes land in the live overlay and flip bits. The base is never written, which is what makes it shareable between every sandbox on the host.

If you have ever traced a qcow2 read through its two-level translation tables, this is the part to appreciate: there are no translation trees here and no allocation metadata in the request path. A flat bitmap, then the data.

A few details do most of the performance work:

The live overlay is mapped into the VMM's address space, so a 4 KiB guest write is a memcpy plus a bitmap update. Not a syscall.

Flushes coalesce adjacent dirty ranges (we merge across gaps up to 8 KiB) before syncing.

Base reads go through a small set-associative cache with sequential readahead of 2 to 32 blocks, so a cold walk through the base, think npm install or a database scan, issues big reads instead of a stream of 4 KiB ones.

There is an io_uring variant of the engine with registered buffers and fixed files. More on that below, because it taught us something.

Overlay layers pile up as a sandbox ages, so the engine can flatten old ones into each other. The kernel copies the blocks directly between files; the data never passes through the VMM.

Guestext4 on one plain virtio-block diskreadwrite · memcpy, no syscallSTORAGE ENGINE · INSIDE THE VMMdirty block trackerbitmaps · one bit per 4 KiB blockknown-zero block: no I/ONVME · FILES ON THE HOSTbase imageread-only · shared by every sandboxclean blocksoverlay filesparse · holds only what changeddirty blocks · coalesced flushRead and write paths through the storage engine, for one sandbox drive.<br>What the benchmarks say

We benchmark with fio inside the guest, through the whole stack: ext4, virtio-block, the storage engine, and the NVMe underneath. Every configuration runs the same binary, warmed, five times over; numbers are p50. The exact commands, hardware, and controls are in the appendix.

workloadstock raw drive, Syncstock raw drive, Asyncnative overlayrandom read, 4 KiB QD6464,100 IOPS62,300 IOPS66,100 IOPS random write, 4 KiB QD6462,400 IOPS49,300 IOPS65,000 IOPS sequential write, 1 MiB2.6 GiB/s3.1 GiB/s4.7...