On-the-fly snapshot compression for elastic inference at scale | Doubleword

When an inference engine starts, it performs several initialization steps before it can handle requests. These include importing runtime dependencies, loading the model weights into GPU memory, allocating the KV cache, running an initial forward pass that triggers just-in-time compilation, and capturing CUDA graphs. Depending on the model size and configuration, this initialization process can take several minutes to complete.

In Cloudburst, we used checkpoint and restore to capture SGLang after this initialization setup had completed. This allows us to create new replicas from that prepared state instead of repeating the same work.

The cost then moves from initialization to storage. A snapshot of a large model can contain tens or hundreds of gigabytes of host memory, which must be written during checkpointing and read during restore. Compression reduces the amount of data stored and read, but it improves restore time only when decompression keeps pace with storage.

CRIU now supports LZ4 compression directly in its memory checkpoint and restore paths. It compresses memory pages while writing the checkpoint and decompresses them during restore, avoiding the need for intermediate storage for uncompressed data and a separate compression pass.

Compressing memory inside CRIU

CRIU stores process memory in pages-*.img files and uses the corresponding pagemap images to describe where those pages belong in the restored address space. To support memory compression, this image format is extended so that CRIU can choose how to store each block of pages.

A block that contains only zeroes is represented in the snapshot without storing any page data. As model weights is the largest component of the memory state and often have a low compression ratio, CRIU stores the compressed bytes as an LZ4 block only if the compressed block size is reduced by more than 12.5%. Otherwise, the original bytes are stored as RAW to avoid the performance overhead of decompression during restore.

This decision is made independently for each block, allowing compressible and low-compression memory to coexist in the same checkpoint.

The pagemap stores the size and representation of each block, while the pages image contains the bytes stored for RAW and LZ4 blocks. During restore, CRIU uses this metadata to locate the stored bytes and reconstruct the original memory pages.

CRIU compressed memory checkpoint and restore pipelineDuring checkpoint, process memory passes through the page pipe and compression step. Non-zero block bytes go to pages image files while stored sizes and pages-per-block metadata go to the pagemap. During restore, CRIU validates the metadata, builds restore batches, and reads only stored bytes. A zero block reads no page data, a raw block reads its full in-memory block size, and an LZ4 block reads fewer stored bytes before decompression. Every path reconstructs the same block size in restored process memory.CHECKPOINTRESTORE1Process memoryprocess mappings2Page pipepages + ranges3Compress blockszero · raw · LZ44pages-*.imgmemory page dataPagemap metadatasizes + pages/blocktotal stored bytesread Nread S1Inventory +pagemapblock metadata2Validatemetadatacounts · sizes · offsets3Build restorebatcheszero · raw · LZ4 pathspages-*.imgmemory page data4Read stored bytessum(block_sizes)ZEROS = 0fill NRAWS = Nno decompressionLZ40 decompress N5RestoredmemoryN-byte blockmetadata / controlimage readmemory write<br>Checkpoint<br>Process memory checkpointed mappings

Page pipe pages + ranges

Compress blocks zero · raw · LZ4

pages-*.img memory page data<br>Pagemap metadata sizes + pages/block

Restore<br>Inventory + pagemap block metadata

Validate image metadata counts · sizes · totals · offsets

Build restore batches choose zero, raw, or LZ4 path

No page data read<br>Zero block S = 0 · fill N

Page data read<br>pages-*.img input memory page data<br>Read stored bytes total for the planned range<br>Raw block S = N · no decompression<br>LZ4 block 0

Restored process memory N bytes from every path

Reading the full flow<br>Metadata controls the work; the pages image supplies stored bytes<br>For stored size S and in-memory block size N, restore reads 0 bytes for zero, N for raw, or S for LZ4. Every path writes N bytes into restored process memory.

Dashed arrows show metadata and control. Solid arrows show reads from the pages image and writes into restored process memory. For each block, S is the stored byte count and N is the number of bytes written to restored process memory.<br>The checkpoint path follows a non-zero block, whose data CRIU writes before recording the corresponding pagemap entry. Zero-filled blocks add no data to the pages image. If every block in an entry is stored raw, CRIU can omit the block metadata and use the standard uncompressed page-image format.

Restoring compressed memory

The three block representations follow different paths during restore. ZERO blocks require no...