ModelExpress: Distributing Model Artifacts at the Speed of Light | NVIDIA Technical Blog

Technical Blog

Subscribe

Related Resources

Agentic AI / Generative AI

English中文

ModelExpress: Distributing Model Artifacts at the Speed of Light

Jul 24, 2026

By Hyunjae Woo, Kavin Krishnan, Nicolas Noble, Zheng Luo and Ganesh Kudleppanavar

Like

Discuss (0)

AI-Generated Summary

Like

Dislike

NVIDIA ModelExpress (MX) efficiently accelerates the model weight lifecycle by selecting the fastest available path for loading model weights, prioritizing direct GPU-to-GPU P2P RDMA transfers via NIXL, and reducing reliance on object storage and host memory.<br>MX employs advanced strategies including multithreaded streaming, atomic distributed caching, GPUDirect Storage, and runtime path selection to optimize cold starts, minimize redundant data movement, and automate optimal transfer methods across diverse cluster environments.<br>The platform enables rapid weight and kernel cache artifact transfer, supports receiver-driven RL refit workflows, integrates with vLLM, SGLang, Dynamo, and llm-d, and incorporates optimizations such as VMM arena registration to significantly reduce startup and registration overheads in production LLM deployments.

AI-generated content may summarize information incompletely. Verify important information. Learn more

Every byte moved has a cost. As model checkpoints grow to hundreds of gigabytes or even a terabyte, that cost adds up quickly. To make things even worse, moving these model weights around the cluster is extremely common. For instance, a cold start may pull weights from remote storage into GPU memory; autoscaling and rolling updates must populate each new replica; and RL post-training continuously moves updated weights from trainers to roll out workers. These may look like different workflows, but they impose the same recurring tax: time spent moving weights before useful work can begin.

ModelExpress: Accelerating the model weight lifecycle

NVIDIA ModelExpress (MX) is built around a simple idea: Before loading a model, first ask where a compatible copy of its weights already lives. Rather than treating every replica as an independent cold start, MX chooses the fastest available source and transfer path.

When a serving peer already holds compatible weights in GPU, MX transfers them directly from GPU to GPU over P2P RDMA via NVIDIA Inference Xfer Library (NIXL), bypassing redundant access to object storage, local disk, and host memory. When no peer is available, MX bootstraps from the fastest supported path by streaming from an object store without landing on disk or reading local files directly into GPU memory.

MX transfers DeepSeek-V4 Pro weights and JIT Kernel cache artifacts from a serving replica into a fresh replica in under 10 seconds, reducing the total startup time to 1 minute 44 seconds from 8 minutes. The rest of the post shows how MX selects the fastest available path to GPU memory, prioritizing P2P RDMA from a serving replica and eliminating redundant downloads and copies along the way. It then extends the same approach to reusing kernel caches and distributing RL weight updates.

Figure 1. Overview of ModelExpress

Accelerating every stage from remote storage to GPU memory

Every new worker must get its weights from one of three places: remote storages (e.g. HF or S3), local storage, or another worker already serving the model. For the first worker, there is no peer yet, so it must bootstrap from storage. MX can stream the checkpoints from object storage or load it from fast local storage, removing avoidable copies along either path.

Once that first worker is serving, the preferred source changes. Its weights are already resident, post-processed, and laid out in GPU memory, so every compatible worker after it should load directly from that peer over P2P RDMA. MX makes this transition automatically: bootstrap once from storage, then scale out GPU to GPU, falling back to storage only when no compatible peer is available.

Starting the first worker: Bootstrap from storage

Remote object storage to GPU: Avoiding local disk

When the checkpoint lives in a cloud bucket and you would rather not provision and manage a disk cache tier, MX uses the Model Streamer to pull safetensors through a reusable CPU staging buffer and into GPU. The checkpoint never lands on local disk, eliminating the intermediate download, reload, and storage volume.

Model Streamer uses a multithreaded tensor reader to fetch tensor ranges concurrently across checkpoint shards. As tensors arrive, it pipelines remote reads with GPU placement: completed tensors are passed to the inference engine while later tensors are still being fetched. This keeps the storage, network, and GPU copy paths busy while reusing a bounded amount of host memory.

In tensor-parallel deployments, the participating ranks divide the remote reads and share the results, typically over NCCL, instead of having every rank download the...