👋" />

RDMA in gVisor: A Deep Dive into GPU Networking | Alessio Toniolo

← Writing<br>It is helpful to think of the Operating System (OS) on a machine as having three distinct spaces:

1. The kernel space, or the OS's core. It manages hardware resources like RAM and CPU.<br>2. The user-space, which is the home of most user processes. These applications communicate with the OS via system calls (syscalls) to the kernel.<br>3. The device driver space, which contains interfaces for programs that run on hardware like GPUs and Network Interface Cards (NICs).

DMA (Direct Memory Access) allows devices to directly access RAM without copying data through the CPU.<br>Reads and writes to DMA-able memory1 bypass the kernel, reducing the latency of data-transfers between RAM and GPUs, NICs, etc.

RDMA (Remote Direct Memory Access) expands DMA to different machines on a network.<br>One peer can read and write into another peer's memory without involving either peer's kernel.<br>The bandwidth required for RDMA payloads can be much larger than the standard 10Gbps Ethernet, especially during large-scale model training2.

Special networking hardware for RDMA exists that can<br>send up to 800 Gbps of data over the wire. The popular transport types are InfiniBand, which is dominated by NVIDIA-acquired Mellanox, and RoCE (RDMA over Converged Ethernet), which implements InfiniBand semantics on upgraded ethernet hardware. To allow the user-space direct InfiniBand/RoCE access, RDMA consumers make requests to "uverbs" devices, which call into a software library called libibverbs.<br>Figure 1: Applications communicate by creating work requests to the low-level hardware. Messages get sent over the network and acknowledged by the receiver host.<br>Successful completion events are propagated back to the application. Pictured are terms like Work Queue Elements (WQE), Completion Queue Elements (CQE), and Queue Pairs (QP). Here is my Anki deck of<br>RDMA jargon for the curious. The gVisor Container Runtime<br>gVisor is an application kernel for containerized applications (see "What is a container?").<br>Using our three-space model, the container is a fourth space, and gVisor acts as an intermediary between the container space and<br>the host space. For example, a containerized application calls memcpy() on a CUDA device. gVisor:

1. Intercepts the syscall via a seccomp filter.<br>2. Checks what CUDA device it's targeting based on the file descriptor (FD) of the call.<br>3. Issues the syscall if the container mounted the device and therefore owns it. Figure 2: The "Sentry" runs as a separate process in the gVisor sandbox and traps syscalls from the containerized application to the host kernel. The Sentry in gVisor is given the same virtual address space, filesystem access, network namespace, and mounted devices that the sandbox gets; only, the Sentry controls what the container can access according to gVisor's security model3. Supporting GPUDirect RDMA in gVisor<br>NVProxy and TPUProxy are two popular features in gVisor that allow pass-through access to GPU and TPU devices respectively. Supporting RDMA with InfiniBand is a highly-requested feature, and the target implementation adds a "RDMAProxy" library interface similar to NVProxy and TPUProxy.

My goal is to match the bandwidth performance of RDMA to within runc, a popular container runtime lacking the security isolation of gVisor.<br>In default operation, CUDA applications use GPUDirect RDMA, which allows for true P2P data transfer4 between GPUs. Memory Registration in RDMA<br>After a gVisor container receives all information about the host's RDMA hardware, the application registers memory for the data transfer, whether from host or GPU RAM. An RDMA-enabled NIC receives a scatter-gather list containing<br>the physically contiguous regions of memory. The "scatter" comes from how the OS page table works: virtual regions of memory may be scattered across many different physical pages. "Gather"<br>refers to a NIC grouping these physical pages into one logical range for transmission or reception of data. Figure 3: The RDMA NICs map 1:1 to every GPU on the host. During memory registration, uverbs programs the NIC with a scatter-gather list of<br>memory regions to be used in RDMA. The scatter-gather list contains virtual addresses and the corresponding length of physically contiguous memory starting at that address. RDMA is designed to bypass the kernel in the data-path (hot-path) of a RDMA transfer. However, an application cannot simply map some memory,<br>program it into the NIC, and start the RDMA transfer. What happens if the kernel realizes that these pages are no longer being referenced or written to<br>and decides to evict the memory pages from RAM? To keep the kernel completely out of the hot path, most RDMA applications "pin" the virtual regions of memory to prevent<br>the kernel from moving the underlying physical backing of memory critical in the RDMA hot-path.

For gVisor, the RDMA application process picks the virtual address regions it wants to...