Xen Grant Tables and dom0 NUMA Blind Spot Explained

Use Cases

Resources

Company

Contact

Products<br>For ContainersFor GPUs<br>Use Cases<br>Multi-Tenant IsolationUntrusted Code ExecutionAI Agent Sandboxing<br>Resources<br>The VineEventsDocs<br>WHO WE ARE<br>Why EderaSocial LovePress & Events<br>BEHIND THE SCENES<br>CareersLegal<br>FOLLOW<br>LinkedInYouTubeBluesky<br>Support<br>Contact EderaEdera GitHub

NUMA Part 3: Xen's PV I/O Path, and Its NUMA Blind Spot<br>July 9, 2026 - Steven Noonan

In the previous part of this series, we walked through how Edera places zones on a host: the algorithm that picks which physical Non-Uniform Memory Access (NUMA) nodes a zone lives on, how its vCPUs are pinned, what topology the guest itself sees at boot. The story ended on a promise that placement is the first layer of a larger design and that each layer above it multiplies the value of getting the first layer right. This part walks through what sits immediately above placement: Xen's paravirtual I/O drivers, the ones that carry every byte of network and storage traffic between domUs and the world. Until recently, that layer was the single biggest reason good placement decisions did not carry through to the bytes hitting hardware.<br>There is a detective story coming later in this post. Two zones, identically configured, on the same host, performing measurably differently. A guess at where the actual work is landing. A diagnosis pointing at a piece of NUMA information that has been silently missing from every Xen dom0 in production for the entire history of the relevant kernel code. We will get to all of it. First, though, the architecture, because the diagnosis only makes sense once the moving parts are on the table.<br>How Does Xen Paravirtual I/O actually work?<br>Xen's split-driver design has been roughly stable since the original paravirtualization work in the early 2000s. The same set of moving parts appears across every PV driver - netfront and netback for networking, blkfront and blkback for storage, several others for less common roles - so once you have the skeleton, every PV driver is a variation on it. What differs between them, and what decides their NUMA cost, is how each one moves the payload.<br>There are four moving parts.<br>The Shared Ring Buffer<br>A small in-memory circular array of request and response slots, with separate producer and consumer indices on each side. The frontend (running inside the domU) pushes requests into the ring; the backend (running in dom0) drains them, processes them, and pushes responses back. The ring is the protocol surface between the two halves of the driver.<br>Grant Tables<br>A controlled mechanism by which a domU can expose a page of its own memory to dom0. The domU allocates a page in its own RAM, fills out a grant table entry saying "dom0 may access this page", and hands the reference number to dom0 via xenstore. Dom0 then asks the hypervisor to map the page into its own address space. On a PVH dom0, the foreign frame appears in dom0's view through the host's second-level address translation (SLAT); the result is that dom0's kernel sees a struct page for the ring as if it were ordinary memory, except that the underlying physical bytes live in the domU's RAM, not dom0's. Grant tables are the mechanism that exposes the ring.<br>Event Channels<br>Per-domain virtual interrupts. When the frontend has added work to the ring, it raises an event channel telling the backend "go check the ring". When the backend has produced a response, it raises an event channel telling the frontend. Without event channels, both sides would have to spin polling the ring to find new work; with them, either side can sleep until there is something to do.<br>Where the Actual Data Lives<br>The ring never carries the payload, only descriptors: which operation, where the result goes, and the grant references that point at the buffers. The payload itself is a separate set of domU-owned pages, granted to dom0 the same way the ring is. What differs from one driver to the next is how dom0 reaches those pages, and that one choice, made differently by block and by network, is what decides the NUMA cost. It gets its own section below.<br>domUblkfrontring bufferpayload buffersthe bytes the I/O movesdom0backend kthreadthe thread that runs blkbackblkbackmapped ring viewbacked by domU's RAMmapped payload viewthe bio is built on itdisk DMAphysical storageevent channelwakes the remote sidegrant table mappingpersistentgrant table mappingpersistent, not copiedXen hypervisor<br>The asymmetry to keep in mind: the frontend allocates the ring (real RAM inside the domU); the backend never owns the ring's underlying memory. The backend's struct page for the ring is a placeholder that lives in dom0's mem_map but is backed by the foreign frame through SLAT. To dom0's kernel the placeholder looks like a normal page - kmap, page_address, the usual accessors all work - but its physical home is in the domU's memory, not dom0's. Hold onto that distinction; it ends up doing the load-bearing work in the...