How Edera Closed the Xen dom0 NUMA I/O Gap for Good

Use Cases

Resources

Company

Contact

Products<br>For ContainersFor GPUs<br>Use Cases<br>Multi-Tenant IsolationUntrusted Code ExecutionAI Agent Sandboxing<br>Resources<br>The VineEventsDocs<br>WHO WE ARE<br>Why EderaSocial LovePress & Events<br>BEHIND THE SCENES<br>CareersLegal<br>FOLLOW<br>LinkedInYouTubeBluesky<br>Support<br>Contact EderaEdera GitHub

NUMA Part 4: Closing the Xen dom0 I/O Gap<br>July 16, 2026 - Steven Noonan

In the Part 3 of this series, we walked through the Xen paravirtual I/O architecture and what makes it expensive on a Non-Uniform Memory Access (NUMA) host: foreign-mapped pages with no NUMA metadata, kthreads landing wherever the scheduler happened to put them, an information gap between the dom0 kernel and the host's actual topology. We closed that part on a promise that the fix has two pieces - a structural one and a per-component one - and that part 4 would deliver both.<br>This part makes good on that - and along the way it covers two things we did not plan to fix. One we hit before the real work could start: dom0's own memory placement was so skewed that synthesising a topology for dom0 without fixing it first would have been an exercise in lying to a kernel that had no memory to back the lie with. That one is a prerequisite, not a detour - everything else in this part is built on top of it. The other we hit after the work was nominally done: a memory-placement bug in our own toolstack that had been silently broken for as long as multi-vnode vNUMA had existed in our tree, and only ever surfaced under a memory-bandwidth benchmark. That one is the genuine detour, and it waits until the end.<br>Let's start with the prerequisite.<br>The First Discovery: dom0's Memory Was in the Wrong Place<br>Stock Xen, when booting dom0, has to trim the host's BIOS memory map (the E820) down to the subset of memory dom0 is supposed to own. The trim walks the host E820 in physical address order, marking regions as dom0's RAM until it has set aside enough to satisfy dom0_mem. On a single-node host, this is harmless; dom0's memory comes from the only node available. On a multi-socket host, the same code is a real problem. The host's RAM regions are typically grouped by NUMA node in physical address order, so walking the E820 in order means dom0 gets all its memory from the lowest-address nodes first.<br>The worst case is severe. On a 128 GiB, 8-node host with dom0_mem=35% - the configuration our lab box runs - dom0 takes 100% of the lowest-address nodes' RAM and 0% of the others. Nodes 0 through 2 are almost entirely consumed by dom0; nodes 3 through 7 have no dom0-owned pages at all. Without dom0 being able to even attempt an allocation on the remote nodes, making dom0 NUMA-aware would have been synthesising a topology the kernel could never act on - the SRAT we generated would have described one node where 60% of dom0's memory lived alongside six nodes where it had nothing at all.<br>Our trim replaces this in-order grab with a proportional one, and it is one of the places our tree diverges from upstream Xen. It runs in two passes: pass one sums total host RAM; pass two gives each region a proportional share of the dom0_mem budget. The arithmetic is Bresenham-style, with rounding remainders accumulating so the total ends up exactly equal to dom0_mem. Dom0's memory ends up spread across every host node in proportion to that node's share of physical RAM.<br>With dom0 actually owning memory on every host node, the rest of the work could begin.<br>Making dom0 NUMA-aware<br>Three Pieces of NUMA Topology, for Any Xen Domain<br>Recall from part 2 that a NUMA-aware Xen domain sees three pieces of topology:<br>The ACPI System Resource Affinity Table (SRAT) , which maps CPUs and memory ranges to NUMA nodes.<br>The ACPI System Locality Information Table (SLIT) , which gives the inter-node distance matrix.<br>The x2APIC IDs in CPUID , which encode the per-CPU package/core/thread identifiers.<br>Any Xen domain that wants accurate NUMA awareness needs all three to be present and consistent. Upstream Xen does most of this work for PVH and HVM domUs when the toolstack asks for a vNUMA layout via XEN_DOMCTL_setvnumainfo: SRAT and SLIT are synthesised to match the requested topology. Upstream Xen does not synthesise the x2APIC IDs in CPUID from the vNUMA layout, however. That third piece has been an Edera extension since before the work this post describes began, and we extended it further along the way to handle non-power-of-two vCPU counts in multi-node domUs.<br>Dom0 is a different story. Until the work described in this post, upstream Xen synthesised none of the three for dom0. Dom0 saw a single flat NUMA node regardless of how many physical nodes the host had. The work this section covers extends the same three-piece synthesis to dom0, under one specific set of conditions.<br>The conditions: dom0 must be PVH; it must boot with dom0_vcpus_pin=1 so each dom0 vCPU is hard-pinned 1:1 to a specific host pCPU; the dom0 vCPU count must equal the host...