Debugging performance regressionsReproducible software deployment for high-performance computing.

About<br>Packages<br>Channels<br>Events<br>Jobs<br>Blog

Debugging performance regressions<br>Ludovic Courtès — July 10, 2026<br>Performance<br>MPI<br>High-performance computing

MPI software stacks are complex beasts that play a key role in<br>application performance. Guix packages Open MPI for a wide range<br>high-speed<br>interconnects—InfiniBand,<br>Omni-Path, Slingshot, ROCE, you name it—with the goal of achieving peak<br>performance for each one of them.<br>The key challenge from a packaging perspective is to ensure no<br>performance regression occur for any of these network types when<br>Open MPI or its dependencies change. Administrators of clusters where<br>Guix is the main deployment tool typically have checks in place for<br>this. But what if a regression does occur?<br>This blog post is a report from the trenches, showing how a performance<br>regression of the MPI stack on Slingshot interconnects was identified.<br>The debugging story is in itself a showcase for the control and<br>transparency that Guix affords.<br>The plot<br>Slingshot support is relatively<br>recent<br>in Open MPI as packaged by Guix and, until now, concerned a relatively<br>small part of Guix usage in HPC. No automated performance regression<br>testing was in place for this interconnect, primarily due to<br>difficulties accessing hardware that would allow us to do that. And so,<br>the inevitable happened: a performance regression struck.<br>The symptom is simple: running an MPI bandwidth/latency benchmark would<br>show poor performance (here we force the use of version 5 of Open MPI<br>with<br>--with-input):<br>$ guix shell intel-mpi-benchmarks --with-input=openmpi=openmpi@5 openmpi@5 -- \<br>mpirun IMB-MPI1 PingPong<br># PingPong

#---------------------------------------------------<br># Benchmarking PingPong<br># #processes = 2<br>#---------------------------------------------------<br>#bytes #repetitions t[usec] Mbytes/sec<br>0 1000 37.94 0.00<br>1 1000 38.30 0.03<br>2 1000 38.41 0.05<br>1048576 40 575.85 1820.92<br>2097152 20 983.58 2132.15<br>4194304 10 1817.18 2308.15Uh-oh—peak bandwidth is 2.3 GB/s instead the expected 25 GB/s or so.<br>The reason this happens is because libcxi (the low-level Slingshot<br>support library) errors out, and then libfabric (which sits right above<br>libcxi) propagates that error to Open MPI, which in turn falls back to<br>plain TCP/IP over Ethernet.<br>We can get debugging info from Open MPI, libfabric, and libcxi by<br>passing the right flags and environment variables:<br>$ FI_LOG_LEVEL=9 guix shell intel-mpi-benchmarks --with-input=openmpi=openmpi@5 openmpi@5 -- mpirun --mca btl_ofi_verbose 3 --mca pml_ucx_verbose 3 --mca btl ofi --mca mtl ofi --mca btl_base_verbose 100 IMB-MPI1 PingPong<br>[c1510:1386001] mca: base: components_register: registering framework btl components<br>[c1510:1386001] mca: base: components_register: found loaded component ofi<br>[c1510:1386001] mca: base: components_register: component ofi register function successful<br>[c1510:1386001] mtl_ofi_component.c:343: mtl:ofi:provider: cxi<br>[c1510:1386001] mtl_ofi_component.c:368: mtl:ofi:provider:domain: cxi0<br>libfabric:1386001:1781018439::cxi:domain:cxip_domain_enable():457 c1510: Security Issue: Using unrestricted service ID 1 for cxi0. Please provide a service ID via auth_key fields.<br>libfabric:1386001:1781018439::cxi:domain:cxip_cp_get():109 c1510: Failed to allocate CP, ret: -22 VNI: 1 TC: BEST_EFFORT TYPE: DEFAULT<br>libfabric:1386001:1781018439::cxi:domain:cxip_cp_get():112 c1510: Remapping original TC from BEST_EFFORT to BEST_EFFORT<br>libfabric:1386001:1781018439::cxi:domain:cxip_cp_get():137 c1510: Failed to allocate CP, ret: -22 VNI: 1 TC: BEST_EFFORT TYPE: DEFAULT<br>libfabric:1386001:1781018439::cxi:domain:cxip_cmdq_alloc():289 c1510: Failed to allocate CP: -22<br>libfabric:1386001:1781018439::cxi:ep_ctrl:cxip_ep_cmdq():95 c1510: Unable to allocate CMDQ, ret: -22<br>libfabric:1386001:1781018439::cxi:ep_ctrl:cxip_ep_ctrl_init():567 c1510: Failed to allocate control TXQ, ret: -28<br>libfabric:1386001:1781018439::cxi:ep_ctrl:cxip_ep_enable():691 c1510: cxip_ep_ctrl_init returned: -262<br>[c1510:1386001] mtl_ofi_component.c:1093:fi_enable failed: Invalid resource domain<br>[c1510:1386001] select: initializing btl component ofi<br>[c1510:1386001] btl_ofi_component.c:318: btl:ofi:provider_include = "(null)"<br>[c1510:1386001] btl_ofi_component.c:320: btl:ofi:provider_exclude = "shm,sockets,tcp,udp,rstream,usnic,net"<br>libfabric:1386001:1781018439::core:core:fi_getinfo_():1391 fi_getinfo: provider efa output empty listThe main clue here are the “Failed to allocate CP” errors that<br>originate in the Slingshot “provider” of<br>libfabric<br>(CP stands for communication profile in libcxi).<br>Bisecting the distro<br>Where do we go from here? Short of being a libcxi expert, a good way<br>forward is to find out what change introduced that regression. There<br>are two good news here: first the MPI stack provided by Guix is entirely<br>self-contained—so software on the machine that is not provided by Guix<br>is unlikely to be the culprit—, and second we...