A Practitioner Guide to AWS EFA Programming

A Practitioner Guide to AWS EFA Programming

By: Yang Zhou (UC Davis), Zhiying Xu (AWS), and the UCCL team

Date: April 13, 2026

We share a practical guide on how to program AWS EFA NICs for GPU communication workloads beyond NCCL.<br>We cover the EFA libibverbs interface, QP creation, address handling, RDMA write operations, and the critical challenges of ordering and atomics on EFA's multi-path SRD transport.<br>We also describe how to address these constraints with software-emulated atomics and receiver-side reordering, as implemented in UCCL .

Code: uccl-project/uccl/experimental/misc/efa_rdma_write.cc (Apache-2.0)

Introduction

AWS uses customized RDMA NICs called EFA (Elastic Fabric Adapter) across their GPU instances: Hopper-based p5, p5e, p5en VMs and Blackwell-based p6 VMs. Under the hood, EFA runs a proprietary multi-path transport protocol called SRD (Scalable Reliable Datagram), described in the SRD paper. SRD supports efficient multi-pathing to avoid single-path network congestion in datacenter networks, without relying on PFC (Priority Flow Control) which is notoriously hard to manage at large scale.

img:first-child]:col-start-2 [&>img:last-child]:col-start-4 [&>img]:!my-0 [&>img]:h-auto [&>img]:max-w-[450px] [&>img]:min-w-0 [&>img]:w-full">

Left: Traditional single-path transport sends all packets along one fixed route, which is vulnerable to congestion at any single hop. Right: EFA's SRD multi-path transport dynamically sprays packets across multiple paths, balancing load and avoiding hotspots without PFC.

However, AWS EFA has different behaviors from conventional RDMA NICs like NVIDIA ConnectX-7 and Broadcom Thor-2, especially when used with GPUs. For a long time, most users did not need to directly program EFA NICs because they simply used NCCL with the aws-ofi-nccl plugin. However, new communication paradigms have recently emerged, such as GPU-initiated communication (e.g., DeepEP), point-to-point transfers (e.g., KV cache transfer for PD disaggregation, RL weight sync), and efficiently supporting them requires tight integration between GPU kernels and RDMA devices. We can no longer rely on one-size-fits-all NCCL collectives.

The UCCL team has done extensive investigation into AWS EFA NICs and built efficient EP and P2P libraries that support heterogeneous RDMA NICs, including EFA. In this blog, we share our experience with a heavy emphasis on how to use EFA for advanced non-NCCL use cases . We cover:

EFA libibverbs programming : QP creation, address handling, write operations

EFA ordering and atomics emulation : How to work around EFA’s lack of ordering guarantees and native atomics

EFA performance characteristics : Strengths and tradeoffs compared to other RDMA NICs

EFA libibverbs Programming

EFA supports both libibverbs and libfabric. From our perspective, libfabric is essentially a wrapper around libibverbs. Most practitioners are more familiar with the libibverbs interface, since other RDMA NICs (NVIDIA ConnectX, Broadcom Thor-2) all use it as their primary programming interface. Thus, this blog focuses on the EFA libibverbs interface.

The full working example is available at efa_rdma_write.cc. You can compile and run it across two EFA-enabled instances to test RDMA write operations.

Creating EFA QPs

Creating an EFA QP differs from standard InfiniBand/RoCE QPs in several important ways. EFA uses the extended QP creation API with a vendor-specific EFA attribute struct:

struct ibv_qp_init_attr_ex qp_attr_ex = {0};<br>struct efadv_qp_init_attr efa_attr = {0};

qp_attr_ex.comp_mask = IBV_QP_INIT_ATTR_PD | IBV_QP_INIT_ATTR_SEND_OPS_FLAGS;<br>qp_attr_ex.send_ops_flags = IBV_QP_EX_WITH_RDMA_WRITE |<br>IBV_QP_EX_WITH_RDMA_WRITE_WITH_IMM |<br>IBV_QP_EX_WITH_RDMA_READ;

qp_attr_ex.cap.max_send_wr = 256;<br>qp_attr_ex.cap.max_recv_wr = 256;<br>qp_attr_ex.cap.max_send_sge = 1;<br>qp_attr_ex.cap.max_recv_sge = 1;

qp_attr_ex.pd = rdma_ctl->pd;<br>qp_attr_ex.sq_sig_all = 1;<br>qp_attr_ex.send_cq = ibv_cq_ex_to_cq(rdma_ctl->cq_ex);<br>qp_attr_ex.recv_cq = ibv_cq_ex_to_cq(rdma_ctl->cq_ex);<br>qp_attr_ex.qp_type = IBV_QPT_DRIVER;

efa_attr.driver_qp_type = EFADV_QP_DRIVER_TYPE_SRD;<br>efa_attr.sl = 8; // low-latency service level

struct ibv_qp* qp = efadv_create_qp_ex(<br>rdma_ctl->ctx, &qp_attr_ex, &efa_attr, sizeof(struct efadv_qp_init_attr));<br>A few key differences from standard RDMA QP creation:

QP type is IBV_QPT_DRIVER , not IBV_QPT_RC or IBV_QPT_UD. EFA uses a custom driver-type QP that runs the SRD protocol underneath.

Extended CQ (ibv_cq_ex) is required. EFA mandates ibv_create_cq_ex with wc_flags = IBV_WC_STANDARD_FLAGS. Using the legacy ibv_create_cq will fail.

No RC-style connection setup. Unlike RC QPs that require exchanging QPN, GID, and LID to transition through INIT → RTR → RTS with remote QP info, EFA QPs transition through these states locally without specifying a remote endpoint:

// INIT - no remote QP info needed<br>attr.qp_state = IBV_QPS_INIT;<br>attr.pkey_index =...