What is RDMA over Converged Ethernet (RoCE)?<br>| Ubuntu

Your submission was sent successfully!<br>Close

Thank you for contacting us. A member of our team will be in touch shortly.<br>Close

You have successfully unsubscribed!<br>Close

Thank you for signing up for our newsletter!

In these regular emails you will find the latest updates about<br>Ubuntu and upcoming events where you can meet our team.Close

Your preferences have been successfully updated. Close notification

Please try again or<br>file a bug report.

Close

Blog

Previous articles walked through RDMA (Remote Direct Memory Access) as a programming model and InfiniBand as the fabric that was built around it. Both led to the same conclusion, even if it was never stated outright: moving data, not compute, becomes the bottleneck once systems scale.

So what happens when you want RDMA, but you’re already running an Ethernet network you’re not keen to replace? That’s usually where RDMA over Converged Ethernet (RoCE) enters the conversation.

At first, it sounds straightforward. Keep the RDMA semantics, keep the verbs model (the low-level RDMA API used to post send/receive and memory operations), just run it over Ethernet. In reality, it works a bit like fitting a racing engine into a standard road car. You can do it, but the rest of the system has to be able to keep up. In this article, we’ll explore what RoCE is, how it works, and when to use it.

What is RoCE?

RDMA over Converged Ethernet (RoCE) runs the RDMA programming model over standard Ethernet networks. Applications use the same verbs interface to read and write directly to remote memory, bypassing the kernel and minimizing CPU involvement. The change is in the transport: instead of a purpose-built InfiniBand fabric, RDMA operations are carried over Ethernet.

RoCE exists in two variants. RoCEv1 operates within a single Layer 2 broadcast domain. RoCEv2 encapsulates RDMA traffic in UDP/IP, which makes it routable across Layer 3 networks. In practice, deployments standardize on RoCEv2 because it fits leaf–spine designs, network segmentation, and multi-rack scale.

While the programming model does not change, the network behavior does. Ethernet does not guarantee lossless delivery by default, so RoCE relies on additional mechanisms to control congestion and avoid drops. That design choice shifts responsibility from the application to the network. Performance and predictability now depend on how the Ethernet fabric is designed, configured, and operated.

This is the key idea to keep in mind. RDMA is the model. RoCE is one way to implement it on top of Ethernet.

Where RoCE fits

RoCE is most relevant in environments where Ethernet is already the dominant networking model and introducing a separate fabric would increase operational complexity, not only because it introduces additional infrastructure, but also because it brings a different networking model, tooling, and operational expertise that teams may not already have in place. It allows RDMA to be introduced incrementally, without changing how the network is provisioned or managed at a high level.

In practice, this shows up in distributed storage systems, database clusters, and accelerator-driven workloads that are deployed on top of standard Ethernet infrastructure. In these environments, the ability to reuse the existing network is often as important as the performance characteristics themselves. In many cases, this direction reflects a practical compromise: pushing beyond application performance limits while avoiding a full redesign of the network stack from scratch.

Ethernet versus InfiniBand behavior

To understand why RoCE behaves differently in practice, it helps to compare the two worlds it bridges: InfiniBand, where RDMA was designed to run, and Ethernet, where it is being adapted.

InfiniBand enforces lossless communication as part of the fabric. Flow control and congestion management are integrated into the transport, which keeps latency stable under load.

Ethernet follows a different model, where packet loss is expected and recovery is handled by higher layers. That choice is not accidental; it comes from how Ethernet evolved. Early Ethernet was designed as a shared medium, with many hosts contending for the same wire. Simplicity and cost mattered more than strict delivery guarantees, so the network pushed complexity up the stack. If a frame collided or a buffer overflowed, it was cheaper to drop it and let higher layers retry than to coordinate every sender and receiver in real time.

That design scaled well as speeds increased and switching replaced hubs. The network stayed simple and fast, while protocols like TCP took on responsibility for reliability, ordering, and congestion control. It works well for web traffic, storage over IP, and most enterprise workloads, where a lost packet can be retransmitted without much consequence.

RDMA changes that assumption. It expects the network to behave predictably and avoid drops altogether,...