Notes on Network Signal Congestion Control – Dip Singh – Network Engineer
Notes on Network Signal Congestion Control
“The test of first-rate intelligence is the ability to hold two opposing ideas in mind at the same time and still retain the ability to<br>function.” - F. Scott Fitzgerald
All congestion control algorithms try to answer the same question: how fast should data be sent? Traditional protocols like TCP Cubic use just one signal: packet loss to decide.<br>NSCC (Network Signal-based Congestion Control), part of the Ultra Ethernet Transport protocol, uses two signals that work on different timescales. In this post, I’ll try to explain<br>how NSCC works from the first principles, and how it stacks up against familiar protocols.
Before diving in, let’s focus on one key number that shapes everything else. For any window-based congestion control, the main physical factor is the Bandwidth-Delay Product (BDP):
\[\hspace{5cm} \text{BDP} = B \times R_0\]
Here, $B$ stands for the bottleneck bandwidth, and $R_0$ is the base round-trip time, which is the delay without any queuing. The BDP shows how many bytes can be sent<br>before the first acknowledgment comes back. For example, at 100 Gbps with a 12μs round-trip time, $\text{BDP}$ = 12.5 GB/s x 12 μs = 150 KB. Assuming 4KB size per packet,<br>that is about 37 MTU-sized packets in flight to keep the connection fully used.
Most of what I know comes from three main sources: the UE design paper, the FASTFLOW research paper (which was previously called SMaRTT-REPS),<br>which describes a similar two-signal design and covers details such as control-law formulas, the WTD mechanism, and parameter scaling. I also use the htsim simulator implementation, specifically uec.cpp in the<br>htsim codebase. I am sure I still have a few things wrong, so if you notice any mistakes, please let me know.
1. The Problem: Why Two Signals?
Traditional congestion control assumes a single path between sender and receiver. Every packet sees the same queue, the same delay, the same congestion. If the delay goes up, that is the network state.
When we begin using packet spraying (for reasons outside the current discussion), this single-path assumption no longer holds.
So if Path B is congested but A and C are not, should you lower your sending rate? If you do, you penalize the flow for congestion<br>on just one of three paths and waste most of your available capacity. If you don’t, you ignore real congestion. There’s no clear right answer.
In terms of BDP, at 100 Gbps and 12us RTT, you need 150 KB in flight to fill the pipe. With three paths, each path carries about 50 KB on average. If you reduce the total window because one path has a full queue, you remove bytes from all three paths, even the two that are working well. This causes the flow to fall below its BDP and wastes capacity.
One solution here is to use two signals instead of one. Queueing delay shows the average congestion across all paths. ECN (Explicit Congestion Notification) tells you if this packet’s path had congestion at that moment. Since the sender tracks which entropy value (EV) was used for each packet, it can link the ECN mark to a specific path. Using both signals, you can tell if only one path is congested or if the whole network is overloaded.
A helpful way to understand NSCC is to compare it to a smoke detector and a thermometer:
ECN works like a smoke detector. It is fast, gives a yes-or-no answer, and can be specific to a path. It tells you that something is wrong, but not how serious it is or exactly where the problem is.
Queueing delay based on RTT is like a thermometer. It is slower, gives a range of values, and shows how severe the problem is. It tells you how hot the whole building is, but not where the heat is coming from, and it reacts with some delay.
There’s also a crucial difference in when each signal arrives. ECN is a leading indicator because a switch marks a packet as soon as congestion happens. ECN gives a warning right away, before queues get big enough to increase RTT past the target. Delay is a trailing indicator. You only see the delay after the packet has gone through the whole path, been processed, and the ACK comes back. Delay can stay high even after queues start to clear and ECN marks stop.
Neither signal gives the full picture on its own. When we use both together, we get four different responses, which we will look next.
Three Loops, Four Clocks
These three loops work across four nested timescales, with each clock running at a different speed from fastest to slowest. To make this clearer, let’s look at each one using the reference setup (100 Gbps, 4 KB MTU, 12 µs base RTT).
Clock 1: Per-ACK. This is the system’s heartbeat, where every ACK triggers a decision. The time between heartbeats depends on serialization delay, which is<br>how long it takes to send one ACK’s worth of data. For example, assuming 4KB size packets on a 100Gbps link will take about 0.33 µs. If the NIC combines...