This Blog Post Runs a TCP Stack — Apoxy Blog<br>All systems operationalApoxy:// Field notes/Engineering, Protocols<br>EngineeringProtocolsJul 17, 202612 min read<br>This Blog Post Runs a TCP Stack<br>An interactive, in-browser TCP congestion control simulation using WebAssembly.
Dmitry Ilyevsky<br>Co-founder & CTO
Our co-founder Matt's home internet provider, who shall remain nameless, is not the best. It tends<br>to lose packets and introduce jitter which occasionally causes frustrating moments where you have to<br>talk to Matt's still image on Zoom. We also run some of our probes from Matt's private lab in his<br>house and so the ISP shenanigans also cause spurious alerts when, in reality, nothing is wrong.
See, the probes are running using our tunnel<br>infrastructure via apoxy tunnel and we designed this to securely tunnel entire IP packets using<br>CONNECT-IP over QUIC, which is commonly referred to as<br>MASQUE. This is different from most other tunnel<br>platforms that implement custom framing and shuttle only higher-level protocol payload. On the agent<br>side of the tunnel (where apoxy tunnel runs), we inject tunneled IP packets into user's network<br>using TAP device, if available, or more frequently, using<br>netstack - a fully userspace TCP/IP stack<br>written in Go for gVisor's networking needs.
Netstack contains an entire TCP/IP stack implementation which includes TCP Congestion<br>Control machinery. It has long been my suspicion that default TCP congestion control algorithms<br>in Linux like CUBIC or Reno are not very good on even mildly lossy networks. I heard from a friend<br>about Google's BBRv3 algorithm, which is supposed to be better at this, so I decided to try it out<br>in netstack.
Below is the explanation of how BBRv3 works and why it is indeed better than CUBIC or Reno (at<br>least on lossy networks). The graph sims are actual network simulations run in netstack within<br>your browser using WebAssembly.
Bottleneck Bandwidth and Round-trip propagation time
First, there is a realization that from the perspective of TCP, any arbitrary complex path behaves<br>as a single link with just two constraints - RTProp (round-trip propagation delay) and BtlBw<br>(bottleneck bandwidth).
FIG. 1 - BOTTLENECK<br>NAIVECUBICBBRV3
loss 0 % jitter 1 ms<br>bottleneck 100 Mbps · base rtt 40 ms · buffer 350 pkt · 30 s run readyRESET
buffer limit · 350 pktSENDERbottleneck · 100 Mbps · tail-drop queueRECEIVERacksROUND-TRIP TIMEbase rtt 40 msDELIVERY RATElink 100 Mbps<br>PAUSEt = 00.00 s0.40×FWD<br>Netstack simulated traffic, aggregated so the particles remain readable. Their timing comes from forward and reverse link activity. CUBIC's growing window appears in the in-flight bar and queue stack, ACKs follow the complete return path, and queue-overflow ✕ marks leave the queue while random-loss marks leave the downstream wire. In the RTT strip the solid trace is the latest raw RTT sample and the dotted one is the sender's own smoothed srtt estimate, both against the dashed base-rtt rule. In the delivery strip the solid trace is goodput (bytes reaching the receiver's app), the dotted one is offered load entering the bottleneck - the gap between them is congestion - and the dashed rule is the configured link rate. Playback cruises at 0.4× and slows to 0.15× around losses and phase changes.
In FIG 1, RTProp is the round-trip time for a ball to traverse the entire path and back without<br>getting queued anywhere along the way. BtlBw is the bottleneck bandwidth - maximum number of<br>balls per second that can move through the bottleneck - a point along the path with a minimum data<br>delivery rate. These two values can be combined to calculate the Bandwidth-Delay Product or<br>BDP = BtlBw x RTProp - which characterizes the maximum amount of data that can be in-flight on<br>this link.
There's another way to think about this - imagine you're driving a delivery truck from Spectrumville<br>to Apoxygard. First, there's a wide highway but then you encounter one or more narrow exits to<br>traverse to another highway. In this metaphor you're also in a rush so you're moving at a constant<br>speed of roughly 2/3rds of the speed of light (fiber optic light propagation speed). BtlBw is the<br>bottleneck bandwidth - maximum number of trucks per second that can move through the narrowest bridge<br>along your path. And the RTProp is the time it takes you to drive the entire distance (and back<br>since one way speed of light is unmeasurable) without<br>any delays along the way.
BBR vs Loss-based congestion control
Now that we have some intuition for the problem space, let's compare BBRv3 to loss-based congestion<br>control algorithms like CUBIC or Reno. In traditional loss-based algorithms, congestion is detected<br>by packet loss, and the sender adjusts its rate by decreasing cwnd (congestion window size) - the<br>number of trucks that can be en route without delivery acknowledgment. This made sense 30 years ago<br>when buffers were limited and it was pretty easy to max out the network.
Today, with much larger buffers and faster networks,...