Moving a Rust WebRTC SFU to thread-per-core: 70ms → 10ms P99.99 latency | PulseBeam Blog← Back to BlogWe recently migrated our open-source Rust WebRTC SFU server from Tokio's default multi-threaded work-stealing model to a thread-per-core (TPC) architecture, reducing P99.99 end-to-end latency from a spiky 70ms to a stable 10ms, while increasing system capacity by 25%.

P99.99 end-to-end latency before and after the thread-per-core migration.

WebRTC SFU Overview

Think of a WebRTC Selective Forwarding Unit (SFU) as a high-performance, low-latency router for real-time media. Acting like a pub/sub broker, it takes in video and data streams from a publisher and forwards them to subscribers using payload-specific transport optimizations. It doesn't do heavy server-side transcoding. It's the core technology that makes massive, Zoom-like video apps work.

An SFU forwarding streams between three participants in a room.

Early Design

PulseBeam is a standalone WebRTC SFU server rather than a library. The first version of our WebRTC SFU was built in Rust on Tokio's multi-threaded scheduler. We used the most intuitive abstraction for the problem: each participant, room, controller, and API got its own task, and these tasks talked to each other over channels, following the CSP concurrency model. That also let us avoid the traditional mutex-based locking that has burned me before.

It worked well under low load. But once we ran a benchmark simulating a fully interactive 4-person conference on 4 CPU cores, cracks showed. WebRTC stats revealed growing, unstable inbound-rtp jitter, which dragged down the congestion control bandwidth estimate and caused stutter (more on this in the Benchmark section below).

CPU usage sat at 80% with no other heavy processes running, and the flamegraph looked unremarkable. Tokio's metrics told a different story. Task steal counts were high.

Digging into Tokio scheduler internals (https://tokio.rs/blog/2019-10-scheduler) pointed to the real culprit. We had spawned too many async tasks in the hot path.

Task-per-connection pipeline. Each hop between demux, per-participant tasks, and fan-out is a separate async task connected by channels.

This hurts latency because every channel send and receive acts as an async yield point where the executor can suspend the current task to run something else. Under the work stealing scheduler in Tokio,<br>that alternative task can be a packet from a different connection that just arrived. This new packet is not more urgent, it is simply newer. Consequently, a packet already most of the way through its pipeline can be set aside for one that has not even started. Being closer to completion does not earn priority. This creates an inversion where the overall work order is determined by scheduling luck rather than how close a packet is to leaving the system.

What we generally want is:

receive a packet from a socket → process it → send it out over a socket

We want to avoid picking up new work before the current packet is fully processed, to avoid that priority inversion.

Each channel hop isn't free, either. Unlike HTTP requests, WebRTC packets are small, roughly 500 to 1,200 bytes, under typical MTU size, so per-packet processing overhead adds up fast.

Since we’re already using str0m, we can easily swap our concurrency model without rewriting the entire media stack. It’s a fantastic sans-I/O WebRTC library that doesn't force any assumptions on your threading or time management.

Removing Unnecessary Task Indirection

After removing channel hops. Ingress, demux, processing, and egress run synchronously on a single task.

To flatten those tail-latency spikes, we removed the async channels entirely. Now, when a packet arrives from the socket, the SFU processes it synchronously all the way through to the egress socket.

This does reduce concurrency, since it pins a connection's whole pipeline to a single CPU core. In exchange, it removes the unpredictable scheduling overhead of context-switching at channel boundaries, giving us a predictable execution path.

WebRTC SFU workloads are stream-based and predictable compared to typical HTTP servers, which makes them a good match for a thread-per-core architecture. Statically pre-load-balancing each connection to a specific core means we eliminate cross-core contention and get higher CPU cache hit rates.

We aren't alone in this approach. High-performance streaming systems like Redpanda and Iggy, and databases like ScyllaDB, rely on this same shard-per-core model to keep latency predictable.

Contention vs. Isolation. Image courtesy of ScyllaDB

Other SFUs make different architectural trade-offs based on their own goals:

LiveKit keeps its pipeline highly synchronous to minimize overhead. It relies on Go's work-stealing scheduler to maximize CPU utilization out of the box, but distributing work across a shared thread pool means managing shared state via locks, which can introduce scheduling jitter under load.

mediasoup...