Caio's Stuff - Work Stealing vs. Executor-Per-Thread: Evaluating different HTTP server workloads with Tokio, Smol and Glommio

Work Stealing vs. Executor-Per-Thread: Evaluating different HTTP server workloads with Tokio, Smol and Glommio

Image by Andreas from Pixabay

90 scenarios based on different balanced and unbalanced workloads were tested against four runtimes1 to evaluate the performance differences between Executor-Per-Thread2 and Work-Stealing architectures.

All tests are based on Rust-written software and Rust-based technologies, although some information can also be applied to other programming languages or similar environments.

All files are available at data.tar.xz. Feel free to reach out if you spot any misunderstandings or misconfigurations.

Motivation

The motivation to test Work-Stealing against Executor-Per-Thread originated from my early attempts to submit the WTX3 project to HttpArena using the default #[tokio::main(flavor = "multi_thread")] macro.

To my surprise, the initial PRs showed a score ~8x lower than the best position in the WebSocket benchmark, which was highly unusual. I was left wondering: Is there a bug in the library or did I do something wrong?

After manually scrolling through other projects in HttpArena as well as in the now-abandoned TechEmpower, it became clear that all Tokio-based libraries were spawning threads where each one had its own WebSocket server instance with a generous backlog, sharing the same address through SO_REUSEADDR.

Another surprise occurred when the same trick was applied to WTX: It became number one in the echo test case. As you might have guessed, the culprit is balanced workloads.

Executor-Per-Thread vs. Work-Stealing

In the Executor-Per-Thread model, the system associates a runtime's executor with a specific thread. All asynchronous work must be contained within that thread, adopting a share-nothing approach. It is often labeled as "Thread-Per-Core".

This approach improves tail latencies by maximizing local CPU cache hits. For Rust specifically it is a big deal because of creeping Send4 bounds that often yield confusing, incomplete and untraceable compile errors.

In Work-Stealing , there is only a single runtime in the whole program that usually orchestrates all the available threads in the system, allowing idle workers to "steal" tasks from the queues of busy workers.

It acts, roughly speaking, as an internal load balancer. Idle workers can pull tasks from the queues of busy workers, though this incurs overhead due to thread synchronization.

The choice between the two usually comes down to the average workload of your specific business or project.

HTTP server

The HTTP servers expose three endpoints based on real-world scenarios:

create_order: Validates user-provided data, creates an order in the database and then signals a gateway payment. Has two awaiting points.

index_page: Represents a chunk of sequential code without external IO calls where the thread is occupied in its entirety. Another analogy could be the processing of in-memory JSON content.

read_order: Fetches an order from the database and then processes the returned data into an user-friendly format. Has a single awaiting point5.

To isolate runtime performance, all IO calls are simulated using asynchronous sleeps (yielding timers) in microseconds and sequential processing is simulated using busy-wait loops.

CREATE_ORDER_DATABASE_LATENCY = 2000us<br>CREATE_ORDER_GATEWAY_LATENCY = 10000us<br>CREATE_ORDER_PROCESSING = 2us<br>INDEX_PAGE_PROCESSING = 2us<br>READ_ORDER_DATABASE_LATENCY = 1000us<br>READ_ORDER_PROCESSING = 2us<br>Unlike HTTP/1.1, where a single connection typically involves several network trips, the web servers in these tests are based on HTTP/2 that can handle long-lived, bi-directional streams, avoiding the need for frequent handshakes and streams also help exercise the system's ability to handle concurrent workloads.

HTTP/2 streams

Workloads

The tests utilized three thread counts, two distribution ratios, three connection scales and five environment types, totaling 90 configurations. Each set was applied to a runtime, resulting in a grand total of 360 performed evaluations.

tokio-ws is the only runtime configured with Work-Stealing (ws suffix), while the remaining runtimes spawn multiple threads in a share-nothing Executor-Per-Thread approach (ept suffix).

tokio-ept: Based on epoll via mio; a LocalRuntime is created on each thread.

tokio-ws: Based on epoll via mio; a single multi-thread Runtime is created in the main thread.

glommio-ept: Based on io_uring via built-in syscalls; a LocalExecutor is created on each thread.

smol-ept: Based on epoll via polling; a LocalExecutor is created on each thread.

All runtime instances have an associated HTTP/2 server that shares the same port through SO_REUSEPORT.

ThreadsBalanceScaleWorkload<br>4, 8, 12even, unevenlow, medium, highcpu_heavy, io_heavy, mixed, pure_io, pure_cpu

Threads

The system has 24 threads with...