Offloading I/O to Dedicated Cores: An Asymmetric io_uring Backend for Seastar and ScyllaDB | ScyllaDB

See all blog posts

Offloading I/O to Dedicated Cores: An Asymmetric io_uring Backend for Seastar and ScyllaDB

By Mikołaj Bilski, Jakub Czyszczoń, Witold Formański, Marcin Szopa<br>July 22, 2026

BlueskyRedditXLinkedIn

RSS

Subscribe to Receive Blog Updates

Subscription Categories

All<br>Posts

Events

Integrations

News

Performance

Releases

ScyllaDB

Seastar

Tutorials

User<br>Stories

Developers<br>Blog

BlueskyRedditXLinkedIn

RSS

We moved I/O off application cores to dedicated cores to give compute-heavy workloads more breathing room

Modern high-throughput systems often run on machines with many cores. At that scale, one practical problem becomes increasingly visible: CPU time spent on low-level I/O handling competes directly with CPU time needed for application-level work.

In this post, we explore how shifting more I/O related work away from cores focused on computations onto a dedicated subset of cores can improve the overall balance of the system. Specifically, we share how we approached Seastar‘s new asymmetric_io_uring backend: the design decisions behind it, what it aims to free up on application cores, and how it compares to linux-aio (Seastar’s existing backend).

Seastar’s Shared-Nothing Architecture

Seastar – the open source C++ framework that underpins high-performance distributed systems like ScyllaDB, Redpanda, and Ceph Crimson – is built from the ground up around a shared-nothing architecture .

To maximize multi-core performance, it usually spawns a single thread (referred to as a shard ) per application core. Each shard runs its own cooperative scheduler (the reactor) managing lightweight tasks and fibers, and owns its memory and data structures. This eliminates the need for expensive cross-shard locking and atomic operations. Explicit message passing relies on inter-shard communication instead.

How the Symmetric reactor_backend_uring Works

In the existing symmetric io_uring backend, every shard initializes its own private io_uring instance. The reactor loop is structured to perform computations first and then invoke pollers to check for I/O events.

To optimize performance, this symmetric model implements a heuristic-based fast track for networking I/O. When a shard initiates a read or write on a network socket, the backend checks speculation flags. If the socket is speculated to be ready (i.e., we expect the operation to be non-blocking), the shard executes a synchronous system call directly on its own core (bypassing the io_uring submission path). If the speculation fails or the socket blocks, it falls back to the slow track (submitting the operation to io_uring).

This design forces application cores to divide their time between running high-level application logic and executing low-level synchronous I/O system calls, resulting in:

Context switching overhead between user space and kernel space.

Cache pollution on application cores due to kernel-level I/O processing.

Higher latency variance and potential scheduler preemption.

What Are Networking Cores?

On systems optimized with Seastar’s tuning script (perftune.py), CPUs can be partitioned into application cores (where Seastar shards run) and networking cores (also known as IRQ cores). This is because modern servers often have more CPUs than hardware queues on network interface cards. The goal of this partitioning is to only allow a subset of cores to handle these queues – to eliminate a situation when multiple cores compete for a single queue.

Using the script, network interface card (NIC) hardware interrupts (IRQs) are routed exclusively to the networking cores. This ensures that application cores are not disturbed by high-priority hardware interrupts and SoftIRQ processing of incoming packets. This however causes the networking stack to run on multiple cores: the application cores (processing syscalls and outgoing packets) and the networking cores (processing interrupts and incoming packets). No application shards are spawned on these networking cores.

The Challenge: Offloading More I/O to Networking Cores

Since the system already contains dedicated networking cores that handle "some part of I/O" (NIC IRQs) and do not run any user-level application shards, we wondered: What if we give them more of the I/O workload?

By utilizing io_uring features like Submission Queue (SQ) polling and kernel worker thread affinity, we can offload the entire asynchronous I/O execution pipeline to these underutilized networking cores. The compute-bound application shards would simply place requests on their submission queues without making synchronous system calls, while kernel workers and polling threads pinned to the networking cores process the requests. This asymmetry promises to offload application cores, improving computation throughput and cache locality.

Additionally, we explore the usage of other features in order to improve...