The .join() That Should Be a Bug | Kronotop Skip to content

The .join() That Should Be a Bug

Jun 29, 2026<br>Burak Sezer

Serving thousands of connections on a backend where every call blocks<br>Section titled “Serving thousands of connections on a backend where every call blocks”

There are two well-known ways to implement connection management in database systems. Each way has a cost. Kronotop takes a third approach. Let’s start with the problem we have.

Kronotop stores its metadata in FoundationDB. It also stores document bodies on the local file system. So almost every operation a client asks for turns into a network call to FDB and a<br>read from the disk. A network call takes milliseconds, not nanoseconds. Reading a document, committing a transaction, looking up an index: all of it waits on I/O. Our command path is<br>not in-memory work. It is dominated by waiting.

Two classic models<br>Section titled “Two classic models”

Redisone thread1 thread✓holds thousands cheaply✕nothing may blockvsPostgresprocess per connectionprocessprocessprocess✓any command may block✕one process per connection, heavy<br>Redis(before 6.0) runs a single thread. That thread listens on every connection and processes commands one at a time. This scales connections well, but it forbids blocking.<br>Nothing in the command path is allowed to wait. If one command waited, every other client would wait with it. That is fine when all your data is in memory. It does not work when<br>a command has to do more. Our commands call a transactional store over the network, and they append data to a file on disk and call fsync. Every one of those steps blocks.

Postgres goes the other way. It gives each connection its own process. Now blocking is not a problem. Each connection runs on its own, so you can write plain, sequential code.<br>But there is a price! A connection is now an operating system process, and that is heavy. A few thousand connections become too much. This is why Postgres deployments almost<br>always sit behind a separate connection pooler.

One model scales connections but cannot wait. The other can wait but does not scale large numbers of idle/open connections cheaply. Kronotop needs both.

Splitting the connection from the work<br>Section titled “Splitting the connection from the work”

The trick is to stop treating “the connection” and “the work” as the same thing.

The connection side works the way Redis does. We build it on Netty. A small number of event loop threads listen on many sockets. They react to whatever socket is ready.<br>They parse the incoming command and write the reply back. This part never blocks and never waits. So a handful of threads can keep thousands of connections alive.

The work side is the part that does disk and network I/O. It runs somewhere else, on a virtual thread. A virtual thread lets us write the I/O call as plain, blocking, top-to-bottom code.<br>It is the Postgres style, but without the Postgres cost. The virtual thread blocks while it waits on FoundationDB or the disk. When it blocks, the Java runtime unmounts it and frees the carrier<br>thread underneath for other work. Thousands of virtual threads can wait on I/O at the same time, while only a few real threads do anything. When the I/O completes, the virtual thread<br>picks up where it left off.

So the network threads stay free. They can serve every other connection. The slow part happens off to the side, where waiting is cheap. The result is then handed back to the connection<br>thread, which writes the reply. We keep the write there by design. That hand-off is the one rule we keep strict.

CONNECTION SIDEWORK SIDEclients…Netty event loopA few threads serve thousands of sockets. Parse and write.Never blocks, never waits.the reply is always written hereVirtual threadtr.get(…).join() blocks on FDB or diskThe runtime parks it. The carrier thread is freed for other work.1offloadsupplyAsync(…, vtExecutor)2resultthenAcceptAsync(…, nettyExecutor)<br>In code, the whole offload comes down to two moving parts:

CompletableFuture

.supplyAsync(supplier, context.getVirtualThreadPerTaskExecutor()) // run on a virtual thread

.thenAcceptAsync(action, response.getCtx().executor()); // resume on the Netty thread

Two executors, two phases. The supplier is the slow part. It runs on the virtual thread executor, where blocking on FoundationDB is allowed. The action is the reply. It runs on response.getCtx().executor(), the Netty event loop that owns this connection. So the work that waits and the write that must not move stay on separate threads. The second only starts once the first is done.

A read command is written against exactly that shape. The first block fetches the value, the second sends it:

public void execute(Request request, Response response) {

AsyncCommandExecutor.supplyAsync(context, response,

() -> {

// virtual thread: open the transaction and wait on FDB

Session session = request.getSession();

Transaction tr = TransactionUtil.getOrCreateTransaction(context,...