Explaining FoundationDB's Architecture Through Compartmentalization | Pierre Zemb's Blog

skip to content

Pierre Zemb

⊕ dark

root@pierrezemb.fr:~/blog$ ./read --quorum posts/foundationdb-architecture-through-compartmentalization

✓ quorum 2/3 · served by node-0

read: quorum

~ / blog / foundationdb-architecture-through-compartmentalization

Explaining FoundationDB's Architecture Through Compartmentalization

2026-08-13<br>read: ~6 min<br>markdown<br>tags: #distributed-systems #foundationdb #consensus #algorithms

The architecture I couldn't explain🔗

One thing always puzzled me about FoundationDB: compared to many distributed databases, its architecture looks almost excessive: GRV proxies, commit proxies, resolvers, log servers, storage servers, and that's only the data plane. When I say that I operate FDB, people often ask, "Isn't that complicated? There are so many processes and roles." After years of operating FDB for Materia at Clever Cloud, I understood what every component did, but I couldn't explain why the system had been split that way. Michael Whittaker's Scaling Replicated State Machines with Compartmentalization (VLDB 2021) finally gave me the vocabulary I was missing.

Start with his talk, it explains the paper better than I could:

A different way to look at distributed systems🔗

When we learn distributed systems, we usually learn to partition data and replicate it. The paper asks a different question: which responsibilities are accidentally coupled inside the bottleneck?

It calls the answer compartmentalization : separate those responsibilities, then scale each one independently.

The MultiPaxos leader is its canonical example, since it sequences commands into the log and handles communication for the whole protocol. For f = 1, each command gives the leader one client message, four messages exchanged with a quorum of two acceptors, and two messages to replicas, seven messages in total, so adding acceptors or replicas only gives it more nodes to talk to.

sequenceDiagram<br>participant C as Client<br>participant L as Leader<br>participant A as Acceptors<br>participant R as Replicas<br>C->>L: x<br>L->>A: replicate x to quorum of 2 acceptors<br>A->>L: acknowledgements from 2 acceptors<br>L->>R: x is chosen, sent to 2 replicas<br>R->>C: result of x<br>Note over L: 7 messages touch leader

There is no fundamental reason those two jobs have to live together. Sequencing is inherently serialized, communication is embarrassingly parallel, so the paper introduces proxy leaders : the leader keeps sequencing and hands each command to a proxy leader that runs the rest of the protocol, dropping the leader to two messages per command. I will not paraphrase the whole construction, the talk does it better, so here is just the paper's result: applied across the protocol, compartmentalization raises MultiPaxos throughput by 6x on a write-only workload and 16x on a workload with 90% reads, without adopting a new protocol.

sequenceDiagram<br>participant C as Client<br>participant L as Leader<br>participant P as Proxy leader<br>participant A as Acceptors<br>participant R as Replicas<br>C->>L: x<br>L->>P: x at position 0<br>P->>A: replicate x to quorum of 2 acceptors<br>A->>P: acknowledgements from 2 acceptors<br>P->>R: x is chosen, sent to 2 replicas<br>R->>C: result of x<br>Note over L: 2 messages touch leader<br>Note over P: 7 messages touch proxy

How I read systems now🔗

Since reading the paper, I read distributed systems through the same short list of questions.

How many RPCs does each component touch per request?

What is this component actually responsible for?

Which steps are inherently serialized, and which are embarrassingly parallel?

When I add instances, does the work per node go down, or does fan-out go up?

Can this work be partitioned across independent groups?

Looking back at FoundationDB, I stopped seeing dozens of processes and started seeing answers to those questions. The RPC flow makes those splits visible:

sequenceDiagram<br>participant C as Client<br>participant G as GRV proxy<br>participant M as Master (singleton sequencer)<br>participant S as Storage servers<br>participant CP as Commit proxy<br>participant R as Resolvers<br>participant T as TLogs<br>C->>G: get read version<br>G->>M: batched request<br>M-->>G: read version<br>G-->>C: read version<br>C->>S: read keys at version<br>S-->>C: values<br>C->>CP: commit transaction<br>CP->>M: batched commit-version request<br>M-->>CP: commit version<br>CP->>R: conflict ranges<br>R-->>CP: conflict decisions<br>CP->>T: mutations and commit version<br>T-->>CP: durable<br>CP->>M: live committed version<br>M-->>CP: acknowledged<br>CP-->>C: commit result

FoundationDB does not minimize the total number of round trips. It keeps the RPC count low at the serialized role. The Master sees batched version requests and live committed version reports, while GRV and commit proxies carry the client traffic. Commit proxies coordinate the fan-out to resolvers and TLogs.

The remaining work is divided along its own boundaries. Resolvers partition conflict checking by key range, TLogs durably...