Reproducing split brain on CloudNativePG | Coroot Blog

Skip to main content<br>Customer PortalToggle menu

← All postsEngineering<br>Reproducing split brain on CloudNativePG<br>Nikolay Sivko<br>July 29, 202618 min read

We run Postgres under an operator for automatic failover. That is a promise about<br>what happens during a failure, so the only way to know you have it is to cause the<br>failure and watch.

The docs tell you what should happen. A config review tells you which knobs are<br>set. Neither tells you how long an isolated primary keeps accepting writes after<br>its replacement has been promoted, and that number decides whether a failover is<br>clean or leaves you with two versions of your data. So we keep chaos<br>testing Postgres<br>operators<br>instead of trusting the datasheet.

This time we tested CloudNativePG, on defaults, with nothing tuned. It should<br>handle this well. Since v1.27 the liveness probe of a primary also asks "can I<br>still reach the API server, and at least one replica?" A primary that cannot say<br>yes fails its own probe and gets killed, and the check is on by default. So we cut<br>the primary off from the API server, the operator and its replicas, and expected a<br>short, boring incident.

Two ways a failover can go wrong #

These two often get treated as the same thing. They are not the same failure, and<br>they do not have the same fix.

Data loss at failover is prefix truncation. The standby was at LSN Y, the primary<br>was at X, you promote, and everything between the two is gone. What survives is<br>still a valid history, just a shorter one, and how much went missing is bounded<br>by how far behind the standby was. That makes it something you can plan for: it<br>is what RPO (recovery point objective) measures, the amount of recent data you<br>have agreed you can afford to lose when something fails. Say "we may lose up to<br>five seconds of writes" and you have set an RPO.

Split brain is forked history. Two nodes both accept writes, and each one looks<br>fine on its own, but put together they could never have come from a single<br>database: the same sequence values handed out twice, the same primary keys<br>holding different rows, side effects firing on both branches. You cannot merge<br>them back. pg_rewind does not even try. It picks a winner and deletes the other<br>branch.

There is exactly one way to avoid the fork: the old primary has to stop accepting<br>writes before anyone else starts. Which is why the experiment has to record two<br>things rather than one, when each side was writing, and what each client was told<br>while it was.

The setup #

A 5-node k3s cluster (v1.34.5+k3s1) running CloudNativePG 1.30.0, PostgreSQL 18.4<br>and Chaos Mesh 2.7.2.

The database is three instances with pod anti-affinity, one per node, and<br>nothing tuned. No probes: section, so the isolation check is on. The three<br>defaults that turn out to matter:

$ kubectl get cluster postgres-splitbrain -o json | jq '.spec | {failoverDelay, smartShutdownTimeout, probes}'<br>"failoverDelay": 0,<br>"smartShutdownTimeout": 180,<br>"probes": {<br>"liveness": {<br>"isolationCheck": {"connectionTimeout": 1000, "enabled": true, "requestTimeout": 1000}

The clients are three small Go processes, written to behave like a normal<br>application. Each connects through the postgres-splitbrain-rw Service and then<br>keeps that connection:

// Established once, then held - deliberately - for the lifetime of the process.<br>conn := connect(ctx, cfg.clusterDSN, "cluster")

for {<br>if err != nil {<br>// Only a broken session gets us a new one. During a partition: as long<br>// as the old primary keeps answering, there is no error, so there is no<br>// reconnect, so we never learn that a new primary exists.<br>conn = reconnect(ctx, cfg.clusterDSN)<br>continue<br>// Postgres said this is durable. Tell the outside world.<br>ledger.Exec(ctx, `INSERT INTO acked_posts (...) VALUES (...)`, ...)

No pool recycling, no max lifetime, nothing that would re-resolve the Service.<br>We did not break it on purpose. This is how a connection pool behaves with<br>default settings, because a Service endpoint change does not tear down a TCP<br>session that is already open.

Each client inserts into a posts table with a bigserial primary key, and<br>after every commit Postgres acknowledges, records what it was told into a<br>separate single-instance Postgres called sb-truth. That second database is<br>how we measure anything at all. It is not part of the cluster under test, so<br>pg_rewind cannot reach it, and it stands in for everything an application<br>touches that a database failover cannot undo: the notification it sent, the row<br>it wrote to another service, the charge it made.

Cutting the primary off #

Picture a multi-AZ cluster, one instance per zone, and one zone loses its links<br>to the other two. Inside that zone nothing looks broken: the node is up, and the<br>application pods next to the primary can still reach it. What the primary cannot<br>reach is the API server, the operator and its two replicas.

A single node losing its uplink does the same thing, as does a control plane<br>outage...