The feedback loops behind Kubernetes — PlanetScaleIntroducing Database Traffic Control™: resource budgets for your Postgres query traffic.Learn more
Navigation<br>Blog|Engineering<br>Table of contents «Close »Table of contents<br>Part 1: running Postgres by handOne container, one machine<br>Pick a node, by hand<br>It needs a real disk<br>One isn't enough<br>They have to find each other<br>The watchdog script<br>Changing a parameter<br>What we actually built
Part 2: how we reinvented KubernetesThe other loops<br>The for-loop translated to Kubernetes<br>Edge-triggered notifications, level-triggered logic<br>Informers, the work queue, and a cache<br>Setpoint and measured variable: spec and status<br>Self-healing by design<br>What "observe" actually means in a real operator<br>Not every edge comes from the API server
Conclusion<br>PlanetScale Postgres is the fastest way to run Postgres in the cloud. Plans start at just $5 per month.<br>Learn more
Get the RSS feed
The feedback loops behind Kubernetes<br>Fatih Arslan [@fatih] | June 16, 2026<br>For the last decade, Kubernetes has been the backdrop to most of my work: operating clusters, helping build hosted Kubernetes, and writing Kubernetes operators. At PlanetScale, that now means running stateful systems like Postgres and MySQL in production. Kubernetes has many faces, but here I want to talk about one face only: why it is so good at running workloads at scale.<br>People ask me what an operator actually does. The canonical answer is: "it reconciles desired state." This is correct, but it also tells you almost nothing.<br>An operator is a feedback controller. It's the same closed loop that runs a thermostat or keeps your car at a fixed speed on cruise control. In our case, the thing being controlled is a database. I have been building these loops for years, and the best way I know to make them click is to ignore Kubernetes at the beginning. Kubernetes is full of control theory, even if we don't call it that in the day-to-day.<br>Before we look at a single line of Kubernetes, we're going to run a production database by hand and slowly let the feedback loop appear on its own. Then we'll map that loop to Kubernetes, with the pieces production needs: a store, watches, queues, retries, and more. At the end, we'll look at what one of these loops looks like in a real operator.<br>Note<br>A working understanding of containers and kubectl helps, but you don't need to be a Kubernetes expert. I'll use terms like idempotent, fan-in, and eventual consistency, and introduce the parts that matter as we go.<br>We're going to start slow and gradually ramp things up. Each part builds on the previous.
Part 1: running Postgres by hand<br>One container, one machine<br>Let's start from scratch. I want to run Postgres on a Linux box, and I need it inside a container. To start it, we run:<br>docker run -d --name pg \<br>-e POSTGRES_PASSWORD=secret \<br>postgres:18
That's it. Postgres is running. My app connects to it, writes some rows, and everything works fine. But then the machine goes away: the cloud provider reclaims the instance (hardware fails, or a spot instance gets taken back), or I ship a new version of my setup, which means stopping the old container and starting a fresh one in its place. Either way, the container is replaced, and my data is gone. The container storage was ephemeral, and I did not attach any persistent volume to it.<br>There is already a gap between what I want (Postgres, running, with my data) and what I have (a container whose storage disappears when the container or node goes away). The rest of this post is about that gap and the machinery we build to close it.<br>Pick a node, by hand<br>Imagine we have hundreds of nodes (servers) we can use. I already have other workloads running on them. I need to decide which one runs this database. So I ssh into the box that looks the least busy and start the container there.<br>ssh node-07 'docker run -d --name pg ... postgres:18'
I picked node-07 because it looked idle enough. I start keeping track of it, save it in some sort of config file, and push it to some repo.<br>It needs a real disk<br>Container storage is ephemeral, so I have to attach a real block device. In the cloud this is an EBS volume (e.g. on AWS); on bare metal it's a physical disk. Assuming it's a block device, this is what we usually do: provision the volume, attach it to the node, format it, mount it, and point Postgres' data directory at the mount.<br># provision + attach first with cloud CLI, then on the node:<br>mkfs.ext4 /dev/nvme1n1<br>mkdir -p /var/lib/pg-data<br>mount /dev/nvme1n1 /var/lib/pg-data<br>docker run -d --name pg \<br>-e POSTGRES_PASSWORD=secret \<br>-v /var/lib/pg-data:/var/lib/postgresql \<br>postgres:18
These are a lot of steps, and each one can fail halfway. And if the disk fills up later, Postgres stops accepting writes and we have to resize the volume by hand: first through the cloud provider, then again inside the filesystem.<br>One isn't enough<br>A single Postgres instance is a single point of failure. We want high availability: one primary and two...