On building scalable control planes | All Things DistributedOn building scalable control planes<br>August 04, 2026 • 3756 words
Zak van der Merwe has spent his entire career at AWS building control planes. First for EC2 and now for DSQL. On the surface, the control plane looks quite boring: it records what should exist and reconciles that with what actually does. Nobody leaves school dreaming of building one, but Zak will be the first to tell you that if you like solving hard problems in distributed systems, there are few better places to be. It’s where many of those hard problems converge, and where the decisions you make determine whether a service survives its own growth.<br>If you’ve been following Marc Brooker’s and Marc Bowes’s writing on DSQL, this is a great companion piece that pulls back the curtain and shows what it means to build a database that was designed from the start with control plane engineers in mind.<br>–W<br>On building scalable control planes<br>I’ve been working at AWS for nearly fourteen years, and for almost all of that time I’ve been building control planes. It’s not the kind of career anyone maps out for themselves. Nobody leaves university thinking “I want to spend the next decade making sure the bookkeeping layer of a cloud service stays up.” But here I am, and I think the reason I’m still here is that control planes turn out to be where many of the interesting problems live, even if it takes a while to see that clearly.<br>Before Amazon, I worked at a telecoms company in Cape Town where we had maybe ten servers, all in a room in the back of the office, and every single one had a name. You’d SSH into them, you’d share them with your colleagues, and if something went wrong you could walk over and deal with it. That was my entire mental model of what it meant to run infrastructure. Servers were things you knew individually, took care of deliberately, and could reason about as a set because there were few enough to fit in your head.<br>I mention this not because it’s an unusual background but because it was so common less than two decades ago, and I think that’s what makes it worth saying out loud. Maybe your version is a small Kubernetes cluster or a handful of RDS instances where you can visualize the whole thing, you can name the parts, and when something breaks you know which part broke. That feeling of knowing your infrastructure is comfortable, and it makes the next part of the story genuinely hard to describe, because what happened when I joined EC2 was that that feeling just evaporated.<br>Honestly, when I started, I didn’t really understand how EC2 worked. I kept trying to map it back to what I knew. If I launch an instance and the underlying server dies, what happens? Does my VM somehow get teleported onto another host? How does the cloud create this illusion that hardware failures don’t matter? I couldn’t square any of it with what I knew about running software.<br>My first job at EC2 was health-checking the fleet, pinging every server and trying to figure out if it was healthy or not, and what I found was the opposite of magic. Things were failing constantly. Hosts were going down, hardware misbehaving, disks dying. I had seen the underbelly of EC2 and it was chaotic. My mental model had gone from “servers are precious things you protect” to “everything is on fire all the time.”<br>It took a while to shake that feeling, but what I would eventually come to realize was that these failures were tiny drops in an enormous ocean of things working fine. The system was just operating at a scale where failures were a constant, a statistical certainty rather than an emergency. And the thing that made it possible to run a service at that scale without a human responding to every failure, the thing keeping everything humming, was the control plane .<br>One way or another, my years at AWS have been spent working on control planes. Every AWS service has one, and I like to think of them as our unsung heroes. The better they work, the less anyone notices them. They’re the reason you don’t have to name your servers, and the reason that when hardware fails, you as a customer never have to deal with it. I’ve gotten to build control planes for two major AWS services: EC2, and DSQL. They’re nearly a decade apart, yet the hard lessons from building one led directly to the design of the other, and that’s the story I want to tell today.<br>What is a control plane anyway?<br>At this point, I probably owe you a better explanation of what I mean by control plane and why I think they’re interesting. I’ll use EC2 as an example, because that’s where I learned most of what I know.<br>The way I think about it is that every service has a data plane and a control plane. The data plane is the set of core capabilities, the raw computing power, the hardware, the networking. The control plane is the...