The Invariant Is Hiding | Yusuf Aytas

Emily's platform team has a cost problem. They run thousands of workloads for application teams, and too much compute sits idle most of the day. Emily has a fairly obvious proposal. Let’s run the clusters at higher utilization, keep less spare capacity around, and let autoscaling add machines when demand increases. It all makes sense, she tested the change and with this optimization, the platform should become considerably cheaper.

SRE wants to review it because it touches capacity at a fundamental level. One of the larger application teams wants to understand what happens during its daily processing peak. The cloud support team wants to be in the loop because they manage the layer between Emily's platform and the cloud provider underneath it. All of a sudden, a proposal that looked like an internal platform optimization now needs four teams to be on the same page.

Why does a team that supposedly owns the platform need everyone else before it can decide how efficiently to run its own clusters? The answer is some combination of history, risk, and everyone’s comfort. In all honesty, this is not a particularly satisfying answer. Hence, the meeting becomes another candidate for simplification. Let Emily's team own the decision, automate whatever checks matter, and stop coordinating something that should sit inside the platform team’s remit.

A lot of the time, that is exactly what should happen. Companies accumulate processes after failures and forget to remove them when the original conditions change. Nevertheless, some meetings are surprisingly hard to kill. You remove the call and Emily finds the same people in Slack. You tell her team they own the capacity model and she still checks with SRE. A large application rollout appears and suddenly the cloud support team is involved again because somebody wants to know how quickly another hundred machines can actually appear.

I used to read that mostly as an ownership problem. Now I think some of these meetings are doing an interesting job that we haven’t named properly. There is an invariant condition the system has to preserve, but the knowledge required to preserve it has been split across several teams. Emily owns the platform, but she does not own everything that determines whether the platform still works when she makes it more efficient.

The meeting is where she puts those pieces back together.

From Meeting to Slack Thread

The Team Owns the Platform

Emily's team owns the clusters, scheduling, autoscaling, capacity management, monitoring, and the tooling application teams use to run their workloads. If you ask who owns the platform, there is a direct and unambiguous answer. This is usually what we mean when we say a team is autonomous.

Emily’s optimization gets support from product and management. Her team changes the capacity model. Instead of keeping a large pool of machines ready, the platform will run at higher utilization and ask the underlying cloud provider for more capacity as demand grows. While testing, they see new capacity arrives quickly enough. From the platform team’s point of view, they are shaving off the fat.

While talking to SRE, Emily realizes they don’t have a good answer for when a node group fails while the platform is already running close to capacity. SRE has a recovery model that assumes workloads can move somewhere else. If twenty machines disappear, the scheduler needs enough space to place those workloads while replacement nodes come online. At lower utilization, that space is already there. Under Emily's new model, recovery may first require the platform to acquire more infrastructure.

The cloud support team makes the problem a bit worse. The platform can ask for another twenty machines, but that does not mean AWS or GCP can always provide the exact instance type immediately. There may be capacity pressure in the availability zone, quota limits, or simply several minutes between the request and a usable node joining the cluster. Unless capacity has been reserved, the layer below Emily's platform offers a capability, not a promise that the machines will be waiting exactly when she needs them.

Emily's optimization still makes sense. Keeping thirty percent of the infrastructure idle forever is expensive. SRE is also right that some of that apparent waste was buying recovery room, and the cloud support team is right that autoscaling cannot manufacture guaranteed cloud capacity out of an API call.

Nobody necessarily made a bad decision inside the thing they own. The problem appears when those assumptions meet.

In distributed systems, an invariant is simply a condition that must remain true for consistency. Independent operations are safe when they can happen separately and still preserve that condition once their results come together. Organizations end up with the same problem because we assign ownership to components, while the guarantees we care about do not always stay inside...