State-Oriented Consistency: Why We Stopped Looking for One Right Answer — Keel

Every distributed system has different kinds of state

The mistake we made, early on, was asking the wrong question.

We kept asking: "Which consistency model should the cluster use?"

The useful question turned out to be: "Which consistency guarantee does this specific piece of state actually need?"

Those sound similar. They aren't. The first one assumes a single answer exists for the whole system. The second one assumes it doesn't — and it's the second one that this whole piece converges on, in the form of one table that, by the end, does most of the explaining for us.

We discovered the difference the hard way while building a clustered message broker. The lesson underneath has very little to do with MQTT. Any distributed system holding more than one kind of state runs into this question eventually. Get it right, and the rest of the design gets simpler almost automatically.

The incident

Two pods, five minutes apart, killed by the kernel's OOM handler. Memory limit: 512Mi. Nothing exotic — a normal container limit for a normal stateless service.

The first hypothesis was the obvious one: load-balancer imbalance, probably compounding into a reconnect cascade. Plausible. Wrong.

Active connections, three pods, same window:

Pod A: 8 ██<br>Pod B: 174 ████████████████<br>Pod C: 1,039 ████████████████████████████████████████

Working-set memory, same three pods, same window:

Pod A: ~270MB ████████████████████████████<br>Pod B: ~310MB ████████████████████████████████<br>Pod C: ~360MB █████████████████████████████████████

That's not the shape imbalance produces. A pod serving 130x more connections than another should not have nearly identical memory. Either the metric was lying, or the mental model was.

The investigation

The metric wasn't lying.

Reading the actual code path responsible for session state turned up the real cause: on every pod boot, a persistence hook loaded every client's stored session state — the entire fleet's, not just the fraction that would ever reconnect to this specific pod. One unfiltered read, called once at startup, and every row it returned became a live in-memory object.

Why would anyone write it that way? In a stateless cluster behind a non-sticky load balancer, there's genuinely no way to know in advance which clients will reconnect to this pod. So the simplest correct-seeming implementation was: load everything, everywhere, and let whatever connects find its state waiting. Not a typo-bug — a design that quietly assumed a node should be ready to serve any client that might show up. A small, local instance of exactly the wrong question from the opening: it optimized for "the cluster can serve anyone," not "this specific piece of state has this specific requirement."

The chain rarely announces itself as an architecture problem:

Wrong abstraction<br>Wrong guarantee<br>Wrong architecture<br>Wrong scaling

Wrong abstraction: treating "who serves this client" as if every node needed the answer, not just one. Wrong guarantee: replicating everywhere, "just in case." Wrong architecture: no boundary on what a node is responsible for. Wrong scaling: cost grows with fleet size, not with actual responsibility.

An OOM is just where the chain happened to become visible. It could just as easily have surfaced as a slow memory leak, a scaling limit nobody could explain, or a rolling update that mysteriously got riskier as the fleet grew.

The question this surfaces

Stated plainly, it stops looking like a memory bug and starts looking like a modeling error:

We had assumed this state needed to be replicated everywhere any client might land. It didn't need that at all — it needed exactly one owner, decided by a rule any node could compute on its own.

That's the question that outlives this one incident: not "how do we fit this in less memory," but "what does this state actually require, and have we been giving it more than that?"

Naming the mistake, before fixing it

The default we'd fallen into deserves a name too, even though nobody chose it on purpose. We started referring to this design habit as Uniform Consistency : apply one consistency strategy to the whole system, and let every piece of state inherit it, regardless of what that specific state actually needs.

To be clear, we don't mean a specific algorithm — Uniform Consistency isn't a technique you'd find in a paper. We mean the habit: reaching for whatever consistency model the system already trusts, everywhere, by default, without asking each piece of state whether it actually needed that much.

Wrong assumption Reality<br>──────────────── ───────<br>Every piece of Each piece of state<br>distributed state has its own semantics —<br>needs the same its own answer to<br>guarantee. ───────► "what happens if two<br>nodes briefly disagree?"

Nobody designs a system by deciding "we will apply Uniform Consistency." It's what happens by default, one small decision at a time, when the question...