The reconnect storm that never touched the broker — Side X Labs

← All writingThere's a particular kind of 2 a.m. that only happens in fleet IoT. A slice of your devices — a few thousand of them — drop off the network at once. Bad uplink, a segment of the field going dark; the reason doesn't matter. Minutes later they all come back. And here's the part that surprised me the first time it happened: the managed broker didn't blink. No connection ceiling hit, no broker to melt, no page from the front door. By every dashboard watching the ingress layer, we were fine.

We were not fine. One number was climbing through the roof: iterator age . The storm never touched the broker. It landed one hop downstream, in the stream consumer — and my first instinct for fixing it dug the hole deeper.

The pipeline, and why steady state lies to you

The shape is textbook, and I'll keep it generic on purpose: a managed MQTT broker → a Kinesis data stream → a Lambda consumer, parallelized maybe ten ways with a batch size around a thousand → writing to MySQL. Call it 10,000 devices in the field. Nothing exotic.

In steady state this pipeline is boring, and boring is the goal. Telemetry trickles in, the consumer keeps up without effort, iterator age sits near zero. You forget it exists.

But "boring" is a property of the arrival rate, not of the architecture. Everything about this pipeline's calm — the near-zero iterator age, the comfortable concurrency, the MySQL connection count you never think about — is quietly assuming devices arrive at the rate they've always arrived. The fleet is about to violate that assumption all at once.

The incident: store-and-forward is a loaded gun

Field devices buffer. When the uplink is down, a well-behaved device doesn't drop its readings — it stores them and forwards them on reconnect. Store-and-forward is exactly what you want from an edge device. It's also a loaded gun pointed at your ingest pipeline.

When a few thousand devices reconnect inside the same minute, they don't resume the gentle trickle. They flush — every buffered reading, all at once, a burst many multiples of steady state. The broker passes it straight through; absorbing connection churn is its entire job. Now that spike hits Kinesis, and Kinesis hands it to a consumer whose drain rate is fixed: shard count × concurrency × per-batch throughput, all bounded by how long each invocation is allowed to run.

Fixed drain, meet variable spike. The backlog grows, and iterator age — the age of the oldest record you haven't processed yet, which is to say exactly how far behind real time you are — starts walking up. Thirty, forty-five minutes behind. Two things break at that lag. Freshness goes first: alerts and downstream actions fire on stale data, or fail to fire when they should. And if the age ever creeps toward the stream's retention window, "late" quietly becomes "lost."

Then it got worse, and this is the part the tutorials skip. Kinesis is ordered per shard — processing is strictly in-order, which means one batch that won't complete blocks every record behind it on that shard. During the flush we hit a record the consumer choked on, and iterator age stopped climbing linearly and went vertical. One poison record, at the worst possible moment, and a whole shard's slice of the fleet was frozen behind it.

The trap: "just scale it out"

Every instinct I had was wrong, and they were all the same instinct: scale out. Add shards. Crank the parallelization factor. Raise concurrency. Throw drain capacity at what looked like a drain problem.

Here's why it backfired. The bottleneck was never Kinesis throughput, and it was never Lambda concurrency. It was the cost of a single record — specifically, that every invocation was paying for a fresh MySQL connection handshake before it did any real work. Thirty, fifty milliseconds of handshake per invocation, invisible at a trickle.

So what happens when you "scale out"? More concurrent Lambdas means more simultaneous cold connections slamming into MySQL. I wasn't adding drain capacity — I was building a denial-of-service attack against my own database, marching straight at its connection ceiling. The faster I tried to drain, the closer I got to knocking over the one component downstream that had stayed perfectly healthy. It was capacity I couldn't actually use.

The real fix: make the record cheap, then make failure cheap

The fix came in three moves, and not one of them was "bigger."

First, make the record cheap. Hoist the MySQL connection out of the handler so warm execution environments reuse it instead of reconnecting every invocation. It's a one-line decision about where a variable lives, and the per-record handshake tax disappears with it — along with the self-inflicted connection storm. Then right-size the batch: large enough to amortize the fixed overhead of an invocation, small enough that each one finishes well inside its duration limit and doesn't itself become a source of...