Anatomy of a One-Minute Outage: An HLS Streaming Post-Mortem
Skip to content
Anatomy of a One-Minute Outage
2026-08-01<br>· Ajitabh
"Some people are saying the audio is broken and the video looks degraded. But it works absolutely fine everywhere I’ve checked."
If you’ve ever run a live service — streaming or otherwise — you know this exact message. It arrives during your most important hour. You check. It works. You check from your phone. It works. And yet the complaints keep coming.
In my over 30 years of operational support, this is the most dangerous kind of problem because the instinct is to dismiss it. I have seen it numerous times that the term "It works for me" is the graveyard of unresolved incidents. As one of the YouTubers I follow says, the truth lies somewhere in between. Both parties are right: the stream was working, and it was broken — for different people, at the same time, for a reason that only made sense once I looked at the whole system.
I run the streaming for our live Satsang (a community worship gathering focused on Arti and prayers) transmissions. This is broadcast to thousands of listeners twice a day. The audience is unusually sensitive to interruptions; a thirty-second silence during a transmission is not a minor bug; it’s a broken experience for people in a moment of devotion.
The architecture is common and unremarkable: an encoder at the venue sends the feed to an origin server, which transcodes it into multiple qualities, and a set of distribution servers deliver small chunks of audio and video — called segments — to every listener’s phone.
The shape of the system: one feed in at the venue, transcoded into several qualities at the origin, then delivered as small segments by a set of distribution servers to every listener’s phone.
This blog post is the story of this simple word, "segments".
Reading the graphs like a detective
When the complaints came in, I pulled up three dashboards: the load balancer, the distribution servers, and the transcoding origin. For the non-technical reader, think of it as checking three witnesses to the same event.
The load balancer said: traffic is climbing steadily, capacity is fine, but a small percentage of requests are failing — and that percentage is growing with the audience. Nothing was overloaded. That ruled out the easy answer ("too many viewers/listeners").
The distribution servers said something far more interesting. At exactly 14:46, all five of them — simultaneously — logged a sharp spike of "file not found" errors. Not for the audio chunks. For the playlists : the small index files that tell every player which chunks to fetch next. Then the errors vanished. The whole event lasted about one minute.
Five independent servers erroring at the same instant means the problem wasn’t on any of them. They were all faithfully reporting the same upstream truth: for one minute, the playlists disappeared at the source.
Reading witnesses. Five independent servers erroring at the same instant don’t share a fault — they’re reporting a single upstream truth. The arrow points back to the origin.
I confirmed it: the venue’s internet connection had blipped. The encoder disconnected, reconnected, and streaming resumed. Sixty seconds, start to finish. So here is the puzzle that makes this worth writing down: why did a one-minute network blip cause a much longer complaint of "broken audio" from some listeners, while everyone else — including me — saw nothing wrong?
The optimization that turned against me
Months earlier, I had made a genuinely good change. Streaming works by having every player download a new chunk every few seconds. Each chunk, once created, never changes — so I told every player and every cache in between: this file is immutable; cache it for a year, never ask again. Request volume dropped, the experience got smoother, everything stabilized. And I was happy and proud.
That optimization had one hidden assumption, and the one-minute blip violated it. When the transcoder restarted after the reconnect, it began numbering its output chunks from zero again: index_0.ts, index_1.ts, index_2.ts — the same filenames it had used hours earlier , now containing completely different audio. But listeners’ phones, and caches along the way, still held the old files under those names, with my own instruction stamped on them: immutable, trust this for a year.
The invisible difference. Identical requests receive different bytes because the two devices remember different things. Hidden state — not a server fault — is why "broken" and "fine" were reported at the same moment.
So a listener’s player would fetch index_3.ts, be handed a stale chunk from hours ago straight out of cache, and produce garbled audio. A fresh listener — or me, checking whether "it works" — had no cached history, fetched everything anew, and heard a perfect stream. Both witnesses were telling the truth.
There was a second, quieter failure stacked on top. Live...