When Good Logging Takes Down Your Whole System

The Forensic Engineer

SubscribeSign in

When Good Logging Takes Down Your Whole System Sometimes doing it "right" isn't enough.

Manpreet Kaur Sasan Jul 08, 2026

Ever been so responsible it backfired? Bluesky’s engineering team knows the feeling. They did the “right” thing - logged every error, exactly like you’re taught to - and it took their whole system down for a weekend. Thanks for reading The Forensic Engineer ! Subscribe for free to receive new posts and support my work.

A blocking syscall, a thread explosion, and a sharp bit of thinking under pressure. Let’s get into it. The Byte

Observability, monitoring, logging - they’re meant to catch failures, not cause them. Or so I thought. Turns out they can take your whole system down too. In April 2026, Bluesky quietly shipped a new internal service. Nothing alarming about it - low traffic, nothing flagged in review. But one endpoint was missing a limit that every other endpoint in the system had. For a week, that went unnoticed. Then, on Saturday, April 4, alerts started trickling in. The team’s first instinct was a network issue - reasonable, but wrong. The system limped along through the weekend. By Monday, April 6, it gave way completely. Half their users were affected for eight hours. The twist: the system’s own error logging was the actual cause of the collapse. Every failed operation was being logged, exactly as good practice says it should be. But in Go, writing a log entry is a “blocking” action - the program stops everything else until that one line is fully written. Normally, a tiny pause nobody notices. But thousands of these pauses, every second, add up faster than a system can clear them. The runtime kept compensating, and the compensation made things worse. A system built to catch failures ended up feeding one. The Breakdown

Here’s why it happened, step by step: The Spark - One endpoint had no limit on concurrent requests. A burst of 20,000 came in at once, and the system tried to open 20,000 connections simultaneously.

The Lock-Up - Closed connections don’t free up instantly; they stay reserved briefly before reuse. With that many opening at once, the system ran out of available connections. New attempts failed outright.

The Block - Every failure got logged. Writing a log line uses a blocking write syscall (true in Go, common in many languages) - it pauses the program until done. Thousands of failures meant thousands of tiny pauses, stacking up fast.

The Overload - With so many tasks stuck waiting, the runtime spun up new, heavier real threads so other work wouldn’t be stuck behind them. This holds true in almost any modern language - lightweight tasks run cheaply on top of a few real threads, until one blocks, and the system compensates by creating more. Do that for thousands of tasks in seconds, and a lean handful of threads becomes an overcrowded mess.

The Freeze - Those extra threads spiked memory pressure. Cleanup had to pause everything just to catch up.

The Loop - The moment things unfroze, every stalled task woke up at once, rushed the same exhausted connections, and triggered the same failure again. A fire the system kept re-lighting.

The Workaround - To buy time, engineers stopped routing every connection through one address. They spread connections across hundreds instead. Slots multiplied, and the cycle broke - for now.

Each step was the correct behavior of one layer, triggering the correct behavior of the next - until the sum of “correct” became a collapse. The team didn’t do the wrong thing by logging failures. What was missing was rigor: a cap on how much could run at once, and separation between the logging path and the system it watched.

That same gap - assuming something is safe at any volume - shows up everywhere: input validation, retries, rate limits, anything on the unhappy path nobody stress-tests. It matters even more in AI-augmented systems, where we’re logging every model call and treating that pipeline as free. Bluesky’s outage wasn’t really about logging. It was about an assumption nobody checked. Somewhere in your codebase, something’s making the same one. Might be worth finding it before production does. Sources: Bluesky’s postmortem on pckt.blog, with additional analysis from Lorin Hochstein on Surfing Complexity. Thanks for reading The Forensic Engineer ! Subscribe for free to receive new posts and support my work.

Discussion about this post CommentsRestacks

Ready for more?

This site requires JavaScript to run correctly. Please turn on JavaScript or unblock scripts

When Good Logging Takes Down Your Whole System

Related Articles

(no title)

Scientists reverse brain aging, with a nasal spray

AI has torched the market for junior programmers

Al Vigier: Canada's AI strategy shouldn't include secret Palantir bills

The Anatomy of an AI-Native Org