How I took down prod - by sysarcher - Blended Labs
Blended Labs
SubscribeSign in
How I took down prod<br>And lived to tell the tale
sysarcher<br>Jun 30, 2026
Share
TL;DR — An AI coding agent deleted our root Kubernetes configuration. We lost prod. Thanks to GitOps, recovery took 11 minutes, three commands, and zero guesswork. Moral: Use GitOps and don’t wing infra in the age of AI. Also, better guardrails.<br>Monday Morning
Our Kubernetes cluster runs on GitOps. Every piece of infrastructure lives in a git repository. A tool called Flux watches that repo and makes sure the cluster matches it — automatically, continuously, without anyone running commands by hand. Edit a file, push to GitHub, and the cluster reshapes itself. It’s fast, it’s auditable.<br>Thanks for reading Blended Labs! Subscribe for free to receive new posts and support my work.
Subscribe
I’d been working with an AI coding agent that morning. The kind you can talk to in plain English. It reads your code, proposes fixes, opens pull requests. I’d been iterating for an hour and it had been genuinely helpful — finding real issues, suggesting smart fixes. The rhythm was smooth. I’d say “fix this,” it would investigate, and I’d say “yes, go ahead.”<br>That rhythm was blinding.<br>The Mistake
The agent was debugging why one of our services kept failing to deploy. It traced the issue through our configuration and surfaced another plausible issue (unrelated) what it thought was a “duplicate” — two blocks of YAML, in different files, that both defined the same thing. Same name, same purpose. Redundant.<br>It told me it was going to remove one of the instances. It created and pushed the commit before I could hit Escape.<br>I caught it. I wrote: “wait, don’t delete that.”<br>But Flux doesn’t wait. It reconciles every few minutes, pulling the latest from git. By the time I objected, the bad commit had already landed. Flux had already processed it. And because our configuration has a rule that says “delete anything from the cluster that isn’t in git,” Flux did exactly that.<br>It deleted the root configuration (flux-system) — the master list that says “production goes here, staging goes there, monitoring, ingress, everything.” The thing doing the deleting deleted itself. And since it was the parent of everything else, all other deployments went silent. The cluster reconciled i.e. deleted everything.<br>The Silence
I ran the equivalent of “show me everything” and saw a single service. One. Out of a dozen. The rest had vanished from the control plane — not crashed, not broken, but frozen, unreachable by any tool that could tell them what to do next.<br>I knew what happened instantly. I had done this before myself. Git history tells the story. The latest commit had removed the root configuration, and Flux had faithfully mirrored that reality onto the cluster. It was working perfectly. It executed the wrong instruction flawlessly.<br>Real log entries: Bad commit identified. flux-system is the root of all configurations<br>1. 2026-06-29 12:12 fix: add arc HelmRepository to flux-system kustomization<br>2. 💀 2026-06-29 12:15 fix: remove duplicate flux-system kustomization and add timeout to arc<br>3. 2026-06-29 12:19 fix: restore flux-system kustomization accidentally removed
The Recovery
Here’s where GitOps saved us.<br>Because everything is in git, we knew exactly what state the cluster was supposed to be in. We didn’t have to guess. We didn’t have to ask “what did the on-call engineer run at 3 AM?” There is no on-call kubectl edit. There is only git.<br>Recovery was three steps:<br>1. Revert the bad commit and push to GitHub<br>2. Apply the restored configuration: “make the cluster match this”<br>3. Wait 90 seconds<br>Production came back. Staging came back. Monitoring, ingress, our feature flag service — all of it resumed reconciliation in under two minutes after the fix landed. Total time from detection to full recovery: roughly 11 minutes.<br>Compare this to the pre-GitOps world: no single source of truth, no audit trail of changes, someone SSH-ing into a node to run half-remembered commands, another person on Slack asking “wait, did you bump that replica count last week?” Recovery would have been hours, not minutes.<br>Instead, it was: revert, push, apply, done.<br>The Moral
AI agents are incredible. They read code faster than any human. They find patterns we miss. They’re tireless. They’re confident. And they have no concept of what they don’t know .<br>This wasn’t a bug in Flux. It wasn’t a YAML syntax error. It was an AI agent making a change it didn’t understand, faster than we could stop it. The bad commit was already reconciled before we finished saying “wait.”<br>If you use AI agents with infrastructure — and you probably will — the lesson is simple: know what they’re doing before they do it. Review every change. If something “looks like it shouldn’t be there,” figure out why it *is* there before you delete it.<br>GitOps gave us a 10-minute recovery. Without it, we’d have been down for hours. But the better...