How a Green Health Check Hid a 50-Day Outage | ClipLatch

The dashboard was green. Everything was broken.<br>We ran a routine audit of our own stack on 24 July. Health endpoint: 200 OK. Redis: 1 ms round trip. Containers healthy, seven weeks of uptime, zero restarts, 14% disk. Load average 0.36 on six cores.<br>Then we pasted an Instagram Reel into our own tool and got an error page.<br>So we pasted another one. Same error. Every content type, every URL shape, same result — and the traffic logs showed 200 to 400 people a day had been hitting that same wall for roughly seven weeks. Nobody emailed. They just left.<br>That last part is worth sitting with. A free tool has no support ticket queue to warn you. Users don't file bugs; they close the tab and open a competitor. Our only signal that the product worked was that we hadn't personally checked.

What our health check was actually checking<br>Here's the endpoint, more or less verbatim as it existed that morning:<br>It pinged Redis. That's the whole check. If Redis answered PONG, the service reported healthy, Docker's healthcheck stayed green, and any uptime monitor pointed at it saw 200 forever.<br>Redis was never the fragile part. The fragile parts are a residential proxy pool leased from a third party, a binary that fights an adversarial API, and Instagram itself — none of which the check touched. We had instrumented the one dependency that never breaks.<br>This is the failure mode worth naming, because it's not laziness — the Redis ping was easy to write and easy to reason about, which is exactly why it got written. A health check that measures only what's convenient to measure doesn't just miss outages. It manufactures confidence while it misses them. Green for fifty days is worse than no dashboard at all, because no dashboard would have made us check manually.

Root cause one: the pool had been 407ing since June<br>Every Instagram fetch we make exits through a residential proxy — datacenter IPs get a 403 from Instagram's CDN before the response headers finish parsing, which we wrote up separately in why datacenter IPs can't fetch Instagram.<br>Testing the pool from the box took one loop. All 100 proxies returned 407 Proxy Authentication Required. Not a subset. Not intermittent. Every single one, instantly.<br>A 407 across an entire pool is never a network problem — it's an account problem. Expired subscription, rotated password, exhausted bandwidth quota. Ours had lapsed and we hadn't noticed, because nothing was watching the thing that mattered.<br>The last container log line proving the proxies still authenticated was dated 2 June. Successful parses don't log, so the exact death date is unrecoverable. Somewhere between 2 June and 24 July, the credentials stopped working and the product stopped existing.

Root cause two: yt-dlp had been frozen since March<br>New credentials went in. The proxy probe went green. Parses still failed — with a different error this time:<br>ERROR: [Instagram] DbB4is3D0GM: Instagram sent an empty media response.<br>The obvious suspicion was that the URL had rotted, so we pulled two fresh public Reels straight off @nasa's profile page and tried those. Same empty response. Not the URL.<br>Then we checked the version in the container. 2026.03.17. Upstream at that moment: 2026.07.04. Four months behind, on a tool whose entire job is tracking an API that actively fights it.<br>The Dockerfile said apk add yt-dlp. That one line is the whole bug. Distro packages lag by design — Alpine's maintainers batch and stabilise, which is correct behaviour for a C library and catastrophic for a scraper. Instagram ships anti-scrape changes every few weeks; yt-dlp patches within days; Alpine picks it up whenever. Pinning the upstream binary and bumping it deliberately is the only sane pattern here, and yt-dlp's own docs have said so for years.

A twenty-minute detour into zero padding<br>Switching to the upstream binary should be four lines of Dockerfile. It cost us a wrong turn worth documenting.<br>PyPI reports the version as 2026.7.4. We built the release URL from that string and got a 404. The tag on GitHub is 2026.07.04 — zero-padded month and day.<br>Two representations of the same release, one of them the only one that resolves. If you script a yt-dlp bump, read the tag from the GitHub releases API rather than composing it from a version string, or you'll ship a Dockerfile that fails at build time and looks like a network flake.<br>We added yt-dlp --version as the last step of that RUN layer. A corrupt or 404'd download now fails the build instead of producing an image that boots fine and returns empty media responses in production.

The new monitor paged us four days later. It was wrong.<br>The obvious lesson from a fifty-day blind spot is to check the thing that actually breaks, so we shipped a second endpoint that fetches through a randomly chosen proxy and returns 503 when that fetch fails. Deliberately separate from the Docker healthcheck — a dead proxy pool should page a human, not restart a...