86,359 Queries in One Request: An Outage Where Every Dashboard Was Green — Vlad Radulescu<br>← all writingThe failure was inside the Ruby process: requests executing tens of thousands of cached ActiveRecord queries, occupying enough Puma threads for the health check to time out. The watchdog saw an unresponsive process, assumed it was dead, and restarted it.
It took four plausible but wrong diagnoses to get there. I am including them because the refutations are the useful part.
Client IPs are redacted and long lines are trimmed for width. Every command, value and stack frame is real output from the box.
The symptom that was not the problem
It started with a deploy that appeared to hang:
➜ games.directory git:(master) ✗ RAILS_ENV=production bin/dev --force-deploy<br>[2026-07-31 08:01:36] Deploying branch master..<br>[2026-07-31 08:01:38] Gemfile.lock unchanged — skipping bundle install.<br>[2026-07-31 08:01:46] ✓ Boot gates passed.<br>[2026-07-31 08:01:46] No db/ changes — skipping migrations.<br>[2026-07-31 08:01:48] No supervisor running — becoming the resident supervisor (pid 25075)..<br>[2026-07-31 08:01:48] WARNING: supervisor is attached to a terminal. Prefer launchd or nohup.<br>[2026-07-31 08:01:53] Starting web..<br>[2026-07-31 08:01:53] Starting sidekiq-regular..<br>[2026-07-31 08:01:53] Starting sidekiq-splash..<br>[2026-07-31 08:02:01] Web is answering http://127.0.0.1:1337/up — stack is up.
Then nothing. No prompt.
That part was expected. bin/dev is both the deploy tool and the process supervisor. The first production invocation becomes the resident supervisor: it starts Puma and both Sidekiq processes under respawn loops, then remains in the foreground so launchd has one process to watch.
become_supervisor() {<br># ...<br>watchdog $$ &<br>release_watcher $$ &
while :; do<br>wait 2>/dev/null || true<br>sleep 1<br>done
Later invocations take the deploy path, cut over the release and exit. The entire fork is one predicate:
supervisor_running() {<br>local pid<br>pid=$(cat "$SUPERVISOR_PIDFILE" 2>/dev/null || true)
if alive_matching "$pid" 'bin/dev'; then<br>return 0<br>fi
rm -f "$SUPERVISOR_PIDFILE"<br>return 1
Alive means deploy and exit. Not alive means take over and stay resident.
I got the second branch, so the previous resident supervisor had disappeared. The first question was why.
Act I: launchd was pointing at a deleted file
➜ sudo launchctl print system/directory.games.supervisor<br>system/directory.games.supervisor = {<br>active count = 0<br>path = /Library/LaunchDaemons/directory.games.supervisor.plist<br>state = spawn scheduled
program = /Users/studio51/Projects/games.directory/daemon_start.sh
active count = 0, state = spawn scheduled, and a program path I did not recognise.
daemon_start.sh was no longer in the repository:
➜ git log --oneline --date=short --pretty='%h %ad %s' -- daemon_start.sh<br>0c8a930fc 2026-07-07 chore: delete dead app plumbing<br>a5c71ace9 2026-07-05 fix(deploy): repair daemon_start.sh launcher bugs<br>d6966b5e4 2026-07-05 chore: fix issue with ruby version when running daemon
The same commit deleted the launcher and rewrote the repository copy of the plist:
ProgramArguments
- /Users/studio51/Projects/games.directory/daemon_start.sh<br>+ /bin/zsh<br>+ -lc<br>+ export PATH="$HOME/.asdf/shims:…:$PATH"; … RAILS_ENV=production exec bin/dev
The repository version was correct. The live definition was not.
launchd was using the copy in /Library/LaunchDaemons, and that definition remains loaded until it is explicitly removed and bootstrapped again. Production pulled the commit, deleted daemon_start.sh, and left launchd trying to execute that path forever.
The failure stayed latent for three weeks because the supervisor started before 7 July was still alive. A running process does not care that its launcher has been deleted. The problem only appeared when the supervisor needed to start again.
Lesson 1. Configuration outside the repository that points to files inside it creates restart-time failures. The deploy that introduces the fault and the incident that reveals it can be weeks apart.
Fixing it was mechanical: copy the current plist, bootout, release the stale pidfile, then bootstrap again.
But this was not the outage. It only explained why launchd failed to bring the supervisor back after something else had killed it.
Act II: the Sidekiq backlog looked guilty
The Sidekiq dashboard showed:
316,395,584 Processed<br>505,619 Failed<br>19 Busy<br>613,238 Enqueued<br>144 Retries<br>1,309 Dead
There were two processes: sidekiq-regular at concurrency 15 and sidekiq-splash at 25.
The first correction was interpretive. Processed and Failed are lifetime cumulative counters, not current gauges. 505,619 failures against 316 million processed is about 0.16% over the application’s history. The live error indicators were Retries: 144 and Dead: 1,309, neither of which described an active failure storm.
The queue distribution was more interesting:
➜ bin/rails runner 'Sidekiq::Queue.all.each { |q| puts "#{q.name}: #{q.size}" }'<br>default: 536542<br>splash:...