When "no healthy upstream" isn't about the upstream you think - Sahan SerasingheSkip to content<br>no healthy upstream is the kind of error that makes you expect wreckage.

Then you open the dashboards and find… almost nothing. CPU is low. No pods have crashed. The last deployment was hours ago. By the time you refresh the page, the service has recovered by itself.

That was the scene a few weeks ago when I started chasing an intermittent failure in a search backend. The eventual fix was only a few lines. The interesting part was getting there.

We already had a confident root-cause analysis (RCA), complete with a tidy explanation and a one-line remedy. It was also pointing at the wrong subsystem. This post is about the gap between a plausible story and the evidence, and about a common failure mode in which a mostly healthy fleet slowly removes itself from service.

The shape of the failure

The setup was ordinary: a search backend behind a load balancer, with a fixed pool of worker processes on each instance.

Every so often, with no obvious schedule, a burst of requests failed. The browser showed a bare no healthy upstream, and a minute or two later everything worked again.

One clue appeared every time. Backend p99 latency climbed to almost exactly the load balancer timeout, stayed flat, and then dropped back to normal:

That shape matters. Organic slowdowns tend to be uneven. This was a cliff, a flat top, and a recovery. Requests were not gradually becoming slower; they were running into a deadline and being cut off.

The theory I inherited was CPU throttling . A heavy periodic job supposedly pegged the pod’s CPU, the scheduler throttled it, request handling starved, and the load balancer eventually evicted the instance.

It was coherent. Better still, it came with a one-line fix: raise the CPU limit. That is an attractive combination during an incident. But a root-cause theory makes predictions, and these predictions did not survive contact with the evidence.

Treating the RCA as a hypothesis, not a conclusion

Instead of treating the existing RCA as a conclusion, I treated it as a hypothesis: if CPU throttling caused the incidents, what else should I be able to observe?

Three checks came back wrong.

There was no CPU-bound work in the serving path. The compute-heavy batch indexers ran on separate machines and reached the datastore over the network. They never ran inside the pods serving requests. A process outside the pod’s cgroup cannot cause that pod to be CPU-throttled. The graphs agreed: container throttling counters stayed flat during every event.

The timing did not fit a scheduled trigger. The batch jobs ran on a coarse schedule and produced one sustained utilisation bump. The incidents arrived at arbitrary minutes and happened far more often than the jobs ran. If a timer were responsible, the failures should have followed the timer. They did not.

The failures ignored instance and version boundaries. The same symptom appeared across different pods and deployment SHAs. A regression usually follows a version. A shared external event usually hits the fleet together. These failures did neither. The pattern looked more like each instance was doing something to itself, triggered by its own traffic.

To keep the CPU theory alive, I would have had to explain away the topology, the timing, and the distribution of failures. At that point the theory was creating more questions than it answered, so I dropped it and went back to the logs.

One line in the logs, and why it changes the model

The event window contained one recurring error:

ConnectionTimeout: Connection timed out<br>The stack ended in the datastore client, blocked on a socket read that never returned.

That one line flipped the model.

A CPU-throttled worker is ready to run but cannot get enough scheduler time. A worker blocked on I/O is off-CPU, waiting on the network while still occupying its worker slot. From the outside, both look like “latency went up, then requests timed out.” Underneath, they are opposites.

Adding CPU to an I/O stall does not unblock the socket. At best, it gives you more workers to park behind the same slow dependency. This is why edge symptoms are a dangerous thing to tune against: resource saturation and dependency blocking can produce the same fever while needing completely different treatment.

Why one slow dependency saturates the whole instance

The mechanism came down to two ordinary client settings that were dangerous in combination:

No timeout on individual datastore calls. One call could occupy a worker long after the user had given up.

Retries on connection timeouts. After waiting too long once, the worker would wait again, with backoff in between.

Nothing exotic. That is partly what makes this failure mode easy to miss.

Little’s Law shows why it is fatal to a fixed worker pool. Concurrency is L = λW: the arrival rate (λ) multiplied by the average time each request spends in the system (W).

Suppose the...