GitHub has another tough day

GitHub has another tough day – Surfing Complexity

Lorin Hochstein

incidents, resilience

August 13, 2026

3 Minutes

On August 6, 2026, GitHub had a pretty rough incident: GitHub Actions was degraded for about nine hours. GitHub posted a public incident write-up. It’s only a few paragraphs long, but there are some interesting details in here.

This was yet another incident that involved saturation. In fact, the write-up even uses the word saturated when describing what happened.

The incident was triggered by a routine deployment to an internal Actions service responsible for processing events and generating Actions jobs. The deployment exposed an existing capacity and concurrency weakness. As pods were replaced during the deployment, remaining capacity became saturated, causing services to crash and triggering a cascading impact across multiple clusters and downstream services.

We often think of deployments as risky because we are changing the code that’s running in production, and the execution of that new code could trigger a behavior change in the system that could lead to an incident. But a deployment is itself also an operational change in the behavior of the system: our system behaves differently during a deployment than it does when nothing is being deployed. Ironically, this is one of the advantages of deploying more frequently: the more often we are deploying, the more that deployment becomes a normal part of the system behavior – we get more experience with the system in deploying state.

In this particular case, running in the deploying state reduces the number of pods available for doing work, as the older pods go online. In this scenario, it sounds like the system was running close enough to the margin that the reduction in capacity due to the deploy pushed the system over the edge, leading to a cascading failure. This is what the resilience folks call a brittle collapse, which is when the system fails in a non-graceful way when it reaches saturation.

As is common when recovering an overloaded the system, they got it back to healthy by shedding load (throttling) and by increasing capacity.

These services recovered at 17:00 after expanding capacity, throttling incoming webhook-triggered work to allow the system to recover, and increasing processing capacity for the backlog of affected events.

I wish the write-up had more details on what was involved in enabling throttling and getting that additional capacity to come online. In particular, I’m curious about whether this was easy to do or difficult. But, alas, you typically don’t get those kinds of details on public writeups.

And, of course, because every incident involves multiple contributing factors, there was a previously undiscovered bug that made things worse by consuming available capacity trying to run invalid jobs:

Due to a latent bug in one of the services responsible for job assignment, runners were getting assigned jobs that were no longer valid and then getting stuck retrying those jobs, preventing them from picking up valid work.

I would love to know more details about how the heck they figured out what was going on with these stuck jobs. I can just imagine being a responder to this incident, trying to get the backlog of work processed, and discovering that there are workers are blocked trying to execute invalid jobs! How did they figure out they were stuck? How did they figure out this was because of a bug?

They deployed a change to work around this problem, but I would love to know what kind of change that was. Was it a quick workaround to get things moving again? I bet it was, but we’ll never know…

This second stage of impact was mitigated by deploying changes to prevent runners from repeatedly attempting to acquire invalid jobs. These mitigations allowed the accumulated queues to drain and Actions to recover to normal operation.

The write-up ends with the typical "here’s what we’re doing to make sure this doesn’t happen again" text, but I am heartened by the last sentence (emphasis mine):

We are also making additional improvements to reduce the risk of cascading failures and accelerate recovery during large-scale Actions disruptions.

Too often, the focus of reliability work is entirely on prevention. I’m happy to see them also focus on preparing to recover more quickly. Because, as we all know, the next big incident is always just around the corner.

GitHub has another tough day

Related Articles

(no title)

Scientists reverse brain aging, with a nasal spray

NHS apologises and admits Palantir have access to identifiable patient data

AI has torched the market for junior programmers

Al Vigier: Canada's AI strategy shouldn't include secret Palantir bills