The Pulse: AI load breaks GitHub – why not other vendors? - The Pragmatic Engineer
Menu
Close
Home
Newsletter
Popular Articles
The Software Engineer's Guidebook
My Books
Early trends
Reading List
Ethics statement
Write a guest article
Sponsors
Investing
Now
Contact me
About
RSS Feed
bluesky
youtube
Subscribe
Home
Newsletter
Popular Articles
The Software Engineer's Guidebook
My Books
Early trends
Reading List
Ethics statement
Write a guest article
Sponsors
Investing
Now
Contact me
About
RSS Feed
bluesky
youtube
Before we start: I'm hosting the first-ever The Pragmatic Summit on 11 February, 2026, in San Francisco. Join 400 top engineers and leaders as we answer the question: How is AI reshaping software engineering, dev workflows, and the modern engineering stack?<br>Spaces are limited - don't miss out! Buy tickets here .
-->
Hi, this is Gergely with a bonus, free issue of the Pragmatic Engineer Newsletter. In every issue, I cover Big Tech and startups through the lens of senior engineers and engineering leaders. Today, we cover one out of four topics from last week’s The Pulse issue. Full subscribers received the article below seven days ago. If you’ve been forwarded this email, you can subscribe here.<br>GitHub’s reliability has been beyond unacceptable recently: last month, third party measurements pinned it at one nine (right at 90%). This month, reliability has been down to zero nines – 86% – as per a third-party tracker, and last week, things got even worse: a frankly embarrassing data integrity incident, more outages, and a partial explanation from GitHub, eventually.<br>Data integrity incident<br>Last Thursday (23 April), this happened: PRs merged via the merge queue using the squash merge method produced incorrect merge commits, when the merge group contained more than one PR. Commits were reverted from subsequent merges: basically, commits were “lost” in the code that was merged!<br>Thanks to a bug GitHub introduced, the service broke its integrity promise that pull requests would be merged as expected when using squash merge, which is a technique typically used to merge multiple small commits into a single, meaningful commit. This is a big deal: as data integrity promises are some of the most important ones, for services like GitHub.<br>A total of 2,092 pull requests were impacted, and companies hit by the outage included Modal and Zipline. Effectively, GitHub pushed a bunch of work on affected customers who had to manually untangle and recover lost commits, which GitHub could offer zero assistance with.<br>Customers had to manually go through their git history and restore missing code. After following manual recovery steps (reverting the squash commit and re-applying commits one by one), all commits should have been recovered.<br>GitHub later emailed the list of affected commits to customers, but it’s odd that GitHub executives seemed to downplay the nature of this outage. After all, an outage that messes with data integrity is a much bigger deal than something like a fall in availability where no data is corrupted.<br>Can Duruk, software engineer at Modal, was unhappy about GitHub’s muted response to the outage:<br>“The COO going out of their way to find a huge denominator to make the impact appear small feels very dishonest; versus a sincere apology about how this invalidates their entire promise to their customers. We had to dig into their status page about this to even realize they just casually f***ed up our repo.”<br>Outages don’t stop<br>On Monday (27 April), pull requests and issues disappeared from GitHub’s web UI:<br>Pull requests go missing. Source: Mario ZechnerIssues also not to be found. Source: David CramerThis had to do with an Elasticsearch outage on GitHub’s backend: the cluster became overloaded and went down. So, while pull requests, issues, and projects didn’t vanish altogether, they also didn’t show up during the 6-hour-long outage.<br>There were other outages this week:<br>Some pull requests not showing up (Tuesday, 28 April)<br>Problems with some GitHub Actions (the same day)<br>Incomplete pull requests in repositories (Wednesday, 29 April)<br>Also on Tuesday (28 April), security firm Wiz disclosed a critical security issue, where a bad actor could get access to all repositories on GitHub and GitHub Enterprise server by using only a git push command. GitHub fixed the issue on GitHub.com within six hours, but GitHub Enterprise servers that were not updated remain vulnerable.<br>Famous open source contributor quits GitHub in frustration<br>On Tuesday, Mitchell Hashimoto, founder of HashiCorp, creator of Ghostty, announced GitHub was unfit for professional work and that he was moving off to Ghostty, the open source terminal that’s his main focus. Mitchell’s reasoning was dead simple: being on GitHub makes him unproductive (emphasis mine:)<br>“The past month I’ve kept a journal where I put an “X” next to every date where a GitHub outage has negatively impacted...