That Test isn't Flaky. It's Broken. · will keleher

That Test isn’t Flaky. It’s Broken.

I’ve seen two different teams delete and redo entire test suites after they became too inconsistent to be useful. Both times the root cause was the same: tests were "flaky," so people reran them when they failed. As tests were added, both suites accumulated more and more debt until starting fresh seemed like the best option.

"Flaky" is a dangerous word for tests because it suggests solutions: rerun the failed test, build a system to automatically retry failed tests, and accept that the suite will occasionally fail. Re-running "flaky" tests might lead to a passing test suite today, but the long-term impact of that policy is a test suite that will get more and more unreliable over time. Without feedback that lets engineers know that they introduced a new problem, the failure rate will only ever increase. Even if nothing changes with the base error rate, simply adding more tests will make failures more likely. And as your engineering team grows, the test suite will be run more often, making it more likely that people will hit problems.

A test suite that doesn’t have intermittently failing tests is easy to keep that way. If an intermittently failing test gets added, it will be relatively simple to debug and fix because you’ll discover it faster.

Why? A 1/100,000 chance to fail doesn’t stay that way

Imagine 5% of your tests are written in a way that will fail 1/100,000 times. If you have 1,000 tests, what are the chances that your test suite as a whole will run successfully? Not too shabby: 99.95%. But as you add more tests, your chances of a successful test run start to go down. At 10,000 tests, your test suite has a 1/200 chance of failing: 1 - (1 − 1/105)(.05 * 10,000) = 0.5%.

1 in 200 test suite runs failing doesn’t sound terrible, but if you think about a team of 10, 100, or 1,000 engineers (and their agents) running tests over the course of a week, many engineers will encounter at least one failed test run over the course of the week. If you assume an engineer runs the full test suite 10 times per day, that means that over the course of the week, they only have a .99550 = 77.8% chance to have every run succeed. Having each engineer have over a 1/5 chance to hit an intermittently failing test suite run over the course of a week is culturally insidious.

Even so, if only 1 in 200 test suite runs failed this way, it’d merely be annoying. But when you have a test suite that is known to be "flaky," it contributes to an engineering culture where people will re-run "flakes." That means that when bad tests are added that fail more often than 1/100,000 times, engineers are more likely to believe that they’re just getting unlucky and hitting errors rather than realize that the suite is now failing on 1/100 or 1/50 full runs.

💬

On a large codebase, I’ve seen plenty of test failures that happen 1/1,000,000 times. A simple example was a test that assumed that the OTP for a user would never be exactly 444444. Another was a test that used a one-in-a-million cutoff to identify an implausibly distributed feature-switch population. In terms of engineering culture, I think it makes sense to treat tests that fail one in a million times as broken because it builds the practice of fixing the other broken tests that are more likely than that to fail.

How do you discover and fix intermittently failing tests?

My favorite tool to flush out intermittently failing tests is running the test suite on an hourly cron. The crux is using a commit that has already passed CI on master/main so that you can be sure that every failure represents a true problem of some sort. Once you have a way of discovering and tracking intermittent failures, it’s relatively straightforward to start driving the failure rate down.1

An hourly cron isn’t enough to surface every 1/1,000,000 or even 1/100,000 test failure, but the failures that it does surface will often represent a class of problem: perhaps the fixture-resetting logic is flawed, perhaps there are a lot of latent time bugs in our system, or perhaps there’s a bad pattern for waiting for an element on the screen. These test failures surfaced by the cron runner are gifts that let you improve the codebase as a whole to make that particular problem less likely. Even when it’s not a codebase-wide problem, a test file that has one questionable pattern in it is likely to have more, so when a single test fails, it often makes sense to scan through the whole file for other tests that could fail in the future and chat with the team about any patterns that could be better.

The goal of the hourly cron runner is to flush out intermittent failures, so you want to make those failures more likely if you can. Run your tests backwards, intentionally try to run these tests on machines with more resource contention, inject random...