If you encounter problems with CI stability, it’s probably due to flaky tests. This has proven to be a recurring theme in our conversations with our CI Analytics and Merge Queue users. So, continuing with our goal to eliminate problems that engineers hate, we embarked on a journey to help developers eliminate flaky tests.
Flaky tests are ubiquitous in teams with automated integration and end-to-end tests, where developers suffer in silence. They’re the biggest challenge to CI stability and a developer pain we’re looking to solve. Even the best teams like , waste countless developer hours due to flaky tests, and wasted engineering hours are expensive.
It’s easy to dismiss flaky tests as a “skill issue” and blame bad test code. But the deeper we dug into test flakiness, the more it looked like a dark, confusing rabbit hole. With our closed-beta partners, we analyzed test results uploaded from 20.2 million CI jobs. We learned that addressing flaky tests in practice requires nuance, you can't just fix all of them because it's impractical. You also can't ignore them because they poison your trust in your test results and slow you down.
While building Flaky Tests, we’ve learned and suffered, and we can’t wait to share our discoveries with you. Both technical and non-technical. If any of what you read resonates with you or you’re also down the rabbit hole with us looking for a better way to tackle flaky tests, we’d love your feedback in the that “over the course of a week, 0.5% of our small tests were flaky, 1.6% of our medium tests were flaky, and 14% of our large tests were flaky.” When you consider that Google has over 4.2 million tests, this is a huge number of flaky tests.
A significant contributor to this phenomenon is that your tests become more complex when you write more realistic tests, like integration tests and end-to-end tests. In an end-to-end test, there can be hundreds of moving parts. Your tests can be flaky for various reasons, like running out of RAM, a service container not spinning up in time, or misconfigured DNS.
Testing is about confidence, and you won't be confident if you don’t have any tests that span your whole system. Dylan Frankland, a lead engineer here at Trunk, puts it like this:
“You can try to make your tests completely un-flakable. But you will have very low confidence in your code. You can mock out every piece and never have a flake, but at the end of the day, then, you're testing mocks, not real code.”
With complexity, writing tests that never flake is practically inevitable.
Suffering in Silence
Software engineers often suffer silently. Tenaciousness, independence, and resourcefulness are often considered standout qualities of great software engineers. However, when tackling flaky tests, these qualities work against them by masking the problem.
When an engineer encounters a flaky test, they might debug it for 30 minutes, decide that this can’t be related to their change, and rerun the test. This sometimes fixes the problem. Since the problem is not critical, they sweep it under the rug, and no one else learns about it. This lets flaky tests accumulate until rerunning no longer clears failures.
If you’re like Uber, who in their Go Monorepo reports 1000+ flaky tests, you’ll rarely see PRs without flaky failures. Even if each test only has a 0.1% flake rate, the chances of having at least 1 flaky failure are 1 - 0.999^1000 or ~63%. That means 63% of PRs will need at least 1 rerun to clear the flaky failure and distract an engineer to debug the failures. If you let the innocent-looking flaky tests accumulate silently, when you finally notice it, it'll already be a monster that paralyzes your PR velocity.
We made a calculator to help you better understand how flaky tests could impact your team, and we recommend that you
We had this problem at Trunk, too. Federico, one of our TLMs, noted after adopting dogfooding flaky tests: "We knew we had flaky tests because we were experiencing failures, but the tool revealed the full extent of the issue." It wasn’t until he saw the numbers in the Flaky Tests dashboard that he realized just how many PRs are being blocked by flaky tests.
For the difficult-to-fix and lower-impact flaky tests, you still want to reduce their impact even if fixing them is impractical. The sweet spot here seems to be have shown that "fixes" submitted by developers to fix flaky tests often don't work anyway, which is more of a reason to have a proper monitoring solution for flaky tests. You need to be able to see that a fix actually worked on your flaky test before marking it healthy again, otherwise, it just ruins your developers' trust in your tests.
Cleaning Under The Bed
One of the main concerns about quarantining tests is that they will be forgotten after the friction is gone. Quarantining fixes the issue of the “boy who cried wolf,” where teams ignored their CI results because the tests were so flaky they lost trust in tests. Quarantine can potentially do the opposite, where flaky tests cause too little friction, and people still ignore them.
To prevent this, we must ensure good reporting and insight to prevent erosion in test coverage over time. We’re currently trying a few ideas. One sees time series data on the number of quarantined test runs over time. You want to see this number go down, not up.
and read part two of this blog here (Coming next week).
SOCIAL SHARE CARD GENERATOR