How to deal with flaky tests that have intermittent failures?

Question

Having flaky tests that sometimes fail and sometimes pass is very frustrating:

• there is no single reason each of the flaky tests fail
• the failures don't seem to happen to real users using the application
• it requires a lot of attention and time to dig into the test failures and try to come up with a fix
• they slow the development&testing process down
• they produce false positives - tests that fail without an actual problem but rather because of timing, network, visual effect issues
• it often results in disabling flaky tests in order to move forward because of lack of time

What are the general strategies to handle flaky tests?

^{If context is needed: we are using Protractor/WebDriverJS for end-to-end browser automatedUI testing.}

Answer 1

Here's the general approach we're currently implementing in our team:

1. Measure flakiness to identify unstable tests. One way is to move suspected tests from the main deployment pipeline into quarantine, repeat execution of those tests multiple times for the same environment conditions and choose tests that were producing mixed results (see Martin Fowler's Eradicating Non-Determinism in Tests).

2. Fix bad test code. This includes fixing obvious bugs and changing test design.

   • Obvious bugs in tests relate to: lack of isolation, asynchronous behaviour, remote service, time issues, resource leaks and global states. See Martin Fowler's Eradicating Non-Determinism in Tests for explanation of those issues. There's also more detailed analysis of root causes and possible fixes in the academic paper An Empirical Analysis of Flaky Tests.
   • Anti-patterns in test design include inverted test pyramid when the team relies primarily on end-to-end tests, using few integration tests and even fewer unit tests. End-to-end tests tend to be not only less stable (and thus less reliable) but also slower and harder at isolating root causes of failures. See Just Say No to More End-to-End Tests from Google Testing Blog for more details on that.
   • There's also evidence that the larger the test, the more likely it will be flaky. Also that certain tools correlate with a higher rate of flaky tests. For example, WebDriver tests (whether written in Java, Python, or JavaScript) have a reputation for being flaky (see Where do our flaky tests come from? from Google Testing Blog). Common solutions to those problems are: do less in the test, shift from out of proc to in-proc and shift from end-to-end to component and unit tests (see Winning with Flaky Test Automation from Microsoft for explanation of those solutions).

3. Use flaky tests for bugs discovery. Automated tests have two purposes: gateway control and finding new bugs. Gateway control is to verify whether a commit can be included or a build can be deployed to a test environment or a product can be released. Gateway control requires stable and fast tests. Unstable end-to-end tests are not fitting here, although they are good at finding more bugs. However, their results require more analysis because, as OP noted, many bugs found with flaky tests can be false positive. Winning with Flaky Test Automation from Microsoft discusses details of this technique.

Answer 2

Here is a good general purpose article that deals with flaky tests:

https://semaphoreci.com/community/tutorials/how-to-deal-with-and-eliminate-flaky-tests

What we find is that with image-driven testing tools (Sikuli, Kantu, Testplant...) flaky tests are easier to diagnose, as the screenshots tell you more easily what goes wrong, as opposed to dissecting HAR files and other network traces. But it still happens. So our approach is to move flaky Selenium tests to Kantu, and flaky Kantu tests to Selenium. This usually eliminates the problem as Selenium tests fail more for timing and network reasons, and image-driven tests fail typically for CSS/font/resolution issues.

More general: Use more than one testing tool for a "second opinion". Usually a test is only flaky in one tool.

Answer 3

I normally follow this flowchart when dealing with a flaky test in general.

Answer 4

I like to setup a re-run strategy, where we rerun any tests that has failed, until it fails three times in a row. If the test is still failing after two times it will be a real failing test and not something caused by network, timing, browser or other infrastructural issues.

Still you should record/log the flaky-ness of the tests as sometimes it is a really issue that users might run into. So always research your flaky tests once in a while, but with the rerun strategy you can postpone this until you have some time.

For protractor have a look at protractor-flake, which might help to re-run your tests.

Random fails will result in losing faith in test suite. Make sure you take time to come-up with a good fix. Either write it down as technical-debt or stop everything else and fix it right away.

Like other already said minimise on the end-2-end tests as they have the highest flakiness ratio.

Other reads:

• https://testing.googleblog.com/2016/05/flaky-tests-at-google-and-how-we.html

Answer 5

Against flaky test the Googlers themselves contend in vain.

(And yes, last link is a reference to a famous sci-fi novel).

Answer 6

From my experience there are 3 steps you should do:

1. Test quarantine. You can read more about it in one of the following blogs:

https://martinfowler.com/articles/nonDeterminism.html

https://qualityengineer.blog/2018/best-practices-1-test-quarantine/

2. Introduce retest strategy. Simply rerun failed tests after each run and check whether second execution helps. You may automate this process.

3. Find a root cause of flakiness in your tests. Here is an article about how Google handle this problem: https://testing.googleblog.com/2016/05/flaky-tests-at-google-and-how-we.html