16

Having flaky tests that sometimes fail and sometimes pass is very frustrating:

  • there is no single reason each of the flaky tests fail
  • the failures don't seem to happen to real users using the application
  • it requires a lot of attention and time to dig into the test failures and try to come up with a fix
  • they slow the development&testing process down
  • they produce false positives - tests that fail without an actual problem but rather because of timing, network, visual effect issues
  • it often results into disabling flaky tests in order to move forward because of lack of time

What are the general strategies to handle flaky tests?

If context is needed: we are using Protractor/WebDriverJS for end-to-end browser automatedUI testing.

  • See also: sqa.stackexchange.com/questions/23172/… – dzieciou Jul 3 '17 at 14:55
  • @dzieciou yeah, just stumbled upon it several minutes ago while researching how bad is re-running the flaky tests. Good info! Thanks! – alecxe Jul 3 '17 at 14:56
  • Sorry @alecxe but this really feels like a duplicate of sqa.stackexchange.com/questions/27923/… However this can't be closed as a duplicate due to the open bounty. – Michael Durrant Jul 8 '17 at 16:21
  • @MichaelDurrant No problem. It is close, but I think Yu Zhang was asking specifically about intermittent bugs - that are not reproducable 100% of the time. Here, I am asking about flaky tests - tests that sometimes fail and sometimes don't..hope that makes sense. Thanks! – alecxe Jul 8 '17 at 16:24
  • ok, after further review i've removed my edits from that other one – Michael Durrant Jul 8 '17 at 16:30
10
+50

Here's the general approach we're currently implementing in our team:

  1. Measure flakiness to identify unstable tests. One way is to move suspected tests from the main deployment pipeline into quarantine, repeat execution of those tests multiple times for the same environment conditions and choose tests that were producing mixed results (see Martin Fowler's Eradicating Non-Determinism in Tests).

  2. Fix bad test code. This includes fixing obvious bugs and changing test design.

    • Obvious bugs in tests relate to: lack of isolation, asynchronous behaviour, remote service, time issues, resource leaks and global states. See Martin Fowler's Eradicating Non-Determinism in Tests for explanation of those issues. There's also more detailed analysis of root causes and possible fixes in the academic paper An Empirical Analysis of Flaky Tests.
    • Anti-patterns in test design include inverted test pyramid when the team relies primarily on end-to-end tests, using few integration tests and even fewer unit tests. End-to-end tests tend to be not only less stable (and thus less reliable) but also slower and harder at isolating root causes of failures. See Just Say No to More End-to-End Tests from Google Testing Blog for more details on that.
    • There's also evidence that the larger the test, the more likely it will be flaky. Also that certain tools correlate with a higher rate of flaky tests. For example, WebDriver tests (whether written in Java, Python, or JavaScript) have a reputation for being flaky (see Where do our flaky tests come from? from Google Testing Blog). Common solutions to those problems are: do less in the test, shift from out of proc to in-proc and shift from end-to-end to component and unit tests (see Winning with Flaky Test Automation from Microsoft for explanation of those solutions).
  3. Use flaky tests for bugs discovery. Automated tests have two purposes: gateway control and finding new bugs. Gateway control is to verify whether a commit can be included or a build can be deployed to a test environment or a product can be released. Gateway control requires stable and fast tests. Unstable end-to-end tests are not fitting here, although they are good at finding more bugs. However, their results require more analysis because, as OP noted, many bugs found with flaky tests can be false positive. Winning with Flaky Test Automation from Microsoft discusses details of this technique.

  • 1
    A very nice collection of follow-up reads, thanks, learned something new today! – alecxe Jul 3 '17 at 0:49
5

Here is a good general purpose article that deals with flaky tests:

https://semaphoreci.com/community/tutorials/how-to-deal-with-and-eliminate-flaky-tests

What we find is that with image-driven testing tools (Sikuli, Kantu, Testplant...) flaky tests are easier to diagnose, as the screenshots tell you more easily what goes wrong, as opposed to dissecting HAR files and other network traces. But it still happens. So our approach is to move flaky Selenium tests to Kantu, and flaky Kantu tests to Selenium. This usually eliminates the problem as Selenium tests fail more for timing and network reasons, and image-driven tests fail typically for CSS/font/resolution issues.

More general: Use more than one testing tool for a "second opinion". Usually a test is only flaky in one tool.

  • 1
    Excellent suggestion on using another test runner. I can think of two reasons this could be valuable: 1) you will probably need to rewrite the test and doing so will potentially reconfirm if the test was a solid test, 2) Each test runner has different characteristics and it would be good information if the test fails in both (meaning possibility you have a bad test). Having the test pass in one and fail in another runner seems like only a temporary victory. But progress still none the less. – John Burley Jul 2 '17 at 4:22
3

I normally follow this flowchart when dealing with a flaky test in general.

enter image description here

3

I like to setup a re-run strategy, where we rerun any tests that has failed, until it fails three times in a row. If the test is still failing after two times it will be a real failing test and not something caused by network, timing, browser or other infrastructural issues.

Still you should record/log the flaky-ness of the tests as sometimes it is a really issue that users might run into. So always research your flaky tests once in a while, but with the rerun strategy you can postpone this until you have some time.

For protractor have a look at protractor-flake, which might help to re-run your tests.

Random fails will result in losing faith in test suite. Make sure you take time to come-up with a good fix. Either write it down as technical-debt or stop everything else and fix it right away.

Like other already said minimise on the end-2-end tests as they have the highest flakiness ratio.

Other reads:

2

Against flaky test the Googlers themselves contend in vain.

(And yes, last link is a reference to a famous sci-fi novel).

1

From my experience there are 3 steps you should do:

  1. Test quarantine. You can read more about it in one of the following blogs:

https://martinfowler.com/articles/nonDeterminism.html

https://qualityengineer.blog/2018/best-practices-1-test-quarantine/

  1. Introduce retest strategy. Simply rerun failed tests after each run and check whether second execution helps. You may automate this process.

  2. Find a root cause of flakiness in your tests. Here is an article about how Google handle this problem: https://testing.googleblog.com/2016/05/flaky-tests-at-google-and-how-we.html

  • 3
    At least the first and third links were already mentioned in other answers, and all 3 of the points you mention were. Avoid posting new answers to old questions unless they provide additional insight over existing answers. – c32hedge Mar 11 '18 at 20:31
  • 2
    Two points- 1) are you part of qualityengineer.blog? 2) Especially when answering older questions, it's expected that answers bring something fresh to the table. Would you be able to describe what this answer brings that's fresh? – corsiKa Mar 12 '18 at 14:30

protected by Kate Paulk Mar 12 '18 at 11:33

Thank you for your interest in this question. Because it has attracted low-quality or spam answers that had to be removed, posting an answer now requires 10 reputation on this site (the association bonus does not count).

Would you like to answer one of these unanswered questions instead?

Not the answer you're looking for? Browse other questions tagged or ask your own question.