Is retrying failed tests good strategy to handle flaky tests?

Question

Like many companies we have flaky tests in our test suites. Google has this problem as well:

Unfortunately, across our entire corpus of tests, we see a continual rate of about 1.5% of all test runs reporting a "flaky" result. We define a "flaky" test result as a test that exhibits both a passing and a failing result with the same code.

They have several mitigation strategies for flaky tests including

the ability to re-run only failing tests, and an option to re-run tests automatically when they fail. We even have a way to denote a test as flaky - causing it to report a failure only if it fails 3 times in a row.

We were thinking about similar solution.

1. However, I am afraid that this way I may confuse a failure due to environmental problems with a failure due to a software bug. A software bug can still succeed 3 out of 5 times.
2. I've read "An Empirical Analysis of Flaky Tests" and learned there might many reasons other reasons for flaky tests than just unstable environment and they can be removed by fixing the tests. I am afraid we will unnecessary extend test execution time instead of fixing the tests.

Is Google making a mistake with their mitigation strategy? Or it is something that I should follow?

Answer 1

I find retrying tests to be extremely risky, unless you diligently investigate every retry.

I have never seen an organization examine any retries. In fact, the reason they use a retry mechanism is to deliberately reduce the amount of time they spend investigating failures. The underlying assumption is that the failures are due to the test, and not the system. This is a very, very risky assumption.

If an automated test gives different results on subsequent runs, that means that some key variable is not under the test's control. The thing that's varying may be in the system, or in the test code and test environment.

Risk 1: Masking real system failures

If the uncontrolled variable is in the system, then each failure is trying to give you information about the system. If you retry these tests, and ignore failures when the subsequent run indicates success, you are deliberately ignoring the very information your test is designed to give you.

If you diligently investigate each failure, you reduce this risk. But as I said earlier, I have never seen any organization diligently investigate failures when a retry passes.

Risk 2: Destroying Trust

If the uncontrolled variable is in the test code or test environment, then the tests itself is, to some extent, unreliable. In my experience, it takes very, very little unreliability to destroy an organization's trust in its automated tests. And it is very, very difficult to regain this trust.

My strong advice: Do not use automated retries. Instead, find the uncontrolled source of variability, and get control of it. If you cannot find the source, or if you cannot gain control of it, mark the test as unreliable, and run unreliable tests in a separate test run.

Reliability is Enormously Important

One trap I see in organizations: "Pass all the tests" as a goal for test automation.

Passing tests is a fine goal when you're developing the system. It's dangerous when you're automating tests. For test automation, the goal is not to pass all the tests. The goal is: Write tests that reliably tell you something you repeatedly want to know.

Reliability is hugely important. Accommodating flaky tests through retries is dangerous. It can mask real system problems. And it can destroy your organization's trust in its automated tests.

If the tests are not trustworthy, fix them, get rid of them, or mark the untrustworthy runs as untrustworthy.

(Why yes, the horse on which I am sitting is very tall.)

Answer 2

I'm with @Kevin- it depends.

The test system I am working with now is highly complicated, it depends on an infrastructure made of a long chain of sub-systems each with high reliability numbers. As an example let's assume 99.9% reliability and 20 sub systems. This gives in total around 98% up time meaning almost half an hour each day ! In a system like this one, when I know that I am well covered with tests below and above my test level repeating flaky tests is a good idea.

On the other hand if you are dealing with a life critical system, I would never consider a flaky test an acceptable one.

On the other other hand I have seen flaky tests being removed without going back to your original assumptions about testing, making things even worst.

The only solid advice I can give is to try and invest some time to understand the source of this instability and get rid of it, not necessarily fix it but maybe go around it or re-think your test environment or code aka refactor it.

Answer 3

As with so many questions like this, the answer is "it depends." At least with the testcases I use, they tend to fail because some other testcase that was running at the same time interfered with them. In an ideal world, you'd run testcases like this in an isolated environment, but often that's not feasible.

So I personally say that the answer is to look at the testcases that tend to be flaky, and understand why they're flaky. You should have logging that you can use to determine why the testcase is flaky. If you determine that the reason it is flaky is because it's running in the wrong environment, you can either figure out how to address the environmental issues, or do something like Google does and rerun them/keep track of how often they fail, and only flag them after they've failed x times in a row.

If you can fix the testcase, though, yes, you should, but as with all things, you have to balance the amount of work fixing the testcase will take vs the amount of risk you're accepting by not fixing it, the amount of time it will take to fix, and so on.

I will say, though, reading their blog entry, that designating 16% of their testcases as flaky sounds far too high to me. But without doing investigation, it's hard to say if that's too high or not.

Answer 4

Researching flaky tests is very time consuming. So coming up with a strategy to make it easier and faster seems like a very good idea.

From experience I think re-running test three times is the magic number. Test that fail after three tries have always been issues with the application under test. Where less often the root-cause is some test-infrastructure issue. (Which are time-consuming to fix often)

Your first points worries are solid. What if something only goes wrong sometimes and now we just ignore it, while users might have the same issues one out of three times. Therefor I think the following comment on the Google article is very important:

Our rerun mechanism is only used for tests that are marked as flaky or when users specifically request it.

This means you have to-do some research and then mark it flaky. Now you can define your own checklist for defining when tests are marked flaky or not. Only the flaky tests get re-run three times. Not just blindly run all tests three times. I think for unit-tests you should probably never need to re-run them. For functional UI tests this will happen more often.

Your second point seems irrelevant in the world of clouds. Run your tests in parallel and make your test-suite is as fast as you need it to be.

Monitoring re-runs

If you do add a default re-run strategy for all your tests. Than you should add test-result monitoring and research flaky tests that are more flaky then others.

Tools like XL TestView let you gather test results from multiple channels and view trends on their results. Now you only have to focus on the tests that are failing every day/week but only succeed the second or third time.

I think you do need to add a safeguard to prevent missing issues that only happen under certain circumstances. For this you have to analyse your test-results once in a while.

Answer 5

When comparing the pros and cons of re-run of failing tests, having a reliable testing environment is a necessary condition, while having a full coverage is only secondary. The real problem with re-run is that it probably won’t solve the flakiness problem. Unless you have a very small set of tests, you will still have a high rate of flakiness that will make your E2E testing layer practically useless.

The “under_test” methodology used by Dropbox is a simple solution that might solve your problem:

1. We developed a simple way to designate the relevant parts of a test as the actual business logic being tested.
2. We modified our test framework behavior to treat failures outside these critical sections differently—as “unable to test,” rather than “failure”.

This has led to a significant reduction in flakiness (90%), in turn reducing maintenance time and increasing code test coverage and developers’ trust in the testing process.

In this example only the business part of the code is designated as "under_test", as a result only failures (also flaky failures) in this part will be treated as failures, failures in other parts will be ignored.

# This test fails before validating business logic.
# Test output: fail_to_verify
def test_flaky_on_setup(self):
    some_flaky_setup_code()
    with under_test()
        do_something()