4

I asked a similar question on Katalon Studio forum, but it didn't get much traction: https://forum.katalon.com/discussion/comment/15556#Comment_15556

Anyway, in order to deal with test flakiness this is what I usually do:

I have a Test Suite consisting of cca 15 Test Cases (obviously, there is more than one TS, but I'm making an example). What usually happens when I run the TS is that two or three TCs fail because of network load issues or varying timeouts but when I run them for the second time they pass.

Now, Katalon Studio console mode has the option of repeating only the failed tests.

So, my idea would be something like this:

  1. run the TS, let's say 3/15 TCs fail
  2. re-run the three failed TCs, say 2/3 fail
  3. re-run the 2 remaining cases, those that fail are really failed

Does this seem like a sound re-test strategy? Are there any best practices?

Improve this question

4 Answers 4

Reset to default
6

It is a common practice, but it should be handled with care.

Some things worth thinking about-

  • keep track of how many retries were made, it could represent a real bug and not a transient problem. For example is the first try is always failing in certain scenario.
  • remember that repetitions will make your results less trustworthy so treat this as a temporary solution and do your best to aim for 0 repetitions. This can be done either by fixing the underlying problems or throwing away complex tests and simplifying the others
  • try to understand and keep track of the reasons for failure, again they could indicate a real problem

good read about the subject

Improve this answer
2
  • Do you know a common way to keep track of the reasons for failure? I can go through the logs and copy/paste the error messages, but it's time consuming.
    – Mate Mrše
    May 23, 2018 at 8:28
  • 1
    add that logic to your tests so a proper error message is presented at the end the more precise the better, although you will need some kind of machine learning algorithm to have good prognoses. Some examples "network failure in server room #2", "known windows file system issue #123456"
    – Rsf
    May 23, 2018 at 8:30
6

Do the test cases test a functionality representing a flow? If yes, then executing only failed test cases the second times does not test the functionality properly. If the test cases are to test one after the other, then testing only failed test cases does not check if the failed test cases run along with other test cases, even if they pass the second time.

In real scenarios, when end-users use the functionality, they will use it in the flow. That functionality is going to fail in that flow. Now, you can't expect the users to use only that functionality, the 2nd time, which the failed test cases represent.

Even if the test cases are not related to each other, all still need to be executed in that scenario. And as that is failing, it is an issue.

So, in my opinion, this is not a good Testing strategy. If the test cases are failing with rest of the test cases, but running standalone, then there must some issue with the functionality which the failing test cases are testing.

Improve this answer
4
  • The test cases don't represent a flow, they can be executed multiple times in any order. I talked to the developers, and they explained there are timing issues because the app was built using different technologies and basically it takes time for the app to "warm-up" first thing in the morning.
    – Mate Mrše
    May 23, 2018 at 8:26
  • @MateMrše - Add a warm-up to the actual application so it is run every time? Otherwise, could the users experience exactly the same failure? Or does it succeed, but slowly; any harm in increasing the expected time? Or can the test run the test twice, but only check the second results?
    – Nigel Touch
    May 23, 2018 at 16:10
  • 1
    The fact that it is expected behavior by the developers doesn't mean it is therefore not bad behavior for a user. Those are different things! If you are focusing on getting your tests to pass you (and your team) might forget to look to the actual problem that people might have with your solution.
    – Ray Oei
    May 23, 2018 at 17:58
  • @NigelTouch - This is (unfortunately) the way the application works in real life situations. We usually run a mock instance (login user, fill out some forms) early in the morning, so real users don't notice it.
    – Mate Mrše
    May 24, 2018 at 6:16
1

My best advice is to find the route cause of all failures, being random or not and fix them once and for all.

I remember a case where a tiny bit of information on a web page was retrieved from an external party. Whenever the external party would randomly return an error, the complete page would not be rendered at all. For a high traffic website this is a bad thing. Especially if you think what would happen if the external party has an outage!

Although the tests were not doing anything with the functionality of the external party, they were still failing on error of the external party because there was no page to interact with at all. Accepting random failures as normal and simply retry all the time, would not have lead to a fix for this problem.

Also, in a different project where we struggled with some instability due to front end/ browser interaction, people did not trust any of the test result at all when the tests were failing: just re-run it, it's probably the tests! The tests are supposed to give fast feedback, but now people were waiting for the tests to run again for 30 minutes only to find out that there were some real blocking issues at the moment we were supposed to deploy to production.

Improve this answer
1
  • Thanks. I'm not trying to sweep the bugs under the rug, I usually know why the tests fail (partly because of the "warm-up", as I told to @Rishikesh). This is all part of the grand scheme of refactoring our old app. :)
    – Mate Mrše
    May 23, 2018 at 8:43
1

I suggest you identify the elements under test which you know lead to flakiness and test those separately from other tests.

If the said tests repeatedly fail, then it's up to the business whether or not a fix should be implemented.

This should avoid the situation whereby the failure of these tests prevents other features being assessed.

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.