How to deal with automated test cases that always fail?

Question

I'm facing a lot of automated GUI tests that fail one or more verifications during their run. All the failings points have been reported in our tracking system and they are known issues but get "ignored" as having low priority.

Development team promised to have those issues fixed "in the next few months", and this was many months ago.

Should I edit the tests to pass (for example, commenting out the verifications)? Or should I keep bashing on the dev team until they fix those minor display issues? What are the "best practices", if any, in this case?

Answer 1

One possibility here is to build into your testing the ability to flag a failure as a "Known issue" which is then reported with each run of your automation.

I've gone into more detail in my answer to the question user246 linked - I'd recommend you check into my comments and the other responses there.

Answer 2

You should not edit the tests to pass. There are still defects in the product, and running the tests and having them fail on those points continues to provide data that the issues are not fixed.

Since the development team has accepted the bugs and scheduled (although not solidly) them to be addressed, modifying the tests to no longer report the defects seems bad practice. If the bugs were rejected and decided as "will not fix", then you should most definitely edit your tests.

Tests should not be modified to mask bugs in the program. It also gives another data point of when the bug was fixed, as you will see the previously failing test cases pass.

Answer 3

If the failing points are isolated solely to these low-priority issues, then you should leave them in place, and deal with these "known issues" showing up as fails.

Make sure these "fails" are still reporting the same low-priority problem and not something bigger than originally reported.

And check them over carefully. Often larger areas are blocked from testing due to these sorts of "known issues". If they are blocking deeper testing, then consider trying to get the priority raised and the untested sections unblocked.

Answer 4

We are building an addon to testng to report tests that are tagged with a @KnownFailure annotation as skips and bundle them into a separate bucket. @KnownFailure annotations will require a ticket # and we will be able to track them via reports.

We're also looking at a way of checking to see if the ticket has been resolved ignoring the @KnownFailure annotation and reporting the true result of the test.

Answer 5

Editing the tests to pass is inappropriate; although it may depend upon the impact of the test failures.

If a test fails quite early and means that other test cases are polluted there may be scope for adding retries or other mechanisms so the test can continue and add more value - but please note this is working around a failure. It should still be reported.

What else you can do depends upon how your results are analysed. The test system I currently use feeds information into a database and requires a tracking identifier against every failure - and a single script can report multiple failures.

Where we encounter a known failure, we have the following options

• Do nothing. Somebody needs to look at the failure every time and assign an identifier.
• Add a tentative Bug identifier in the test. Somebody needs to confirm that the failure mechanism matches.
• Add a Bug identifier in the test. Fully automatic.

Fully automatic bug identifiers are useful but how confident are you that the script can determine the exact failure mode ? We have missed new problems in the past because of a badly coded method of identifying an issue.

Alongside this the technique of grouping tests together can be useful. This allows you to split results into

• regression: or should work
• broken: completely unreliable
• unreliable: tests that fail >10% of the time.

Separating the tests in this way allows you to see new failures (regression campaign), unexpected fixes (broken campaign) and yet still spot major regressions in unreliable scripts (unreliable campaign suddenly fails more than 25% of the time)

These techniques really require some form of simplified analysis front end (a database would do - but that word can scare people)

Answer 6

In this case, your UI tests are acting as real users, what your tests are experiencing are also what real users are experiencing. So, bugs like this, no matter they're known or not, should always be exposed to your team since users are also suffering until they are fixed.

Answer 7

Differentiate & Implement soft assertions to signal these low priority issues which simply shows up in test results as step failure without impacting the overall test results.