Take the 2-minute tour ×
Software Quality Assurance & Testing Stack Exchange is a question and answer site for software quality control experts, automation engineers, and software testers. It's 100% free, no registration required.

I'm facing a lot of automated GUI tests that fail one or more verifications during their run. All the failings points have been reported in our tracking system and they are known issues but get "ignored" as having low priority.

Development team promised to have those issues fixed "in the next few months", and this was many months ago.

Should I edit the tests to pass (for example, commenting out the verifications)? Or should I keep bashing on the dev team until they fix those minor display issues? What are the "best practices", if any, in this case?

share|improve this question
2  
This may be useful to you: sqa.stackexchange.com/questions/2998/… –  user246 Sep 25 '12 at 14:08
add comment

5 Answers

up vote 6 down vote accepted

One possibility here is to build into your testing the ability to flag a failure as a "Known issue" which is then reported with each run of your automation.

I've gone into more detail in my answer to the question user246 linked - I'd recommend you check into my comments and the other responses there.

share|improve this answer
add comment

You should not edit the tests to pass. There are still defects in the product, and running the tests and having them fail on those points continues to provide data that the issues are not fixed.

Since the development team has accepted the bugs and scheduled (although not solidly) them to be addressed, modifying the tests to no longer report the defects seems bad practice. If the bugs were rejected and decided as "will not fix", then you should most definitely edit your tests.

Tests should not be modified to mask bugs in the program. It also gives another data point of when the bug was fixed, as you will see the previously failing test cases pass.

share|improve this answer
add comment

If the failing points are isolated solely to these low-priority issues, then you should leave them in place, and deal with these "known issues" showing up as fails.

Make sure these "fails" are still reporting the same low-priority problem and not something bigger than originally reported.

And check them over carefully. Often larger areas are blocked from testing due to these sorts of "known issues". If they are blocking deeper testing, then consider trying to get the priority raised and the untested sections unblocked.

share|improve this answer
add comment

We are building an addon to testng to report tests that are tagged with a @KnownFailure annotation as skips and bundle them into a separate bucket. @KnownFailure annotations will require a ticket # and we will be able to track them via reports.

We're also looking at a way of checking to see if the ticket has been resolved ignoring the @KnownFailure annotation and reporting the true result of the test.

share|improve this answer
add comment

Editing the tests to pass is inappropriate; although it may depend upon the impact of the test failures.

If a test fails quite early and means that other test cases are polluted there may be scope for adding retries or other mechanisms so the test can continue and add more value - but please note this is working around a failure. It should still be reported.

What else you can do depends upon how your results are analysed. The test system I currently use feeds information into a database and requires a tracking identifier against every failure - and a single script can report multiple failures.

Where we encounter a known failure, we have the following options

  • Do nothing. Somebody needs to look at the failure every time and assign an identifier.
  • Add a tentative Bug identifier in the test. Somebody needs to confirm that the failure mechanism matches.
  • Add a Bug identifier in the test. Fully automatic.

Fully automatic bug identifiers are useful but how confident are you that the script can determine the exact failure mode ? We have missed new problems in the past because of a badly coded method of identifying an issue.

Alongside this the technique of grouping tests together can be useful. This allows you to split results into

  • regression: or should work
  • broken: completely unreliable
  • unreliable: tests that fail >10% of the time.

Separating the tests in this way allows you to see new failures (regression campaign), unexpected fixes (broken campaign) and yet still spot major regressions in unreliable scripts (unreliable campaign suddenly fails more than 25% of the time)

These techniques really require some form of simplified analysis front end (a database would do - but that word can scare people)

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.