Editing the tests to pass is inappropriate; although it may depend upon the impact of the test failures.
If a test fails quite early and means that other test cases are polluted there may be scope for adding retries or other mechanisms so the test can continue and add more value - but please note this is working around a failure. It should still be reported.
What else you can do depends upon how your results are analysed. The test system I currently use feeds information into a database and requires a tracking identifier against every failure - and a single script can report multiple failures.
Where we encounter a known failure, we have the following options
- Do nothing. Somebody needs to look at the failure every time and assign an identifier.
- Add a tentative Bug identifier in the test. Somebody needs to confirm that the failure mechanism matches.
- Add a Bug identifier in the test. Fully automatic.
Fully automatic bug identifiers are useful but how confident are you that the script can determine the exact failure mode ? We have missed new problems in the past because of a badly coded method of identifying an issue.
Alongside this the technique of grouping tests together can be useful. This allows you to split results into
- regression: or should work
- broken: completely unreliable
- unreliable: tests that fail >10% of the time.
Separating the tests in this way allows you to see new failures (regression campaign), unexpected fixes (broken campaign) and yet still spot major regressions in unreliable scripts (unreliable campaign suddenly fails more than 25% of the time)
These techniques really require some form of simplified analysis front end (a database would do - but that word can scare people)