3

I have a bit of a dilemma, I have recently taken over an automated suite of tests written in Selenium/Java for a legacy project.

The trouble is, the suite has consistently had a 40% Fail rate and as such has historically been ignored. What I mean by that is if a new test fails then people have simply stopped looking at why and they are ignored.

Am I better off commenting out/@ignoring/removing the failing tests to have the suite pass and re-enable the tests as I fix them over time? Or is it preferable to leave the failing tests and slowly fix them?

Improve this question
1
  • Update for everyone, I follows the advice on this board and now have a 100% passing suite integrated into the CI build. Now just the small task of getting the coverage back up!
    – ECiurleo
    Commented Nov 30, 2015 at 15:43

3 Answers 3

Reset to default
6

First priority should be for everyone to trust the test suite again, failing tests suites will not give this feeling.

Disable all the broken tests and run all the tests a hundred times to also get rid of all the flickering tests (that work sometimes, but not always).

Start running the tests on each commit or at-least daily to make sure the tests can be trusted to work and find issues and not fail because of any other thing.

Now you can start working on all the disabled tests and enable them again one by one. But maybe focus on more critical parts of the application that have high risk or are known to be brittle. Maybe the old tests are not really worth your time. Decided that on a case by case basis.

Improve this answer
1
  • 1
    This is what worked for us. Better two or three key cases that are stable than lots of coverage that the devs ignore.
    – Ethel Evans
    Commented Nov 25, 2015 at 21:25
2

We experienced the same situation when we started creating the automated test suite indeed the team have created hundreds of automated tests (you know, if you are a passionate tester it is a funny activity) but the suite was not structured and many tests was similar, sometimes duplicate or simply too complex.

In the end, as you said, the suite was not reliable even if it contained several really useful tests because it was always unclear if the result of each test run should have been evaluated or not.

So we decided to (and we had the opportunity to do it) comment all of the failing tests and then we performed a general refactoring of the suite (involving one of our software architect) with the aim of reducing the amount of tests and improving the maintenability.

Improve this answer
1
  • How did you solve duplicate test issue?
    – Mate Mrše
    Commented Aug 27, 2021 at 14:31
1

Better you separate tests which are running 100% successful.So at least your 60% test cases get ready to automate. After that you can concentrate on failed tests and fix them time by time.

Do not completely remove/delete failed tests because it is better to fix already created things rather than to create completely from beginning.

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.