Should release management accept anything less than 100% pass on automated tests?

Question

Should release management accept anything less than 100% pass on automated tests? Is a 97% pass on automated tests a typical thing or is it a sign of test automation problems that need to be fixed? If you get less than a 100% pass, how do you get release operations to look at the report and consider the test results when all they do is complain that the tests failed? What do other companies do when they have tests suites that do not run perfectly?

I am hoping for an answer that provides insight on how I should tackle this problem. Should I spend more time on perfecting tests or should I spend time educating people to read the accurate test reports and make educated estimates?

Answer 1

There are two sides of this question:

1. Your result is not 100% because there is an issue with the Automation test designed by you.

If this is the problem, then definitely you need to improve your automated test cases, so that they execute perfectly (not in terms of approach) but in terms of requirement and coverage. Once they are corrected your result will be 100% and same will be reflected in your reports which can then be shared with the release management team.

2. Now the other part is 'Is there a problem with your application, which is throwing some errors because of which result is not 100%'.

In this situation normally we refer to the test plan or the SLA which should be there in the contract. Test plan or SLA (shared with management and client) should provide you the value that under which release is acceptable like in our case one of the client which I worked has clearly said that anything below 100% is not acceptable (whether it is related to functionality, UI or Language requirement). Hence for us until all test cases are passed we never give a 'GO' for the release.

So, something should also be there is your case formal or informal communication, if your test plan says than with 'Low' priority bugs and above 95% of Quality index (i.e. passed test cases) your application is ready for deployment then, you are good to go and there you need to highlight this thing in your Test Report and Release notes (if any) and communicate the same to the release management team too. At this point you will need to educate them that your pass criteria matches the agreed upon condition and so it is a Green light. Create your report in same way where on first look the impression goes that you are meeting the agreed SLA and hence, further explanation may not be required.

And, if your test plan or SLA says it should be 100% then you can't go ahead. This time you will hold on the release and inform the client about the condition that you have 97% pass and 3% fail (with details), then let him decide, because sometimes for meeting the deadline you can compromise over 1 small feature but rest of the X-1 features should work perfectly fine, which will change your 97% to 100%, but keep this decision with Client or Management as you should not delay the release from your end.

Answer 2

You should spend more time understanding your application and importance of different features.

It does not make sense to postpone important release, which delivers important new features which give you advantage over competition because of a bug in some obscure feature which rarely anyone uses. But you would not want to knowingly release software with bug which can compromise safety of people or their lives.

When our tests do not run perfectly, we follow the same triage process: investigate the bug, how big a problem it is for how many users, find any workarounds if possible, and assess the danger of new bugs created by fixing this one, then put the bug in the pipeline.

Remember that tests are programs, and therefore can contain bugs: reporting bug when there is none, and non reporting when bug is present.

Of course regression bugs have usually higher priority, because they should be easier to fix, and they deliver functionality users expect.