What are guidelines to use for exit criteria when releasing?

Question

When I am in ship mode I normally apply a set of criteria before I will "Ship". They do vary, however they are typically along the lines of the following:

• Zarro Boogs (Zero active issues). This normally translates to critical bugs fixed, all other bugs marked as "won't fix" and moved to the next release.
• All test cases executed against a single release candidate build.
• All test failures reviewed and agreement reached that the failures are acceptable, and bugs have been raised.

I am assuimg a large, system that has taken 12 months to develop with a team of say 30-50 team members.

Are these types of criteria common? What criteria would be considered general guidelines or sample exit criteria be?

Answer 1

I would sum up your bullet points as this:

1. Reported bugs are prioritized for level of severity. All bugs considered "must be fixed for release" are then fixed and verified. All other bugs are re-prioritized and scheduled for future releases.
2. A final release candidate build is created and all test cases deemed critically necessary have been executed and verified as passed. If there are any bugs found during this process, it must be repeated to ensure a "clean" release candidate.
3. Risk assessment is performed against all "new" bugs found during final test verification. Bugs are prioritized and evaluated based upon risk and severity and you go back up to item 1.

In an ideal world, when you get down to release time, 1 and 2 should be "clean" and you have nothing else to worry about. But this is not necessarily an ideal world so an iterative approach needs to be taken to make sure that, if something critical is found in the final run through, you have a place to go back to in order to start over and try again without having to re-do the universe.

Answer 2

The "best practice" project exit criteria - would be the one which enables business to take a subjective call on whether to ship or not.

Here is what I have experienced to be useful items on an exit criteria

--No bugs in "open" state - i.e. not triaged,not reviewed,not differed(if a candidate) or not verified(after a fix)

--Candidates for test execution identified . Reason I say candidates it is usually not possible to execute all your tests on each candidate build.Hence the team needs to get together,analyze the risk of the change/fixes(since the last run) and agree upon a list of test activties they would be performed.Once agreed publish this with the project owner (and other relevant stakeholders) to that they know and align their expectations around what is planned to be tested and what not and why ?

--Execute the above tests

-- Test results published,analyzed and triaged

-- Any resulting(from 4) fixing/retesting activties completed

-- Relevant documentation published - release notes,known issues,EULA,legal etc etc

-- Any other explicit and implcit expectations(atrefacts) that business has from the software release in question...

Answer 3

That sound about right. Few questions though:

• by "won't fix" did you mean "won't fix in this release" ?
• "All test cases"- what happens if some test cases didn't run, for example due to a broken test equipment. Where I work we review the risks involved, and decide on a course of action.
• "Single release candidate"- what happens if minor fixes are added ? do you re-run all of the test cases ? we tend to combine several releases if the changes are not major (I know... it's risky)
• "All test failures reviewed and agreement reached that the failures are acceptable"- what happens if a failure is still being investigated ? we have bugs that stays open for weeks or months in rare cases. A known issue can be added to the release notes in this case.