Take the 2-minute tour ×
Software Quality Assurance & Testing Stack Exchange is a question and answer site for software quality control experts, automation engineers, and software testers. It's 100% free, no registration required.

When I am in ship mode I normally apply a set of criteria before I will "Ship". They do vary, however they are typically along the lines of the following:

  • Zarro Boogs (Zero active issues). This normally translates to critical bugs fixed, all other bugs marked as "won't fix" and moved to the next release.
  • All test cases executed against a single release candidate build.
  • All test failures reviewed and agreement reached that the failures are acceptable, and bugs have been raised.

I am assuimg a large, system that has taken 12 months to develop with a team of say 30-50 team members.

Are these types of criteria common? What criteria would be considered general guidelines or sample exit criteria be?

share|improve this question
    
What does "ship" mean for you? If the cost of a release is very large, that's an entirely different situation to something where the cost of doing a release is low. Also - do you mean that you make the ship or no ship decision? How do you take the commercial environment into consideration? (E.g. shipping now might cost us $X in support and fix rollout, but not shipping now may lose us $5X in lost revenue if we miss the trade show...) –  testerab Jun 1 '11 at 0:45
    
I am assuimg a large, system that has taken 12 months to develop with a team of say 30-50 team members –  Bruce McLeod Jun 1 '11 at 4:56
    
Bruce, it's a good question but I recommend amending it to describe the project size as in the above comment. Of course there are many projects that are smaller than that. (My company: 3 developers, 8-12 weeks to develop.) –  user246 Jun 1 '11 at 13:41
1  
Are we missing an F the title? :-) –  corsiKa Jun 17 '11 at 21:24
add comment

3 Answers

up vote 3 down vote accepted

I would sum up your bullet points as this:

  1. Reported bugs are prioritized for level of severity. All bugs considered "must be fixed for release" are then fixed and verified. All other bugs are re-prioritized and scheduled for future releases.
  2. A final release candidate build is created and all test cases deemed critically necessary have been executed and verified as passed. If there are any bugs found during this process, it must be repeated to ensure a "clean" release candidate.
  3. Risk assessment is performed against all "new" bugs found during final test verification. Bugs are prioritized and evaluated based upon risk and severity and you go back up to item 1.

In an ideal world, when you get down to release time, 1 and 2 should be "clean" and you have nothing else to worry about. But this is not necessarily an ideal world so an iterative approach needs to be taken to make sure that, if something critical is found in the final run through, you have a place to go back to in order to start over and try again without having to re-do the universe.

share|improve this answer
add comment

That sound about right. Few questions though:

  • by "won't fix" did you mean "won't fix in this release" ?
  • "All test cases"- what happens if some test cases didn't run, for example due to a broken test equipment. Where I work we review the risks involved, and decide on a course of action.
  • "Single release candidate"- what happens if minor fixes are added ? do you re-run all of the test cases ? we tend to combine several releases if the changes are not major (I know... it's risky)
  • "All test failures reviewed and agreement reached that the failures are acceptable"- what happens if a failure is still being investigated ? we have bugs that stays open for weeks or months in rare cases. A known issue can be added to the release notes in this case.
share|improve this answer
    
As a general rule, appeals for more information on a question belong in a comment, not an answer. :) I'm not going to transition this, because there is some more to this than just clarification requests, and it's quite long and formatted so it wouldn't fit anyway! :) Just figured I'd put that out there. –  corsiKa Jun 2 '11 at 22:32
add comment

The "best practice" project exit criteria - would be the one which enables business to take a subjective call on whether to ship or not.

Here is what I have experienced to be useful items on an exit criteria

--No bugs in "open" state - i.e. not triaged,not reviewed,not differed(if a candidate) or not verified(after a fix)

--Candidates for test execution identified . Reason I say candidates it is usually not possible to execute all your tests on each candidate build.Hence the team needs to get together,analyze the risk of the change/fixes(since the last run) and agree upon a list of test activties they would be performed.Once agreed publish this with the project owner (and other relevant stakeholders) to that they know and align their expectations around what is planned to be tested and what not and why ?

--Execute the above tests

-- Test results published,analyzed and triaged

-- Any resulting(from 4) fixing/retesting activties completed

-- Relevant documentation published - release notes,known issues,EULA,legal etc etc

-- Any other explicit and implcit expectations(atrefacts) that business has from the software release in question...

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.