Is release ready for production?

Question

Every time after we stage our release, we usually have some time to run manual and automated tests against the release. This time period typically lasts for about a week in our project.

To the end of the period, my boss asks me a question which is usually stated in a broad form:

Is the release ready for production?

This has usually been a difficult question for me to answer because I find it hard to understand at what point can I be sure that all the new features are working as expected and there were no new uncaught bugs introduced. Even if there are no new critical bugs, I still cannot get rid of the feeling that we've missed something.

What considerations should I take into account to provide an educated answer to my boss? In what form would you respond?

_{Please let me know if any more context needed.}

Answer 1

That feeling is a common one to testers; I don't think you can get away from it.

That said, and the terminology is going to differ depend on how you arrange the development process, I'm assuming you have the following things:

1) Some sort of test plan, ie, a list of scenarios you'll attempt, or function test cases you'll run, or whatever. Ideally, these will have ties to new functions in the code, or are marked as being done for regression purposes. The test plan should have been reviewed by the stakeholders, including development, support, and possibly the client/end user.

2) A problem list, with problems marked as open, closed/fixed, closed/unreproducible, with some sort of customer associated severity associated with each problem.

Given the above, at any given point during the test cycle, you should be able to have some idea of what percentage of your testing has been attempted (test executed but unsuccessful), completed (executed successfully), and bypassed for some reason (can't be executed for some reason).

At the beginning of a development cycle, you should have a discussion with stakeholders over what done means. Typically, this is something along the lines of 100% attempted, 95% complete, with no open severity 1 or 2 problems. Or maybe it's you've run a stress test for 24 hours without any failures. Or you haven't found any severity 1 or 2 problems in a week. Whatever makes sense for the testing you're doing.And once you've met those requirements, you're done.

You can adjust these criteria over time, but this is the best way, at least in my opinion, of declaring you're ready for production. First, you have buy-in from the other stakeholders that this is the right set of things to do to determine you're ready to release, and second, you have a defined set of metrics to use to make that determination.

If for some reason those metrics need to be tweaked, or additional types of testing needs to be done, or whatever, that's fine, and good, but by having a set of metrics you have predefined, you remove the human factor, which is good, possibly necessary, because as a tester, you're always going to think there's something more you could do, one more problem you could find.

Answer 2

Like continuous deployment, you can you also continuous improvement, with these two crucial steps:

• Release readiness checklist
• post-release "lesson learned" meeting.

Many of members our our team are former military, air force, marines, navy. And they are used to checklists, and to post-mortem analysis. They say that all these lessons are "written in blood" because we learned them by making mistakes and paying for them. It is expensive way to learn, so you want to learn the lesson first time you make the mistake.

Before every release we have "Release readiness" meeting where we go over the release with interested parties, assign responsibilities, and follow the checklist. Exactly like when jet is preparing for takeoff.

After every release we have "lessons learned" session, where anyone can suggests what went wrong (or nearly wrong), and how our process can be improved to prevent it from happening next time. Sometimes we add new questions to checklist, if we feel that some check would prevent something preventable. Sometimes teams add entries to their own internal checklists (so check is done internally and does not appear on final checkist).

So decision "is release ready" is not made by QA team, but by readiness review team, after going over checklist. QA just summons the meeting, business managers make the decision after evaluating the evidence.

It is a decision based not only on manual testing and running automated e2e test, but also unit tests, preparedness for all other teams (documentation, sales, training), etc. Some teams are involved in every release, some (like marketing) only as necessary.

Answer 3

Sounds like you are missing coverage analysis. Whenever we build manual or automated test coverage we should have some way to track the coverage we have created. Historically this could be a simple excel sheet listing test cases by functional area, e.g.:

Login_001 - Enter valid sign in and click login button

Login_002- Enter invalid sign in and click login button etc

In this way we track completion and can understand if our coverage is adequate or not > this drives our go\no go decision and should answer the question "are we ready to release".

This coverage needs to be updated with each new release so that we maintain our test set at optimal efficiency. Even automated coverage should be tracked(often it isn't) and these days there are tools to help(specflow\gherkin layer, pickles etc.).

Any test run is only as effective as its coverage analysis :)