2

One of the most frequently asked questions of QA by team leads or other business representatives is:

We're getting close to the planned release date. What do you think - is the software ready? Can we move forward with the release?

As far as we know, no software can be made 100% bug free. What should be the answer given by QA in this case?

  • 3
    The answer to this depends completely on context and assessment of risk. There is never a generic answer. If you don't feel confident answering then you should educate yourself to the point to make the decision. If you still can't then maybe the role of Team Lead isn't for you. – ECiurleo Oct 27 '15 at 9:04
  • The question is not about QA should give answer or not Or not to judge him on it! It's all about what should be the answer that some one can better share his experience. – Disha Oct 27 '15 at 9:17
  • The point is the QA Should be judged on their answer. Their role/responsibility is to inform the decision of if the app should go live and if it is considered stable. This may be via exit criteria, risk assessments or other metrics. If they are unable to give that answer, what benefit are they adding to the release process? – ECiurleo Oct 27 '15 at 9:54
  • I'm truly not getting your views why you are interested to judge QA? Why you are talking about the point when QA won't give the answer? Is this pertaining to the question asked? The terms exit criteria,risk analysis etc. are very generic & seems to provide some documentary impacts only. It's better if we discuss the Real & respective contents over here & not out of the track! – Disha Oct 27 '15 at 11:26
  • 1
    It's not so much judging QA as making sure QA is part of the communications when the business decides whether or not to release - see my answer to this question. – Kate Paulk Oct 27 '15 at 11:36
4

The question of "Is this application/feature/fix ready for release?" is one that can't be answered accurately, because - as the OP said - there's no way to be sure that there are no bugs in the software.

More than that, depending on the situation, the business could have reasons for releasing despite the concerns of QA or testers that the software is not ready for production (I've been there).

Our role in this situation is simply this: Give the business the information they need to make an informed decision whether or not to release. That can be in the form "Yes, I think it's good to go - there are no critical bugs and I've tested everything I can think of", it can be a matrix of test case results, or it can be something else entirely.

Ultimately, we give the business a risk assessment from our perspective, ideally looking at these areas:

  • What has been tested - features and functions that have been tested and in how much detail. This includes the outcome of tests and bugs that have been reported and fixed.
  • What has not been tested - any feature or function that wasn't tested and why it wasn't tested, as well as the risk of releasing with that part of the software not tested.
  • Known bugs - any reported bugs that were not fixed, why they weren't fixed, and if you know it, when the team intends to fix them. If there are workarounds for the known bugs, they get included here as well.
  • Any caveats - any quirks of the software that you've come across which aren't actual bugs but could cause support calls - this information often winds up in readme files and goes to the support staff so they know how to advise customers who call about these items (some examples include the software requiring a specific version of a supporting framework, or that it's not compatible with some piece of hardware).
  • Risk assessment - the summary of how comfortable we are with releasing given all the information above.

The format you use to communicate the information isn't relevant: what matters is that it's something everyone involved understands.

  • 3
    +1. Also: what is burn rate of the bugs: how many existing bugs are being closed per day/week, how many new opened. Also, developers should also contribute to risk assessment - they know better about possible risks and consequences of last-minute changes. Quality is a team effort. QA cannot be solely responsible for quality, quality cannot be "assured", it has to be designed and engineered in. – Peter M. Oct 27 '15 at 13:26
  • @PeterMasiar - absolutely, and thank you for the comment. You should add an answer emphasizing this - we're the test/quality-focused specialists, not the sole arbiter of quality. – Kate Paulk Oct 27 '15 at 15:59
  • @KatePaulk +1 Well said! The real picture is somewhat different than what we pretend to do so. Even the team knows about the bugs, the circumstances are forced to release the software. Despite of these, on the other side, its our duty to show the real picture of about to release software to team lead/representatives!! – Disha Oct 31 '15 at 6:42
2

The answer on if a product / website / software is ready to ship should come from the entire team and not just the software tester or QA team.

  • A developer will most likely have unit tests and these will give a good indication on if the software is stable. (The test coverage will of course determine how much weight these have in any final decision)

  • A developer will also have a gut feeling as to how good / stable the code is. They are the ones who have seen the inner workings of the product and know if there are some possible areas that could cause a problem.

  • Any client or product owner should also be involved. Have their requirements been met, have they done any uat testing.

  • What external factors (e.g. an ad campaign) are happening which could make you want to wait until its over before releasing any new update and reduce risk during high traffic periods.

  • The designer will also have an opinion. They will always be able to spot small but important things which are not right which more than often can be put right very quickly.

Of course a software tester will have a very good feeling as to how stable the software is and confidence will grow with the more time they can spend testing, but their feedback shouldn't be combined with the entire teams thoughts.

  • +1 Well mentioned all aspects! I do agree with you that the decision should be combined from all individuals played a significant role in making the app. Still at some level client/leader/Business representative has some perspectives that tester would give unbiased review on how software is stable! – Disha Nov 20 '15 at 13:27
1

That question is the entire reason that the QA team exists. As QA engineers it is our job to test the application and get an idea of its quality and whether or not it is fit for purpose. As such, there are three answers possible:

  • Yes, I think the application is ready for release
  • No, I think we shouldn't release because the quality is insufficient
  • I can't give you a good answer because insufficient testing has been performed to be able to give an accurate assessment, so I would advise against releasing

You should always make it clear to your client and/or team lead what exactly you will be testing, which areas of the application you will be focussing on more heavily, etc. so that they can know what to expect.

Having said that, inevitably there will be issues with the application that you did not catch. When this happens, it is important to analyse the situation: Is this an issue that you should have caught? If yes, is it something that should normally have come out in the tests you did or was there a gap in the testing?

It's in the name: QA is about Quality Assurance. This means that not only should you continuously evaluate the quality of the application, you should also keep evaluating the quality (and quantity) of your own testing to ensure you are still covering what needs to be covered and that the depth of your testing corresponds with what's required in each area.

  • 1
    QA is "quality assistance". You cannot "assure" quality of a product which is badly designed and engineered, with management blaming QA for bugs in production. There are always known bugs/improvements. If you will wait until 100% bugs is fixed, you will never release. Sometimes you have to release (regulation requirement), and role of QA is to document known bugs and workarounds, and prepare follow-up release with fixes. In reality, life is quite messy. – Peter M. Oct 27 '15 at 13:32
  • I'm not sure how my answer would lead you to think I am saying that you should wait for 100% of all issues to be fixed? I'm saying that QA should give their impression of the quality of the application and making it very clear what that impression is based on. I'm also explaining that it is inevitable that issues will occur in the live environment and how to handle that. – Cronax Nov 2 '15 at 8:55
  • QA means Quality Asurance though it may often feel more like Quality Assistance. However the formal definition is Quality Assurance as you are assuring that the product meets the definition of working that you define. That does not mean zero bugs, just that, for a defined set of parameters the product can be assured to x% to be working, e.g. works 99.9% of the time for the basic case. – Michael Durrant Nov 20 '15 at 15:08
  • "That question is the entire reason that the QA team exists." - that would only be true in shops that view QA as a "gatekeeper". That's not the case in shops I manage. While QA provides input, it's up to the business to decide what is "ready for release" or not, in companies where I manage the QA function. – Joe Strazzere Mar 5 '16 at 19:23
1

When the software meets the definition of done it is ready.

So the answer to your question is that all the folks involved need to agree on the definition of done. Formally called 'requirements gathering' before Agile took over the world.

Get together and, for for better or worse agree on how the system should perform in terms of features, usability, performance and bugs.

One example:

  • Initial page should should be under 10 seconds
  • Subsequent pages should load in under 5 seconds
  • User should be able to buy a widget with their visa card
  • User should get an email confirmation that they bought something
  • Page should allow for browsing and searching products
  • Search for products should show images and price
  • etc...

In an Agile world these will be quite minimal as the idea is that you will discover what to work on each week with a meeting of minds of developers, qa and product ("3 amigos").

1

I have been asked this question a countless times during countless releases by nervous and excited project managers. Theoretically, this question does not make sense. If the PM well and truly diligently reads the daily testing status report sent by the QA team (which PM's rarely do), he should be well aware of the 'state' the application is in. But being the project manager(especially in a service based company where he/she is handling multiple projects), he/she is always surrounded by countless issues and rarely gets the time to read reports in full detail. In that case, I kind of forgive the PM and tell my interpretation.

My criteria is based on how close the application meets the test closure criteria at the end stages:

  • Critical / Major functionality is tested and works fine.
  • No major open Sev 1 / Sev 2 issues or blockers
  • No last minute scope changes or change requests which impact the testing work

If all the above hold true, then I would say that the app looks good.

  • + 1. Unfortunately what you say is what happens daily in the real world. QA team is often underestimated and underdimensioned nonetheless is requested to tell if the product is ready or not even if the last code check-in is 5 minutes old. – L123 Nov 20 '15 at 6:51

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.