2

Our release cycles are not strictly periodical but happen about every month. Sometimes, after a regular testing phase, we, a QA team, are not satisfied with the amount of testing and feel that this particular release needs to be examined more. This may happen if, for instance, we release some complicated or "big" feature that, we feel, was not fully tested from all the possible perspectives.

How can we convince our boss to allow us extra time for testing and potentially delay a release? What kind of prove or data can we show to help defend our position?

|improve this question
2

Document weekly rate (count and seriousness) of newly detected bugs, fixed bugs, and fixes which failed and were returned back to development for additional fixes. Then you can extrapolate how many would be detected in the week after release, and let managers to decide if it is OK to release this many bugs to production, or longer testing is required to make product more stable.

Compare these rates with rates for previous releases, and with known bugs detected in production after previous releases, so you can show the trends to management.

Remember you cannot "assure" there are no bugs. QA is "Quality assistance": assist management to make best decisions for the business by providing up-to-date info about current status, and trends, of the quality of the product. Sometimes management is willing to accept more risk to release version at certain date, and sometimes is willing to postpone to get better quality. It is business decision, QA just provides inputs for it (as does marketing, production, finance dept etc).

I would never try to convince boss to delay the release. I would note that rate of detecting new bugs is higher that during previous releases, and let him to make the decision.

We work in aviation, and aviators are strong believers for checklists (and acronyms). So we have a "Cutover Readiness Review" (CRR) checklist, where we have all the information comparable with previous cutovers. After release, we have "Release Lessons Learned" where we can add new questions to CRR, to improve the process for next release One of the questions we ask is "what is your gut feeling about the upcoming release", because once in RLL many people noted they had a strange gut feeling about the release which was hard to verbalize (and it was less that stellar success), so - a lesson was learned. :-)

|improve this answer
  • 2
    Gah. Just what I was writing. I'd also add, though, that you should correlate the weekly rate data with prior release data once released. At least in theory, you should be able to demonstrate that for version X, at the end of the release, we still had a defect rate of Y even in the week right before release, and then they found Z problems in the field. With enough of those data points, you should be able to at least demonstrate to management that they're likely to have a repeat of those field problems if they release the code as scheduled. – Kevin McKenzie Sep 1 '17 at 14:45
2

From a manual perspective, I'd suggest trying something along these lines:

  • List critical functionality tested and level of confidence
  • List happy path/steel thread functionality tested and level of confidence
  • List other functionality tested and level of confidence
  • List all configuration combinations used in testing, and note the percentage of configuration combinations you were unable to test
  • List areas of the software covered by regression testing, and note the areas you were not able to cover
  • List everything you can think of that was not tested, and where possible estimate the risk of releasing with those areas not tested.

Your goal here is not to stand at the gate shouting "You shall not pass!". It's to ensure that the business stakeholders have enough information to make an informed decision about whether to release or not.

|improve this answer
1

The good objective way of proof is a kind of coverage analysis. The way how to do such the analysis is highly dependent on the platform or technology you use when develop an application. For example in Java world there are the tools which can build a code coverage report where one can see which lines were executed with the tests and which were not.

Such the tools are mostly used in UnitTest development however in my practice they proved the efficiency for manual tests as well. The approach is basically following:

  1. You build your application
  2. You run a tool that "isntruments" built application and injects the coverage monitoring code.
  3. You run the testing
  4. You build a coverage report.

Often it turns out that the tests covered only a small part of all the code. So, as to me, such the way is the only objective way to show that there is still a lot of work to do.

|improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.