What is a reasonable expectation for bugs upon release?

Question

This isn't really meant to be subjective, but I guess it is slightly.....but im trying to figure out what is "reasonable" in terms of bugs found post release.

Now lets assume you do your normal QA/Dev cycle, and release a product. Customer finds a few small bugs (Lets say visual or low impact issues)....and they are mad!

"Why did we find bugs???? Isn't QA supposed to find the bugs"

What's an appropriate response to this? Unless im living in a fantasy land.....Im guessing everyone (even if QA is doing a good job) runs into some minor bugs post release? Isn't that supposed to be built into the "cost" of the project? or am I wrong?

Answer 1

That's not really answerable. What's reasonable depends on the expected use of the software, and the time/money that's being invested. For example, for aeronautic software, what's viewed as reasonable is perfection, and they spend a huge amount of time, and money, and engineering resource, on making the flight software perfect. And they fail. Same thing with software for space exploration, and they lost a probe because one programmer used metric measurements while another used imperial measurements. That wasn't a minor bug; that was a catastrophic one.

Now, a valid thing to do, certainly, when there are bugs, is analyze them, and figure out how the development process (not test process, but the entire development process) can be improved. Look at things like the environment the defect was found in, the impact it had, the effort it took to fix, and, crucially, the type of bug it was. One model for doing this is Orthogonal Defect Classification, where you look at where in the process the bug was introduced (design, development, post-release, etc), and what it would have taken to find the bug in test. You can also compare signatures during different parts of the release cycle to determine if you're finding the sorts of problems you expect to, although this requires a fair amount of historical data to do with any accuracy.

You can also explain the Halting problem to them.

Answer 2

Role of QA is not to find all the bugs. QA is there to establish a process of finding bugs, collecting and prioritizing them, to be fixed fixed by developers in order of business priorities. And provide business side with info about current and projected stability of the codebase, and if release is stable enough to be released withing the deadlines.

Quality cannot be "assured" by more testing - quality has to be designed and coded in. In famous saying, QA cannot prove there are no bugs - can only show where known bugs are, using the limited time and resources QA has to do the testing.

Quality is a team effort. Developers, QA and business side all have integral part in assuring the quality, and finger-pointing is the fastest way to hell.

So if customer found a minor bug, and QA was blamed for it, business side failed to do their job. Proper response would be emailing the customer:

Thank you for notifying us about the problem. Your comments were entered to bug database and they will be evaluated and prioritized by severity and how many customers they affect.

While I cannot tell you right now when your complaint will be fixed, I can promise it will be taken seriously.

I would also like to thank you for our using our product and for taking time to enter the feedback. Feedback from customers like you helps us to improve our product and make it work as best as we can for as many customers as we can.

If you have more concerns, please do not hesitate to contact me again.

This is the role of business in the situation like yours (with a minor bug). Cover your back, report the bug, not throw you under the bus.

Answer 3

There is an old gag about acceptable failure rates. The joke goes that an american company ordered some manufactured parts from a company in (a foreign country) The spec called for there to be "no more defective parts than 5 in 1000"

Some time later the parts arrive at the american company. 5 containers of 1000 finished parts each. then the driver of one of the trucks delivers a smaller crate and asks the american company to receive the shipment. The american representative signs for it and reads the invoice for "5000 parts. The requested 25 defective parts are shipped separately"

My point here is that you cant set an expectation of absolute number of failures. You can set expectations of classes of failure based on the expected product specification. As noted, avionics would be very intolerant of failure. A personal ego web site, more tolerant.

Also this is the purpose of User Acceptance Testing. The user has a chance to run those scenarios they really care about and verify that they are getting what they are paying for.

Answer 4

The most straightforward answer is: those that were found and were not fixed, those that were found and were labeled as features plus all that were not found. The ones in the first and second group should be documented and the customers should be aware of them and the ones in the last group should not be the elephant in the room.

QA is supposed to find bugs, and given time it will find every one of them, but the costs and the time to find the next bug increase for every bug found, to reach a point where you need to spend days, weeks or months to find a single one, that affect maybe one customer. At this point QA is no more effective.

When a costumer will report a defect, there is no if, no complex software is bug free, put yourself in their shoes. Is the defect a subtle one? Do you need to do specific steps to make it happen? Or it happens every time you execute the primary process?

How will you react? Will you be angry to find the defect?

In particular visual bug, if not subtle, are never reasonable in my opinion. Think of printing an invoice with the logo of another company, will you not be angry as the customer?

After that thanks the customer, add some test cases in your test suite for every reported issue and be sure to run them before the next release.

Answer 5

There is, of course, a substantial difference between a customer finding a defect that the Test team had previously found (but which Management were happy to release into Live) and those which were missed by the Test team. As a tester, it's the latter which most concern me.

"Reasonable" is inherently subjective. For the sake of argument, I take "Reasonable" to mean a level of defects which I'd be willing to say was unavoidable, given the circumstances.

"Reasonable" is also not the same as "Acceptable". "Acceptable" is the easier approach, as it's something which can be agreed in advance with your management structure. "No high or critical severity defects, no more than X normal severity..." That sort of thing.

"Reasonable", though... Much harder.

The issues that control code quality (prior to and post-testing) are a huge subject, so I'll try to bundle it up a bit. I think if I started writing about it, I'd still be here this time tomorrow.

A "reasonable" level of defects would be determined by: the amount of resource available measured against the code quality and level of bug-fixing support.

I can't think of a better way to sum it up. Bad code and no time = lots of bugs. Good code and no time = few bugs. Bad code and lots of time = few bugs. It's not very scientific but I think it stands scrutiny.