Fixes hard to verify because of poor quality bug reports

Question

Working with programmers and customers I find it is sometimes harder for me to verify the fix for a defect, than to fix a defect for a developer.

Just a simple example is when there is NullPointerException thrown by a method. A bug report contains no steps to reproduce the issue, just a stacktrace. Given such a poor report, a developer corrects the code so the method swallows such exceptions or re-write the method. Then he marks the defect as fixed.

As a tester I usually care more about the context. Why null values arrived to the method? Which component produced it and why? So simply swallowing the exception is not a solution: it probably did't removed the error. Rather it masked it.

This kind of problems usually happens, when someone else reported a bug (not me), for instance a customer representative, and I see the bug report only when verifying the fix.

So it all boils down to the quality of bug report. If steps to reproduce weren't written before fixing the bug, they must be identified after the fix. And then it is often hard to do it, because I need to check-out/setup two system versions from SVN: one before the fix, and another after the fix was committed.

So how can we improve the process?

1. Shouldn't testers and devs complete the bug report with steps to reproduce before fixing it? I don't care who will do it, but I think such a step should be present in the bug lifecycle here.
2. If devs and testers cannot collect information from customer to reproduce the bug, should they patch it (as in the example above) or reject a bug report as "cannot reproduce"?

EDIT: Maybe I wasn't clear enough. I haven't asked how to write a good bug report. I'm rather looking for practical ways to make devs, customer representative, business analysts, me, etc. complete a bug report as it goes through our hands. Some solutions other have proposed so far:

• introduce templates in bug reports
• review good and bad reports with stakeholders to demonstrate how they help or hinder reproducing bugs and fixing them
• gently ask customer for more details (talking to a customer seems a separate art to me), particularly for what they did before the failure occurred.

Particularly, I haven't seen any answer what a dev should provide in a bug report after fixing the bug? Root cause? What areas were touched? How were they changed? In which revision change was committed?

Answer 1

First, I would like to say that often just taking care of exception is enough. Maybe it's a forgotten corner case (sort a list, but forget it might be empty). Of course you know better when you can trust the developers to make sure that fix is correct.

Then to your questions:

1. It would be really nice to have the steps in each bug. It would be even better to have a test case reproducing the bug before fixing it. That said, I also write bug reports that don't have exact steps but do explain the situation as well as possible. And sometimes I have to file a bug even though I don't even know what it exactly was that broke the system. Of course in these cases I help the developers with whatever questions they might have.

   Maybe the developer doesn't know how to reproduce it exactly either, but does a code review for the broken part and finds some problems. It's fully possible to know that something would be problematic without knowing how to actually get fully running system to that exact state.

   If the problem clearly is in the communication and not in the difficulties of reproducing the problem, then I can't suggest other than telling everyone why writing a good bug report matters. Point out that there are potentially lot of people reading the case is widely different times (developer fixing the bug, tester testing the fix, technical support helping customers, other people reporting bugs, product owner and other shareholders trying to understand state of the product, ...).

2. I don't think it's fruitful to just reject the bugs. It may easily sound arrogant and only hinder discussion with the reporter. Leave the bug open, but ask clarifying questions or steps to reproduce and still be prepared to fix the problem without ever getting these. If there clearly is a problem, it should be fixed even if the person pointing it out can't tell how they got it exactly. They have seen it anyway. Maybe they don't even know the product well enough to understand what it was that caused the problem or are just too busy to come back to it. That's no good, but different people have different priorities and bug reporting may not be near the top for some.

All in all I think we often have to live with less information than we would like to have. Take pride in being able to work also in those conditions but keep pushing people to better process.

Answer 2

I like to think of "bugs" as having a three-part anatomy:

• Failure. An observed difference between the results that the system produces and the results we want it to produce.
• Fault. The error in the system that, under certain conditions, produces a failure.
• Conditions. The conditions under which the fault leads to a failure.

Of these three pieces, the one testers (and others) first encounter is the failure. In the OP's case, you see the NullPointerException.

Sometimes the nature of the failure gives enough information to identify the fault. That might be what happened in the OP's case. So the developer fixes the fault... as far as we know.

What's missing in the OP's scenario (and often in defect reports) are the conditions under which the fault leads to a failure.

To identify the conditions, someone typically has to do some investigation. That someone might be a tester, or a developer, or a team of people. The idea is to isolate the variables involved. A quick overview of a technique I use:

1. Brainstorm all of the variables you can think of that might be involved.
   • Work forward from the test actions and inputs to identify the variables (preconditions, actions, inputs, sequences of events) that might be affected by the actions and inputs.
   • Work backward from the failure (or even the fault, if you know it) to identify the variables that might lead to that failure.
   • Work forward and backward from each variable to identify others that might be involved.
2. Categorize the variables. I like to use a mind map just to cluster them. You could use an fishbone diagram. Or you could build some sort of cause-and-effect map among the variables.
3. Identify the relevant conditions by picking the most likely category or variable and experimenting with it to see whether it has any effect on the failure. If it does not seem to have an effect, move on to the next variable.

This investigation can be costly. It may require people with detailed knowledge of the internals of the system. It may require people who are currently giving higher priority to other activities.

But a defect report that includes not only the failure but also the conditions is more helpful to find the fault, more helpful for reproducing failures, and more helpful in testing and retesting.

Answer 3

It's possible the bug writers aren't collecting the right information because they don't understand what to include and when to include it. Common mistake for people who think writing bug reports is easy but don't understand the elements of a good bug report.

As it happens I just wrote an article on How to Write a Good Bug Report. In the article I tried to distill the elements of a good bug report from the BBST Bug Advocacy class. Read it. You can also look at the source information from Bug Advocacy to get more information.

So how do you fix your problem:

You can review existing bug reports with your team members (programmers, other testers) with an eye for understanding the various elements of their reports and how you can improve them. How are the titles and summaries written? Short, concise and accurate? Show them examples of bad and good reports. See if they can identify ways to improve their own reports if they had to rewrite them. You can even start fixing bug reports you find to be of low reporting quality. This will focus the entire team on making sure reports, whether they wrote them or not, are of high quality.

Now, if customers are reporting the problems directly you might need to set up some ground rules and/or templates to make sure they include valuable information. Examples work well here. There's always the chance they won't pay attention, in which case you can respond back for more information. If someone else (internal person) records the bugs for your customers include them in the same training as your programmers.

I like to look at it like this: The reason to file a bug report is to get the bug fixed. If you can't get enough concise information to identify the problem, there's no reason to open a bug report - you won't be able to do anything with it.

Answer 4

A few thoughts to add to the suggestions others have made:

• Reproduction matters: Wherever possible I try to reproduce any customer-reported issues because there may be other problems involved that the customer didn't report - especially if the bug report itself contains only a stack dump. It might be that this scenario occurs only if a particular configuration is used, or that it's the visible end-point of some other instability that's been surfaced by the sequence of actions the customer is performing.
• Guidance in the bug tool helps: If at all possible, it helps to have a template for customer reported bugs that contains something like this (my wording is pretty crude, but you get the idea):

Where in the system did the problem happen? (several lines space so it's clear that an answer is expected)

What did you expect to happen? (again, add several lines of space)

What actually happened (please be as specific as you can)?

Does this happen every time? (that is, can the problem be reproduced)

Please describe the steps you take to reproduced the problem: (and lots of white space below)

• Update the bug report if necessary: Chances are that with customer-reported issues, the initial information isn't going to be enough. Customers tend to forget that their configuration isn't the only one in the universe, so unless you have an accurate database of customer configurations, someone else is likely to need to add this information. It doesn't matter who makes the updates, so long as they get made so that when the issue needs to be retested two months later all the information the tester needs is right there in the bug report.
• Document testing steps and test configurations: This should be obvious, but it's worth repeating anyway. Early in my testing career I had one customer-reported bug come back to me multiple times because there were actually multiple bugs with identical symptoms, each displaying with a slightly different configuration. In the first pass, the developer fixed and I verified the most obvious manifestation. The next pass, we got a bit more information about the customer's configuration which added a condition that hadn't been present in the first iteration. So that was fixed and verified - but there were other conditions and configurations involved as well, which came from the customer one by one as "Oh, yes, we're doing X" when they were asked. If I'd known to document exactly what I'd done to set up, reproduce, and verify, there would hopefully not have been as many rounds of back and forth (after that bug, I learned. Boy did I learn). Of course, if I had, I might not have found all the interrelated bugs by accident, so who knows?

Answer 5

As a tester I always like to report a bug in such efficient manner so that it can reduce the cost of fixing the bug (the time and the effort) at developers end by giving as much information as possible. I would like to tell few point about the improving of the process of bug reporting :

While reporting the new BUG we should try to mentioned the following:

1. Short description: Each bug must have one short description consists of as much as possible information about the problem in shortest possible way.

2.Severity:Term itself suggests that it is measure of the impact of the found bug on the whole system or system operation. Severity of the bug must be mentioned.

3.Priority:How important is to fix this bug before another? Priority must be marked when new bug has been reported.

4. Subsystem: It is helpful in scrutinizing the bug while fixing or assessment of the developed/developing module, if we mention the subsystem (module/sub module) the current bug belongs to.

5. Description: Description is the also very important part in the bug reporting. In the description part we should provide as much info as possible related to the BUG in all aspect. Some are following:

• Reported By:Helps the developer to come back to the person in case he might not be able to reproduce it while fixing it.
• Reported On date
• Testing Server Info(if any)
• Tested version/build

All mentioned three helps in root cause analysis while fixing the bug.

If possible provide page link as reference by hitting the link directly can see the bug.

6. Attachments: Attachments basically includes the screen shot, and any other file which is required while reproducing the bug.

While taking the screen shot we should focus on following: -Should not include unnecessary part in screen shot but be careful to include everything that is necessary. -The area(s) where actual problem lies should be marked in red color -Some message should be provided in the screen shot itself about the bug

7. Steps to reproduce: This is most important step while reporting bug Here we should write exact step step by action which was followed while reproducing the bug at QA end. This must include: Current/actual behavior, and expected behavior (it is not always compulsory as I feel, but is is mandatory while reporting a bug).

8. Error Log: Even we include the Error message which we get while reproducing the bug which we get at UI, but that does not give the complete and detailed idea about bug. so we should provide the error log with the bug. Error log is some thing which is part of code included in the code by the developer itself in the way so that it should be captured when any crash or some failure occurs in the application. Error log gives a lot of idea about why failure occurred and looking at this log developer can easily figure it out in which area the problem is.

It it possible case that some times we have not all the information mentioned above but we can try as much as possible to provide info.

This is very important not only to reduce the cost of fixing but while verifying the fixes also as you mentioned in your case above. It is not always the case that the person reported the bug will be verifying the bug after it has been fixed, so it is good practice to give the detailed info while reporting the bug.

About your second question: "If devs and testers cannot collect information from customer to reproduce the bug, should they patch it (as in the example above) or reject a bug report as "cannot reproduce"?"

If the bug is reported by Customer itself, you can not expect always that they will provide you step by step action to reproduce the bug by devs or tester, in this case a tester should try to reproduce the bug in all possible way, if still not able to reproduce then you can ask the customer to provide some more info to reproduce the bug. But in any case I would not suggest you to go for rejecting a bug before once trying to collect some more info from customer.