19

I recently ran into an almost-consistently reproducible bug that I was trying to reduce. At the time, I was tired and simply couldn't think of what to do next after about an hour of work (this bug was time-consuming to hit). I know I've read some great lists of steps for reducing and reproing bugs before, but Google didn't find what I was looking for this time.

Just to be clear, I know that you can't just walk through a checklist or else we'd automate the process - but a good list can help a tester come up with ideas when tired and stumped.

Does anyone have any great lists of steps to try to reduce a bug to the smallest set of steps that will generate a consistent repro?

Improve this question
1
  • 4
    Great question - bug isolation is a much neglected key skill for testers. And having a prompt sheet/checklist for something doesn't in any way imply that you've managed to take the skill and judgement out of it - just that you're smart enough to realise that they help.
    – testerab
    May 23, 2011 at 20:50

6 Answers 6

Reset to default
14

Note this is community wiki, please add if you have additional resources. I normally don't answer my own questions, but I thought if I could track down one example, maybe others would get a better idea of what I'm looking for.

Lesson 77 out of "Lessons Learned in Software Testing", titled "Nonreproducible bugs are reproducible", has a short, non-exhaustive list of common types of problems, including brief strategies. The lesson includes resources at the end for more information: Nguyen (2000), Kaner et al. (1993), and Telles and Hsieh (2001). I haven't looked in to these yet, but will once I get some free time.

Danny Faught's "How to make your bugs lonely" is a great walkthrough of bug reduction, demonstrating top-down and bottom-up approaches.

James Bach has a great blog post: How to Investigate Intermittent Problems, which gives some principles (I love "It’s easy to fall in love with a theory of a problem that is sensible, clever, wise, and just happens to be wrong."), suggestions for how to investigate, and a whole heap of guideword heuristics for potential causes at the end.

There are also similar questions on SQA like: How to isolate a bug? or more specific like: Isolating defect in distributed event-driven system.

Improve this answer
2
  • Thank you very much to those who have added to this post!
    – Ethel Evans
    May 23, 2011 at 22:25
  • I did not find lesson 77 in Kaner's book very useful. It does not give detailed examples for the bugs therein and strategies to investigate such bugs. Here is a sample item from 77 that illustrates my point - "The bug might depend on specific data values or on a corrupted database.". This book might be useful in audiobook form. One can hear his testing suggestions while driving, cleaning etc. instead of picking up a book, i.e. not worth the time.
    – JohnSink
    Sep 20, 2017 at 18:33
8

There are a couple of senses in which developers ask testers to reduce a bug. First, it's often handy to be told which values of which inputs trigger the bug. And second, sometimes the tester tries to narrow down where in the product the bug arises. I assume you are asking about the first.

Rather than asking for generally-applicable checklists, perhaps a better question would be, "How do I write a bug reduction checklist for my own product/feature?" I think this comes down to a list of dependencies and a list of strategies for varying manipulating those dependencies.

Some dependencies will be environmental, e.g. how the operating system is configured, which hardware you are using, and how your software is configured. Other dependencies will be unique to your software. If you think broadly enough, you can some up with an infinite number of dependencies, but of course you can't investigate that many. Instead, you might start with ones that have triggered bugs in the past.

In addition to the dependencies, you also need strategies for investigating those dependencies. Some strategies will be easy, e.g. if you are dependent on a browser, try using a different browser, or a different version of a browser. Other dependencies may require special measures, e.g. special software or an undocumented feature that a developer added specifically for diagnosing or debugging a problem. For example, if you are dependent on a fast network connection, you may be able to use a proxy or a hardware device that forces your network connection to run slowly.

Improve this answer
1
  • +1 - your last paragraph is exactly the kind of thing I'm looking for. My own "product / feature" is huge, encompassing multiple services, applications, and web pages (small company, each employee has a large area to cover), so almost every tip will apply to one product or another.
    – Ethel Evans
    May 24, 2011 at 17:03
4

Depending on the context of your bug the heuristics vary. You can also try the below things to find exactly how to reproduce.
1. Ask your peers to execute the flow to see if they can reproduce the issue
2. Repeat the test with a fresh user account
3. Delete all the cache and cookies from the browser and repeat it If the impact of the bug is severe take the help of the DEV in analyzing it

Improve this answer
1

I'm gonna go ahead and say it, even if i get a down vote... But man.. Are you kidding me? :) That's like asking how can I reduce the creation of say.. A cookie. Or how can make milk faster, or bread, or flower grow faster.

If there are certain ways to reproduce a bug, then you have a great chance that the bug can't be reproduced somehow else.

Like... You have a text field that will drop you a nullpointer exception when you don't enter any data. You can't reduce that.

Of course if you have a large, huge bug, let's say... You have a train that goes from DEN to JFK and everyone that sits in the second row will have a ticket number of zero, on every third day of a month in summer. I would try to catch of it is maybe every day, not just summer, or every plain etc etc.

I guess your only option would be elimination. Build a matrix and try to reduce every step, or leave out a few and see if the bug is still reproducible.

But your question can really be answer by common sense. :) Use your brain. Backtrack your steps. Try to leave out a few and see what happens.

Good Luck! :)

Improve this answer
11
  • 1
    You misunderstand the purpose of the checklist. The goal is to convey experience with testing. Newer testers are more likely to think about functional issues when reducing a bug (in fact, you do precisely this!). More experienced testers know that these bugs are fairly easy, but what about intermittent bugs where you can follow the exact same steps and reproduce it once, but not the next time? Memory, CPU, race conditions, timeouts, etc. can all play into these bugs. How do you reduce them? That's where a list of steps / tricks comes in handy.
    – Ethel Evans
    May 23, 2011 at 17:45
  • I don't think so. There are many bugs out there that are either not reproducable but once, or just plain hard to catch. You can't put a list on that. The reason that you didn't find any is that there are none. You have to be intuitive. Or know the code a little bit perhaps.
    – Hannibal
    May 23, 2011 at 20:57
  • 2
    Ethel wasn't asking for a guarantee. She was asking for a set of heuristics - tools that an experienced and skilled tester can apply to help them. If you think that checklists are for unskilled work only, you absolutely should read Atul Gawande's book: gawande.com/the-checklist-manifesto
    – testerab
    May 23, 2011 at 21:29
  • 2
    Ok, so I'm not sure if we have a nomenclature clash here. Regardless of what we're calling it, the list of resources in the first answer is something that a skilled tester can apply when reducing a bug. Do you find those useful? The shortest thing is probably the Bach article - I'd challenge you to read that and say it's not generally applicable!
    – testerab
    May 24, 2011 at 12:30
  • 2
    Hrm . . . actually, I could give a list of ideas for reduction of a cookie recipe (or an issue in a cookie recipe), if I knew the domain well enough. Someone does it right here: allrecipes.com/HowTo/perfect-cookies/Detail.aspx This is the cookie equivalent of what I'm looking for in bug reduction.
    – Ethel Evans
    May 24, 2011 at 17:42
1

I find these sort of things are subjective and depend entirely on the environment that you are working in, steps for A will not always work for B. Still this sort of thing is a long list, it basically entails checking for functionality at all points, such as is the Service installed? is it running? is it updating? is it still running? That get's long easy. If you REALLY need to create something like this, make a wiki and then document things like this, I've done it when starting work on new products and its expanding over time as more and more people added issues they've encountered. After a few years we had a huge number of steps, hints and troubleshooting tools noted.

In the kind of situation you are in, and when you are tired, I start a document on what I am doing, otherwise I can't rely on my ability to recall all my steps. So perhaps if I was going to make a list like you want I would have one item on it:

  1. Document my Test Steps

Then use that for the defect.

Improve this answer
1
  • Step 2 of the list could be "trim down the list removing all steps that do not actually matter in reproducing the problem".
    – philant
    Jul 4, 2012 at 13:21
1

If the failure is induced by large inputs, you may want to take a look at delta-debugging: http://pag-www.gtisc.gatech.edu/courses/common/zeller-tse02.pdf

In short, it takes a failure-inducing input and it tests its subsets, with the goal of finding a (substantially smaller) one that still induces the failure.

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.