A common question from the testers I mentor is "When do I stop investigating on an issue, log a defect with everything I've found/determined so far, and hand it over to dev?" I have yet to come up with an answer I think is reasonable - a heuristic that I feel will work well enough enough of the time. Any suggestions?
Tell me more
×
Software Quality Assurance & Testing Stack Exchange is a question and answer site for
software quality control experts, automation engineers, and software testers. It's 100% free, no registration required.
|
|
|
In the section "Where to draw the line?" of "How to Make your Bugs Lonely: Tips on Bug Isolation" the author says:
The authors lists then:
I, as a tester, particularly agree that "testers may have access to more resources, such as staff and lab equipment, than the programmers". In our project, devs often work more on component level. At the same time testers tend to know more about points of integration of our system with other systems and about operational environment, because they participate more in end-to-end testing. |
||||
|
|
|
The primary concern for the tester should be "Have I found and written enough so that the developers can pick it up from here and successfully debug and fix the problem?" To determine the specific stopping point requires that you know the developers' abilities, and the overall expectations in your shop. Some developers/companies require much more detail from you if they are to have a chance to be successful. Others require far less. When the developers are new, or contractors, or are on a different shore, we may need to provide more details. On the other hand when the developers have been around a long time, and perhaps sit within shouting distance, far less detail is needed. Bug reports are about communication. As with all communication you need to tailor your presentation to your audience. And as with all communication, your feedback loop will tell you if you have been successful or not. And if not, you need to modify your presentation for the next time. |
|||
|
|
|
Think about your role in the bigger picture. You're trying to get the best software in to the hands of your users as soon as possible. The most important thing is to provide enough information that the programmer can reproduce the issue, and that they can be sure they've fixed the issue. When you discover and document a bug, you are the expert on that issue. You may be in a better position than anyone else to gather useful information about the bug. That information can be very helpful in deciding who should work on the bug how urgent it is vs. other bugs. (I used to work on Microsoft Visual Studio, with over 2000 engineers. Getting the bug to the right person was often quite hard!) Certain types of bugs benefit from certain types of information. For example, I want a callstack for any crash report. On many teams, programmer hours are slightly more precious than tester hours. Skilled programmers may be harder to hire than skilled testers. Increasing the number of programmers has more negative consequences than increasing the number of testers. Programmers often get paid more than testers. For these reasons, if you can spend 1 hour of your time to save 1 hour of a programmer's time, it's good for the team and good for the customer. That said, if you're gathering information the programmer doesn't need, then it's a waste of time. The difference is highly contextual; with experience you will get better at knowing when to stop. Because this is all contextual, seek out feedback. Show your bug report to a friendly programmer and ask if they wish for more information, or if anything you provided was unnecessary. Keep in mind that programmers are usually biased to save their own time at your expense, so don't take their word as gospel. |
|||
|
|
|
I'd argue, that when you're thinking that question, it's the right time. But wait, if you can, bring them over and go through it with them - you've got it in a debugger - if they're in the building they'll understand and value the bug far more if the pair of you spend 10 mins debugging through together. Feel free to open up an issue for it, but comunication is best done face to face where possible. What I'm really arguing for is pair programming / pair testing. Once you're sure there's something wrong, walk it through with them. |
|||
|
|
|
Yea, Good Question. Answer to this questions depends on multiple factors. There is no golden rule to say this is the level of investigation required. I will share my experience on this
Hope this provides some clarity and direction. |
||||
|
|
|
I'm not sure there is a single heuristic to apply here. Some of the things I do (assuming this isn't something super-obvious like "open this form and get an access violation") are:
In general, I try to triage first, looking for consistency, seriousness, and impact. Something with high rankings on all three (say, a particular kind of product with tax configuration X always miscalculates tax on returns) will get more detailed information than the bug where you have to hop three times with your left hand on the top of your head and... One thing to remember is that unless your organization enforces a "wall", your tester simply asking a developer what information they need to find a problem makes a big difference. "Enough" information varies for each bug: if I can point the developer to the source of the problem, that's great, but if I can't then I give what I need to do to reproduce it and cross my fingers. |
|||
|
|
|
Something else I've not seen mentioned here is the experience of the tester. |
|||
|
|
|
The general rule I try to follow is "Respect everyone's Time" - one of the lessons I remember from the great book "Lessons Learned in SW testing". This is what that comes in mind when I advice my team members whether to continue investigation or not and when negotiating the amount of help I give to development team on debugging. |
|||
|
|
Without knowing more about you, your organization, and your project, it is very difficult for an outsider to propose a heuristic that you will think is good enough. However, if you are willing to work closely with the relevant parties in an honest, open-minded way, and if you are willing to challenge all of your premises, you may eventually arrive at a conclusion that satisfies you. |
|||
|
|
|
I found Jerry Weinberg's comments about pinpointing in "Perfect Software: And other illusions about testing" really helpful. (My copy's at work, so this is from memory). Jerry comments that responsibility for pinpointing a bug's exact location sometimes lies with the programmers, and sometimes with the testers. In my experience, the most trouble occurs when it isn't clear who should be doing what, especially when that means a management battle over who should be spending their team's budget on that work. I think that it's difficult to come up with a heuristic that works well for an individual tester, because deciding where responsibility for pinpointing should lie isn't a decision that an individual tester should be making: it's a management decision. The amount of time that pinpointing takes can be very unpredictable, and it can take considerable time. (Yes, if you have a team that's empowered to make their own decisions about where they allocate their resources, it can be a team decision. In my current environment, it hasn't actually been an issue yet - in my opinion that's because we don't have separate programming and testing teams, we all work in the same room, and we work pretty closely together on tracking down tricky bugs.) If I did have to discuss where responsibility for pinpointing lay, I'd want to get some programmers, some testers, and whoever's responsible for resource allocation (the PM, product owner, dev/test team managers), and consider the following factors:
There's also some very thoughtful discussion on this blog post about pinpointing, which mentions some of the factors above and more. |
|||
|
|
