A common question from the testers I mentor is "When do I stop investigating on an issue, log a defect with everything I've found/determined so far, and hand it over to dev?" I have yet to come up with an answer I think is reasonable - a heuristic that I feel will work well enough enough of the time. Any suggestions?
I found Jerry Weinberg's comments about pinpointing in "Perfect Software: And other illusions about testing" really helpful. (My copy's at work, so this is from memory).
Jerry comments that responsibility for pinpointing a bug's exact location sometimes lies with the programmers, and sometimes with the testers. In my experience, the most trouble occurs when it isn't clear who should be doing what, especially when that means a management battle over who should be spending their team's budget on that work.
I think that it's difficult to come up with a heuristic that works well for an individual tester, because deciding where responsibility for pinpointing should lie isn't a decision that an individual tester should be making: it's a management decision. The amount of time that pinpointing takes can be very unpredictable, and it can take considerable time.
(Yes, if you have a team that's empowered to make their own decisions about where they allocate their resources, it can be a team decision. In my current environment, it hasn't actually been an issue yet - in my opinion that's because we don't have separate programming and testing teams, we all work in the same room, and we work pretty closely together on tracking down tricky bugs.)
If I did have to discuss where responsibility for pinpointing lay, I'd want to get some programmers, some testers, and whoever's responsible for resource allocation (the PM, product owner, dev/test team managers), and consider the following factors:
- Testability of the product: how easy is it to track down the bug - do you have good logging, enough control over your test environment to control some variables, ability to set up test data easily?
- Ratio of programmers/testers: if you have one tester to six programmers, and they spend all their testing time pinpointing bugs, they're not going to discover very many new bugs.
- Relative skill levels of programmers and testers (do you have a high turnover in programming, and a relatively experienced test team? Or vice versa?)
- Relative levels of understanding of the whole system in the programming and test teams. (Do the programmers work only on their own component and have very little exposure to the systems that interacts with? What about the testers?)
- How long is the feedback loop between testers and programmers? Can you talk directly to build up a better understanding of what information you both need? How long on average is it between a tester raising an issue, and a programmer picking it up to fix it? Is the programmer out of context by then? Is the tester?
There's also some very thoughtful discussion on this blog post about pinpointing, which mentions some of the factors above and more.
The general rule I try to follow is "Respect everyone's Time" - one of the lessons I remember from the great book "Lessons Learned in SW testing". This is what that comes in mind when I advice my team members whether to continue investigation or not and when negotiating the amount of help I give to development team on debugging.
Something else I've not seen mentioned here is the experience of the tester.
I've been in my current job over 10 years, I'm going to understand the application, log files and database much better than someone we hired 1 year ago.
My ability to dig further & provide more infomation is just a function of my time. A newbie may not know about debugging options or log files or other things I do. And there are other testers much more skilled and experienced than I who's bug reports will be better than mine.
I'd argue, that when you're thinking that question, it's the right time.
But wait, if you can, bring them over and go through it with them - you've got it in a debugger - if they're in the building they'll understand and value the bug far more if the pair of you spend 10 mins debugging through together. Feel free to open up an issue for it, but comunication is best done face to face where possible.
What I'm really arguing for is pair programming / pair testing. Once you're sure there's something wrong, walk it through with them.
I'm not sure there is a single heuristic to apply here. Some of the things I do (assuming this isn't something super-obvious like "open this form and get an access violation") are:
- Can I make it happen every time?
- Have I streamlined the sequence of actions to make the problem happen to the bare minimum?
- Have I ruled out incorrect configuration or corrupt data?
- Does the application log give me any useful information?
- Can I increase the logging available to get more information?
- Has anyone else reported this problem? (this is often the first check - if there's already a bug report, there's no need to spend time determining all of the above, unless the existing bug report doesn't provide enough information in my opinion).
- How critical is it to the system? (Particularly when I'm testing development work in progress, I'll simply ask the developers to take a look at something that's not critical if/when they get time rather than put in a detailed bug report - and save the bug reporting for released software)
- How much time do I have to spend tracking this down? (always cross-referenced against how critical the problem is)
- How much time do the developers have to spend tracking this down? (again, cross-referenced against how critical it is)
- How likely is it a user would encounter this problem? (I'll admit, this is usually a partially-educated guess)
- Do I need to include a screenshot?
- Do I need to include a video showing the sequence of events?
- Is there active development in this part of the software (quite often if something is impacted by a development in progress, the bug will go away without any further action)
In general, I try to triage first, looking for consistency, seriousness, and impact. Something with high rankings on all three (say, a particular kind of product with tax configuration X always miscalculates tax on returns) will get more detailed information than the bug where you have to hop three times with your left hand on the top of your head and...
One thing to remember is that unless your organization enforces a "wall", your tester simply asking a developer what information they need to find a problem makes a big difference.
"Enough" information varies for each bug: if I can point the developer to the source of the problem, that's great, but if I can't then I give what I need to do to reproduce it and cross my fingers.
Yea, Good Question. Answer to this questions depends on multiple factors. There is no golden rule to say this is the level of investigation required. I will share my experience on this
As a Tester, Do you agree if the steps provided, details provided are good enough for your peer to repro/verify and agree if this is a bug. It is more of a personal satisfaction. In case if details are not complete it is going to be in a to-and-fro communication again between Dev and Steps before actually arriving at root cause/ bug repro steps
If the number of bugs opened by a tester are closed as 'Not Repro', 'Not a bug / By Design', 'Environmental Issue' then it is certainly an indication of 'Improvement' from QA to provide more details and not to miss a 'Valid Defect' released to production
A bug if it has clear mapping to functional spec for the missing functionality, clear repro steps, Snapshots if it is not consistent, log file entries in debug mode, envrionment in which application is deployed would be helpful for the developer to investigate the issue. End goal is to identify / investigate the issue to identify the issue. If your steps provided are good enough for developer to get started with repro and code fix this would save time required to investigate and confirm if it is a bug
You observe some behavior which is not inline with expected results for a test case
- Check whether the issue is consistently occuring, Classify whether it is a design issue / requirement issue / environmental issue
Depending on which bucket it falls it do further investigation
Example #1 - If it is environmetal issue, you need to check in 32-bit/62 bit if there is any change in behavior, Depending on settings on software installation in system does this behavior change
- Example #2 - requirement issue - Verify for this particular scenario what is expected in requirements, If it does not explictly call out any behavior. Based on other scenarios if you can arrive at what is expected vs what is actual that should be a good place to log defect
Hope this provides some clarity and direction.
In the section "Where to draw the line?" of "How to Make your Bugs Lonely: Tips on Bug Isolation" the author says:
There is no clear dividing line between the bug isolation and reporting that the tester does and the debugging that the programmer does.
The authors lists then:
- factors in favor of having testers put significant effort into isolating the bugs they report, and
- factors that could indicate that it's best to put most of the weight of bug isolation on the programmers
I, as a tester, particularly agree that "testers may have access to more resources, such as staff and lab equipment, than the programmers". In our project, devs often work more on component level. At the same time testers tend to know more about points of integration of our system with other systems and about operational environment, because they participate more in end-to-end testing.
Without knowing more about you, your organization, and your project, it is very difficult for an outsider to propose a heuristic that you will think is good enough. However, if you are willing to work closely with the relevant parties in an honest, open-minded way, and if you are willing to challenge all of your premises, you may eventually arrive at a conclusion that satisfies you.
Think about your role in the bigger picture. You're trying to get the best software in to the hands of your users as soon as possible.
The most important thing is to provide enough information that the programmer can reproduce the issue, and that they can be sure they've fixed the issue.
When you discover and document a bug, you are the expert on that issue. You may be in a better position than anyone else to gather useful information about the bug. That information can be very helpful in deciding who should work on the bug how urgent it is vs. other bugs. (I used to work on Microsoft Visual Studio, with over 2000 engineers. Getting the bug to the right person was often quite hard!)
Certain types of bugs benefit from certain types of information. For example, I want a callstack for any crash report.
On many teams, programmer hours are slightly more precious than tester hours. Skilled programmers may be harder to hire than skilled testers. Increasing the number of programmers has more negative consequences than increasing the number of testers. Programmers often get paid more than testers. For these reasons, if you can spend 1 hour of your time to save 1 hour of a programmer's time, it's good for the team and good for the customer. That said, if you're gathering information the programmer doesn't need, then it's a waste of time. The difference is highly contextual; with experience you will get better at knowing when to stop.
Because this is all contextual, seek out feedback. Show your bug report to a friendly programmer and ask if they wish for more information, or if anything you provided was unnecessary. Keep in mind that programmers are usually biased to save their own time at your expense, so don't take their word as gospel.
The primary concern for the tester should be "Have I found and written enough so that the developers can pick it up from here and successfully debug and fix the problem?"
To determine the specific stopping point requires that you know the developers' abilities, and the overall expectations in your shop.
Some developers/companies require much more detail from you if they are to have a chance to be successful. Others require far less.
When the developers are new, or contractors, or are on a different shore, we may need to provide more details. On the other hand when the developers have been around a long time, and perhaps sit within shouting distance, far less detail is needed.
Bug reports are about communication. As with all communication you need to tailor your presentation to your audience. And as with all communication, your feedback loop will tell you if you have been successful or not. And if not, you need to modify your presentation for the next time.