Is it fair to report a bug discovered during review without performing a test?

Question

I'm QA with some programming experience, so I am able to debug and review the code of the system under test. I also often read the code to understand how to define request or what response should I expect, if documentation is rudimentary (yea, I know system under test should not be treated as a expected result reference...).

Reporting potential bugs

It happens that during such a review or debugging of some problem found during tests I find another bug. For instance, I find that parameters passed to the client stub are not forwarded in Web service request. From Web service documentation I conclude this may cause integration defect (for instance, it may change the semantics of the requests or cause service failure for a mandatory parameter).

I report this as a bug and describe potential effect, but I have no time to implement full test to provoke such failure. Still, developers keep asking me to specify steps to reproduce the bug and an actual effect and impact.

Is it fair to report a bug discovered during review without performing a test?

Reporting API quality

A subset of problems I find do not impact application behavior but refers to API quality. For instance, I find subsystem API with superfluous parameter that is never used by the component, but confuses both developers and testers using this API ("what value should I put here?").

Should I report such problems in bug tracking system?

Answer 1

I don't think fairness is the issue. More important: Would a report be useful?

For the web service usage, the usefulness of your report depends partly on the scope of your review. Given the code you reviewed, you have a concern. But perhaps other code (outside the scope of your review) prevents the problematic scenarios you're concerned about.

Unfortunately, at this point we don't know whether the problem is a problem, and we won't know until someone investigates further. But everyone who could investigate is attending to other priorities. This is a common issue with bug reports: Who is responsible for investigating? You want the developers to do it; they want you to do it.

What you're learning is that this bug report is not sufficient to inspire others to investigate. At this point I'm inclined to invoke The Responsibility Razor: If I want something to happen, I'm responsible to make it happen. You'll have to either investigate yourself, or do something more to persuade others to investigate.

It's possible that your investigation need not involve running a test. Perhaps walking through the code with some specific values will suffice. Whether this will be persuasive depends on the plausibility of the values you choose, and various people's assumptions about what the web service will do. But I'd start there.

If you don't have time even for that, that's a sign of how important this is for you. So make your report, and let it go.

For the API, the usefulness of a bug report depends partly on the API's audience. If the API is used only internally, and only by a relatively small group of people, I would leave it up to the internal users to decide whether it meets their needs, and to change it if they wish. If the API is used by people outside your team, perhaps you can gather information about their use of the API. If it causes trouble, it's a bug.

Answer 2

Yes, it is fair to report bugs, no matter how you have detected them.

You may find that the developers need the steps to reproduce the problem, in which case you may have to do more work before you can expect the bug to be fixed.

A bug found by reading the code is still a bug. (Unless it's not actually a bug because you misread the code, or didn't understand it - but that's a different issue.)

Consider the counter argument - what if you don't report the bug you know to be there? What could be the consequences? This is not exactly the same scenario, but gets to a similar point: http://www.allthingsquality.com/2010/04/non-reproducible-bugs.html

Answer 3

I think you should ask the developers and business owners rather than us. :-) Ask them how to handle these kind of border cases. Do they want bug reports also on unclear things to remind them or do they want to talk about them first and decide case by case whether to file a bug? Remember it's your task to provide them information on what is or could be wrong, but in the end it's their decision what to do with the information.

One quite nice way of handling the API quality kind of issues is to have separate category for things that should be improved, but won't be done just now. At least if someone at some point comes back and does something about them. If nothing is done, it's actually worse to have this kind of list than just bug/no bug decision.

Answer 4

From your explanation two things I can infer

• Code review shows parameters not handled / passed
• No time to test it

My Questions

• I would challenge why you do not have time to test it. You can check for Free tools like SOAP UI which can help to test the methods directly without writing single line of code

• Secondly reporting an issue without actually validating it is not right practice in my opinion. I would strongly suggest reproducing an issue before filing a bug on it

Your good will can help developer during Unit Testing, Code review sessions. Quality conscious developer would definitely consider your inputs during code review sessions