11

I'm QA with some programming experience, so I am able to debug and review the code of the system under test. I also often read the code to understand how to define request or what response should I expect if documentation is rudimentary (yea, I know system under test should not be treated as an expected result reference...).

Reporting potential bugs

It happens that during such a review or debugging of some problem found during tests I find another bug. For instance, I find that parameters passed to the client stub are not forwarded in Web service request. From Web service documentation I conclude this may cause integration defect (for instance, it may change the semantics of the requests or cause service failure for a mandatory parameter).

I report this as a bug and describe the potential effect, but I have no time to implement a full test to provoke such failure. Still, developers keep asking me to specify steps to reproduce the bug and an actual effect and impact.

Is it fair to report a bug discovered during review without performing a test?

Reporting API quality

A subset of problems I find do not impact application behavior but refers to API quality. For instance, I find subsystem API with a superfluous parameter that is never used by the component but confuses both developers and testers using this API ("what value should I put here?").

Should I report such problems in a bug tracking system?

Improve this question
2
  • 2
    When you do report a "potential bug" but haven't actually performed a test, what happens when the bug is fixed? Will you test it then? Or just read the code and call it good enough?
    – Joe Strazzere
    Commented Nov 13, 2012 at 22:18
  • Those I found I would re-review, because they were rather trivial and obvious. I want also to focus system tests (that I will perform, when test environment is ready) in that area.
    – dzieciou
    Commented Nov 13, 2012 at 22:30

5 Answers 5

Reset to default
9

I don't think fairness is the issue. More important: Would a report be useful?

For the web service usage, the usefulness of your report depends partly on the scope of your review. Given the code you reviewed, you have a concern. But perhaps other code (outside the scope of your review) prevents the problematic scenarios you're concerned about.

Unfortunately, at this point we don't know whether the problem is a problem, and we won't know until someone investigates further. But everyone who could investigate is attending to other priorities. This is a common issue with bug reports: Who is responsible for investigating? You want the developers to do it; they want you to do it.

What you're learning is that this bug report is not sufficient to inspire others to investigate. At this point I'm inclined to invoke The Responsibility Razor: If I want something to happen, I'm responsible to make it happen. You'll have to either investigate yourself, or do something more to persuade others to investigate.

It's possible that your investigation need not involve running a test. Perhaps walking through the code with some specific values will suffice. Whether this will be persuasive depends on the plausibility of the values you choose, and various people's assumptions about what the web service will do. But I'd start there.

If you don't have time even for that, that's a sign of how important this is for you. So make your report, and let it go.

For the API, the usefulness of a bug report depends partly on the API's audience. If the API is used only internally, and only by a relatively small group of people, I would leave it up to the internal users to decide whether it meets their needs, and to change it if they wish. If the API is used by people outside your team, perhaps you can gather information about their use of the API. If it causes trouble, it's a bug.

Improve this answer
1
  • All answers are helpful, but this one names the problem the problem precisely "Would a report be useful?" and explains that someone need to find time and investigate whether it really is problem.
    – dzieciou
    Commented Nov 15, 2012 at 6:44
10

Yes, it is fair to report bugs, no matter how you have detected them.

You may find that the developers need the steps to reproduce the problem, in which case you may have to do more work before you can expect the bug to be fixed.

A bug found by reading the code is still a bug. (Unless it's not actually a bug because you misread the code, or didn't understand it - but that's a different issue.)

Consider the counter-argument - what if you don't report the bug you know to be there? What could be the consequences? This is not exactly the same scenario, but gets to a similar point: Non-reproducible Bugs.

Improve this answer
5

I think you should ask the developers and business owners rather than us. :-) Ask them how to handle these kinds of border cases. Do they want bug reports also on unclear things to remind them or do they want to talk about them first and decide case by case whether to file a bug? Remember it's your task to provide them information on what is or could be wrong, but in the end, it's their decision what to do with the information.

One quite nice way of handling the API quality kind of issues is to have a separate category for things that should be improved, but won't be done just now. At least if someone at some point comes back and does something about them. If nothing is done, it's actually worse to have this kind of list than just bug/no bug decision.

Improve this answer
2
  • Thanks, Edu. In your practice, where you put such a new category for things to improve? In bug tracking system? Or you have some separate system to track technical debt? One options I see could be tools for collaborative code review that let you annotate the code...
    – dzieciou
    Commented Nov 13, 2012 at 19:55
  • All those could work fine. Bug tracking system would be probably easy place, if you can enter them in a way that doesn't bother everyone every day.
    – Edu
    Commented Nov 13, 2012 at 20:05
4

From your explanation two things I can infer

  • Code review shows parameters not handled / passed
  • No time to test it

My Questions

  • I would challenge why you do not have time to test it. You can check for Free tools like SOAP UI which can help to test the methods directly without writing a single line of code

  • Secondly reporting an issue without actually validating it is not the right practice in my opinion. I would strongly suggest reproducing an issue before filing a bug on it

Your goodwill can help developer during Unit Testing, Code review sessions. A quality conscious developer would definitely consider your inputs during code review sessions

Improve this answer
1
  • SOAP UI is just a part of the game (on test driver part). The system is complex and relies on external components, so setting up environment just for this single test is expensive. Also, since the request is forwarded by many subsystems, I would need to verify it in the very last one. But you're right, I may challenge myself how to setup whole environment for tests faster/easier.
    – dzieciou
    Commented Nov 13, 2012 at 19:46
2

Is it fair? I suppose that depends on the context - who you ask and what their definition of fair is.

I think an important consideration of opening bugs without evidence of a problem is that it could reduce your credibility in the eyes of the bug report readers.

Bug Reports are the primary work product for testers. They are the tangible work most seen by others (programmers, analysts, managers, etc.) and because of this its important, when we write bug reports, to make sure they reflect something we want others seeing. That they are well written and contain the appropriate elements to get the bug fixed. If we write bad reports we lose credibility. If we write a bunch of good reports we might gain credibility.

If you don't have time to test something to prove (evidence) that its a bug (as opposed to an issue, enhancement, etc.) you can always make a note of it and return to it later. If you open it in your bug tracker you might be able to find a way to keep it assigned to yourself until you have time to verify / find evidence of the problem.

If you have questions about an aspect of quality (is this a bug?) but are unsure you can ask in a more informal way (assuming opening in a bug tracker is formal) - in person, email, instant message, whatever. When you have evidence that it is a bug is when I generally recommend writing it up formally. Then again this all depends on the organization.

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.