How to respond to requests to retest, in hope that the bug is gone?

Question

We sometimes get requests from developers to re-run a failing test on the latest build, since "we did a lot of changes" and they hope the bug is gone.

On one hand, one can claim that they should know what they are doing and not hope that some side-effect removed a bug. It also projects lack of respect for our time, if they can run the test easily on their system. On the other hand, if they did do a lot of changes, then it's actually possible they fixed the bug.

What should be the response?

While at it: if the bug is indeed gone, how should it be closed? (we use: "Rejected" with a Reason field = "no longer seen").

Any other approaches?

Answer 1

As a developer, not a QAist, I have a responsibility to provide software that meets the specs, with as few errors as possible. From my perspective, QA has the responsibility to inform me of any errors that I have made. I place them in extremely high regard for that.

When QA tells me that "they found a bug" they found that bug in a particular commit. The code was different before that commit, and the code is different now. Often, bugs are found in developed features, which will change considerably within just a few commits. It is very, very likely that the code they tested against last week has changed, even if just the line number for error messages has changed.

When we ask you to retest, it is not because we value your time less than ours. It is because the code no longer exists in that file line, or that file, or it is no longer coupled to that class, or the business requirement have made that feature obsolete, or the Python version was updated, or a million other things. Code under development is dynamic, and any bug filed against a specific slice of the development process is likely outdated after just a few more slices to that part of the application.

Answer 2

I see several possible problems here:

• There are reported bugs that the test team can apparently reproduce but the development team has trouble finding/fixing. This suggests that there is insufficient information in the bug report. In situations like this, it can be helpful to provide an exact set of reproduction instructions. If you are testing a standalone application, this can include providing config files, database backups, and even local data files that can be used. For a web application, the test server and if needed the user login can be given.
• The system is apparently sufficiently complex that side effects of changes can remove other issues. In my experience when a developer says "the changes we've been making here could have fixed defect X" I need to do a decent amount of regression testing around the area that's been changed, and around any features that connect to the area that's been changed.
  • Ideally I'd have automated regression running at minimum on a nightly basis (preferably running as part of continuous integration) to cover this, but life is not always ideal.
• Your developers are not sure enough of their work to simply tell you that they have fixed something as part of another work item (or possibly this is not covered in the regular process).
  • Alternatively, your developers trust your team's evaluation more than they trust their own.
  • If any of these are the case, you may want to work more closely with your dev team to build up confidence and trust on both sides.

Regardless of the problem that's leading to the situation, you definitely should recheck the bug.

What I generally do if changes for other work fix a bug is to set it to whatever status the tool uses for "fixed" status, and make a note that changes for another work item fixed the bug. If I know the work item in question, I'll list it so that it's known that defect X was fixed by changes for user story Y.

Then I'll see if there is anything I can do to mitigate whatever underlying problem led to the situation - sometimes there is nothing to do. Sometimes there isn't a problem: it's simply an old bug that got refactored out during other work.

Answer 3

Most issue trackers have a closed reason of "obsolete". This kind of situation is exactly what that status is for: the bug was reproducible, but no longer is, and you don't know exactly when or how it was fixed. This shouldn't happen often, but you can expect it to occur when competent developers are cleaning up a project that is poorly structured. Tight coupling causes changes to have unexpected side effects in distant parts of the system, and it's not always worth the time to fully understand how those side effects occurred when the end goal is to decouple everything and eliminate them.

(Shout out to Google's Android developers who abuse the "obsolete" close reason to mean "we ignored this issue for five years" or "the person who reported this bug got fed up and started developing for another platform.")

Answer 4

James Bach says,

"Testing is the process of evaluating a product by learning about it through exploration and experimentation, which includes some degree: questioning, study, modeling, observation, inference, etc."

I would use "re-run a ... test" as a synonym to "looking to learn something in a new context".

"if they did do a lot of changes"

This implies that the context has changed considerably; the chances of finding some new interesting information are higher, marginally. Therefore, I may be interesting to explore some particular area again.

Now, for two points you've raised that I think are important for you in prioritizing/optimizing this exploration:

they hope the bug is gone

It seems to imply that the developers did not understand the problem and its consequences to the system in general. The root cause of this may be a lack of some crucial information that your previous exploration didn't uncover (or wasn't communicated) or due to the absence of a deeper investigation of these consequences. It would go from case to case, bug to bug. Either way, it's important to reflect on this root cause for improving team communication and effectiveness.

It also projects lack of respect for our time, if they can run the test easily on their system.

If that's the case, you have a deeper problem of team or company culture. Any analysis would be lenghty and not tackle your particular case anyway, but the problem needs to be addressed in order to allow any sort of continuous improvement. It's also good to remember that there is another side in this coin (the developer), so, there may be the case they feel your testing feedback (bug reports e.g.) are not prepared in such a way that allows them to rapidly tackle the problems. Again, team reflection and experimentation it's necessary so you all can uncover how to work better.

In summary:

• Ask about why they think the fix the problem;
• Ask how they understood the problem and identified the problem;
• Have a conversation about how to optimize the bug identification to fix the confirmation process, on the details, not only regarding documents (bug reports, tickets, etc) handovers.

Answer 5

Although the question portraits it as a problem, and the answers imply to immature processes or procedures, my experience is somewhat different.

For some bugs it's simply difficult to come up with consistent "steps to reproduce", and the the few occasions where the bug was observed required complex execution environments, special conditions or simply "a tester's touch" (remember, you are still the team's expert on testing)

lack of respect for our time

It's not your time, it's the team's time. If we assume a mature team following semi-decent development procedures then such a decision is taken by the team, or team leader, based on work load and capabilities.

"we did a lot of changes" and they hope the bug is gone.

As a developer I must admit that this is simply reality, you can waste invest time investigating complex bugs in complex systems but if you are not in a life critical devices businesses (aerospace, medical or ammunition) then sometimes it is more efficient to simply retest while remembering the associated risks- for example if the bug was a major data loss on a enterprise storage system you should do some investigation nevertheless.

There might be another benefit for retesting, it could be a hint for understanding the original problem for example by comparing logs or behaviors.

What should be the response?

Understand why are you asked to retest and continue from there, possibly with a followup question.

how should it be closed? (we use: "Rejected" with a Reason field = "no longer seen").

Sounds good it explains the actual reason for closing it, considering adding some free text explaining what was exactly retested and on what version and environment. Just be careful if someone ever tries to collect statistics about bugs, they will see a lot of rejected ones.

Final Disclaimer- my answer is relevant mainly to complex systems and non-trivial bugs

Answer 6

Programming is complex. Stop for a minute and imagine how complex it is. No, it's ten times as complex as that. No, you just underestimated it again, it's another ten times more complex than that! Every feature adds complexity and can cause unforeseen interactions and side-effects, some of which are bugs (some are happy accidents).

Why would a programmer "hope" a bug has been fixed? Because they don't have the time or the domain knowledge to be more certain. Any programmer who tells you there is a 100% chance that a particular bug is fixed is lying. There is always a degree of uncertainty, and a good programmer will be honest about this uncertainty. This doesn't mean that they're not doing their jobs properly! Their manager might demand a quick turnaround on bug fixing, so they can work on "higher priority" stuff. They might need more time to become familiar with the software or the domain, anywhere from hours to months.

Some production bugs are really hard to reproduce in a development or testing environment. In this case, you (the programmers and/or testers) either spend more time investigating, or "hope" that you can fix the production error without reproducing it. There needs to be a clear sense of ROI: Return On Investment (of time).

Who should spend the time understanding the bug? It's the testers' responsibility to provide clear steps to reproduce the bug. If it's an intermittent bug, they should say how often it occurs. It is not the testers' responsibility to diagnose the cause or pinpoint a line of code causing the bug, although a comment like "this only seems to happen between 11:00 am and midday" can very useful if it helps narrow down where the root cause of the bug lies. I have found that testers often get this root cause analysis wrong, because it's not the part of the project they work with every day. This is not to say that the testers should have no input into the programmers' work, or that programmers should have no opinion on UI, etc., only that they aren't the ones who have the responsibility to make decisions in other people's areas, they can only advise and their advice can be overruled by other concerns. Testers shouldn't try to be expert programmers and programmers shouldn't try to be expert testers, or decide which bugs are "important" and which aren't. Both programmers and testers should work together since they have a common goal: releasing a product that's fit for purpose.

Once the conditions to trigger the bug are known, it's the programmers' responsibility to track down the root cause, fix it, make the fix ready for testing and notify the testers. There is a clear line of division of responsibility. Programmers are responsible for writing maintainable code that is fit for purpose. Testers are responsible for verifying the programmers' code, including exploring edge cases that the programmers haven't thought of. The whole team (including programmers and testers) is responsible for delivering a working product.

What the programmers shouldn't do is make a token change without trying to understand the root cause and "throw it over the fence" for the testers to verify. That is indeed disrespectful. The programmers and testers need to be equally respectful of each other's time. Management plays a part in this too. Some teams are in a high pressure environment where they try to blame others for their lack of time when it's their managers setting unrealistic expectations. It's easy to judge someone else's behaviour without trying to understand it first. I always try to make sure everybody in a discussion is on the same page before trying to move forward.

Importantly, every change, big and small needs to be tested, because you have no idea what it could have accidentally broken without fully (re)testing it. Hopefully, you have automated tests that do the bulk of your testing. Programmers should have a suite of unit tests, QA should have a suite of acceptance tests. Each step should be sanity checked as well. New bugs should have tests written for them and included in the testing suite.

Lastly, if the bug is gone, it shouldn't be rejected. The programmers made some changes to make the bug disappear, therefore the bug is "fixed" not "rejected". A bug should only be rejected if it turns out it was incorrectly reported, or can't be reproduced in any environment, including production. Sometimes bug can be closed as "won't fix", i.e. it's either not considered a bug or it's not considered worth fixing.

Answer 7

What I can see from your case is a poor coding discipline of your dev team. I am not sure what methodology you are using for your product development but I would talk to a person who is responsible for both your teams (qa and dev) results.

If you have a sort of efficiency indicators (for example some use defect rejection ratio as the measure of how effective QA team works) I would propose to the management to introduce some kind of "fix rejection by QA" metric for dev guys. Any "hope the bug is gone" verified and reverted back to dev would negatively impact dev score.

Answer 8

The point of testing is to provide feedback

Generally, the quicker the feedback the better (shift-left)

Pair with your developers until you get to the point that they can run the tests.

Separating development and testing is anti-agile in my experience.

Development and QA involvement is a two-way street at the end of the day. dev needs to make code testable and QA needs to make testing easily runnable. More usable. More valuable.

However (to be clear) this does NOT mean eliminate QA - as has happened at many companies. I still believe in a tester mindset and quality focus and for QA/QE folks to meet that needs. When I personally switch (within 1 day) from dev to testing I need to mentally 'put on a different hat (persona)' and start focus on breaking stuff instead of focusing on proving that it works.

A more traditional approach is that QA/QE carves out its own space. It's own fiefdom. Which it then defends rigorously. This is often not in the best company interest, but more in the interest of current individuals who are employed there.

At the end of the day if people are making changes can break tests but don't know about the tests breaking...

Respond to requests to test by enabling the requestor to run the tests

Put another way, 'hope' is not a strategy for future and constant improvement

Answer 9

if the bug is indeed gone, how should it be closed? (we use: "Rejected" with a Reason field = "no longer seen").

Honest to goodness, do you not have a "Not Reproducible" Close status?