27

We sometimes get requests from developers to re-run a failing test on the latest build, since "we did a lot of changes" and they hope the bug is gone.

On one hand, one can claim that they should know what they are doing and not hope that some side-effect removed a bug. It also projects lack of respect for our time, if they can run the test easily on their system. On the other hand, if they did do a lot of changes, then it's actually possible they fixed the bug.

What should be the response?

While at it: if the bug is indeed gone, how should it be closed? (we use: "Rejected" with a Reason field = "no longer seen").

Any other approaches?

  • 2
    Just a hint: Do you use e.g. deployment version or release version in your bug description? Because this is required. Afterwards then I wouldn't use the term "Rejected" - since this bug was actually available in version x.x. I would hence use "Retested" (because you do a retest afterwards for checking that the bug is not available anymore) and then you can close it as "Retested". This also leads to a better reputation for your test team instead of "Rejected" – Daniel Boehm Nov 12 at 7:30
  • 11
    How is the bug rejected ?! The bug was seen and was reproducible with version 1, so in version 2 if the bug is no longer seen then it should be marked as retested and verified to be fixed. – PDHide Nov 12 at 22:15
  • 5
    I think this question needs a lot of clarifications. Are you just complaining that they aren't reporting specific bug fixes, but rather make a lot of changes and ask for a blanket retest? Are you complaining that they have the ability to test, yet ask the testers to do it instead? – Mars Nov 13 at 6:08
  • 12
    Re: your assertion the devs have a lack of respect for your time: it's far easier for someone who's already seen what the bug looks like and has reproduced it to test to see if the bug is gone than it is for someone else, working from just a description of the bug, to try to test to see if the bug is gone. In the latter case, the developer can't know if the bug is fixed or if they just aren't doing the right thing to reproduce it.Thus, even if they try to reproduce it, and fail, they still have to ask you to verify that it's gone. Thus, it's not unreasonable for them to ask you to test. – Makyen Nov 13 at 9:42
  • 5
    Sometimes you van classify such defect as fixed upstream. Maybe there was a big refactorization done in poor code area and a lot of defects were fixed. If you think they waste your time go and automate test execution - it is the easiest way to make developers run tests. – Asia Wawrzyczek Nov 13 at 19:37
28

As a developer, not a QAist, I have a responsibility to provide software that meets the specs, with as few errors as possible. From my perspective, QA has the responsibility to inform me of any errors that I have made. I place them in extremely high regard for that.

When QA tells me that "they found a bug" they found that bug in a particular commit. The code was different before that commit, and the code is different now. Often, bugs are found in developed features, which will change considerably within just a few commits. It is very, very likely that the code they tested against last week has changed, even if just the line number for error messages has changed.

When we ask you to retest, it is not because we value your time less than ours. It is because the code no longer exists in that file line, or that file, or it is no longer coupled to that class, or the business requirement have made that feature obsolete, or the Python version was updated, or a million other things. Code under development is dynamic, and any bug filed against a specific slice of the development process is likely outdated after just a few more slices to that part of the application.

  • 3
    I would hope that my answer didn't come across as bikeshedding. I was looking more towards suggesting potential reasons for the situation the OP described, without blaming devs or testers. I can say that most of the time in situations like the OP described, there's no ill-will on the dev team side of affairs - it's often a case of it's a better use of resources for the tester to check than for the dev to do it. – Kate Paulk Nov 13 at 12:45
  • 1
    @KatePaulk: Your answer was fine! – dotancohen Nov 13 at 13:24
  • 2
    While I like this answer as a general Hanlon's Razor type answer, it doesn't actually answer either of the two questions being posed... – Mars Nov 14 at 0:40
  • 5
    Also, OP knows that the code is different now. OP's complaint is that dev isn't reporting a specific fix, they just changed (possibly unrelated) things and said "check again please!". At least, that's my interpretation of OP's situation. – Mars Nov 14 at 0:42
  • 6
    @Mars: right. That's what it is. If Dev can give us a good explanation why they think the bug is gone, then I accept it. It's when it looks like clutching for straws that I get frustrated. – user2978183 Nov 14 at 11:05
19

I see several possible problems here:

  • There are reported bugs that the test team can apparently reproduce but the development team has trouble finding/fixing. This suggests that there is insufficient information in the bug report. In situations like this, it can be helpful to provide an exact set of reproduction instructions. If you are testing a standalone application, this can include providing config files, database backups, and even local data files that can be used. For a web application, the test server and if needed the user login can be given.
  • The system is apparently sufficiently complex that side effects of changes can remove other issues. In my experience when a developer says "the changes we've been making here could have fixed defect X" I need to do a decent amount of regression testing around the area that's been changed, and around any features that connect to the area that's been changed.
    • Ideally I'd have automated regression running at minimum on a nightly basis (preferably running as part of continuous integration) to cover this, but life is not always ideal.
  • Your developers are not sure enough of their work to simply tell you that they have fixed something as part of another work item (or possibly this is not covered in the regular process).
    • Alternatively, your developers trust your team's evaluation more than they trust their own.
    • If any of these are the case, you may want to work more closely with your dev team to build up confidence and trust on both sides.

Regardless of the problem that's leading to the situation, you definitely should recheck the bug.

What I generally do if changes for other work fix a bug is to set it to whatever status the tool uses for "fixed" status, and make a note that changes for another work item fixed the bug. If I know the work item in question, I'll list it so that it's known that defect X was fixed by changes for user story Y.

Then I'll see if there is anything I can do to mitigate whatever underlying problem led to the situation - sometimes there is nothing to do. Sometimes there isn't a problem: it's simply an old bug that got refactored out during other work.

  • 10
    From the dev side, it can be a hassle to get a clean environment set up. I think the first bullet point may come from a place of "You already have the environment and can test in 5 minutes whereas it might take me a few hours just to set aside my dev environment and setup your test environment" – Mars Nov 13 at 5:57
  • 2
    @Mars, you are absolutely correct. Most often problems like the OP describes come down to communication misfires between test and dev. If the test team isn't aware that the devs need more time to set up their environment than the testers do, they'll have a different response to being asked to check something than if they do know. – Kate Paulk Nov 13 at 12:28
  • 3
    @Mars Not to mention that if the dev finds out the bug doesn't seem to appear in the new version, the QA will need to retest anyway. – Luaan Nov 13 at 14:12
  • 3
    A variation on #1: Another possible cause is that the test failure was reported on some setup that the dev team doesn't even have. Test teams generally have much larger budgets for buying a wide variety of hardware to test with. A dev might not be confident that the original problem is possible to reproduce on their limited setup, so they're asking the test team to re-run it on theirs. – bta Nov 14 at 0:54
  • 1
    @bta - even when hardware configuration isn't involved, test teams often have many more software configuration setups that aren't available - or easily available - to dev teams. When I was testing desktop applications, I used to maintain a set of configurations to cover several major customer configurations, and as many of the payment gateway configurations as I could. Some of them involved several hours of setup - the test team used to joke about tests being 2 hours setup, 5 minutes test time. – Kate Paulk Nov 14 at 12:20
7

Most issue trackers have a closed reason of "obsolete". This kind of situation is exactly what that status is for: the bug was reproducible, but no longer is, and you don't know exactly when or how it was fixed. This shouldn't happen often, but you can expect it to occur when competent developers are cleaning up a project that is poorly structured. Tight coupling causes changes to have unexpected side effects in distant parts of the system, and it's not always worth the time to fully understand how those side effects occurred when the end goal is to decouple everything and eliminate them.

(Shout out to Google's Android developers who abuse the "obsolete" close reason to mean "we ignored this issue for five years" or "the person who reported this bug got fed up and started developing for another platform.")

5

James Bach says,

"Testing is the process of evaluating a product by learning about it through exploration and experimentation, which includes some degree: questioning, study, modeling, observation, inference, etc."

I would use "re-run a ... test" as a synonym to "looking to learn something in a new context".

"if they did do a lot of changes"

This implies that the context has changed considerably; the chances of finding some new interesting information are higher, marginally. Therefore, I may be interesting to explore some particular area again.

Now, for two points you've raised that I think are important for you in prioritizing/optimizing this exploration:

they hope the bug is gone

It seems to imply that the developers did not understand the problem and its consequences to the system in general. The root cause of this may be a lack of some crucial information that your previous exploration didn't uncover (or wasn't communicated) or due to the absence of a deeper investigation of these consequences. It would go from case to case, bug to bug. Either way, it's important to reflect on this root cause for improving team communication and effectiveness.

It also projects lack of respect for our time, if they can run the test easily on their system.

If that's the case, you have a deeper problem of team or company culture. Any analysis would be lenghty and not tackle your particular case anyway, but the problem needs to be addressed in order to allow any sort of continuous improvement. It's also good to remember that there is another side in this coin (the developer), so, there may be the case they feel your testing feedback (bug reports e.g.) are not prepared in such a way that allows them to rapidly tackle the problems. Again, team reflection and experimentation it's necessary so you all can uncover how to work better.

In summary:

  • Ask about why they think the fix the problem;
  • Ask how they understood the problem and identified the problem;
  • Have a conversation about how to optimize the bug identification to fix the confirmation process, on the details, not only regarding documents (bug reports, tickets, etc) handovers.
5

Although the question portraits it as a problem, and the answers imply to immature processes or procedures, my experience is somewhat different.

For some bugs it's simply difficult to come up with consistent "steps to reproduce", and the the few occasions where the bug was observed required complex execution environments, special conditions or simply "a tester's touch" (remember, you are still the team's expert on testing)

lack of respect for our time

It's not your time, it's the team's time. If we assume a mature team following semi-decent development procedures then such a decision is taken by the team, or team leader, based on work load and capabilities.

"we did a lot of changes" and they hope the bug is gone.

As a developer I must admit that this is simply reality, you can waste invest time investigating complex bugs in complex systems but if you are not in a life critical devices businesses (aerospace, medical or ammunition) then sometimes it is more efficient to simply retest while remembering the associated risks- for example if the bug was a major data loss on a enterprise storage system you should do some investigation nevertheless.

There might be another benefit for retesting, it could be a hint for understanding the original problem for example by comparing logs or behaviors.

What should be the response?

Understand why are you asked to retest and continue from there, possibly with a followup question.

how should it be closed? (we use: "Rejected" with a Reason field = "no longer seen").

Sounds good it explains the actual reason for closing it, considering adding some free text explaining what was exactly retested and on what version and environment. Just be careful if someone ever tries to collect statistics about bugs, they will see a lot of rejected ones.

Final Disclaimer- my answer is relevant mainly to complex systems and non-trivial bugs

  • 2
    For an example of the "we did a lot of changes" situation, the software I'm working on has a number of known issues in the window-management code (in short, it was originally designed around the needs of a single-window application running on Windows 3.1 or OS/2). In the next version, that entire chunk of code is being replaced with something better suited for our needs, so rather than trying to understand each bug individually, we're simply going to re-test them all and confirm they're gone. – Mark Nov 15 at 2:40
4

Programming is complex. Stop for a minute and imagine how complex it is. No, it's ten times as complex as that. No, you just underestimated it again, it's another ten times more complex than that! Every feature adds complexity and can cause unforeseen interactions and side-effects, some of which are bugs (some are happy accidents).

Why would a programmer "hope" a bug has been fixed? Because they don't have the time or the domain knowledge to be more certain. Any programmer who tells you there is a 100% chance that a particular bug is fixed is lying. There is always a degree of uncertainty, and a good programmer will be honest about this uncertainty. This doesn't mean that they're not doing their jobs properly! Their manager might demand a quick turnaround on bug fixing, so they can work on "higher priority" stuff. They might need more time to become familiar with the software or the domain, anywhere from hours to months.

Some production bugs are really hard to reproduce in a development or testing environment. In this case, you (the programmers and/or testers) either spend more time investigating, or "hope" that you can fix the production error without reproducing it. There needs to be a clear sense of ROI: Return On Investment (of time).

Who should spend the time understanding the bug? It's the testers' responsibility to provide clear steps to reproduce the bug. If it's an intermittent bug, they should say how often it occurs. It is not the testers' responsibility to diagnose the cause or pinpoint a line of code causing the bug, although a comment like "this only seems to happen between 11:00 am and midday" can very useful if it helps narrow down where the root cause of the bug lies. I have found that testers often get this root cause analysis wrong, because it's not the part of the project they work with every day. This is not to say that the testers should have no input into the programmers' work, or that programmers should have no opinion on UI, etc., only that they aren't the ones who have the responsibility to make decisions in other people's areas, they can only advise and their advice can be overruled by other concerns. Testers shouldn't try to be expert programmers and programmers shouldn't try to be expert testers, or decide which bugs are "important" and which aren't. Both programmers and testers should work together since they have a common goal: releasing a product that's fit for purpose.

Once the conditions to trigger the bug are known, it's the programmers' responsibility to track down the root cause, fix it, make the fix ready for testing and notify the testers. There is a clear line of division of responsibility. Programmers are responsible for writing maintainable code that is fit for purpose. Testers are responsible for verifying the programmers' code, including exploring edge cases that the programmers haven't thought of. The whole team (including programmers and testers) is responsible for delivering a working product.

What the programmers shouldn't do is make a token change without trying to understand the root cause and "throw it over the fence" for the testers to verify. That is indeed disrespectful. The programmers and testers need to be equally respectful of each other's time. Management plays a part in this too. Some teams are in a high pressure environment where they try to blame others for their lack of time when it's their managers setting unrealistic expectations. It's easy to judge someone else's behaviour without trying to understand it first. I always try to make sure everybody in a discussion is on the same page before trying to move forward.

Importantly, every change, big and small needs to be tested, because you have no idea what it could have accidentally broken without fully (re)testing it. Hopefully, you have automated tests that do the bulk of your testing. Programmers should have a suite of unit tests, QA should have a suite of acceptance tests. Each step should be sanity checked as well. New bugs should have tests written for them and included in the testing suite.

Lastly, if the bug is gone, it shouldn't be rejected. The programmers made some changes to make the bug disappear, therefore the bug is "fixed" not "rejected". A bug should only be rejected if it turns out it was incorrectly reported, or can't be reproduced in any environment, including production. Sometimes bug can be closed as "won't fix", i.e. it's either not considered a bug or it's not considered worth fixing.

  • "Any programmer who tells you there is a 100% chance that a particular bug is fixed is lying" No, there are bugs you can be certain of, and others you can't. It depends on the complexity & how reproducible the bug was. If it reproduced 100% of the time and now it doesn't happen in multiple attempts, I know I have fixed it. – Dragonel Nov 14 at 21:51
  • @Dragonel I'll give you an example. I fixed a simple bug and proved it in one environment. I merged the code without conflicts, tagged it, and deployed the tagged commit to another environment. The bug was not fixed. It took a while to discover I'd accidentally tagged the pre-merge commit instead of the post-merge commit. Human error can occur at any time, even on simple tasks. – CJ Dennis Nov 14 at 22:03
  • The bug was fixed - you had a secondary issue that the deployment was incorrect, but that doesn't mean you lied about fixing it. – Dragonel Nov 17 at 18:39
3

What I can see from your case is a poor coding discipline of your dev team. I am not sure what methodology you are using for your product development but I would talk to a person who is responsible for both your teams (qa and dev) results.

If you have a sort of efficiency indicators (for example some use defect rejection ratio as the measure of how effective QA team works) I would propose to the management to introduce some kind of "fix rejection by QA" metric for dev guys. Any "hope the bug is gone" verified and reverted back to dev would negatively impact dev score.

  • 1
    The metric for QA is the ratio of the number of bugs they prevent going to production to the number of bugs reported by customers (higher is better). The metric for developers is how many bugs are found (by both QA or customers) per change (lower is better). – CJ Dennis Nov 14 at 22:52
1

if the bug is indeed gone, how should it be closed? (we use: "Rejected" with a Reason field = "no longer seen").

Honest to goodness, do you not have a "Not Reproducible" Close status?

  • 1
    This. Closed: Fix (#Commit Num) and Closed: Fix (No longer reproducible) – Mars Nov 15 at 0:17
  • 2
    I'd make a distinction between "Not reproducible" and "No longer reproducible". Not reproducible means Testing did something weird and Dev can't confirm the bug – Mars Nov 15 at 0:18
  • Of course we have "not reproducible"; but as @Mars indicated, it's something else. It means the bug is not reproducible on the same setup and version that was reported. My question is about cases where the bug is gone when trying to reproduce on a new build. – user2978183 Nov 15 at 12:09
1

The point of testing is to provide feedback

Generally, the quicker the feedback the better (shift-left)

Pair with your developers until you get to the point that they can run the tests.

Separating development and testing is anti-agile in my experience.

Development and QA involvement is a two-way street at the end of the day. dev needs to make code testable and QA needs to make testing easily runnable. More usable. More valuable.

However (to be clear) this does NOT mean eliminate QA - as has happened at many companies. I still believe in a tester mindset and quality focus and for QA/QE folks to meet that needs. When I personally switch (within 1 day) from dev to testing I need to mentally 'put on a different hat (persona)' and start focus on breaking stuff instead of focusing on proving that it works.

A more traditional approach is that QA/QE carves out its own space. It's own fiefdom. Which it then defends rigorously. This is often not in the best company interest, but more in the interest of current individuals who are employed there.

At the end of the day if people are making changes can break tests but don't know about the tests breaking...

Respond to requests to test by enabling the requestor to run the tests

Put another way, 'hope' is not a strategy for future and constant improvement

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.