Asking a developer to be explicit about how a bug was fixed

Question

Whenever our developer fixes a bug, he or she just posts "Fixed in version x.x.x.x, please test".

We, the QA team, have requested developers to be as detailed as possible about what caused a problem and how he or she fixed it, but they still keep doing it (we don't want to think they are lazy or that this is not our business, of course). It does not always make sense for a developer to do it, but we think that knowing the strategy behind a fix, what caused it, can really be beneficial for the overall quality of the product and, we think, this is something that both teams have to contribute to.

This additional information, for instance, may help a tester to find similar issues in other parts of the application or can provide ideas for exploratory testing of a fix.

How can we convince developers to explain a bugfix in a more detailed way?

Answer 1

I would operate under the assumption that your developers are documenting this somewhere, since I would also assume that they are good developers who care about good development practices. As a developer myself, I strongly believe that developers should be expected to describe what they've done in a language other than code. It doesn't have to be comprehensive, and I believe they should, if they don't already, care to take that time and effort to ensure others understand what they changed without having to read the code. As the saying goes "if you can't explain it simply, you still don't understand it", and that's a dangerous place to be if you're touching code.

This may be as simple as writing a good commit message, or if there's a code review involved, providing a short summary of what was changed and why along with the code review. The code review artifact would then be a useful tool for testers to reference.

I would also be surprised if, for all but the most trivial bugs (e.g. "The label for field X is spelled wrong"), a good developer didn't put any information about root cause in the bug report after investigating and finding the bug, perhaps along with an explanation of the correct fix. (If there's a change control board involved, this is often necessary to justify fixing the bug in the first place--often it's worth describing several possible fixes based on the rough effort for each.)

If your product has written design documents, I would also expect that some bugs would require updates to the design, especially if something major had to change to eliminate a failure mode.

I would have to imagine they are capturing the desired information in one of the above (commit messages, code reviews, or design docs, though your question implies they are not putting it in the bug reports). It may be as simple as doing some spelunking of their other artifacts to find this information, or talking to the developers see where they're putting such information.

Personally, I would go to them under the assumption that they are putting this information somewhere other than their heads, and just ask where you can find it. If your developers truly aren't leaving any breadcrumbs about what they're changing other than the code itself, I believe they are poor developers. It's cowboy coding. In that case, then it seems some introductions to basic development practices are in order, but it will be hard to drive that unless they want to be good developers.

Answer 2

Request a meeting between upper management of QA and DEV and explain the goals of what you are asking. I think the description you provided in the question is a good start at explaining the why of the ask.

This additional information, for instance, may help a tester to find similar issues in other parts of the application or can provide ideas for exploratory testing of a fix.

QA wants to better understand the fixes for bugs so that we can:

1. Better test the original issue
2. Better discern and test the other areas that might be affected
3. Better understand the systems in play
4. Be more efficient in our testing

There's always two sides... what the dev intended to fix and what actually changed. Ideally those two things are the same but QA wouldn't have a job if it was. That's not to say that DEVs are bad or dumb. Many of the systems are very complicated and we're trying to move at a brisk pace. Mistakes are made and DEV and QA need to work together to increase the number of defects found and reduce the number of defects that escape.

Sometimes having to explain something to someone else makes you think of things you didn't think of before. Explaining it to QA educates them more about the various systems in play and how they interact which makes them better testers.

Overall, the goal is to better educate QA on the fix so they can test more effectively and efficiently and detect more bugs. I think everyone would get behind that. The problem, I think, comes with the phrasing of the ask.

to be as detailed as possible about what caused a problem and how he or she fixed it

I think in some/many minds that is asking for a lot of "extra work" to be put into each bug. I would ask for a brief description in the bug. If QA looks at the brief description and needs more info or has more questions, they contact the dev and get that info. I think that's a good compromise between too little and too much info in every bug.

If I were to phrase it, it would be something like

Please briefly explain:

1. What the fix was
2. What other areas were affected by the fix
3. What other areas might be affected by the fix
4. Any special guidance for testing? Are there any concerns over the fix? Special areas of focus for testing?

I don't think this is a big ask of DEV. I think it's something they work into their process. When they finish fixing a bug, they look back at the files they changed and summarize the fixes into answers to the 4 questions above.

I think after v1.0 of what gets put into each defect, you meet back a few months after it's implemented and review. Are DEVs doing this consistently? Is it too much work? How can we minimize the documentation DEVs need to enter vs how much info QA needs to do their job well.

I DON'T think the answer is to have QA look at the code changes themselves. You are then assuming that QA knows all aspects of these complicated systems that DEV works in daily and I don't think that's a reasonable ask.

Answer 3

Most other answers suggest that requirement to be better in testing is understanding the code - but that requires skills what most QA testers do not possess. Deep understanding the code is not required. In most cases, code changes are way too complicated and obscure for QA to be able to glimpse anything relevant from just reading the code in detail.

What IS relevant for QA is to be able (but not required) to browse through the code changes not to understand the changes, but to see what parts of the system were affected by the changes (which webpages, libraries handling which functions, etc), which are good candidates for testing. This would be only as addition to the description of changes/test in bug ticket.

In my own experience, while I was able in few cases to read the code (as a former developer for our system) and build a test case to fail the fix and prove developer that fix is wrong, it is very rare situation.

Your developers should already use a tool for code review, so use it too (if you feel description of changes in ticket is not sufficient) - and if they are not, code review is the cheapest way to improve the quality of the code.

Not sure how it is in your organization, but in ours, every bug has a champion ("internal customer") who works with other experts to write how exactly to test the changes (in addition to unit tests added by developers). It is this "bug champion" responsibility to identify what and how should be tested (not QA's responsibility). This information is preserved and sometimes used in lieu of documentation to better understand system behavior in specific corner cases.

Remember QA is there not to "assure" the quality after the fact, but to maintain the consistent process and communication between all involved parties to design the quality into the product, and provide information about the status of the quality.

Responsibility of QA is to make sure that agreed-upon process was followed (code was reviewed by senior developers, unit tests were written and passed, regression test passed, developer and champion determined what should be tested), and requested test was performed as described. Task of QA is not to replace all that skills and knowledge what developers and problem are expert have. Task of QA is to make sure all relevant parties participated in the process, not to solve problems for those parties.

Quality cannot be bought, but has to be paid for.

Answer 4

It seems that this question has two answers depending on the qualities of your QA team.

If your QA team can read code, they can look at the commit(s) where the bugs are fixed - if you use a version control system. In this way your team can analyse the changes themselves, meaning you don't need any information or time from someone else.

If your QA team cannot read code (or at least only a minor percentage of the whole), you should keep asking the developers to provide more information so that you can spot the bugs earlier from now on. It will help if you keep giving good arguments for the developers to provide you with more information, as it could feel for them as wasting time because the bug is already fixed. Possible arguments are:

• Improving regression testing. If you know a bug is caused by f.e. a for-loop not being executed for the last element, you will add more test cases to your regression test that cover that.
• Improve automated testing. The same argumentation holds as above.
• It will help you spot these bugs earlier the next them, resulting in the bugs being fixed sooner the next time, meaning the product can be released sooner.
• You can perform "white-box testing" next to "black-box testing". If you know what parts of the code has changed, you can use different testing techniques that are more applicable. It will take more time (most of the times) if you use black-box testing techniques, because you don't know what parts of the codebase have been changed. If you use white-box testing techniques, you can test the more precisely at those parts of the codebase that have been changed.

Of course, it could still be that the developers are not convinced and they do not provide the extra information the next time. But trying to get such a rhythm into the process will hopefully result in them doing it upfront (before you need to ask about it) in the future.

Answer 5

As a counterpoint to the many answers saying that testers can review the changes in source, I have worked in more than one place where testers have no access to source code. Their only source of information on what has changed is whatever information is in the ticket.

In this situation, developers absolutely must provide testers with information on what changes they made, and cover things like:

• What configuration settings the changes impact or apply to. Many bugs only occur under specific configuration settings, but I've seen first pass fixes cause issues with different configurations due to the assumptions made by the code.
• Any related parts of the application the developer is aware could be impacted. Experienced testers will have a general idea what modules in the software are interrelated, but there is no guarantee there will not be subtle dependencies the developer can see but the testers wouldn't know about.
• Any related issues the developer may have fixed or refactored as part of the bug fix: if there is an ongoing refactoring effort, this is crucial for testers to know so they can check the other changes as well.

It's also much faster for a developer to add a set of notes to a ticket with this information than it is for the tester to dig through the code to work out what might have been affected, simply because the developer can make those notes while working on the item.

In my opinion it comes down to all parties respecting each others time and effort. The developers I've worked best with have been the ones who give me the information I need to effectively target my testing and then work with me when I find things that don't match my expectations.

Answer 6

I'm a developer, and I do add additional information in our bug/feature tracking system on completing an item whenever I think it might be useful. I target this mainly to testers, but much of the same information might also be useful to another developer who comes across some code I touched and wants to know more (after reading source comments) about why it is that way.

For some simple fixes, there really is not much more to say. (Problem description: "The foobar icon is green, but per requirements it should be cyan." Resolution: "Changed the default color for the foobar icon to cyan.") But for maybe around half of all Defect fixes, something from my analysis and/or fixes is worth noting. After all, I'm the one who invested the most time in understanding the original issue, deciding exactly how to fix it, and analyzing whether my changes might potentially affect anything else.

I don't expect a tester to be able to understand code, but even for someone who does read code, a brief English explanation of the bug and fix can be quite helpful. Sometimes in complex code, the sequence of events involved could run through many different functions and stored data, and yet have a single line fix. Looking at that single line (and any added comment), you might be able to see why the change is "correct", but it wouldn't always be obvious how that relates to the observed bug.

So here are some questions you might try posing, to be more exact about what you need than just asking for "more information":

What conditions cause the bug?

This is slightly different than "How can the bug be reproduced?". A bug report will normally include at least one way to reproduce a bug, and possibly a few scenarios that seem to cause the same problem. But quite often when I discover the cause of a problem, I'll realize it's actually more general than the reported scenario(s), and/or would only happen depending on something else not described in the report. For example, a bug report might note the issue is reproduced when you "Do actions A and then B to widget C", but there's actually a problem whenever you "Do any particular-class-of-action A' and then action B to any widget with property D, while at least one E is active".

Or even better, I might start with an issue that seemed to happen frequently but not reliably, and come up with a way to reproduce it consistently.

This is critical information for a tester to properly test the breadth of the change. For example, if I described the original bug as above, a tester might want to also verify that the problem is fixed for a sampling of actions A' and widgets with property D, and that the system continues to work correctly in the same scenarios except where no E is active.

You might rephrase this question, "Are there other ways of reproducing the bug?". A subcategory question is, "Were there any existing configuration settings that would have affected the bug?"

Was anything besides the observed bug fixed?

Of course entirely unrelated changes should not be attached to the same ticket. But maybe two different features were using a common function that had an error that manifested in different ways in each feature. Or maybe the fix involved refactoring some code in a way that solved more than one issue identified during the work.

The test team should definitely be made aware if any positive "side effect" like that was noted, so they can test that out as well.

Which features exercise any of the changed, added, or removed code?

This is the flip side of the previous question. Sometimes a code change has happy side effects, but there's almost always a risk any code change might break something else. Developers should make reasonable efforts at analyzing changes to try to determine what else might be incorrectly impacted, but might miss something, or simply conclude that the remaining risk is acceptable.

When a few other features specifically use some of the same changed code, it may be worth keeping an eye on any automated tests for those features, and/or running a few simple regression tests for those features.

At one extreme, the answer may be "modified a core library function, so effects could be seen anywhere". (And maybe the modification is "obviously" correct, but some other code accidentally relied on the old incorrect behavior...) It's probably not feasible to run all the manual regression tests you have documented and/or can think of when this sort of thing happens, but at least you'll be aware of it as a possible issue.

Are there any non-user tools relevant to the bug or fix?

We developers like to create or find ways to get all sorts of information about what the software is doing that are not meant to be visible to the end user. This includes things like log output, execution traces, and means of inspecting internal data live.

Many of these things would need to be configured or otherwise activated, or might even require an external program. But they're often relatively easy to learn to use, just because the main purpose is to make things easier for us.

Only user-visible behavior truly defines whether software works or not, but sometimes peeking at the software internals in these ways can help verify more exactly that a fix or feature is working as intended. In particular, maybe in a certain scenario you would expect some variation in the output, but there seems to be an undesired pattern in that output. Since it may be impossible to tell from one try if anything's wrong, it may be more useful to investigate the steps in computing the output.

As a bonus, knowing a few of these sorts of tools for your software system can come in handy when writing a new bug report or failing a fix or feature. If you include some data from such a tool that appears, in your best estimation, to be highly relevant to the actual observed bug, the developer assigned to investigating it might appreciate it.

---

Hopefully explaining exactly what sort of information would be useful to you and why will convince developers to start attempting to provide it.

If developers still resist changing, you might want to discuss these same reasons for wanting more information with a manager, so expectations on the process can be established. This might be more or less formal, depending on your team's size and style.

Answer 7

You can't expect developers to repeat what they coded in natural language. It would take time and effort they don't care about. At best you will get two sentences which reveal hardly anything (usually - I have had cases where a one-line comment from a developer triggered a few alarm bells).

The good news is: there is a fairly easy manner to check fixes as a tester. (Of course, the fact that you're asking the question makes me somewhat fear you don't have this possibility). One caveat is that you must be able to read code.

Assuming that your team works with code versioning, every check-in by a developer (changesets in TFS, commits in GIT) gets a unique number. If the developer makes the link with the defect during check-in, you'll even see the link appear in the defect item as well. If that's not the case, you can always consult the commit history, where you will see check-ins related to fixes.

Once you know the number, you can simply open it and see the code changes that occurred. Looking at before/after comparisons in this way can really give you excellent insights as tester. It can also help you greatly in reducing and focusing testing effort, e.g. if you see a list of hard-coded values in code, you'll know if a certain value is missing before you even tried it via the GUI.

Another plus is that this approach takes no extra effort from developers. Of course, after looking at the changeset you can always go and ask a developer a more specific question which will get answered more readily than Tell me everything you did for this defect and how and why.

Answer 8

There are two questions here:

1) How do we convince developers that testers need to know what they've changed?

2) What's the best way for them to convey that information?

I'll focus on 2, as this is what we're doing. I may come back and rant about 1, though there are many good answers here.

For 2, I'm assuming, hopefully correctly, that there is some sort of code review being done, and some sort of function test. (If you're doing function test and don't understand what changed, that's a bigger problem then just this.) I'm assuming you're doing integration test/system test/something where you're using the system as an end-user was.

So what we've done is have everyone involved in the testing invited to code reviews, and they restructured the code review so that the first few minutes are devoted to an overview of what changed, and then the rest of the code review is diving into the code. So we can be there for the first few minutes, understand what was wrong, at a level we understand, and what changed. This both helps us understand the system a bit better, but also what we need to do to verify the fix, because sometimes, the external symptom and the internal cause are not easily related.

Answer 9

I do not think it is reasonable to ask the developers to always add a very detailed report.

Working software over comprehensive documentation

Are you sure you are always going to use the data? This just sounds as a documentation waste. I read it as we the QA want you to log all the information possible, because just in case we might want to use it sometime...

Do you really need it for re-testing? If yes:

• Ask the developer in person (or over chat) when you a really interested or if you are going to do something with the information. Explaining why you need it is better than just asking to supply it always. Good reasons are risks estimation and or doing a root-cause analysis.
• Do a root-cause analysis with the developer together. Now you can log the cause and actions to prevent similar issues in the future. Handover data like you are requesting loses 50% of the information even if the developer really tried to give everything.
• Link commits to bug-tracker-id. Most if not all bugfixes there are only a couple of lines changed. A tester should be able to scan for dependencies or risks for other systems now.

Sure I do understand why you want to information for further testing, but more data in a bug-tracker is not the solution. Sitting with the DEV team is. Talking to the DEV team is.

Answer 10

I understand where you are coming from, asking this question.

It is fairly important for us as testers to understand the fix. Not necessarily from a coding pov but to understand it as much as we know how to assess if the change will or will not be impactful on other areas of the application. We`ve had success in just talking to the developer and stating the above(some understand others are a bit more hard-headed, cannot win them all unfortunately)

On the other hand, there are means, we as testers, can understand if a certain fix on a specific area of the SUT will have potential to cause regressions on the code.

Some specific means we can inspect the changes by ourselves:

• Continuous integration means that we should have access to the commits/branches/deploys that the DEV team are doing, and we can see what the changes have been and what areas are affected
• Certain tools, like TFS and MTM scan the change made, and can suggest you a set of tests that cover the impacted area.

On how you should convince devs to share that, my answer would be that you are self sufficient enough to find out by yourselves. Do not rely on a DEV to babyfeed you stuff when you can find it out by yourself, and to me this feels a bit more satisfactory(and you can learn a bunch).

Answer 11

So the actual question was:

How can we convince developers to explain a bugfix in a more detailed way?

In my opinion the best way to convince somebody is not trying to convince, just put them in your shoes. For example, once in a while, before version release for example, you may ask the managers to get help in verifying closed bugs. Managers will assign developers to verify the bugs in order to help QA Team. When developers will try to reproduce and test bugs they will understand what details are missing in the ticket or in the commit message.

So my message here is,

Don't convince, let them join you and be in your shoes.

Answer 12

This is a communication problem.

As defined in the Agile Manifesto

Individuals over processes and tools

Most of the answers point to tools and code and process. I would push for:

regular in person conversations about the fix details so you understand the fix.

If you get too much push back about having the time to go into the details then as a team you need to have a talk about why you need to do that. This is the role of retrospectives in Agile environments. If, after attempts that are not leading to things improving, it's time for a discussion with management - but be careful: if they "fix" the problem you may have also just enabled command and control which is a big problem in most workplaces.

The ironic thing you will find over time is that as the QA person asks questions of the "fixer" it will start to turn up questions that the developer hadn't thought about... conditions they had not consider... and sometimes it will lead to a "oh yeah, let me address that too".

This is the path to quality

Answer 13

"The essence of strategy is choosing what not to do"- Michael Porter

What QA need here is insight into underlying pattern/s for bugs, not the nitty gritty details of Individual fixes. Abstract vs Concrete

There can be established a solid common ground( probably by a meeting) mutually beneficial for both parties involved by setting the right expectations of what kind of information is needed AND in which precision for exactly WHAT purpose.

This will help in NOT wasting effort from either side as there is much useful and precise information can be provided by developer with least effort(assuming he already has it by fixing the bug) for which tester will take considerable effort by reading the code(not justifiable in most cases) or in some cases not possible at all( testers cannot read code). Similarly what EXACTLY tester is looking for and for what purpose will save developer lot of time by not putting any detailed information of actual individual fixes which might not be exactly useful for testing purposes and keeping specific information signaling the patterns of bugs as specified by the test team.

This in turn will help testers to come up with overall better effective test strategy by root cause analysis with less efforts and improving overall quality of the product under testing helping developers as well by reducing incoming unplanned production issues in sprints.

If done correctly, this will mutually benefit both the developers & testers and essentially the whole team.

Answer 14

I'll start with the issues, then work into how to fix it.

The issue here is bidirectional communication. Both parties are failing to communicate with each other. You're going to need to do a better job of communicating to the devs what you actually want. For example, let's take your example of "Fixed in version x.x.x.x, please test" You say you want more details. For sake of argument, I'll play the dev and play along. My next reply looks something like:

Fixed an off-by-one issue in the UnderlyingServerManager decorator which caused it to produce an incorrect hash on Widgets whose memory address was a prime number - 1.

Now you have details. There's only one problem. None of those details are useful to you. You don't know what an UnderlyingServerManager is (hint: it's underlying, buried deep in the code where non-coders cannot see it), you don't know what Widgets are, and you may or may not know what the implications of the incorrect hash could be. You have the details, but they didn't help all that much.

What you are really asking them to do is to help you do your job. They know that they made a fix, and you know that there may be implications associated with that fix, but you don't know what those implications are. In fact, since you mentioned many testers don't know the language it is coded in, many testers don't even know where to start! You need the developers to do the first half of your job, so you can do the last half.

Now this is not an unreasonable desire. It has been said time and time again that developers cannot just code and throw the results over the wall to test. However, I chose my words to point out that the handover of information you are looking for is not on a clean border between code and test. You're looking for them to distill the code related information into something that's already fully in the language of test. That's a lot more work, and the developers may or may not be up for it.

The fix needs to start with you or someone above you in QA learning what tasks the developers have on their plate and how stressed they are. You need this information for two reasons. One is the obvious one: you're going to be asking them to do more work to make your life easier. It's good to know how much bandwidth they have. The second reason, which is more important, is subtle. You're going to need a communications channel opened to properly discuss what needs to be done. A channel that is capable of gleaning their work load is one which is in a position to find ways to help them out.

"You scratch my back, I'll scratch your" should be the name of the game. Perhaps you find out that there's a QA step that's actually getting in the way of them doing their job well. See if you can offer a trade: simplify that QA step for them, in exchange for opening the dialogue on how to get "more detail." Get them to want to help you out, and the process is easier.

The next step is to realize that what you are asking for is not "more detail," but "less detail, in a human readable form." They gave you the detail -- it's in the code. What you want is a version of that which has already been paraphrased into a form that your testers can work with. I point this out because the developers are not going to know what you need. You're going to have to work with them, back and forth, until they understand what information is meaningful to testers. You need to train them to speak your language (or learn code, so that you can speak theirs). Generally speaking, they wont want to, which is why we started with the first step. Sweeten the deal by finding ways to make their lives easier.

Make sure management knows that you're doing this, and make sure to laud the developer's efforts once they start doing it right. Cooperation is not easy in a corporate environment, especially a remote environment. Make sure their bosses know that the devs are pulling their own weight.

Answer 15

It is information valuable to other coders so why not share it with test.

If I am pretty sure the fix is limited and does not require a full set of regression testing I will share that.

If it was a database change that could affect other programs I share that.

Test could be using some manual queries that need to be changed.

I had a case where it did not work in test but it did in development. Turns out test did not copy a config file. If I had told them there was a change to the config file time could have been saved.

Answer 16

Let's be completely honest here: The most probable reason the bug got in was that a developer made a mistake. No problem, we're all human, and if developers didn't make mistakes there would be no jobs for QA engineers. You're asking people to put down in writing, in a permanent record, "I made a mistake", and it will probably also be a mistake that was very obvious after the fact, embarrassingly so. And you are surprised that people push back on this? Let me flip this question around, if a customer discovered a bug would you consider it reasonable to be expected to explain to that customer how you personally, with responsibility for quality, allowed such an obvious bug to slip past you?

Things like this are best discussed face to face in meetings at which no fingers are pointed and no names are written down. You can get all the information you need, and people will be willing to provide it, but you must go about it in a sensitive way, rather than gathering ammunition for a witchhunt.