What to do when a bug is fixed, unfixed and fixed

Question

A blocker bug I raised last week had been fixed and ready for re-testing.

• This morning, I re-tested it, it passed and subsequently closed.
• In the afternoon, by accident, I noticed this feature was unfixed again.
• After I reported it to my project manager, it was fixed again.

Any suggestion on how to manage this kind of bugs?

--More background--

• We are contractors working on a short contract, 6 weeks in total.
• Pure manual testing, our employers are not interested in any forms of automation.
• Developers are contractors as well, they work offsite. We communicate via conference calls. Developers have their own project manager and we have our own project manager.
• There are only 10 days left in our contract.
• Raised bugs 104, fixed bugs 26, as of 27th June 2017

Answer 1

This is a project management issue, not testing. Your actions depends on your specific situation, for example-

A big company, huge code repository, a lot of changes- I'll ping the relevant teams, reopen the bug, check the fix and forget about it. If I see the same issue again and again I'll make a bigger issue out of it, supporting data can probably come from the bug system queries.

A small company, no decent quality culture- this would be a great opportunity to educate others

Anyway, usually it wouldn't be wise to add a test per such bug as a rule. Those bugs are re introduced by complex branching schemes and sloppy developers, they are usually not "clustered" around sensitive areas.

Answer 2

Add an automated test for each defect. Make sure they never return.

Now it is a short cycle, you caught it fast. I have seen long cycles. One client asking to remove/change/fix something. Release some months later, one client complains something is missing. The team fixes it, until the other clients comes back.

You need someway to document the changes. I would say write a test-case that gets repeated before every release. Preferable automated.

Answer 3

It looks like developers need to produce release notes and inform everyone that a patch is applied or unapplied.

• The developers had fixed it and for some reason unfixed it, but they had never informed anyone as you found out this issue was unfixed by accident.
• As a general practice, a release note should be sent out to everyone before a patch is applied. In this release note, it should address which issues are addressed by this patch.
• A notification email should be sent as well.

My personal recommendation:

• Escalate this issue to your project manager and ask for a release note and notification email

• Make sure you have recorded how you re-tested it, how it became unfixed and etc. Show them to your project manager if it is necessary.

• Keep your own personal backlog of important blocker tickets which have been fixed. Regression test all of them in case they become unfixed again.

Answer 4

Not high volume of such issues:

So, from my experience, I believe that such issues generally exist in projects but they are not huge in numbers.

Track them separately and re-test periodically

The best way is to keep the Ids of such defects handy and add them to your watch list. Once bug surfaces again from Closed state, it should be "Re-opened". These bugs should be re-tested couple of times (once a week or once a fortnight) even after getting fixed (since they are history-sheeters :-))

Be the whistle blower

Moreover, in case you feel that you need to escalate this issue to your manager or anyone else, then you can pull the data related to the number of iterations for such issue, in order to highlight the uncertainty associated to such issues.

Cure for such issues

Generally, such issues are caused if there are issues at design level. So, these issues are deep rooted and developers are unable to resolve them merely by making changes to the code. Such issues, generally require the attention of the person playing the role technical architect (or equivalent). And the team should sit together in order to finalize a fix for such issues. Impact analysis is very important when a solution is proposed because you want to get the bug fixed permanently and no transformed from one form to another.

Answer 5

These things happen and possible reasons are:

1. junior developers
2. senior developers which work out of their domain (take a look at enter link description here)
3. under estimated project: people are rushing to get things done so they don't regression test new code
4. huge code base: developers don't know exactly how to add a feature without breaking something else
5. legacy code base: it might be that the software architecture is not so modern, unit tests are not present, etc. Therefore it's difficult to add new code.

From your question, the reason might be bullet 3, but obviously it can be something else.

There are some options to avoid regressions like the one you are describing:

• Pair programming so that developers with low experience can be assisted by senior developers
• Code review. Instead of just adding new lines of code, each changeset is peer reviewed by at least another people, ensuring that mistakes are prevented (not excluded though!)
• Unit tests can be written by both developers and testers, depending on how they are structured. The aim is to black box testing of a piece of functionality. This ensures that it matches the actual requirements, but it's hugely helpful to avoid regressions. Indeed, if the tests are run as part of each build, and you accidentally change the behaviour of a piece of software, then a failure in a unit tests will highlight the problem.
• End to end tests: these are helpful when the application interacts with other components (also third party drivers/software), and again it ensures that the behaviour exposed to the customer is well tested. E2E tests are expensive to write (normally), but they save time in manual testing. You need to decide what can be e2e tested and what it's better to just delegate to manual test. Normally if you don't plan to change the functionality in a particular area, manual testing is a choice.
• Build email: developers should write an email every time they release a new build to testers, highlighting new features, bug fixing, etc. This helps testers to know what has been fixed, what are the new features, etc. It also helps to understand things like bugs rate, velocity of the project, etc.
• Issue tracker: Bugzilla is a good example. It helps to organize software development and testing. In your case, the same bug would have been: raised in build 1 --> fixed in build 2 --> verified in build 2 --> reopened in build 3. Again, this tool helps to have an idea of the quality of the software and regressions.
• Tests tracker: TestLink is a good example. It allows you to write/run/document manual test cases based on builds. It allows you to generate reports. In your case, you could have generated a report for a specific build stating which tests succeeded and which tests failed. If there's a regression in a further build, that's not your direct problem.
• System test: when the project manager thinks the software is good, the last build can be system-tested instead of functional-tested. System test should highlight different types of issues, for example related to networking or memory leaking, but obviously regressions can be discovered.
• Project management systems: Trello is a good example. It allows project manager, developers and testers to be at the same page. It also helps the project manager to prioritize, measuring velocity, and select what features go in a release and what features go back to backlog.

In your case I think it's worth writing an email to the project manager saying that the regression was found in a build after the same bug was already verified in a previous build.

Your behaviour also depends and what you contract states, i.e. what the company expects from testers.