Is defect driven test automation an good idea?

Question

Defect driven Test automation: Test strategy in place to consider candidates for test automation primarily as earlier found and fixed defects in the application. With the thought process that it will give the most bang for the bucks as regression test strategy.

Question: When teams build tests based on reported defects many times the strategy is to build a test for every defect. The concept is when the team picks up a defect, they write a test for the expected behavior and see that fail. Next they write a fix that causes the test to pass. This approach will definitely increase test coverage and will yield an automated test suite that can be used. However how do you know that you have made the best investment in writing automated tests?

My view:In my view, it is an good idea for legacy applications which are stable and having large defects data from past but not as good for new application under development. Do you agree with my viewpoint?

Answer 1

I slightly disagree with this approach in general. But this approach can be used over and above existing automation strategy to improve the regression suite.

Firstly, this approach doesn't consider "unknown" side effects of bug fixing. So, solely relying on this strategy would not bear fruits. But this approach can complement the automation strategy that is based on ensuring complete code coverage.

If a bug is fixed but as a consequence of this fix, what if another functionality is broken for which there were no bugs reported in the past? and somehow, the developer had no clue about this impact?

So, it is a good approach to have an automation strategy which is independent of reported bugs. But, having said that this approach can be applied over and above the automated regression suite to improve the coverage. Because, "practically", it is very difficult to ensure 100% code coverage :-). It is a continuous improvement process.

If a bug is found in UAT that was missed then there are chances that it was not covered in the automated regression suite. So, it is good to add such missed functionalities in the regression suite to make it more robust.

Answer 2

As most things in life: It depends.

A bug can be seen as a failure in planning, where the proper verifications for that behavior were not placed. And since it happened in the past, it is likely to happen in the future - thus automated checking would prevent regressions in high risk areas.

On the other hand, when you decide to go through one path, you are discarding all other paths. An opportunity cost analysis may be necessary. E.g., an edge case bug can be resulted from to much manual focus on re-validation of simpler behavior. Automated checks for the latter may free people to investigate more the edge cases.

Answer 3

I wouldn't say it's a good idea for general automation since the future defects distribution in the code is hardly dependent on where previous defects were located and fixed.

As a general test automation strategy I would pick the one where you can maximize code coverage of your AUT.

However as a side strategy it still can be used. For example I would automate the defects which come from UAT phase because they often lean on how the end user or stake holder use your application and the user paths which they value from their business perspective that might often be hidden from engineers.