Do we need to add automation coverage for customer bugs?

Question

As currently, we are having coverage for P1 test cases for an end to end scenarios. We are following the test pyramid strategy. But now we are planning to cover the customer bugs. So just want to know whether we need to provide automation coverage for all bugs or not.

Answer 1

Recently been in similar situation and after analysis found an interesting co- relation of bugs found.

The result of the analysis was bugs hide in clusters mostly in few specific areas only.

In other words bugs are not scattered all across the application but mostly exist in clusters in few area only.

So instead of mapping 1 to 1 to each production bug , we added/updated few end to end scenarios in these areas covering most of these issues path + few additional related validations.

This is working very well now as we are finding most of these bugs (if any) in regression test runs only. And automation is being considered very effective by business analysts as well.

Answer 2

It depends

It depends on your system functionality, your current test case coverage, your current test type allocation (based on the testing pyramid and the Agile Testing Quadrants

Your basic choice is always: Update an existing unit/integrated/UI tests or create a new unit/integrated/UI test. Yo should do that in the following order which prioritizes updates existing tests over creating new ones and also prioritizes unit over integration and both over UI testing - the slowest and least reliable form. The order means that if you can do a low level tests instead of a high level one, you should do that

• Review the existing Unit cases and see if a specific example is missing

else

• Add a brand new Unit test case

else

• Review the existing Integration tests and see if a specific example is missing

else

• Add a brand new integration test

else

• Review the existing UI tests and see if a specific example is missing

else

• Add a brand new UI case

else

• Add a new manual test case

else

• Add new items to the exploratory testing charter

Answer 3

Reported defects are not covered specifications, it is behaviour not anticipated. I would always add an automated test for each found defect, as it is probably is the brittle part of the application. It also documents the new expected behaviour.

Now what kind of test you write depends on the defect. Preferably you first write a failing test to reproduce the defect and then fix it. I would try to start reproducing it in a unit-test and slowly go up the pyramid if needed.

Answer 4

1. My first question to you - what is your criteria for regression tests? How do you choose what to automate and what to not? I guess you don't have it, because it should answer your own question. It is not a problem to automate something, it is challenge to automate the right test cases. We learn from bugs raised by customers, but only in few cases we really need to automate them.

2. Since we cannot automate everything typical criteria for e2e is to automate 1) only standard features 2) high risk 3) most used

3. If you are living in agile world, I would suggest you to read a book "Agile testing" by Janet Gregory & Lisa Crispin. In their book they have several chapters on Agile Quadrants and chapter on Test Automation Strategy. They suggest to write e2e tests for epics, integration tests for stories and unit tests for tasks.

4. Another idea would be to take free training on Test Automation Strategy at Test Automation University.

5. More test automation strategy modules, collected by Lisa Crispin you will find on JoeColantonio website.