4

As currently, we are having coverage for P1 test cases for an end to end scenarios. We are following the test pyramid strategy. But now we are planning to cover the customer bugs. So just want to know whether we need to provide automation coverage for all bugs or not.

  • 2
    The test pyramid does not specificy which level a bug would be fixed at - it could be any. – Michael Durrant Apr 6 at 14:57
3

Recently been in similar situation and after analysis found an interesting co- relation of bugs found.

The result of the analysis was bugs hide in clusters mostly in few specific areas only.

In other words bugs are not scattered all across the application but mostly exist in clusters in few area only.

So instead of mapping 1 to 1 to each production bug , we added/updated few end to end scenarios in these areas covering most of these issues path + few additional related validations.

This is working very well now as we are finding most of these bugs (if any) in regression test runs only. And automation is being considered very effective by business analysts as well.

1

It depends

It depends on your system functionality, your current test case coverage, your current test type allocation (based on the testing pyramid and the Agile Testing Quadrants

enter image description here

enter image description here

Your basic choice is always: Update an existing unit/integrated/UI tests or create a new unit/integrated/UI test. Yo should do that in the following order which prioritizes updates existing tests over creating new ones and also prioritizes unit over integration and both over UI testing - the slowest and least reliable form. The order means that if you can do a low level tests instead of a high level one, you should do that

  • Review the existing Unit cases and see if a specific example is missing

else

  • Add a brand new Unit test case

else

  • Review the existing Integration tests and see if a specific example is missing

else

  • Add a brand new integration test

else

  • Review the existing UI tests and see if a specific example is missing

else

  • Add a brand new UI case

else

  • Add a new manual test case

else

  • Add new items to the exploratory testing charter
  • downvoter - you have a different approach? or you think this is actually wrong? I am most curious. – Michael Durrant Apr 6 at 15:01
  • I am very confused with my downvoters. I guess people are not used to design test automation from logical side. Other explanation I don't have. – kriscorbus Apr 17 at 19:11
1

Reported defects are not covered specifications, it is behaviour not anticipated. I would always add an automated test for each found defect, as it is probably is the brittle part of the application. It also documents the new expected behaviour.

Now what kind of test you write depends on the defect. Preferably you first write a failing test to reproduce the defect and then fix it. I would try to start reproducing it in a unit-test and slowly go up the pyramid if needed.

-3
  1. My first question to you - what is your criteria for regression tests? How do you choose what to automate and what to not? I guess you don't have it, because it should answer your own question. It is not a problem to automate something, it is challenge to automate the right test cases. We learn from bugs raised by customers, but only in few cases we really need to automate them.

  2. Since we cannot automate everything typical criteria for e2e is to automate 1) only standard features 2) high risk 3) most used

  3. If you are living in agile world, I would suggest you to read a book "Agile testing" by Janet Gregory & Lisa Crispin. In their book they have several chapters on Agile Quadrants and chapter on Test Automation Strategy. They suggest to write e2e tests for epics, integration tests for stories and unit tests for tasks.

  4. Another idea would be to take free training on Test Automation Strategy at Test Automation University.

  5. More test automation strategy modules, collected by Lisa Crispin you will find on JoeColantonio website.

  • Hey, downvoter, I am curious and would like to know why you did not like my answer. – kriscorbus Apr 8 at 21:24
  • I wasn't the downvoter, but I see two reasons why someone would: 1) self promotion without full disclosure in the post is against policy, so that's a big one. 2) the answer doesn't seem to address the question. Those will probably need to be addressed for the community to view it favorably. – corsiKa Apr 8 at 22:14
  • As an addition (also not the downvoter): "Janet & Lisa" is not really helpful if you have no clue who they are and what to look for. Full names, titles and a reference to the actual book(s) is much more helpful. – Ray Oei Apr 9 at 21:00
  • 1
    Thank you, @RayOei, for comment. It was not helpful to assume that everyone knows who are Janet & Lisa. – kriscorbus Apr 10 at 23:40
  • @corsiKa I removed the link. According answering the question - in testing you never do automation for bugs, but for features. I described both ways how to do it. – kriscorbus Apr 11 at 0:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.