0

Details:

We had a discussion on this very topic in our team today, some brought in a factor of 30% which I also know, but I am of the opinion that a much higher automation factor should always be achieved.

What automation factors do you work with?

  • Full automation, i.e. 100%, or a different factor depending on the project?
  • Is the 30% automation mentioned a known and verifiable value that can be adopted in this way?
8
  • What is an "automation factor"? Is it test coverage? If so, how would you measure tacit knowledge that wasn't uncovered? Isn't trying to objectively measure subjective perspectives (which is what testing evaluates) a misleading simplification? Commented Feb 16, 2021 at 10:31
  • It is a question of how a project can be measured at all, and how it can be measured in the direction of automation. There are factors that cannot be automated. Here are hardware tests that cannot be automated in part.
    – Mornon
    Commented Feb 16, 2021 at 11:12
  • 1
    The point is that testing is a social activity, of evaluating risks to a product. Just one doesn't say that a scientific research is X% done, one cannot say x% of the the possible testing is done. You can try to move your goal from measuring to assessment: Understand your context, explain to your clients what you will do to learn about certain risks and perform your exploration of these areas. And you can add tools that you will use to enhance your capability of such exploration. But saying "x% of the testing is xyz" is misleading because you cannot evaluate all testing space. Commented Feb 16, 2021 at 11:13
  • The question then is why many teams set this 30% automation factor as a target?
    – Mornon
    Commented Feb 16, 2021 at 11:14
  • What do you mean by "many teams"? In which context are we talking about? Commented Feb 16, 2021 at 11:19

3 Answers 3

1

Assuming we are relating to system tests, empirical data shows that the effectiveness of unit tests above 80% coverage is decreasing while the complexity significantly increase, I am not aware of any magic number for coverage.

But let's take a step back and analyze your question.

First we need to understand what coverage is. Having one test per requirement doesn't take into consideration ranges and timing for example, measuring the actual code coverage of your system tests is usually not practical and while better it still doesn't guarantee that your business logic is covered.

Next you ask about "what should", to answer that you need to know first why do you care. Do you think that higher coverage is better? why? Hard scripted Tests in general, and automated tests even more, tend to have a "pesticide effect" meaning that they are not really good at finding new issues, and while they are doing a better job finding regressions even this is quite limited to bigger problems.

The last question needed to be answered is the cost of achieving higher coverage (cost means effort, resources, time etc.), writing the first automated test is hard and costly since it requires setting up of a test infrastructure. From there the cost is usually low, especially when adding related tests using the same mechanism, think Selenium and POM- most of the tests are about manipulation and interaction with locators. But the low cost trend stops somewhere when complexity increases, the test's runtime increases or you need a sophisticated infrastructure to simulate special conditions. In a typical Pareto principle the cost of the last added percentages towards 100% will be significantly higher than the lower percentages of coverage.

Now you are probably asking what should your team do. The answer depends on the structure and way of working in you company and team. A fast pace, Agile team with a new product and few people responsible for automation can be satisfied with few valuable, fast and stable regression tests. A legacy product with a a lot of available test resources might want to achieve higher coverage, I really like the SQlite example with "Millions and millions of test cases"

2

Your question is very general and the answer depends on many factors such as organization rules, AUT's (Application Under Test) business, Test methodologies, type of tests you want to automate and ...

But there is one key concept in test automation named Test Pyramid introduced by Mike Cohn. enter image description here

You should climb up the triangle as necessary as you want and for each step upward the percentage of test coverage becomes lower.

Dzone has a good article about Automated Testing Strategy

1
  • 1
    That's exactly what I was really looking for, in the direction of a test strategy. That already helps me. Yes, unfortunately the question is very broad, but these strategies are also very extensive.
    – Mornon
    Commented Feb 16, 2021 at 11:50
1

enter image description here

To add to above answer , i don't belive test automation pyramid refer to "How many test cases" but about the test coverage

THe higher you go in the pyramid less likely you will be able to hit all 100% of the branches in your code. The effort and investigation required to reach such a 100% code coverage would be tremendous and costly in higher levels of the pyramid like the UI.

So when talking about test pyramid we should talk about test scope than test numbers.

The strategy that i would follow would be:

Unit Test:

100 % branch and condition coverage:

meaning unit tests should be able to test all conditions of a component.

Integration test/API test

Validate 100 % of contract coverage , test that all contracts are tested and verified.

UI tests/ System test

100% user story coverage , ensuring all user interactions are tested and covered.

THis is why BDD becomes important in this level as it can make it easy for tracking Scenario/User story coverage and shares coverage responsibility among team than just QA

6
  • 1
    research from real world projects shows that the benefit from adding unit tests above 80% coverage is decreasing fast. How do you define Validate 100 % of contract coverage? it's one thing to have one test per API and another thing to actually cover everything.
    – Rsf
    Commented Feb 17, 2021 at 9:18
  • @Rsf contract defines the behavior of the endpoint for different use cases , API testing is not just about testing status code , scehma and response. Its also about testing the senarios in which these are generated .
    – PDHide
    Commented Feb 17, 2021 at 9:21
  • 1
    just that, you can't cover 100% of the scenarios and usually it's not really measurable
    – Rsf
    Commented Feb 17, 2021 at 9:25
  • @Rsf why cannot we cover 100% ?
    – PDHide
    Commented Feb 17, 2021 at 9:42
  • 100% of not the exhaustive use cases but the decided acceptance criteria
    – PDHide
    Commented Feb 17, 2021 at 9:44

Not the answer you're looking for? Browse other questions tagged or ask your own question.