How can I measure how effective my testing is? What metrics should I use?

Question

I am working as SQA and mostly do smoke testing and regression testing manually. But I think it is consuming more time and I am not certain about if it's completely tested or not.

Answer 1

Measuring test effectiveness is a great question.

So many in our industry focus on "Tests, number of tests, how long they run, how often they fail", all without looking at the key question "How effective is my testing?"

In order to evaluate tests, start with the business value. Ask questions such as:

• What is the software release cycle? Once every 6 months vs once every two weeks vs. many times a day will require dramatically different strategies to testing and test cases

• What failure level does the business consider acceptable? In some businesses, such as banking there may be little tolerance for failure which may require more testing. In others such as a startup, consumer growth is everything and even if 10% of customers have errors, this is ok for the business model under operation

Actual KPI (measures) of test effectiveness are hard. Immature organizations (in terms of quality) will tend to look more to

• number of tests
• number of bugs*
• failure rate

A more mature organization (in terms of testing and QA) may tend to look more at measures for:

• Revenue affected
• Customers affected
• Production testing
• AB and Canary testing
• Speed to market of new features
• An appropriate test pyramid
• Mean time between failure in production
• Mean time to Recover from failure in production
• An appropriate division of Unit, Integrated and UI tests

* the problem with number of bugs is that it can easily end up being a very objective measure, easily gamed and manipulated within the desire of showing improvement. Measure of quality should be relevant to the development process at hand. Measuring outcomes by number of bugs found later by other people can lead to fixing individual issues as they come up but not necessarily the underlying processes that need improvements and lead to these issues being able to happen in the first place.

Answer 2

EDIT: This answer is better suited for measuring automated test suites. But I will keep it here anyways as the question has the automated-testing tag

---

One concept which blew my mind when I heard about it is Mutation Testing:

Mutation testing involves modifying a program in small ways. Each mutated version is called a mutant and tests detect and reject mutants by causing the behavior of the original version to differ from the mutant. This is called killing the mutant.

Test suites are measured by the percentage of mutants that they kill.

Depending on your compilation cost, this metric can be very expensive to calculate, though.

Answer 3

Below are the testing efficiency metrics used in qa testing company to evaluate QA team:

• Number of defects reported in each stages of application
• Percentage coverage in functional testing
• Number of defects validated
• Number of automated test cases written