Measuring test effectiveness is a great question.

So many in our industry focus on "Tests, number of tests, how long they run, how often they fail", all without looking at the key question "How effective is my testing?"

In order to evaluate tests, start with the business value. Ask questions such as:

• What is the software release cycle? Once every 6 months vs once every two weeks vs. many times a day will require dramatically different strategies to testing and test cases

• What failure level does the business consider acceptable? In some businesses, such as banking there may be little tolerance for failure which may require more testing. In others such as a startup, consumer growth is everything and even if 10% of customers have errors, this is ok for the business model under operation

Actual KPI (measures) of test effectiveness are hard. Immature organizations (in terms of quality) will tend to look more to

• number of tests
• number of bugs*
• failure rate

A more mature organization (in terms of testing and QA) may tend to look more at measures for:

• Revenue affected
• Customers affected
• Production testing
• AB and Canary testing
• Speed to market of new features
• An appropriate test pyramid
• Mean time between failure in production
• Mean time to Recover from failure in production
• An appropriate division of Unit, Integrated and UI tests

* the problem with number of bugs is that it can easily end up being a very objective measure, easily gamed and manipulated within the desire of showing improvement. Measure of quality should be relevant to the development process under hand. Measuring outcomes by number of bugs found later by other people can lead to fixing individual issues as they come up but not necessarily the underlying processes that need improvements and lead to these issues being able to happen in the first place.

Measuring test effectiveness is a great question.

So many in our industry focus on "Tests, number of tests, how long they run, how often they fail", all without looking at the key question "How effective is my testing?"

In order to evaluate tests, start with the business value. Ask questions such as:

• What is the software release cycle? Once every 6 months vs once every two weeks vs. many times a day will require dramatically different strategies to testing and test cases

• What failure level does the business consider acceptable? In some businesses, such as banking there may be little tolerance for failure which may require more testing. In others such as a startup, consumer growth is everything and even if 10% of customers have errors, this is ok for the business model under operation

Actual KPI (measures) of test effectiveness are hard. Immature organizations (in terms of quality) will tend to look more to

• number of tests
• number of bugs*
• failure rate

A more mature organization (in terms of testing and QA) may tend to look more at measures for:

• Revenue affected
• Customers affected
• Production testing
• AB and Canary testing
• Speed to market of new features
• An appropriate test pyramid
• Mean time between failure in production
• Mean time to Recover from failure in production
• An appropriate division of Unit, Integrated and UI tests

* the problem with number of bugs is that it can easily end up being a very objective measure, easily gamed and manipulated within the desire of showing improvement. Measure of quality should be relevant to the development process at hand. Measuring outcomes by number of bugs found later by other people can lead to fixing individual issues as they come up but not necessarily the underlying processes that need improvements and lead to these issues being able to happen in the first place.

Measuring test effectiveness is a great question.

So many in our industry focus on "Tests, number of tests, how long they run, how often they fail", all without looking at the key question "How effective is my testing?"

In order to evaluate tests, start with the business value. Ask questions such as:

• What is the software release cycle? Once every 6 months vs once every two weeks vs. many times a day will require dramatically different strategies to testing and test cases

• What failure level does the business consider acceptable? In some businesses, such as banking there may be little tolerance for failure which may require more testing. In others such as a startup, consumer growth is everything and even if 10% of customers have errors, this is ok for the business model under operation

Actual KPI (measures) of test effectiveness are hard. Immature organizations (in terms of quality) will tend to look more to

• number of tests
• failure rate

A more mature organization (in terms of testing and QA) may tend to look more at measures for:

• Revenue affected
• Customers affected
• Production testing
• AB and Canary testing
• Speed to market of new features
• An appropriate test pyramid
• Mean time between failure in production
• Mean time to Recover from failure in production
• An appropriate division of Unit, Integrated and UI tests

Measuring test effectiveness is a great question.

So many in our industry focus on "Tests, number of tests, how long they run, how often they fail", all without looking at the key question "How effective is my testing?"

In order to evaluate tests, start with the business value. Ask questions such as:

• What is the software release cycle? Once every 6 months vs once every two weeks vs. many times a day will require dramatically different strategies to testing and test cases

• What failure level does the business consider acceptable? In some businesses, such as banking there may be little tolerance for failure which may require more testing. In others such as a startup, consumer growth is everything and even if 10% of customers have errors, this is ok for the business model under operation

Actual KPI (measures) of test effectiveness are hard. Immature organizations (in terms of quality) will tend to look more to

• number of tests
• number of bugs*
• failure rate

A more mature organization (in terms of testing and QA) may tend to look more at measures for:

• Revenue affected
• Customers affected
• Production testing
• AB and Canary testing
• Speed to market of new features
• An appropriate test pyramid
• Mean time between failure in production
• Mean time to Recover from failure in production
• An appropriate division of Unit, Integrated and UI tests

* the problem with number of bugs is that it can easily end up being a very objective measure, easily gamed and manipulated within the desire of showing improvement. Measure of quality should be relevant to the development process under hand. Measuring outcomes by number of bugs found later by other people can lead to fixing individual issues as they come up but not necessarily the underlying processes that need improvements and lead to these issues being able to happen in the first place.