I am working as SQA and mostly do smoke testing and regression testing manually. But I think it is consuming more time and I am not certain about if it's completely tested or not.

  • 4
    Possible duplicate of What is a good KPI for software QA?
    – Prome
    Nov 18, 2019 at 12:57
  • 1
    Additionally to that Prome said, I would say that this question is "Too Broad". Context and goals are essential to any useful answer here - there are no best practices, only good practices in certain contexts. Nov 18, 2019 at 13:42
  • 3
    Are you looking for an absolute measure of how effective your testing is? In that case the question Prome linked is close to a duplicate. If you're asking how to use metricts to decide if you need to use automated smoke testing and automated regression testing, that's a different and equally important question
    – Kate Paulk
    Nov 18, 2019 at 14:49
  • As usual @KatePaulk asks the best questions. Thank You Kate. Nov 18, 2019 at 15:33

3 Answers 3


Measuring test effectiveness is a great question.

So many in our industry focus on "Tests, number of tests, how long they run, how often they fail", all without looking at the key question "How effective is my testing?"

In order to evaluate tests, start with the business value. Ask questions such as:

  • What is the software release cycle? Once every 6 months vs once every two weeks vs. many times a day will require dramatically different strategies to testing and test cases

  • What failure level does the business consider acceptable? In some businesses, such as banking there may be little tolerance for failure which may require more testing. In others such as a startup, consumer growth is everything and even if 10% of customers have errors, this is ok for the business model under operation

Actual KPI (measures) of test effectiveness are hard. Immature organizations (in terms of quality) will tend to look more to

  • number of tests
  • number of bugs*
  • failure rate

A more mature organization (in terms of testing and QA) may tend to look more at measures for:

  • Revenue affected
  • Customers affected
  • Production testing
  • AB and Canary testing
  • Speed to market of new features
  • An appropriate test pyramid
  • Mean time between failure in production
  • Mean time to Recover from failure in production
  • An appropriate division of Unit, Integrated and UI tests

* the problem with number of bugs is that it can easily end up being a very objective measure, easily gamed and manipulated within the desire of showing improvement. Measure of quality should be relevant to the development process at hand. Measuring outcomes by number of bugs found later by other people can lead to fixing individual issues as they come up but not necessarily the underlying processes that need improvements and lead to these issues being able to happen in the first place.

  • Thanks for edits. Note that some rejected as they changed meaning. When I write 'mean time' i mean the average time between failures. I do not mean 'meantime' as in "in the meantime" that is different. Thank You. Dec 30, 2019 at 12:30

EDIT: This answer is better suited for measuring automated test suites. But I will keep it here anyways as the question has the automated-testing tag

One concept which blew my mind when I heard about it is Mutation Testing:

Mutation testing involves modifying a program in small ways. Each mutated version is called a mutant and tests detect and reject mutants by causing the behavior of the original version to differ from the mutant. This is called killing the mutant.

Test suites are measured by the percentage of mutants that they kill.

Depending on your compilation cost, this metric can be very expensive to calculate, though.


Below are the testing efficiency metrics used in qa testing company to evaluate QA team:

  • Number of defects reported in each stages of application
  • Percentage coverage in functional testing
  • Number of defects validated
  • Number of automated test cases written
  • Do you consider these good metrics? I guess it depends on the people as well, but what if this becomes some sort of competition of who can fill in more bugs (because well we have such a metric, right?); similar with the automation tests - just a number doesn't say much about how appropriate they are (do they check what they should?).
    – pavelsaman
    Jan 15, 2020 at 11:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.