How to know if an existing set of automated tests are effective?

Question

I came to a new project and it was already written 100 autotests. The product is new. How to determine whether the tests are effective? What is the best way?

Answer 1

Learn how to use the product as an end user. Become familiar with all the functions, error pages and workflows.
Then study the tests. Observe them run and/or look at their code and ask yourself:

• Is there functionality in the UI that you do not see UI tests for ?
• Are there automated UI tests for things you cannot do in the UI ?
• Do feature tests cover happy, sad and optional workflows ?
• Do unit and integration tests exist for the backend functions that support the ui ?
• Is security tested?

Answer 2

• Determine coverage (of source code) and add test exercising parts not covered yet
• Found most often used "paths" through the system, and different kinds of users would use it. Check if most common are covered.

"Best" way depends what you want to cover. What is critical. What business sees as most important. There are many ways to measure it.

Answer 3

How to determine whether the tests are effective? What is the best way?

It is a new product, so there is little historical test results to compare against, but can you

• Find a similar product, how is this similar product tested? Can this similar product be tested effectively? How are your test cases compared to those for the similar product?
• Over time, there will be bugs. Can your test cases catch those bugs or not? If an easy to catch bug manages to slip into production, that means your test cases need improvement.
• The best way to define "effectiveness" for a given product is to measure against its own historical product quality.

Answer 4

The version control systems for both the tests and for the product code should be able to quickly answer the following questions:

• Were there any changes to the code due to the tests finding issues?
• Were there issues found that were not found by the tests?
• Were tests introduced for such issues and when run on code from before the fix would they fail?
• Are there tests that currently fail?

You can also look at test coverage of the code & specification, traceability between specification and tests will give a good indication of this.

Answer 5

How to know if an existing set of automated tests are effective?

Effective tests can be hard to gauge. The problem with automated tests is that they find less bugs over time.

Having said that, I find the regression testing mnemonic RCRCRC proposed by Karen Johnson useful when evaluating existing tests. Although specifically designed for regression testing I still think they work well.

• Recent. How recent are the tests? Do they cover new features, areas or changes? Or are they primarily old tests?
• Core. What functionality do these tests cover? Are they the critical or essential functionality?
• Risky. What features are riskier than others? What features are more likely to be broken?
• Configuration Sensitive. What code is dependent on environment settings? How might this change based on different environments (Staging, Production, etc)?
• Repaired. What features or code has been fixed or repaired recently and might still have more bugs around it?
• Chronic. Features or code that always seem to be breaking or having issues.

Answer 6

Now a days computers are fast enough to automate mutation testing.

You run all the tests, make sure they are green/successful. Then introduce a mutation and run all the tests again and expect it to be red/failing.

Any mutation that is not caught by a test, means you lack a test.

Mutation testing is based on two hypotheses. The first is the competent programmer hypothesis. This hypothesis states that most software faults introduced by experienced programmers are due to small syntactic errors. The second hypothesis is called the coupling effect. The coupling effect asserts that simple faults can cascade or couple to form other emergent faults.

General code coverage could also help, combining both coverage and mutation testing will go a long way to proof that your test suite is effective in catching unintended changes or small defects.