How do you know if you are writing high quality tests?

Question

Inspired by Exercises on software testing

We can learn to write automated tests and we can learn to do tdd and bdd but what tools / resources / guides are available to tell us if the actual tests we are writing are of high quality?
What specific standards or guidelines exist for writing high quality tests?
What specific approaches (within tdd/bdd) result in higher quality tests?
Conversely, what tests are usually of a lower quality level?

Answer 1

The goal is to find bugs while leaving time and money to do other things too. So a high-quality test is a test that finds bugs but doesn't require a disproportionate expense to write, maintain, or operate.

Sometimes you can tell whether a test is low-quality just by looking at it. For example, if the test is buggy or written in a brittle way so that every change to the system requires a change to the test, it's a low-quality test.

You can't tell whether a test is high-quality without measuring the test's impact, i.e. measuring whether it finds bugs. A test can be really well-written but if it doesn't find bugs, it's not a good use of resources.

Of course, it's hard to know in advance whether a test will be high-quality because it's hard to know where the bugs will be.

Answer 2

It depends on the kinds of tests.

High quality unit tests have the following characteristics

• They mock and stub out external dependencies such as the database
• They run fast
• They test a single piece of functionality

High quality UI feature tests have the following characteristics

• They reflect real user usage, actions and workflows
• They do not fail intermittently
• Failures represent real user issues
• Real user issues break the tests
• They are robust in their use of HTML element selectors
• They use a Page Object approach for selectors, actions and services

All high quality tests share the following characteristics

• They are readable and also act as documentation of the system
• They are run for each change to the software
• They are reliable
• They do not have false positives or negatives
• They are written in plain English and generally avoid obscure terms and acronyms
• They do not test obsolete or removed functionality

Answer 3

Low quality tests

When I think of low quality tests, I think of teams mis-applying or over-applying a single test techniques in a way that doesn't yield adequate information.

I've provided an example below along with high quality tests.

High quality tests

I don't know of any summarized guidelines, although I could see a lot of value in this.

I am trying to learn how to write higher quality tests by finding materials (reading books, publications, etc.) on individual test techniques and trying to isolate what the most important attributes are.

Thanks to the Domain Testing Workbook I now know one of Domain Testing's (also called Boundary Analysis + Equivalence Class Partitioning) most important attributes is Power. Power (aka statistical power) in this case is selecting a small number of tests (values) that would have a high probability of causing or revealing failures. All domain tests should be powerful. If they aren't they are of low quality and wasteful.

Answer 4

When you are working to write high-quality tests, there are a few important factors that one must keep in mind to attain perfection. These include:

• Define naming standards for the tests simplifying the intent of the test

• Arrange your tests to clearly separate what is being tested from the arrange and assert steps.

• The input must be simple to ensure viable results are obtained through the test initiatives.

• Variables must be named to avoid any special values causing confusion.

• Avoid logic in tests to prevent bugs

• Create separate act for each test