9

Inspired by Exercises on software testing

We can learn to write automated tests and we can learn to do tdd and bdd but what tools / resources / guides are available to tell us if the actual tests we are writing are of high quality?
What specific standards or guidelines exist for writing high quality tests?
What specific approaches (within tdd/bdd) result in higher quality tests?
Conversely, what tests are usually of a lower quality level?

  • 1
    When the application you're testing does well in the field. There is no other genuine proof. – Bookeater May 13 '17 at 11:26
5

The goal is to find bugs while leaving time and money to do other things too. So a high-quality test is a test that finds bugs but doesn't require a disproportionate expense to write, maintain, or operate.

Sometimes you can tell whether a test is low-quality just by looking at it. For example, if the test is buggy or written in a brittle way so that every change to the system requires a change to the test, it's a low-quality test.

You can't tell whether a test is high-quality without measuring the test's impact, i.e. measuring whether it finds bugs. A test can be really well-written but if it doesn't find bugs, it's not a good use of resources.

Of course, it's hard to know in advance whether a test will be high-quality because it's hard to know where the bugs will be.

3

Low quality tests

When I think of low quality tests, I think of teams mis-applying or over-applying a single test techniques in a way that doesn't yield adequate information.

I've provided an example below along with high quality tests.

High quality tests

I don't know of any summarized guidelines, although I could see a lot of value in this.

I am trying to learn how to write higher quality tests by finding materials (reading books, publications, etc.) on individual test techniques and trying to isolate what the most important attributes are.

Thanks to the Domain Testing Workbook I now know one of Domain Testing's (also called Boundary Analysis + Equivalence Class Partitioning) most important attributes is Power. Power (aka statistical power) in this case is selecting a small number of tests (values) that would have a high probability of causing or revealing failures. All domain tests should be powerful. If they aren't they are of low quality and wasteful.

3

It depends on the kinds of tests.

High quality unit tests have the following characteristics

  • They mock and stub out external dependencies such as the database
  • They run fast
  • They test a single piece of functionality

High quality UI feature tests have the following characteristics

  • They reflect real user usage, actions and workflows
  • They do not fail intermittently
  • Failures represent real user issues
  • Real user issues break the tests
  • They are robust in their use of HTML element selectors
  • They use a Page Object approach for selectors, actions and services

All high quality tests share the following characteristics

  • They are readable and also act as documentation of the system
  • They are run for each change to the software
  • They are reliable
  • They do not have false positives or negatives
  • They are written in plain English and generally avoid obscure terms and acronyms
  • They do not test obsolete or removed functionality
  • What would be considered "a single piece of functionality"? E.g. Filling out a web form or filling out a single field inside that form? – Mate Mrše Jul 4 '18 at 11:15
  • 1
    @MateMrše 'a single piece of functionality' is most applicable to unit tests as indicated in the list. UI tests have different priorities although they should still generally be testing a specific thing. In that context happy UI tests are testing one thing - that the entire form flow works. Sad UI tests are much more specific though, frequently testing that a specific field - or specific form - is working as intended. Thus that categorization kinda breaks down badly for UI tests. – Michael Durrant Jul 4 '18 at 11:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.