33

Please don't burn me for this simple question. I know, it may be very simple but for me it's complicated.

"A Good Test" or "A Bad Test" - every tester comes across these two terms.

Sometimes it becomes very difficult to figure out for an experienced tester whether it's a good test or a bad test. So here I have a very simple question:

What's the difference between a good test and a bad test?

Improve this question
9
  • 1
    What is your current understanding of the same? What is making complicated? Please edit the question and add the details.
    – Rao
    Nov 27, 2017 at 12:32
  • 1
    and in what context..
    – Rsf
    Nov 27, 2017 at 12:41
  • 4
    But context does matter, the answer will be different for a regulated medical device tested by a dedicated team then for a small startup building a proof of concept for a mobile app.
    – Rsf
    Nov 27, 2017 at 12:49
  • 2
    I disagree, and my answer attempts to explain why.
    – Kate Paulk ♦
    Nov 27, 2017 at 13:04
  • 4
    Unless you have a goal in mind, you shouldn't be creating a test at all. A good test achieves it's goal. A bad test does not.
    – Joe Strazzere
    Nov 28, 2017 at 18:55

15 Answers 15

Reset to default
35

It depends

There are many factors that decide whether a test is "good" or "bad" (useful or not) - some examples are:

  • A test is more likely to be good if it surfaces new information about the performance of the software
    • The happy path tests you do when you're exploring new functionality become much less good once that functionality is stable, but can be useful to use for regression checking.
    • Tests for throwaway software (such as proof of concept apps) will probably be less useful than tests for software that is planned to be deployed.
    • Tests for high-risk scenarios (that is, with a high chance of occurring and a large impact if they happen) are likely to be more useful than test for low-risk scenarios.
  • A regression test is more likely to be good if it covers core functionality or functionality in the 20% of the software that is used by 80% of users.
  • A test is more likely to be useful if it echoes user behavior (just don't forget to include malicious users in that criteria).
  • A test is more likely to be useful if it covers parts of the application that aren't covered by other tests.

Beyond the general principles, there really aren't any "hard" rules. Modern software is often responsive and non-deterministic in that you can only predict behavior for small subsets of the total functionality (Because the full behavior set approaches infinite).

An experienced tester will look at how the software is used, the problems it is solving and the people it is solving them for, and test accordingly.

Improve this answer
4
  • 1
    Tests are also more likely to be good if they are easier and faster to run.
    – chrylis -cautiouslyoptimistic-
    Nov 27, 2017 at 17:19
  • 1
    @chrylis - not necessarily. I can think of any number of useless tests that are easy and fast to run. They're useless because they don't provide any meaningful information.
    – Kate Paulk ♦
    Nov 27, 2017 at 19:55
  • 3
    A bad test will be flakey and give lots of false-positive failures. This will cause you to ignore the test failures, even if they are valid
    – David says Reinstate Monica
    Nov 28, 2017 at 1:41
  • @KatePaulk Yes, but functionally equivalent tests that are fast (direct dependency injection instead of starting up a captive container, for example) are more likely to be run frequently.
    – chrylis -cautiouslyoptimistic-
    Nov 28, 2017 at 14:59
31

Great question.

Every test has a value and a cost. Its value is its ability to reduce risk. Here are some ways that a test can be valuable:

  • Impact. Detecting high-impact bugs is more valuable than detecting low-impact bugs, which is more valuable than not detecting any bugs.
  • Diagnostic value. A test that narrows down the cause of a bug is more valuable than a test that just reports "something went wrong".
  • Consistency. A test that yields consistent results is more valuable than one that is inconsistent.
  • Success rate. A test that detects bugs that actually occur is more valuable than a test that detects bugs that have never happened.

Here are some ways that a test can be expensive:

  • Resource-intensive. A test that requires a lot of compute-resource, or a lot of money, is more expensive than one that does not.
  • Time-intensive. A long-running test is more expensive that a short-running test.
  • Fragile. A test that requires a lot of maintenance is more expensive than a test you never have to update.

If you know the value and cost of a test, you can judge its goodness.

  • A test with high value and low cost is a great test.
  • A test with low value and high cost is a bad test.
  • A test with high value and high cost is just ok. For example, a Selenium test that covers the entire user interface but requires a lot of maintenance might be just ok. You might turn it into a better test by rewriting the test (or the UI) so that the test is less fragile.
  • A test with low value and low cost is just ok.

This exercise is hard because value and cost change over time. Some testers recommend reviewing test suites periodically to remove or rewrite bad tests.

Improve this answer
3
  • Excellent answer; I'd just say that I'd look at tests as value - cost, not with value and cost on different axis. A test can be high cost, but really, really high impact, and be a great test.
    – Kevin McKenzie
    Dec 5, 2017 at 20:39
  • Thanks. I put them on different axes because the units are different. Example: what is the quantity and units of the result of subtracting the 30 minutes of runtime from the ability to detect a high-risk bug?
    – user246
    Dec 5, 2017 at 21:18
  • Stable tests yielding consistent results are good for discovering small regressions, but incosistent, fragile, complex tests are often the ones discovering new unexpected bugs. We need both types of tests but we need to remember they serve different goals. See research from Microsoft on that: uploads.pnsqc.org/2016/papers/….
    – dzieciou
    Dec 8, 2017 at 19:17
8

A good test produces information whose value outweighs the cost of the test.

Improve this answer
2
  • +1. Great, succinct answer. All other answer says the same in so much more words. and details. Cost of test: writing it, running it, maintaining it as tested system changes, analyzing failures. All these are not free.
    – Peter M. - stands for Monica
    Nov 30, 2017 at 15:18
  • +1 There might be long, fragile tests that produce information for finding new bugs. Even if their maitenance cost is high, running them from time to time and spending more time on analysis of failures might be worth.
    – dzieciou
    Dec 8, 2017 at 19:54
6

Note that goodness and badness tends to be on a sliding scale but the more of the characteristics on the good list the better.

Characteristics of "Good" tests:

  • Has clear instructions on how & when to run it
  • Tests that desired behaviour is present
  • Tests that undesired behaviour is not present
  • Provides good isolation - if it fails you know where the problem is!
  • Is based on the requirements not the implementation.
  • Is maintainable and has good tracability so that if the specification changes you can quickly match it.
  • No False positives or negatives - i.e. If it passes then the desired behaviour is present and the undesirable is missing, if it fails then there really is a problem.
  • Provides rapid results at low cost.
  • Tests the most significant behaviour first
  • Either terminates on the first or nth problem or provides ongoing feedback on status so far.
  • Is interruptable but still provides useful if partial results
  • Provides good coverage of the item under test, possibly at the cost of extra time.
  • Is implemented with low cost, stable, long life, tools with good support
  • Is not platform dependant
  • Follows recognised good practices for the type of testing underway
  • Is non-invasive i.e. can be run on the deploy-able code without any changes

Characteristics of Bad Tests

  • Is undocumented or can only be run by the expert
  • Doesn't ensure that the desired behaviour is present
  • Doesn't ensure that the undesired behaviour is absent
  • Is based on the implementation not the requirements
  • Is hard to maintain &/or lacks traceability so that you don't know the impact of specification changes
  • Tells you that there is a problem but not what or where
  • Sometimes passes on bad items &/or fails on good.
  • Takes a long time &/or has a high cost
  • Doesn't give you any results or feedback until the end (then maybe you find out that there was a problem 3 weeks ago that could have been rectified).
  • If you interrupt it you can't tell how far it got and what the results so far were.
  • Only covers the "Good" path(s) through the code so gives low coverage.
  • Depends on expensive tools that have high annual costs and that change rapidly
  • Only runs on one specific machine
  • Doesn't match any recognised methodology
  • Requires a special build of the code that doesn't match the production code.
Improve this answer
3

An automated test is an impediment to change.

A good test is a test that only fails on breaking/undesirable change: broken feature, regression, something that a user would consider a bug.

A bad test fails on random or desirable change: randomly, whenever someone refactors the code with no behavior change, when the environment it is executed in changes, when a new feature is introduced, etc. Too many bad tests means that development wastes time fixing tests any time they want to get anything done.

There are other practical aspects to consider: how fast does the test run, and how much data does it give about the origin of the failure (which change triggered the breakage) - but in my opinion they're secondary.

Improve this answer
3

Characteristics of good tests are:

  • Behavior: They test behavior not implementation.
  • Understandable: When they fail you understand why they fail. The name is a description that helps when the test fails. Same goes for error/assert messages. A good failing test explains why it fails so non developers/testers can understand it as well.
  • One thing: A good test tests one thing, behaviour, use-case or unit. Not multiple. This is not the same as one assert, multiple asserts can test one thing.
  • Independent: A good test can be run on its own, making if it easier/faster to verify it has been fixed.
  • Fast: They are fast to execute.
  • Readable: They are short enough so it easy to spot what it tests, structured in a clear way. Use GivenWhenThen or AAA-pattern.
  • Maintainable: They need minimal updating when the application under test changes. You should not hate your tests.
  • Documentation: They can be used as a form of documentation.
  • No-flickering: They never fail for the wrong reason, e.g. it is repeatable.

Bad tests do the opposite of good tests.

I think these things count for manual as well as automated tests.

Improve this answer
2
  • 1
    I'd put Automatable on here as well. If it's automatic, you can run it every time a developer commits to your development branch, which will catch errors much faster than if you have to push it to testing / staging / try to push to prod for it to be tested.
    – anon
    Nov 29, 2017 at 9:21
  • Short tests might be good for regression suites but often long tests find more unexpected bugs. See my answer and the paper from Microsoft: uploads.pnsqc.org/2016/papers/….
    – dzieciou
    Dec 8, 2017 at 19:45
3

There are some great answers here.

Mark Winteringham wrote about this over on the Dojo recently in What's The Difference Between A Good And A Bad Test.

The important thing to note with any test is what risk is it addressing. This can open the floor to many other questions which will then expand your tests and ideas to strengthen the approach.

Improve this answer
2

As to me a good test is a test that catches errors and does not produce false positive or false negative results.

Improve this answer
3
  • And what is a bad test? And what is the difference between these two?
    – Embedded
    Nov 27, 2017 at 14:24
  • 1
    your answer is not wrong but over simplistic. For example I am ok with tests not detecting some, or even a lot, of failures as long as it reliably detects the important ones for me.
    – Rsf
    Nov 27, 2017 at 15:18
  • I deliberately answered in simplified manner since otherwise the naswer is primarily opinion-based. There are to many conditions and circumstances which can be arguable from case-to-case and from experience-to-experience. There couldn't be the objectively corect answer.
    – Alexey R.
    Nov 27, 2017 at 15:21
2

Good tests fail on bugs in one unit they are written for. They are not impacted by bugs in other units. These other units have they own tests. Good tests are also not as fragile as less good tests that may take significant effort to keep up to date as the legitimate behavior develops.

Bad tests are often better than no tests at all.

Improve this answer
1
  • I dare to you to write system level tests that will direct to you to an issue in one module and only in one module. Actually a lot of times bad tests waste your time and efforts leading to less quality, on the other hand simple tests or tests with less coverage are many time better than no tests at all
    – Rsf
    Nov 28, 2017 at 12:42
2

Automated or Manual/Explorative Tests?

A good automated test, often a unit test, is a piece of software. To be a good test, it must be programmed well -- readable, maintainable, reasonably bug-free, fast enough to run it often.

Some people argue that a good test makes few or no assumptions about the internal working of the system under test (often a class) and only test public methods, but few classes function in complete isolation. In order to provide mocks, the tester has to understand how the methods work. The automated test should cover the contract of the system under test and a reasonable selection of the paths through the methods. Covering all of them is usually too expensive.

A good test automation engineer has the training and mindset to look at the code and to select the data which provides a good enough coverage. Many programmers who are told "write tests for your classes" simply bring unit test coverage to 100% -- each instruction executed at least once.

A good manual test makes good use of (expensive) human labor. It covers the most critical parts of the system from a business viewpoint, and it uses the professional judgement of the tester.

For instance, I found writing automated test for responsive web design impractical. So instead I made a note that we need a manual test, "call the site on all test devices and resize the browser window where applicable. Decide if it looks acceptable, otherwise document the device and settings." And some poor schmuck got paid to do it. But not with every release, because that would be too expensive.

A good manual test tests many things at once. Where an automated test might log an user in and then close the browser, and the next test might log another user in and out and then close the browser, the manual test would cover login and logout as part of some other test case which requires a logged-in user.

Improve this answer
2

To me a bad test is any 'test' that has no value what so ever. Any other test or check you can perform that has value, e.g. provides useful information is a good test.

And whether your test is useful depends on context. I can think of all kinds of very useful tests that are totally meaningless in a different context.

Improve this answer
2

I believe good test development strategy resembles that for building up a quality regression test suite. Devising this test suite, testing teams apply Risk-based approach to selecting regression test cases in Agile projects. The teams usually choose test cases that :

  1. cover key product functions
  2. are not duplicate
  3. stably detected bugs in the previous iterations

I think the same principles can be applied to devising any test. A good test should fully cover the product functions, include test cases that hardly miss bugs and that are unique, i.e. test cases are not repeated throughout the suite. The last, but not the least: a good test should be maintainable, as the applications functionality may change oftentimes.

Improve this answer
2

Automated tests serve two goals:

  • gate control -- they decide when to release new software or move the software from one test environment to another
  • discovering new bugs -- they help identifying new unexpected bugs

Each goal has different requirements and thus tests that are often good for discovering new bugs might be bad for gate control.

Most of the answers here discuss only tests for gate control, called regression tests, so I will only summarize what has been written here. Tests for gate control need to be precise, deterministic, stable, and fast, with assertions written explicitly.

Tests for discovering new bugs are different. They have higher recall but lower precision: they can have more failures but those failures might be false negatives. Hence, failures require more analysis and troubleshooting. Tests for discovering new bugs are usually longer, touching multiple parts of the system and non-deterministic (e.g., can have an element of randomness like fuzzy tests). They fail when something unexpected occurs, but unlike in regression tests, you don't have all assertions written explicitly.

Wayne Matthias Roseberry from Microsoft writes about this in his article "Winning with Flaky Test Automation".

Improve this answer
2

The truth is that writing good tests is as hard as writing good production code. It takes practice and it requires careful thought, but so many times I have seen how tests are treated as second class citizens, which usually means that we do not reap the whole benefits of a good test suite, ending up with suites that are more of a burden than an asset.

A good test suite:

  • Gives you confidence.
  • Finds bugs.
  • Is cheap to maintain.
  • Document the system.
  • Robustness.
  • Does not have bugs(test it self).
Improve this answer
1

In my view, a good test provides a uniquely valuable(to the business) information about the state of the system under test.

It is good, till the provided information is valuable and unique(not redundant).

Improve this answer

Not the answer you're looking for? Browse other questions tagged or ask your own question.