31

Please don't burn me for this simple question. I know, it may be very simple but for me it's complicated.

"A Good Test" or "A Bad Test" - every tester comes across these two terms.

Sometimes it becomes very difficult to figure out for an experienced tester whether it's a good test or a bad test. So here I have a very simple question:

What's the difference between a good test and a bad test?

  • 1
    What is your current understanding of the same? What is making complicated? Please edit the question and add the details. – Rao Nov 27 '17 at 12:32
  • 1
    and in what context.. – Rsf Nov 27 '17 at 12:41
  • 4
    But context does matter, the answer will be different for a regulated medical device tested by a dedicated team then for a small startup building a proof of concept for a mobile app. – Rsf Nov 27 '17 at 12:49
  • 2
    I disagree, and my answer attempts to explain why. – Kate Paulk Nov 27 '17 at 13:04
  • 4
    Unless you have a goal in mind, you shouldn't be creating a test at all. A good test achieves it's goal. A bad test does not. – Joe Strazzere Nov 28 '17 at 18:55

16 Answers 16

35

It depends

There are many factors that decide whether a test is "good" or "bad" (useful or not) - some examples are:

  • A test is more likely to be good if it surfaces new information about the performance of the software
    • The happy path tests you do when you're exploring new functionality become much less good once that functionality is stable, but can be useful to use for regression checking.
    • Tests for throwaway software (such as proof of concept apps) will probably be less useful than tests for software that is planned to be deployed.
    • Tests for high-risk scenarios (that is, with a high chance of occurring and a large impact if they happen) are likely to be more useful than test for low-risk scenarios.
  • A regression test is more likely to be good if it covers core functionality or functionality in the 20% of the software that is used by 80% of users.
  • A test is more likely to be useful if it echoes user behavior (just don't forget to include malicious users in that criteria).
  • A test is more likely to be useful if it covers parts of the application that aren't covered by other tests.

Beyond the general principles, there really aren't any "hard" rules. Modern software is often responsive and non-deterministic in that you can only predict behavior for small subsets of the total functionality (Because the full behavior set approaches infinite).

An experienced tester will look at how the software is used, the problems it is solving and the people it is solving them for, and test accordingly.

  • 1
    Tests are also more likely to be good if they are easier and faster to run. – chrylis Nov 27 '17 at 17:19
  • 1
    @chrylis - not necessarily. I can think of any number of useless tests that are easy and fast to run. They're useless because they don't provide any meaningful information. – Kate Paulk Nov 27 '17 at 19:55
  • 2
    A bad test will be flakey and give lots of false-positive failures. This will cause you to ignore the test failures, even if they are valid – David Grinberg Nov 28 '17 at 1:41
  • @KatePaulk Yes, but functionally equivalent tests that are fast (direct dependency injection instead of starting up a captive container, for example) are more likely to be run frequently. – chrylis Nov 28 '17 at 14:59
28

Great question.

Every test has a value and a cost. Its value is its ability to reduce risk. Here are some ways that a test can be valuable:

  • Impact. Detecting high-impact bugs is more valuable than detecting low-impact bugs, which is more valuable than not detecting any bugs.
  • Diagnostic value. A test that narrows down the cause of a bug is more valuable than a test that just reports "something went wrong".
  • Consistency. A test that yields consistent results is more valuable than one that is inconsistent.
  • Success rate. A test that detects bugs that actually occur is more valuable than a test that detects bugs that have never happened.

Here are some ways that a test can be expensive:

  • Resource-intensive. A test that requires a lot of compute-resource, or a lot of money, is more expensive than one that does not.
  • Time-intensive. A long-running test is more expensive that a short-running test.
  • Fragile. A test that requires a lot of maintenance is more expensive than a test you never have to update.

If you know the value and cost of a test, you can judge its goodness.

  • A test with high value and low cost is a great test.
  • A test with low value and high cost is a bad test.
  • A test with high value and high cost is just ok. For example, a Selenium test that covers the entire user interface but requires a lot of maintenance might be just ok. You might turn it into a better test by rewriting the test (or the UI) so that the test is less fragile.
  • A test with low value and low cost is just ok.

This exercise is hard because value and cost change over time. Some testers recommend reviewing test suites periodically to remove or rewrite bad tests.

  • Excellent answer; I'd just say that I'd look at tests as value - cost, not with value and cost on different axis. A test can be high cost, but really, really high impact, and be a great test. – Kevin McKenzie Dec 5 '17 at 20:39
  • Thanks. I put them on different axes because the units are different. Example: what is the quantity and units of the result of subtracting the 30 minutes of runtime from the ability to detect a high-risk bug? – user246 Dec 5 '17 at 21:18
  • Stable tests yielding consistent results are good for discovering small regressions, but incosistent, fragile, complex tests are often the ones discovering new unexpected bugs. We need both types of tests but we need to remember they serve different goals. See research from Microsoft on that: uploads.pnsqc.org/2016/papers/…. – dzieciou Dec 8 '17 at 19:17
6

A good test produces information whose value outweighs the cost of the test.

  • +1. Great, succinct answer. All other answer says the same in so much more words. and details. Cost of test: writing it, running it, maintaining it as tested system changes, analyzing failures. All these are not free. – Peter M. Nov 30 '17 at 15:18
  • +1 There might be long, fragile tests that produce information for finding new bugs. Even if their maitenance cost is high, running them from time to time and spending more time on analysis of failures might be worth. – dzieciou Dec 8 '17 at 19:54
4

Note that goodness and badness tends to be on a sliding scale but the more of the characteristics on the good list the better.

Characteristics of "Good" tests:

  • Has clear instructions on how & when to run it
  • Tests that desired behaviour is present
  • Tests that undesired behaviour is not present
  • Provides good isolation - if it fails you know where the problem is!
  • Is based on the requirements not the implementation.
  • Is maintainable and has good tracability so that if the specification changes you can quickly match it.
  • No False positives or negatives - i.e. If it passes then the desired behaviour is present and the undesirable is missing, if it fails then there really is a problem.
  • Provides rapid results at low cost.
  • Tests the most significant behaviour first
  • Either terminates on the first or nth problem or provides ongoing feedback on status so far.
  • Is interruptable but still provides useful if partial results
  • Provides good coverage of the item under test, possibly at the cost of extra time.
  • Is implemented with low cost, stable, long life, tools with good support
  • Is not platform dependant
  • Follows recognised good practices for the type of testing underway
  • Is non-invasive i.e. can be run on the deploy-able code without any changes

Characteristics of Bad Tests

  • Is undocumented or can only be run by the expert
  • Doesn't ensure that the desired behaviour is present
  • Doesn't ensure that the undesired behaviour is absent
  • Is based on the implementation not the requirements
  • Is hard to maintain &/or lacks traceability so that you don't know the impact of specification changes
  • Tells you that there is a problem but not what or where
  • Sometimes passes on bad items &/or fails on good.
  • Takes a long time &/or has a high cost
  • Doesn't give you any results or feedback until the end (then maybe you find out that there was a problem 3 weeks ago that could have been rectified).
  • If you interrupt it you can't tell how far it got and what the results so far were.
  • Only covers the "Good" path(s) through the code so gives low coverage.
  • Depends on expensive tools that have high annual costs and that change rapidly
  • Only runs on one specific machine
  • Doesn't match any recognised methodology
  • Requires a special build of the code that doesn't match the production code.
3

An automated test is an impediment to change.

A good test is a test that only fails on breaking/undesirable change: broken feature, regression, something that a user would consider a bug.

A bad test fails on random or desirable change: randomly, whenever someone refactors the code with no behavior change, when the environment it is executed in changes, when a new feature is introduced, etc. Too many bad tests means that development wastes time fixing tests any time they want to get anything done.

There are other practical aspects to consider: how fast does the test run, and how much data does it give about the origin of the failure (which change triggered the breakage) - but in my opinion they're secondary.

3

Characteristics of good tests are:

  • Behavior: They test behavior not implementation.
  • Understandable: When they fail you understand why they fail. The name is a description that helps when the test fails. Same goes for error/assert messages. A good failing test explains why it fails so non developers/testers can understand it as well.
  • One thing: A good test tests one thing, behaviour, use-case or unit. Not multiple. This is not the same as one assert, multiple asserts can test one thing.
  • Independent: A good test can be run on its own, making if it easier/faster to verify it has been fixed.
  • Fast: They are fast to execute.
  • Readable: They are short enough so it easy to spot what it tests, structured in a clear way. Use GivenWhenThen or AAA-pattern.
  • Maintainable: They need minimal updating when the application under test changes. You should not hate your tests.
  • Documentation: They can be used as a form of documentation.
  • No-flickering: They never fail for the wrong reason, e.g. it is repeatable.

Bad tests do the opposite of good tests.

I think these things count for manual as well as automated tests.

  • 1
    I'd put Automatable on here as well. If it's automatic, you can run it every time a developer commits to your development branch, which will catch errors much faster than if you have to push it to testing / staging / try to push to prod for it to be tested. – Nic Hartley Nov 29 '17 at 9:21
  • Short tests might be good for regression suites but often long tests find more unexpected bugs. See my answer and the paper from Microsoft: uploads.pnsqc.org/2016/papers/…. – dzieciou Dec 8 '17 at 19:45
2

As to me a good test is a test that catches errors and does not produce false positive or false negative results.

  • And what is a bad test? And what is the difference between these two? – Embedded Nov 27 '17 at 14:24
  • 1
    your answer is not wrong but over simplistic. For example I am ok with tests not detecting some, or even a lot, of failures as long as it reliably detects the important ones for me. – Rsf Nov 27 '17 at 15:18
  • I deliberately answered in simplified manner since otherwise the naswer is primarily opinion-based. There are to many conditions and circumstances which can be arguable from case-to-case and from experience-to-experience. There couldn't be the objectively corect answer. – Alexey R. Nov 27 '17 at 15:21
2

Good tests fail on bugs in one unit they are written for. They are not impacted by bugs in other units. These other units have they own tests. Good tests are also not as fragile as less good tests that may take significant effort to keep up to date as the legitimate behavior develops.

Bad tests are often better than no tests at all.

  • I dare to you to write system level tests that will direct to you to an issue in one module and only in one module. Actually a lot of times bad tests waste your time and efforts leading to less quality, on the other hand simple tests or tests with less coverage are many time better than no tests at all – Rsf Nov 28 '17 at 12:42
2

Automated or Manual/Explorative Tests?

A good automated test, often a unit test, is a piece of software. To be a good test, it must be programmed well -- readable, maintainable, reasonably bug-free, fast enough to run it often.

Some people argue that a good test makes few or no assumptions about the internal working of the system under test (often a class) and only test public methods, but few classes function in complete isolation. In order to provide mocks, the tester has to understand how the methods work. The automated test should cover the contract of the system under test and a reasonable selection of the paths through the methods. Covering all of them is usually too expensive.

A good test automation engineer has the training and mindset to look at the code and to select the data which provides a good enough coverage. Many programmers who are told "write tests for your classes" simply bring unit test coverage to 100% -- each instruction executed at least once.

A good manual test makes good use of (expensive) human labor. It covers the most critical parts of the system from a business viewpoint, and it uses the professional judgement of the tester.

For instance, I found writing automated test for responsive web design impractical. So instead I made a note that we need a manual test, "call the site on all test devices and resize the browser window where applicable. Decide if it looks acceptable, otherwise document the device and settings." And some poor schmuck got paid to do it. But not with every release, because that would be too expensive.

A good manual test tests many things at once. Where an automated test might log an user in and then close the browser, and the next test might log another user in and out and then close the browser, the manual test would cover login and logout as part of some other test case which requires a logged-in user.

2

To me a bad test is any 'test' that has no value what so ever. Any other test or check you can perform that has value, e.g. provides useful information is a good test.

And whether your test is useful depends on context. I can think of all kinds of very useful tests that are totally meaningless in a different context.

2

I believe good test development strategy resembles that for building up a quality regression test suite. Devising this test suite, testing teams apply Risk-based approach to selecting regression test cases in Agile projects. The teams usually choose test cases that :

  1. cover key product functions
  2. are not duplicate
  3. stably detected bugs in the previous iterations

I think the same principles can be applied to devising any test. A good test should fully cover the product functions, include test cases that hardly miss bugs and that are unique, i.e. test cases are not repeated throughout the suite. The last, but not the least: a good test should be maintainable, as the applications functionality may change oftentimes.

2

Automated tests serve two goals:

  • gate control -- they decide when to release new software or move the software from one test environment to another
  • discovering new bugs -- they help identifying new unexpected bugs

Each goal has different requirements and thus tests that are often good for discovering new bugs might be bad for gate control.

Most of the answers here discuss only tests for gate control, called regression tests, so I will only summarize what has been written here. Tests for gate control need to be precise, deterministic, stable, and fast, with assertions written explicitly.

Tests for discovering new bugs are different. They have higher recall but lower precision: they can have more failures but those failures might be false negatives. Hence, failures require more analysis and troubleshooting. Tests for discovering new bugs are usually longer, touching multiple parts of the system and non-deterministic (e.g., can have an element of randomness like fuzzy tests). They fail when something unexpected occurs, but unlike in regression tests, you don't have all assertions written explicitly.

Wayne Matthias Roseberry from Microsoft writes about this in his article "Winning with Flaky Test Automation".

2

The truth is that writing good tests is as hard as writing good production code. It takes practice and it requires careful thought, but so many times I have seen how tests are treated as second class citizens, which usually means that we do not reap the whole benefits of a good test suite, ending up with suites that are more of a burden than an asset.

A good test suite:

  • Gives you confidence.
  • Finds bugs.
  • Is cheap to maintain.
  • Document the system.
  • Robustness.
  • Does not have bugs(test it self).
2

There are some great answers here.

Mark Winteringham wrote about this over on the Dojo recently in What's The Difference Between A Good And A Bad Test.

The important thing to note with any test is what risk is it addressing. This can open the floor to many other questions which will then expand your tests and ideas to strengthen the approach.

1

In my view, a good test provides a uniquely valuable(to the business) information about the state of the system under test.

It is good, till the provided information is valuable and unique(not redundant).

-3

For example, you wan to test any ecommerce website. So you just need to understand 2 type of user activity and actions.

Bad Test : Bad user, Dont want to puchase product but want to break your product, negative activity, Too Many Assumption, Confusion flow, Complex flow, Bad user inputs, Hackers

Good Test : User stories, Business Rules, Good User, Want to purchase product, Business Scenarios etc

So in my eyes, it is important to put both Good and Bad Test Hat !!! If you can make bad test case then it is also important because it is coverage for useless or bad activity but it is still some kind of action in real world.

  • 1
    I'm not sure you answering the right question, the OP asked about if a test is good for its purpose, not the test's point of view – Rsf Nov 27 '17 at 15:16
  • 1
    Test for bad user inputs is a very good test! – Embedded Nov 27 '17 at 15:40
  • 2
    I think you are confused with positive and negative tests. – Niels van Reijmersdal Nov 29 '17 at 7:16

protected by alecxe Dec 8 '17 at 23:50

Thank you for your interest in this question. Because it has attracted low-quality or spam answers that had to be removed, posting an answer now requires 10 reputation on this site (the association bonus does not count).

Would you like to answer one of these unanswered questions instead?

Not the answer you're looking for? Browse other questions tagged or ask your own question.