Detecting tests that will never fail

Question

We have a rather big test codebase (~1500 test cases) for a Python/Django application.

From time to time we stumble upon existing tests that will never fail:

• tests with missing assertions
• tests that assert not what they supposed to assert
• logical errors resulting into assertions not being applied

Is there an effective (preferably automated) way to find these kind of tests?

Answer 1

I've recently opined, provocatively, that a test that never fails, adds no value.

https://durrantm.wordpress.com/2017/05/30/a-new-way-to-think-about-tests/

At a more practical level I look to use the following practices.

• When writing code, start with a failing tests as recommended in TDD. Really do this on an ongoing basis.
• When hacking around in code, occasionally 'force' the test in question to fail using a known value or piece of logic that will do that. This is a key practice for me to make sure I have a good test. You'll get a sense after a while when a test is feeling 'suspiciously green'.
• Similar to the previous point, when working on a test and application code for a long period and you've finally finished it, force the test to fail one last time manually and then change it back to passing.
• Be aware and alert for both false negatives (test fails but should actually pass) and false positives (test passes but should fail)
• Be careful when using negative assertions to see that something 'does not' exist. Be sure it include a positive assertion about something else that should exist otherwise error pages such as 404's will still show the test as passing (incorrectly).
• Code Review of all code with review by at least two additional parties for each ticket. You mention issues that you stumble across. Maybe you can add more eyes on the code to try and find them more up-front.
• If there is an issue with intermittent failures, consider running the test thousands of times to examine stability.

Ideas such as 'delete tests that haven't failed in a year' are provocative and have some attraction but in practice I've never seen it and really can't imagine actually doing that. I have reviewed test suites and decided to remove tests due to duplication, slowness, fragility, not representing user functions, etc. but the mere fact they haven't failed in x period of time doesn't some like a criteria I'd be in favor of or able to convince my pair

Other ideas to address your specific points:

• Tests with missing assertions
  You should be able to use your test framework to identify these. You may need to break out a tool like sed, awk or other search tools to help you.
• Tests that assert not what they supposed to assert
  Make sure you have code review
• Logical errors resulting into assertions not being applied
  Make sure tests fail first

Answer 2

Another idea which we have not yet applied in practice is Mutation Testing.

Mutation testing (or Mutation analysis or Program mutation) is used to design new software tests and evaluate the quality of existing software tests. Mutation testing involves modifying a program in small ways.

Which in practice means that if there was a mutation/change (typically a common programming error) introduced in a code and a test did not react to a change (or set of these changes) - it is a sign of a test that potentially is not properly testing a module under test.

In Python world, there is the cosmic-ray mutation testing tool.

Answer 3

Your situation is there because of Technical Debt. Everything you pointed out is controlled easily enough via Continual Review of your test suites. The programming world has words for this "Refactoring", "Code Reviews", "Test Result Reviews".

Here's the steps which will allow your team to start actively maintaining your test cases, suites and ultimately the test plans. The end result is absolute confidence in your product and a shift into automation.

1. Determine which test suites, test cases are the most important to your product. Rate them in priority order.
2. For each new iteration schedule some time for test case reviews. Open the code and or steps and run through them manually to determine if they are correct.
3. For each Test Case that isn't correct create a bug and put it in the backlog for now. The bug should list an acceptance criteria.
4. After you have enough bugs in the queue, rank them all. Pay attention to the concept "Most Value to the Product".
5. Each new iteration planning session pick off the top set of area you want to focus (from the bug queue), allowing for the proper effort based on available resources.
6. Within one year's time all tests of most value to the product would have been certified/reviewed to be proper.
7. Start an automation effort to automate the Tests, the automation effort will consolidate many tests into one DLL and implement re-usable patterns.
8. Hold code reviews for the automation work and get team approval of content.
9. Add new manual and or automated tests as the need presents itself, but now make sure your process for integration is one that addresses any additional Technical Debt.
10. Continuously Improve always!

Answer 4

I would like to present a generic system under test and its test suite as below:

• Test suite injects inputs into system under test
• System under test generates outputs based on inputs from test suite
• Test suite applies Assertions onto the system outputs
• Based on Assertions results, tests pass / fail

The symptoms of tests with missing assertions and logical errors resulting into assertions not being applied is

• No conclusive test result, an assertion produces True or False, or binary results in other forms.

Introduce an automated test result scan may help, e.g. Look for tests that produce no conclusive binary results.

Tests that assert not what they supposed to assert

• Introduce an automated script that inverts outputs under test from the system under tests; tests that are not asserting what they are supposed to assert should produce the same result as before. But I am not sure if it is possible to modify the system under test this way, or how much additional effort it is needed to do so.

Answer 5

I read somewhere a suggestion that any test which did not failed for a year should be a candidate for elimination.

Not automatically eliminated - maybe for some critical functionality you want multiple tests and multiple checks (defense in depth - assuming that one of the duplicate tests checking same condition might fail to detect the error) but "a" good candidate for elimination.

Answer 6

One system I worked on had a 100% failure framework, (actually it was a set of stubs output by the design tool), that was run as a part of the automated test suite, (Jenkins), with an evaluation criteria that the number of failures had to exactly equal the number of tests, (there was also a 1 test 1 result policy). Whenever the criteria was not met the test that did not give a fail result was revised.

Answer 7

Based on a number of questions you have submitted I think your best course of action is to engage with a company that specializes in Quality Assurance.

This question, like many of the questions you have asked, are indicative of much deeper systemic issues. You're trying to apply band-aids, but in reality it's time to amputate.