5

It is known that 100% test/code coverage does not mean your code "works" as expected.

What is generally meant by this statement and what use cases could demonstrate that 100% coverage is not enough?

7
+100

In addition to the other answers, a few extra thoughts:

  • 100% coverage does not mean it meets user expectations - Your tests could cover 100% of the code, but that does not mean that the application satisfies potential users. The application could:
    • have performance issues under load
    • have workflows that users find too awkward or too difficult to use
    • not include information users expect to see
    • be insecure or not secure enough
    • be missing functionality users want or need
    • have issues handling data outside the test data parameters
    • have memory leaks causing crashes after a period of time
  • 100% unit test coverage does not mean integration tests will run - If MethodA does not provide MethodB with the correct parameters, the unit tests for both could succeed where integration tests fail.
  • 100% unit test coverage does not mean the application will run - Unit tests run inside a harness that exercises specifically those units. It's possible to miss setting a reference to be bundled in with the deployment package and have an application with 100% unit test coverage (all passing) that can't run.
  • 100% coverage does not prevent display issues - You can run into display issues when the user's screen size isn't what you expected, or the user runs the application on a system that doesn't have the fonts you used, or doesn't use the same encoding as you. These are not things that get caught with unit tests - or integration tests, or most automated functional tests for that matter.
  • 100% coverage is not 100% path coverage - Most applications have an infinite or near-infinite number of paths through the application. It's not practical to test for a user opening and closing a screen ten times before they start entering data, then 11 times, then 12, then... It's not practical to test using the back and forward browser buttons in every possible combination while entering data.
  • Test coverage only catches what's been coded - You can't test functionality that hasn't been implemented. If an error scenario isn't coded for, it can't be covered in test cases - but an experienced tester could realize it's not there.

Of course, these are all also reasons why it's impossible to test everything, which is at least in part the point. As I see it, 100% test coverage is like driving over every street in a town. You might be able to drive over each street at least once, but that doesn't mean you'll be able to find your way from where you are to a random address in the town, and every user will be going from a different starting point to a different end point, by a different route, at a different speed, in a different vehicle, with a different amount of fuel in the tank.

  • Really like that driving through streets in a town analogy, very nice! Thank you, great answer, as usual! – alecxe Dec 19 '18 at 17:13
  • Well explained. – Joe Dec 20 '18 at 3:44
8

Thinking of 100% test coverage as a holy grail of testing is a common misconception which leads to over-confidence in the testing strategy and a false sense of security:

  • code coverage metric only tells you what percentage of your code was executed. That's it - this is all that the coverage metric determines. The actual value of this depends on how your tests are written, what input values are passed in etc.
  • one of the many examples of when the 100% code coverage is not enough could be any code that uses sub-languages like regular expressions where your pattern behaves differently depending on the input. Another example could be concurrent code with non-deterministic behavior changing from execution to execution
  • there are also partially covered lines like ternary operators which may appear covered if one of the branches was covered (at least, in Python coverage)
  • 100% code coverage could also potentially be a warning sign of a wrong approach to the metric - it could mean that the last few percents were closed down with an idea of chasing the "ideal 100%" cutting corners and over-mocking while trying to hit the difficult-to-reach lines

What else could you do?

  • prioritize achieving higher business requirements coverage than literal unit-test code coverage
  • instead of example-based approach to thinking about your test inputs, explore Property Based Testing where you could define the properties of the desired inputs and let the tools generate inputs:

    Instead of asking “what is the expected result of this program given these example inputs?”, we ask the question: “what is the property of the result and program that doesn’t change given inputs?

  • for certain type of projects (e.g. compilers), ideas like Random Testing could help reveal difficult-to-find issues

  • Mutation testing is yet another technique to check the quality of your tests where the code under test is changed/mutated for the purpose of seeing if the tests would change its behavior


Other resources on the subject:

  • 2
    My favorite analogy is spices - if you're a chef and you've got 100% of test coverage on your spices, all that's really said is you've added them to a dish and it didn't ruin the dish. What happens when you add multiple spices to the same dish? You might have testing of "multiple spices" with 100% coverage but you won't have 100% coverage of all combinations of spices. Adding onion and garlic is different than adding jalapeno and chili powder! – corsiKa Dec 18 '18 at 21:22
  • 3
    @Corsika,this spice analogy is actually more confusing then clarifying in this context. – Vishal Aggarwal Dec 19 '18 at 1:19
3

Normally the app is not just the code, but infrastructure, resources, protocols, databases, etc. Usability, performance, stability aspects are not usually a matter of code coverage as well.

1

Low quality test cases could result high coverage. One could write a test case to cover a piece of source code but this does not mean necessarily that this test case actually verifies the correct and expected behavior of that specific code part.

So unless a test case is specified, designed and then implemented thoroughly, the value of the coverage it actually introduces could be uncertain.

1

When you begin without unit tests, the inclusion metric can help. You'll feel a feeling of achievement as you cure the circumstance and go from zero percent to 25 percent to 70 percent. Stamp your advancement and praise your successes.

However, as you begin coming to the heart of the matter where you're discussing whether to stop at 85 percent, 95 percent, or 100 percent, you're beginning to ask the wrong inquiry. You're beginning to ask how far you ought to go to satisfy a measurement producing apparatus. Rather you ought to ask how you can legitimize each line of code you make and how you can keep a circumstance in which your clients pioneer striking new trails through your code. How would you in any event come to them with the affirmation that you've executed the majority of the code in your codebase? That is unquestionably not all that a lot to inquire.

1

On top of other excellent answers ,I want to share an real story depicting where 100% coverage is not enough:

I came across a situation in a start-up project, where a well tested(unit/integration) page was brought into consideration first time to Business owner.Due to some project constraints it was entirely designed by a Sr. developer.

The page was working fine as fully covered by detailed tests on multiple levels as per the functionality described by the developer.

However as discovered in business review ,that's not how a real user would use the application on the production floor in busy hours.Entire page was scrapped.

Moral of the story: When requirement itself is incorrect, no amount of coverage will suffice , even 100%.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.