How can 100% test coverage be "not enough to prove working code"?

Question

It is known that 100% test/code coverage does not mean your code "works" as expected.

What is generally meant by this statement and what use cases could demonstrate that 100% coverage is not enough?

Answer 1

In addition to the other answers, a few extra thoughts:

• 100% coverage does not mean it meets user expectations - Your tests could cover 100% of the code, but that does not mean that the application satisfies potential users. The application could:
  • have performance issues under load
  • have workflows that users find too awkward or too difficult to use
  • not include information users expect to see
  • be insecure or not secure enough
  • be missing functionality users want or need
  • have issues handling data outside the test data parameters
  • have memory leaks causing crashes after a period of time
• 100% unit test coverage does not mean integration tests will run - If MethodA does not provide MethodB with the correct parameters, the unit tests for both could succeed where integration tests fail.
• 100% unit test coverage does not mean the application will run - Unit tests run inside a harness that exercises specifically those units. It's possible to miss setting a reference to be bundled in with the deployment package and have an application with 100% unit test coverage (all passing) that can't run.
• 100% coverage does not prevent display issues - You can run into display issues when the user's screen size isn't what you expected, or the user runs the application on a system that doesn't have the fonts you used, or doesn't use the same encoding as you. These are not things that get caught with unit tests - or integration tests, or most automated functional tests for that matter.
• 100% coverage is not 100% path coverage - Most applications have an infinite or near-infinite number of paths through the application. It's not practical to test for a user opening and closing a screen ten times before they start entering data, then 11 times, then 12, then... It's not practical to test using the back and forward browser buttons in every possible combination while entering data.
• Test coverage only catches what's been coded - You can't test functionality that hasn't been implemented. If an error scenario isn't coded for, it can't be covered in test cases - but an experienced tester could realize it's not there.

Of course, these are all also reasons why it's impossible to test everything, which is at least in part the point. As I see it, 100% test coverage is like driving over every street in a town. You might be able to drive over each street at least once, but that doesn't mean you'll be able to find your way from where you are to a random address in the town, and every user will be going from a different starting point to a different end point, by a different route, at a different speed, in a different vehicle, with a different amount of fuel in the tank.

Answer 2

Thinking of 100% test coverage as a holy grail of testing is a common misconception which leads to over-confidence in the testing strategy and a false sense of security:

• code coverage metric only tells you what percentage of your code was executed. That's it - this is all that the coverage metric determines. The actual value of this depends on how your tests are written, what input values are passed in etc.
• one of the many examples of when the 100% code coverage is not enough could be any code that uses sub-languages like regular expressions where your pattern behaves differently depending on the input. Another example could be concurrent code with non-deterministic behavior changing from execution to execution
• there are also partially covered lines like ternary operators which may appear covered if one of the branches was covered (at least, in Python coverage)
• 100% code coverage could also potentially be a warning sign of a wrong approach to the metric - it could mean that the last few percents were closed down with an idea of chasing the "ideal 100%" cutting corners and over-mocking while trying to hit the difficult-to-reach lines

What else could you do?

• prioritize achieving higher business requirements coverage than literal unit-test code coverage

• instead of example-based approach to thinking about your test inputs, explore Property Based Testing where you could define the properties of the desired inputs and let the tools generate inputs:

  Instead of asking “what is the expected result of this program given these example inputs?”, we ask the question: “what is the property of the result and program that doesn’t change given inputs?

• for certain type of projects (e.g. compilers), ideas like Random Testing could help reveal difficult-to-find issues

• Mutation testing is yet another technique to check the quality of your tests where the code under test is changed/mutated for the purpose of seeing if the tests would change its behavior

---

Other resources on the subject:

• Is 100% code coverage a really good thing when doing unit tests?
• Test&Code episode: When 100% test coverage just isn't enough - Mahmoud Hashemi
• Is 100% code coverage a pipe dream?

Answer 3

Normally the app is not just the code, but infrastructure, resources, protocols, databases, etc. Usability, performance, stability aspects are not usually a matter of code coverage as well.

Answer 4

Low quality test cases could result high coverage. One could write a test case to cover a piece of source code but this does not mean necessarily that this test case actually verifies the correct and expected behavior of that specific code part.

So unless a test case is specified, designed and then implemented thoroughly, the value of the coverage it actually introduces could be uncertain.

Answer 5

When you begin without unit tests, the inclusion metric can help. You'll feel a feeling of achievement as you cure the circumstance and go from zero percent to 25 percent to 70 percent. Stamp your advancement and praise your successes.

However, as you begin coming to the heart of the matter where you're discussing whether to stop at 85 percent, 95 percent, or 100 percent, you're beginning to ask the wrong inquiry. You're beginning to ask how far you ought to go to satisfy a measurement producing apparatus. Rather you ought to ask how you can legitimize each line of code you make and how you can keep a circumstance in which your clients pioneer striking new trails through your code. How would you in any event come to them with the affirmation that you've executed the majority of the code in your codebase? That is unquestionably not all that a lot to inquire.

Answer 6

On top of other excellent answers ,I want to share an real story depicting where 100% coverage is not enough:

I came across a situation in a start-up project, where a well tested(unit/integration) page was brought into consideration first time to Business owner.Due to some project constraints it was entirely designed by a Sr. developer.

The page was working fine as fully covered by detailed tests on multiple levels as per the functionality described by the developer.

However as discovered in business review ,that's not how a real user would use the application on the production floor in busy hours.Entire page was scrapped.

Moral of the story: When requirement itself is incorrect, no amount of coverage will suffice , even 100%.