If you can test every line of [your product's] code via unit tests, wouldn't unit testing alone (theoretically) be sufficient?
Are there other "necessary" benchmarks of proper test coverage other than simply covering (ideally) all of your LOC?
Unit testing only isolates units in isolation. In these tests all dependencies to other units are mocked or stubbed out. So how do you know those units together do what they are supposed to do? Code tends to grow hierarchically in complexity, and with that growth comes more and more units working together, more and more groups of units (modules) working together, and along the growth things need to checked.
Hence you need integration tests for this. These tests check the configured set of working components work together correctly.
Finally, some sort of acceptance, validation, or verification tests should be written to check the software in the end does what it's supposed to do. Business (usually) drives the creation of software, so these are an important metric to any organization. For most web apps/sites this is usually the selenium UI tests.
I'm not aware of any metrics for Integration tests.
But surely acceptance/verification tests should always be 100% passing to ensure a functional software product according to a specification. But even then, their could be bugs in the specification. I find writing these tests really helps catch these bugs.
There are many cases where unit testing as you have described would not be "sufficient".
(And you haven't really defined what you mean by "sufficient" in this case. Good enough to move the code to Production? Good enough to pass it on to QA? Good enough to please your boss? Good enough to feel like you did a good job? Something else?)
In most practical cases, Unit Testing tests that the code the developer wrote actually does what the developer thinks it should do.
Even theoretically, unit testing isn't sufficient because it doesn't cover the paths through the system. You can cover every line of code in a system, but not cover every potential way through it.
For instance, can you launch the application? Does it run on all the target operating systems? Does it render correctly? These are all out of the scope of unit tests. In addition,
To take a very simple example, let's suppose your application is a calculator, similar to the basic version of the Microsoft calculator. You can unit test the math calculation - but not completely. A complete test of the calculations is impossible - just to test single digit basic operations you're looking at 400 distinct test cases (I think. It's possible my mental math is off here).
That doesn't even cover the quality of the unit tests - do they cover only what the code is supposed to do, or do they test the edge cases and potential failures?
Unit tests are invaluable, but they aren't all that's needed. As Joe and Craastad have said, they need to be part of a larger set of tests including integration tests, functional tests, user acceptance tests, compatibility tests, performance tests... and so forth.
I'll throw out a couple of definitions so I can answer the question with an esoteric and abstract response:
So clearly, if we have n independent functional pieces, then it suffices that we can perform n tests on these functional pieces of code. A test here is defined as the minimum number of actions that are needed to accurately determine the algorithm of each functional piece of code. (In other words, we only need to test independent components individually.)
Unfortunately, it is rarely the case that functional pieces of code are completely independent. Typically, there is some shared overlap that is involved between these functional pieces. However, we can transform each of these regions into independent regions by breaking them down into components that overlap, and components that don't overlap.
To see this via set theory notation, let AB be the intersection of two sets, and A' be the complement of A. We can then form ABC, A'BC, AB'C, ABC', A'B'C, A'BC', AB'C', A'B'C'. We can always perform this decomposition.
Anything that has 'all but one prime' is what people typically think of when they think of a unit test: AB'C' is the functionality of A that does not involve B or C. Integration tests typically involve more than one component, and so forth.
If you have n functional pieces of code, then there up to 2^n possible ways they could all overlap with each other. So if you manage to break these into 2^n independent functional pieces, and then test every piece, you'll have fully tested your product.
Good luck doing this in practice, however, for the number of possible arrangements will be quite large, and they won't always be easy to identify (and often not easy at all).
|show 1 more comment|