Is unit and component testing sufficient?

Question

If you can test every line of [your product's] code via unit tests, wouldn't unit testing alone (theoretically) be sufficient?

Are there other "necessary" benchmarks of proper test coverage other than simply covering (ideally) all of your LOC?

Answer 1

Not exactly.

Unit testing only tests units in isolation. In these tests all dependencies to other units are mocked or stubbed out. So how do you know those units together do what they are supposed to do? Code tends to grow hierarchically in complexity, and with that growth comes more and more units working together, more and more groups of units (modules) working together, and along the growth things need to checked.

Hence you need integration tests for this. These tests check the configured set of working components work together correctly.

Finally, some sort of acceptance, validation, or verification tests should be written to check the software in the end does what it's supposed to do. Business (usually) drives the creation of software, so these are an important metric to any organization. For most web apps/sites this is usually the selenium UI tests.

Metrics are always dangerous, you can always find a way to beat them. It's very easy to write a bunch of code with 100% test coverage that does nothing useful together. So the difference between quality of 60%, 70%, 95%, or 100% test coverage is tough to define. Some argue code should always be 100% covered, while some say 70-80% is sufficient, it's all about project/personal preference. The amount of coverage for each test type depends on situation and project. I find myself with some tasks writing 100% unit tested code (production code, new features), 0% unit tested code (for writing UI tests), ~50% integration tested code (refactoring legacy code, javascript), etc.

I'm not aware of any metrics for Integration tests.

But surely acceptance/verification tests should always be 100% passing to ensure a functional software product according to a specification. But even then, their could be bugs in the specification. I find writing these tests really helps catch these bugs.

Answer 2

There are many cases where unit testing as you have described would not be "sufficient".

(And you haven't really defined what you mean by "sufficient" in this case. Good enough to move the code to Production? Good enough to pass it on to QA? Good enough to please your boss? Good enough to feel like you did a good job? Something else?)

In most practical cases, Unit Testing tests that the code the developer wrote actually does what the developer thinks it should do.

• What if the developer misinterpreted the Requirements? The code might well pass all Unit Tests, but be completely incorrect.
• Testing every line isn't the same as testing every line under all conditions. If you test a line of code where the input data is "1", what happens to that same line when the input data is "-23456789"?
• Refer to https://en.wikipedia.org/wiki/Code_coverage for a deeper discussion about code coverage (beyond just "testing every line")
• What about performance and load? Most unit tests don't cover non-functional aspects.
• What about integration issues that arise during the interaction among several units? Unit testing usually doesn't cover that.
• etc, etc.

Answer 3

Even theoretically, unit testing isn't sufficient because it doesn't cover the paths through the system. You can cover every line of code in a system, but not cover every potential way through it.

For instance, can you launch the application? Does it run on all the target operating systems? Does it render correctly? These are all out of the scope of unit tests. In addition,

To take a very simple example, let's suppose your application is a calculator, similar to the basic version of the Microsoft calculator. You can unit test the math calculation - but not completely. A complete test of the calculations is impossible - just to test single digit basic operations you're looking at 400 distinct test cases (I think. It's possible my mental math is off here).

That doesn't even cover the quality of the unit tests - do they cover only what the code is supposed to do, or do they test the edge cases and potential failures?

Unit tests are invaluable, but they aren't all that's needed. As Joe and Craastad have said, they need to be part of a larger set of tests including integration tests, functional tests, user acceptance tests, compatibility tests, performance tests... and so forth.

Answer 4

I'll throw out a couple of definitions so I can answer the question with an esoteric and abstract response:

1. A functional piece of code is any operation that makes changes to a finite system (in other words, a computer.) It is important to realize that there are only a finite number of configurations that can occur on a computer. Obviously, a functional piece of code will not operate on the entire computer, so this finite system is often restricted to be some subset of the computer. This restriction is typically dependent on the functional piece of code itself, and not any grouping of functional pieces. The changes that a functional piece of code makes is referred to as an algorithm.

2. Two functional pieces of code are said to be independent if the restricted systems above are disjoint. This comes out of the definition of the restriction, and this definition should be immediately clear.

So clearly, if we have n independent functional pieces, then it suffices that we can perform n tests on these functional pieces of code. A test here is defined as the minimum number of actions that are needed to accurately determine the algorithm of each functional piece of code. (In other words, we only need to test independent components individually.)

Unfortunately, it is rarely the case that functional pieces of code are completely independent. Typically, there is some shared overlap that is involved between these functional pieces. However, we can transform each of these regions into independent regions by breaking them down into components that overlap, and components that don't overlap.

To see this via set theory notation, let AB be the intersection of two sets, and A' be the complement of A. We can then form ABC, A'BC, AB'C, ABC', A'B'C, A'BC', AB'C', A'B'C'. We can always perform this decomposition.

Anything that has 'all but one prime' is what people typically think of when they think of a unit test: AB'C' is the functionality of A that does not involve B or C. Integration tests typically involve more than one component, and so forth.

If you have n functional pieces of code, then there up to 2^n possible ways they could all overlap with each other. So if you manage to break these into 2^n independent functional pieces, and then test every piece, you'll have fully tested your product.

Good luck doing this in practice, however, for the number of possible arrangements will be quite large, and they won't always be easy to identify (and often not easy at all).