Take the 2-minute tour ×
Software Quality Assurance & Testing Stack Exchange is a question and answer site for software quality control experts, automation engineers, and software testers. It's 100% free, no registration required.

If you can test every line of [your product's] code via unit tests, wouldn't unit testing alone (theoretically) be sufficient?

Are there other "necessary" benchmarks of proper test coverage other than simply covering (ideally) all of your LOC?

share|improve this question
add comment

4 Answers 4

Not exactly.

Unit testing only isolates units in isolation. In these tests all dependencies to other units are mocked or stubbed out. So how do you know those units together do what they are supposed to do? Code tends to grow hierarchically in complexity, and with that growth comes more and more units working together, more and more groups of units (modules) working together, and along the growth things need to checked.

Hence you need integration tests for this. These tests check the configured set of working components work together correctly.

Finally, some sort of acceptance, validation, or verification tests should be written to check the software in the end does what it's supposed to do. Business (usually) drives the creation of software, so these are an important metric to any organization. For most web apps/sites this is usually the selenium UI tests.

Metrics are always dangerous, you can always find a way to beat them. It's very easy to write a bunch of code with 100% test coverage that does nothing useful together. So the difference between quality of 60%, 70%, 95%, or 100% test coverage is tough to define. Some argue code should always be 100% covered, while some say 70-80% is sufficient, it's all about project/personal preference. The amount of coverage for each test type depends on situation and project. I find myself with some tasks writing 100% unit tested code (production code, new features), 0% unit tested code (for writing UI tests), ~50% integration tested code (refactoring legacy code, javascript), etc.

I'm not aware of any metrics for Integration tests.

But surely acceptance/verification tests should always be 100% passing to ensure a functional software product according to a specification. But even then, their could be bugs in the specification. I find writing these tests really helps catch these bugs.

share|improve this answer
    
+1 for "How do you know those units together do what they are supposed to do?" There's no point in all the cogwheels spinning properly if there's an odd number instead of an even one and the last one turns the wrong way. –  user867 May 30 '13 at 5:17
add comment

There are many cases where unit testing as you have described would not be "sufficient".

(And you haven't really defined what you mean by "sufficient" in this case. Good enough to move the code to Production? Good enough to pass it on to QA? Good enough to please your boss? Good enough to feel like you did a good job? Something else?)

In most practical cases, Unit Testing tests that the code the developer wrote actually does what the developer thinks it should do.

  • What if the developer misinterpreted the Requirements? The code might well pass all Unit Tests, but be completely incorrect.
  • Testing every line isn't the same as testing every line under all conditions. If you test a line of code where the input data is "1", what happens to that same line when the input data is "-23456789"?
  • Refer to https://en.wikipedia.org/wiki/Code_coverage for a deeper discussion about code coverage (beyond just "testing every line")
  • What about performance and load? Most unit tests don't cover non-functional aspects.
  • What about integration issues that arise during the interaction among several units? Unit testing usually doesn't cover that.
  • etc, etc.
share|improve this answer
    
Excellent point about asking what 'sufficient' means –  Phil Kirkham May 23 '13 at 11:40
    
If you read my OP by "sufficient" I mean attaining "proper test coverage" –  ManRow May 23 '13 at 23:34
1  
And what does 'proper test coverage' mean? It will be different dependent on the app. Would a medical app and a game have the same? –  Phil Kirkham May 24 '13 at 3:09
    
Ok, so then I guess there are no universal rules for "proper test coverage" then! –  ManRow May 24 '13 at 18:06
    
Definitely not. "Proper" depends almost entirely on context. What's proper in my shop probably isn't the same as what's proper in your shop. (And at least in my shop it's often different with each project) –  Joe Strazzere May 24 '13 at 19:16
add comment

Even theoretically, unit testing isn't sufficient because it doesn't cover the paths through the system. You can cover every line of code in a system, but not cover every potential way through it.

For instance, can you launch the application? Does it run on all the target operating systems? Does it render correctly? These are all out of the scope of unit tests. In addition,

To take a very simple example, let's suppose your application is a calculator, similar to the basic version of the Microsoft calculator. You can unit test the math calculation - but not completely. A complete test of the calculations is impossible - just to test single digit basic operations you're looking at 400 distinct test cases (I think. It's possible my mental math is off here).

That doesn't even cover the quality of the unit tests - do they cover only what the code is supposed to do, or do they test the edge cases and potential failures?

Unit tests are invaluable, but they aren't all that's needed. As Joe and Craastad have said, they need to be part of a larger set of tests including integration tests, functional tests, user acceptance tests, compatibility tests, performance tests... and so forth.

share|improve this answer
    
Not sure I agree. Practically speaking, you generally don't have to test functionality outside of your code. For example, do you have to test that your calculator app works after say, closing the app and rebooting the computer, even if your code is just a simple calculator not doing anything special with the OS's shutdown/restart events/procedures/hooks? –  ManRow May 23 '13 at 23:48
1  
@ManRow - I've tested applications that had bugs in very similar areas. Under certain unusual circumstances, one application left behind corrupt data in a temp folder, then read it next time the system rebooted. The error was only apparent after "closing the app and rebooting the computer". Perhaps really good unit testing would have uncovered that, perhaps not. –  Joe Strazzere May 24 '13 at 9:59
1  
Even if your code isn't doing anything special with the OS, you still need to test it with the OS - because changes to how the OS works can still disable your application. Vista broke many applications because of the changes to how Microsoft handled system security. Applications not doing anything special can still fail to handle network and/or database outages gracefully. –  Kate Paulk May 24 '13 at 11:37
add comment

I'll throw out a couple of definitions so I can answer the question with an esoteric and abstract response:

  1. A functional piece of code is any operation that makes changes to a finite system (in other words, a computer.) It is important to realize that there are only a finite number of configurations that can occur on a computer. Obviously, a functional piece of code will not operate on the entire computer, so this finite system is often restricted to be some subset of the computer. This restriction is typically dependent on the functional piece of code itself, and not any grouping of functional pieces. The changes that a functional piece of code makes is referred to as an algorithm.

  2. Two functional pieces of code are said to be independent if the restricted systems above are disjoint. This comes out of the definition of the restriction, and this definition should be immediately clear.

So clearly, if we have n independent functional pieces, then it suffices that we can perform n tests on these functional pieces of code. A test here is defined as the minimum number of actions that are needed to accurately determine the algorithm of each functional piece of code. (In other words, we only need to test independent components individually.)

Unfortunately, it is rarely the case that functional pieces of code are completely independent. Typically, there is some shared overlap that is involved between these functional pieces. However, we can transform each of these regions into independent regions by breaking them down into components that overlap, and components that don't overlap.

To see this via set theory notation, let AB be the intersection of two sets, and A' be the complement of A. We can then form ABC, A'BC, AB'C, ABC', A'B'C, A'BC', AB'C', A'B'C'. We can always perform this decomposition.

Anything that has 'all but one prime' is what people typically think of when they think of a unit test: AB'C' is the functionality of A that does not involve B or C. Integration tests typically involve more than one component, and so forth.

If you have n functional pieces of code, then there up to 2^n possible ways they could all overlap with each other. So if you manage to break these into 2^n independent functional pieces, and then test every piece, you'll have fully tested your product.

Good luck doing this in practice, however, for the number of possible arrangements will be quite large, and they won't always be easy to identify (and often not easy at all).

share|improve this answer
    
You don't need to test 2^n pieces, in fact you don't need a test for every "combination" of pieces thereof. From the thread of responses I'm receiving, it seems the testing goal is to have 100% code coverage (which is just accomplished by having those 'N' tests), plus some general "acceptance" tests that test the product as a whole. –  ManRow May 30 '13 at 0:06
    
The N pieces you're referring to, and the 2^N pieces referred to in that post, are the same thing. It's just those N pieces are bigger chunks with coupled pieces inside of it, and the 2^N pieces are the decoupled pieces. (Many of those 2^N pieces can be nothing, which means that a piece cannot be fully decoupled.) –  Jim Pedid May 30 '13 at 0:20
    
Not necessarily. I can take a simple example of 'N' classes, which require 'N' unit tests for complete code coverage. Not 2^N... –  ManRow May 30 '13 at 1:31
    
You can have N classes that work perfectly and have perfect unit tests for those classes, but there's nothing in these unit tests that can ever prevent or catch a bug that results from you having a bug that results from using a typo from using one class inside of another, or even using those classes incorrectly (say a typo or lack of understanding of requirements) at the next level up. It is precisely these kinds of scenarios that are included in that upper bound of 2^N. –  Jim Pedid May 30 '13 at 1:43
1  
While interesting, does any of this answer the question "If you can test every line of [your product's] code via unit tests, wouldn't unit testing alone (theoretically) be sufficient?" –  Joe Strazzere May 30 '13 at 12:16
show 1 more comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.