Should coverage analysis be performed on test code?

Question

I'm of the opinion that it should. Test code is meant to be deterministic. This means that 100% of it should be executed. If it's not 100% (and assuming the analysis tool isn't under-reporting), either there's dead code that ought to be deleted or the code is complicated enough that it ought to be extracted into helper code and that helper code ought to have tests.

Answer 1

I wouldn't necessarily put it high on the ToDo list, but I think it's beneficial to measure test code coverage to find dead tests. You probably won't get to 100%**, but you can find dead functions and binaries - which makes a big difference when you have a 20-hour automation run you're trying to whittle down to an overnight run.

** note - test code often has more code in error (or failure) conditions than production code, so unless your tests are failing, you probably won't hit the error paths. Of course, this begs the question of whether you should induce failures to ensure that the diagnostics and error handling in the test code is valid, but I'll leave that answer for another question.

Answer 2

I'm with Alan on this one - a lot of test automation code will cover error handling which may not get exercised often - and if you're really lucky, it won't ever get exercised (you do include error handling for every routine, right?)

Also, if you have a lot of libraried routines, you might find that one set of tests uses a relatively small subset of your library, where different test sets will use a different but overlapping subset. For instance, a library routine handling say user configuration could be used in multiple test runs. Libraries of utility functions are particularly prone to this.

I'd also suggest that this is a much lower priority than finding older, less effective test automation code and refactoring it to a cleaner state, which in turn is a lower priority than keeping up with new feature development.

Of course, it's possible that the software you're testing is stable and your automation code is nice and clean. In which case, more power to you! You're doing better than I am.

Answer 3

Personally I don't run code coverage on test code, as a tester with limited time, I simply have too many other things that need my limited time and attention.

What I do run though is static code analysis with Visual Studio or fxCop as a way to keep test code quality high.

I actually joke that visual studio Pro should be called Visual Studio un-Professional as it does not include code analysis, and it forces you to turn off the code analysis policy on TFS if you have a single copy being used as they can't check in with the policy on.

Answer 4

I assume by "deterministic" you mean "under the same conditions, always does the same thing". I am not sure that deterministic test code equates to 100% test code coverage. For example, a test may be deterministic and at the same time contain error-handling code for reporting anticipated conditions that rarely or never arise in practice.

Do you need 100% test code coverage? I think that depends on the needs of your organization. For example, if you publish your test code, or if your test code is a deliverable to your customer, it may be important to know that every line of your test code has been exercised. In my organization, tests are purely internal tools. I need a high level of confidence that my tests exercise the product in the way I intended and that my tests report failures and success in the way I intended. On the other hand, I am content with a lower level of confidence that my test will behave the way I intended if the network connection between my test and my database server times out.

Your suggestion about extracting uncovered test code into helpers that are then tested by other tests (meta-tests?) raises an interesting question: do you need to ensure 100% coverage for your meta-tests too? And if they contain uncovered code, should you extract them into meta-helpers that are tested with meta-meta-tests? In my organization, this would not be considered a good use of time.

It may be tempting to ask, "If I do not have 100% coverage, how do I know that my tests are correct?" You are wise to ask this question, because we tend to place a great deal of faith in the value of automated tests. However, code coverage is just one of a myriad of possible measurements of quality, all useful and all imperfect. There are other ways to raise your confidence about your tests, e.g. code reviews. Give code coverage its due, but do not allow your enthusiasm for code coverage interfere to with your good judgement.

In a subsequent comment, you said the deeper question is whether test code is as important as production code. "Important" can mean a lot of things; I assume you mean "necessary". It is necessary for a software company to have production code, and it sounds as if it is also necessary for your organization to have test code. Both are software, perhaps written in the same language and perhaps even written by the same people. Nonetheless, they exist to solve different problems, and so it is reasonable that they may be treated differently.