What's the recommended practice for separating unit tests from integration tests for automated testing?

Question

On a project I've been working on, we've been steadily increasing the code coverage through tests. Some of these tests were done as integration tests and some as unit tests. The integration tests can sometimes take a long time, thus discouraging users from running the entire test suite prior to checking in (and understandably so).

What are some recommended strategies for separating unit tests from integration tests in such a way that we can easily run both from Hudson, while allowing code coverage statistics for both the unit tests alone and both sets of tests combined? (I'm assuming there's no value in getting the code coverage statistics from the integration tests alone, but if you think that's a bad assumption, please let me know that as well.)

Specifically, this is a Java application using Maven as the build manager and Subversion as the version control system, with most developers using a NetBeans IDE, and also using Spring heavily along with some new JMockit and AspectJ code.

The types of things I'm wanting to know are:

1. Currently, NetBeans has a tool for automatically creating Unit Tests for you. It puts them in a src\test directory hierarchy that mirrors the src\main directory hierarchy for your main project. Should I create a src\integrationTest directory or something to specifically house integration tests to make for easy separation, or is it typical to use a naming convention to distinguish between the two types of tests, or perhaps an Annotation?
2. Along the same lines as #1, are there any Maven patterns for separating integration tests from unit tests?
3. Is there a standard check-in hook for Subversion to force developers to run the unit tests, but not the integration tests, prior to check-in? (This last one is less important than #1 or #2, but would still be nice.)

Alternatively, if you've got experience doing this with different tools, I'd like to hear what your practices are there as well. It could be that a good practice with another set of tools maps to my set of tools fairly simply.

Answer 1

> The integration tests can sometimes take a long time, 
> thus discouraging users from running the entire test 
> suite prior to checking in 

For the checkin-runs you can mark the long-running tests with their own category and tell the test-runner to exclude those long-runners

You may also look at Is there a way to separate long running (e.g. stress tests) out so they're not run by default in Maven 2?

Answer 2

To answer question 1: Yes, I'd separate the unit test code from the integration test code. They're logically different, are probably built differently, and you're planning on running them differently. I'd also name them differently too for extra clarity. For example, our unit tests are in a ut_* directory and named ut_classname, so I'd put the integration tests in an it_* directory and name them it_something.

With regard to code coverage, I'm not familiar with Maven or Hudson, but one strategy is to run only the unit tests for every commit (= fast) after the main build loop. Of course, the code coverage will be lower. So you have a second build loop that is triggered after the first that will run both fast unit and slow integration tests to give you the combined coverage. Or if the integration tests take a very long time, you could move them from a triggered build loop to an hourly/nightly schedule, although the more frequent the better.

(I'm assuming there's no value in getting the code coverage statistics from the integration tests alone, but if you think that's a bad assumption, please let me know that as well.)

I don't see any value in this either. The unit tests should be very fast to run, so there's no real reason to exclude them when running the integration tests.

Answer 3

I personally would seperate the test in what way makes logical sense. If you have a system that allows test catagorisation test lists in the execution tool or attributes on the code then I would leverage that.

As an overall approach I would follow a gated principle. Run the unit tests first and then if they pass allow the developers to check in, once checked in, then run the intergration tests and if they passs then commit the code to the main branch or trunk.

You want to keep your developers agile and allow them to work as quickly and efficently as possible, but still have the intergration tests as a quality "gate".