Take the 2-minute tour ×
Software Quality Assurance & Testing Stack Exchange is a question and answer site for software quality control experts, automation engineers, and software testers. It's 100% free, no registration required.

On a project I've been working on, we've been steadily increasing the code coverage through tests. Some of these tests were done as integration tests, and some as unit tests. The integration tests can sometimes take a long time, thus discouraging users from running the entire test suite prior to checking in (and understandably so). What are some recommended strategies for separating unit tests from integration tests in such a way that we can easily run both from Hudson, while allowing code coverage statistics for both the unit tests alone and both sets of tests combined? (I'm assuming there's no value in getting the code coverage statistics rom the integration tests alone, but if you think that's a bad assumption, please let me know that as well.)

Specifically, this is a Java application using Maven as the build manager and Subversion as the version control system, with most developers using a NetBeans IDE, and also using Spring heavily along with some new JMockit and AspectJ code.

The types of things I'm wanting to know are:

  1. Currently, NetBeans has a tool for automatically creating Unit Tests for you. It puts them in a src\test directory hierarchy that mirrors the src\main directory hierarchy for your main project. Should I create a src\integrationTest directory or something to specifically house integration tests to make for easy separation, or is it typical to use a naming convention to distinguish between the two types of tests, or perhaps an Annotation?
  2. Along the same lines as #1, are there any Maven patterns for separating integration tests from unit tests?
  3. Is there a standard check-in hook for Subversion to force developers to run the unit tests, but not the integration tests, prior to check-in? (This last one is less important than #1 or #2, but would still be nice.)

Alternatively, if you've got experience doing this with different tools, I'd like to hear what your practices are there as well. It could be that a good practice with another set of tools maps to my set of tools fairly simply.

share|improve this question
add comment

3 Answers 3

up vote 14 down vote accepted
> The integration tests can sometimes take a long time, 
> thus discouraging users from running the entire test 
> suite prior to checking in 

For the checkin-runs you can mark the long-running tests with their own category and tell the test-runner to exclude those long-runners

You may also look at Is there a way to separate long running (e.g. stress tests) out so they're not run by default in Maven 2?

share|improve this answer
    
Nice. I really like both of those pieces of advice. –  Ben Hocking Jun 28 '11 at 10:52
add comment

To answer question 1: Yes, I'd separate the unit test code from the integration test code. They're logically different, are probably built differently, and you're planning on running them differently. I'd also name them differently too for extra clarity. For example our unit tests are in a ut_* directory and named ut_classname, so I'd put the integration tests in an it_* directory and name them it_something.

With regard to code coverage, I'm not familiar with Maven or Hudson, but one strategy is to run only the unit tests for every commit (= fast) after the main build loop. Of course, the code coverage will be lower. So you have a second build loop that is triggered after the first that will run both fast unit and slow integration tests to give you the combined coverage. Or if the integration tests take a very long time, you could move them from a triggered build loop to a hourly/nightly schedule, although the more frequent the better.

(I'm assuming there's no value in getting the code coverage statistics rom the integration tests alone, but if you think that's a bad assumption, please let me know that as well.)

I don't see any value in this either. The unit tests should be very fast to run, so there's no real reason to exclude them when running the integration tests.

share|improve this answer
add comment

I personally would seperate the test in what way makes logical sense. If you have a system that allows test catagorisation test lists in the execution tool or attributes on the code then I would leverage that.

As an overall approach I would follow a gated principle. Run the unit tests first and then if they pass allow the developers to check in, once checked in, then run the intergration tests and if they passs then commit the code to the main branch or trunk.

You want to keep your developers agile and allow them to work as quickly and efficently as possible, but still have the intergration tests as a quality "gate".

share|improve this answer
    
That is what I want to do, but what I'm wanting to know is what is the recommended practice for doing it? I'll rewrite my question to make it clearer. –  Ben Hocking Jun 23 '11 at 10:24
    
@Ben your comment there is the story of my firm to a T. My boss's boss says "We all have great ideas; there is no short supply. But how can we actually implement them?" probably three times a month. –  corsiKa Jun 23 '11 at 17:20
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.