Should unit tests be executed as part of a build?

Question

I worked at a company where unit tests ran as part of the build. The build failed if the unit tests failed. What are the benefits and disadvantages of that practice?

Answer 1

We run unit tests prior to check-in and rerun them as part of a larger test suite on each daily build (feature branch) and weekly build (main branch; multiple feature branches aggregated).

Some advantages we have found include:

• Fewer build breaks, and virtually zero build breaks in main
• Additional test coverage for component/integration level testing
• Helps improve unit tests by identifying tests that pass in isolation but fail in lab runs, or on different language versions, (test case reliability)
• And proactively engages developers as part of the testing process
• Makes unit testing transparent to entire team (if a UT fails, a bug gets logged)

Some potential cautions

• May provide false sense of good test coverage if assume increase code coverage == better testing, and may mask analysis of areas that are marked as covered (if doing white box structural analysis)
• Added test run time (may be a factor in smaller shops with limited resources)
• Overreliance on unit tests may lead to false sense of confidence and cause testers not to develop more robust tests in some areas

Answer 2

For the sake of discussion... there are some real disadvantages for unit tests when done properly-

• New tests should be added for every new feature, and you can't release the code before the test is ready (I know, you can argue that you can bend the rules if needed, and that having unit tests is a good thing).

• When doing major API changes you will have to rewrite your tests, this might take a while. see "you can argue above")

• From my experience CI tends to be fragile enough even without unit tests.

Answer 3

Unit tests should always be run as a part of build. We have been doing that for over 2 years in our project and the benefits have always outweighed the disadvantages. Not only these are run with the build but we have checks in place which breaks the build when the coverage falls below a certain threshold (although I agree with Bj above on code covergae != good testing). Some of the benefits we see :

1. It brings a sense of urgency in the team to keep the unit tests always green else the build breaks. This leads to fewer breaking build over period of time with devs becomimg more disciplined w.r.t to executing unit tests before committing there code.

2. A failing test will always lead to a fix (test or code).

3. The code coverage and unit test execution trends can be analyzed over a period of time using tools like Sonar to detect grey spots and identify areas of improvement.

4. Refactoring will be always covered by the safety net of unit tests running continuously as a part of CI.

5. It acts as a first line of defence in case you aim for something like 'continuous delivery' where the software will pass through many automated gates/checks before being pushed to production (and major part of the complete process is automatic).

6. Its more transparent and effective this way.(think of distributed teams spread across the planet working on the same codebase in different timezones)

7. Its your unit level regression suite (a bug found should be added as test at unit level) .

Some things to take care for:

1. Code coverage is misleading in a sense that a covergae of 80% only means that 20% of the code has never been tested. So the unit tests should be meaningful and only than it would be useful to run them with the build.

2. This should not lead to an excuse for not doing system/integeration testing.

3. It should not slow down the build so much that it negates any meaningful aka quick feedback.

Answer 4

In a Team comprised of multiple developers.

• Before checking in the Code, Every Developer is expected to unit test their module
• After the code is integrated, smoke testing / basic test cases need to be executed to verify integrated code works fine

Below are advantages of running unit test cases before build is provided to QA Team

• Build might succeed even and unit tests might fail. This situation is potentially identified before build is provided to QA Team
• This would help to identify if there are failures due to files missed in build
• Running unit tests is fine, You can also run basic P1 cases to ensure functionality works fine
• Earlier the bugs are detected lesser the cost required to fix it