I worked at a company where unit tests ran as part of the build. The build failed if the unit tests failed. What are the benefits and disadvantages of that practice?


4 Answers 4


We run unit tests prior to check-in and rerun them as part of a larger test suite on each daily build (feature branch) and weekly build (main branch; multiple feature branches aggregated).

Some advantages we have found include:

  • Fewer build breaks, and virtually zero build breaks in main
  • Additional test coverage for component/integration level testing
  • Helps improve unit tests by identifying tests that pass in isolation but fail in lab runs, or on different language versions, (test case reliability)
  • And proactively engages developers as part of the testing process
  • Makes unit testing transparent to the entire team (if a UT fails, a bug gets logged)

Some potential cautions

  • May cause a false sense of good test coverage if you assume that increasing code coverage == better testing, and may mask analysis of areas that are marked as covered (if doing white box structural analysis)
  • Added test run time (may be an issue in smaller shops with limited resources)
  • Overreliance on unit tests may lead to false sense of confidence and cause testers not to develop more robust tests in some areas
  • Good coverage of the process! This covers many of the issues I've seen from time to time with broken builds.
    – MichaelF
    Commented Jun 20, 2011 at 17:23
  • I would add to the advantages: fewer functionality regressions if someone reverts code by accident, or changes for a new feature are made to existing code and end up breaking existing functionality. Commented Jun 23, 2011 at 5:05

Unit tests should always be run as a part of build. We have been doing that for over 2 years in our project and the benefits have always outweighed the disadvantages. Not only these are run with the build but we have checks in place which breaks the build when the coverage falls below a certain threshold (although I agree with Bj above on code covergae != good testing). Some of the benefits we see :

  1. It brings a sense of urgency in the team to keep the unit tests always green else the build breaks. This leads to fewer breaking build over period of time with devs becomimg more disciplined w.r.t to executing unit tests before committing there code.

  2. A failing test will always lead to a fix (test or code).

  3. The code coverage and unit test execution trends can be analyzed over a period of time using tools like Sonar to detect grey spots and identify areas of improvement.

  4. Refactoring will be always covered by the safety net of unit tests running continuously as a part of CI.

  5. It acts as a first line of defence in case you aim for something like 'continuous delivery' where the software will pass through many automated gates/checks before being pushed to production (and major part of the complete process is automatic).

  6. Its more transparent and effective this way.(think of distributed teams spread across the planet working on the same codebase in different timezones)

  7. Its your unit level regression suite (a bug found should be added as test at unit level) .

Some things to take care for:

  1. Code coverage is misleading in a sense that a covergae of 80% only means that 20% of the code has never been tested. So the unit tests should be meaningful and only than it would be useful to run them with the build.

  2. This should not lead to an excuse for not doing system/integeration testing.

  3. It should not slow down the build so much that it negates any meaningful aka quick feedback.


In a Team comprised of multiple developers.

  • Before checking in the Code, Every Developer is expected to unit test their module
  • After the code is integrated, smoke testing / basic test cases need to be executed to verify integrated code works fine

Below are advantages of running unit test cases before build is provided to QA Team

  • Build might succeed even and unit tests might fail. This situation is potentially identified before build is provided to QA Team
  • This would help to identify if there are failures due to files missed in build
  • Running unit tests is fine, You can also run basic P1 cases to ensure functionality works fine
  • Earlier the bugs are detected lesser the cost required to fix it
  • I don't understand "Build might succeed even and unit tests might fail." Can you clarify that please?
    – user246
    Commented Jun 20, 2011 at 13:20
  • Sure, Suppose you have modified 10 database files, Only 8 was checked in the build. Your build got successfully deployed but 2 files which were required did not go in the build. In this scenario your build would pass but test cases would fail. Build getting successfully deployed does not mean basic p1 cases would pass.
    – Siva
    Commented Jun 20, 2011 at 13:44
  • I guess it depends on how you define "build would pass", if you define a successful build to be one that compiles and passes all the tests then no this would not happen.
    – MichaelF
    Commented Jun 20, 2011 at 17:25
  • it might be better or clearer to say that the 'code can successfully be compiled and built' meaning that process itself does not generate an error, however unit tests (which verify the code is performing the correct behaviors) fail.. Commented Jun 23, 2011 at 4:59

For the sake of discussion... there are some real disadvantages for unit tests when done properly-

  • New tests should be added for every new feature, and you can't release the code before the test is ready (I know, you can argue that you can bend the rules if needed, and that having unit tests is a good thing).

  • When doing major API changes you will have to rewrite your tests, this might take a while. see "you can argue above")

  • From my experience CI tends to be fragile enough even without unit tests.

  • There are means by which developers can tag unit tests as 'pending' if the unit test itself is not complete, or the code that would make it pass is not complete, but the developer needs to checkin their code. The key thing is keep those to a minimum, and make sure that once a dev thinks their feature is complete, there are no tests still in that condition. Commented Jun 23, 2011 at 5:02
  • Totally disagree. Unit tests are intended to find API changes, behavioral changes. If you do not test your code, it will not work. Simple law of software development. Testing hours are expensive. The simplest and cheapest regression testing is unit testing. Let your testers do the hard in depth testing.
    – ReneS
    Commented Jun 24, 2011 at 14:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.