Take the 2-minute tour ×
Software Quality Assurance & Testing Stack Exchange is a question and answer site for software quality control experts, automation engineers, and software testers. It's 100% free, no registration required.

I worked at a company where unit tests ran as part of the build. The build failed if the unit tests failed. What are the benefits and disadvantages of that practice?

share|improve this question
    
Google's view on the matter: googletesting.blogspot.com/2011/06/… –  Rsf Jun 21 '11 at 14:46

4 Answers 4

up vote 16 down vote accepted

We run unit tests prior to check-in and rerun them as part of a larger test suite on each daily build (feature branch) and weekly build (main branch; multiple feature branches aggregated).

Some advantages we have found include:

  • Fewer build breaks, and virtually zero build breaks in main
  • Additional test coverage for component/integration level testing
  • Helps improve unit tests by identifying tests that pass in isolation but fail in lab runs, or on different language versions, (test case reliability)
  • And proactively engages developers as part of the testing process
  • Makes unit testing transparent to the entire team (if a UT fails, a bug gets logged)

Some potential cautions

  • May cause a false sense of good test coverage if you assume that increasing code coverage == better testing, and may mask analysis of areas that are marked as covered (if doing white box structural analysis)
  • Added test run time (may be an issue in smaller shops with limited resources)
  • Overreliance on unit tests may lead to false sense of confidence and cause testers not to develop more robust tests in some areas
share|improve this answer
    
+1 for the points, and for 'cautions' and not 'disadvantages' –  Suchit Parikh Jun 18 '11 at 19:14
    
Good coverage of the process! This covers many of the issues I've seen from time to time with broken builds. –  MichaelF Jun 20 '11 at 17:23
    
I would add to the advantages: fewer functionality regressions if someone reverts code by accident, or changes for a new feature are made to existing code and end up breaking existing functionality. –  Chuck van der Linden Jun 23 '11 at 5:05

In a Team comprised of multiple developers.

  • Before checking in the Code, Every Developer is expected to unit test their module
  • After the code is integrated, smoke testing / basic test cases need to be executed to verify integrated code works fine

Below are advantages of running unit test cases before build is provided to QA Team

  • Build might succeed even and unit tests might fail. This situation is potentially identified before build is provided to QA Team
  • This would help to identify if there are failures due to files missed in build
  • Running unit tests is fine, You can also run basic P1 cases to ensure functionality works fine
  • Earlier the bugs are detected lesser the cost required to fix it
share|improve this answer
    
I don't understand "Build might succeed even and unit tests might fail." Can you clarify that please? –  user246 Jun 20 '11 at 13:20
    
Sure, Suppose you have modified 10 database files, Only 8 was checked in the build. Your build got successfully deployed but 2 files which were required did not go in the build. In this scenario your build would pass but test cases would fail. Build getting successfully deployed does not mean basic p1 cases would pass. –  Siva Jun 20 '11 at 13:44
    
I guess it depends on how you define "build would pass", if you define a successful build to be one that compiles and passes all the tests then no this would not happen. –  MichaelF Jun 20 '11 at 17:25
    
True Agreed. Thanks Micheal. –  Siva Jun 20 '11 at 18:10
    
it might be better or clearer to say that the 'code can successfully be compiled and built' meaning that process itself does not generate an error, however unit tests (which verify the code is performing the correct behaviors) fail.. –  Chuck van der Linden Jun 23 '11 at 4:59

Unit tests should always be run as a part of build. We have been doing that for over 2 years in our project and the benefits have always outweighed the disadvantages. Not only these are run with the build but we have checks in place which breaks the build when the coverage falls below a certain threshold (although I agree with Bj above on code covergae != good testing). Some of the benefits we see :

  1. It brings a sense of urgency in the team to keep the unit tests always green else the build breaks. This leads to fewer breaking build over period of time with devs becomimg more disciplined w.r.t to executing unit tests before committing there code.

  2. A failing test will always lead to a fix (test or code).

  3. The code coverage and unit test execution trends can be analyzed over a period of time using tools like Sonar to detect grey spots and identify areas of improvement.

  4. Refactoring will be always covered by the safety net of unit tests running continuously as a part of CI.

  5. It acts as a first line of defence in case you aim for something like 'continuous delivery' where the software will pass through many automated gates/checks before being pushed to production (and major part of the complete process is automatic).

  6. Its more transparent and effective this way.(think of distributed teams spread across the planet working on the same codebase in different timezones)

  7. Its your unit level regression suite (a bug found should be added as test at unit level) .

Some things to take care for:

  1. Code coverage is misleading in a sense that a covergae of 80% only means that 20% of the code has never been tested. So the unit tests should be meaningful and only than it would be useful to run them with the build.

  2. This should not lead to an excuse for not doing system/integeration testing.

  3. It should not slow down the build so much that it negates any meaningful aka quick feedback.

share|improve this answer

For the sake of discussion... there are some real disadvantages for unit tests when done properly-

  • New tests should be added for every new feature, and you can't release the code before the test is ready (I know, you can argue that you can bend the rules if needed, and that having unit tests is a good thing).

  • When doing major API changes you will have to rewrite your tests, this might take a while. see "you can argue above")

  • From my experience CI tends to be fragile enough even without unit tests.

share|improve this answer
    
There are means by which developers can tag unit tests as 'pending' if the unit test itself is not complete, or the code that would make it pass is not complete, but the developer needs to checkin their code. The key thing is keep those to a minimum, and make sure that once a dev thinks their feature is complete, there are no tests still in that condition. –  Chuck van der Linden Jun 23 '11 at 5:02
    
Totally disagree. Unit tests are intended to find API changes, behavioral changes. If you do not test your code, it will not work. Simple law of software development. Testing hours are expensive. The simplest and cheapest regression testing is unit testing. Let your testers do the hard in depth testing. –  ReneS Jun 24 '11 at 14:05

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.