Take the 2-minute tour ×
Software Quality Assurance & Testing Stack Exchange is a question and answer site for software quality control experts, automation engineers, and software testers. It's 100% free, no registration required.

Christmas List:

  • Manually I nark tests as being in category "long running". I'm lazy and don't want people to have to do this.

Are there any testing or CI frameworks out there that can run only tests that take less than 1s for example?

I know there's a down side that a test that should be quick takes an exceptionally long time which may not get highlighted - but I think CI servers should be able to highlight this too.

(JetBrains if your listening....)

share|improve this question
1  
It would seem a prerequisite of doing this automatically would be to running the tests to see if they take a "long time" or not. It also would require periodic updates to the "long time"ness of a test to ensure it has not since lost the "long time" state. –  corsiKa Aug 3 '11 at 15:57
    
Wouldn't it mean the framework would be able to distinguish between long running and short running tests? Seems like you might need some kind of property to either run specific types or tests, or set up scenarios to run. –  MichaelF Aug 4 '11 at 15:18
add comment

4 Answers

If you are talking about unit tests I would consider moving the long running tests to a different test suite. If they are taking a long time they probably aren't unit tests and should not be treated as such. These tests should be moved to the location where you run integration tests.

If that's not what your looking for you could also look into running tests in parallel. Testng supports this and I've read that is this is possible with junit as well (although I've never seen it work in junit).

share|improve this answer
add comment

This is one reason why it is important to know how long each test takes, and to refactor if necessary. A lot of times folks will just continue to pile on functionality into a test, use unnecessary sleeps throughout their code instead of events or polling loops, or just add new (sometimes redundant tests) to an already over-bloated automated test suite.

In any high volume test automation used in continuous integration cycles it is important to manage those tests via categories/suites. Here is a brief overview of how we do it in my team.

  • Pre-check-in suite - a subset of unit tests and a functional tests ran before each check-in (we capped the total run time of this suite to less than 15 minutes).
  • Integration test suite - All unit tests and a subset of critical functional tests that would likely block self-hosting or integration into the main branch (this takes less than 2 hours). The purpose of this suite is to identify any critical issues as quickly as possible that are introduced due to code churn.
  • Functional test suite - All remaining functional tests not run in the integration test suite (this can be further sub-categorized by priority)
  • Non-functional test suite - All non-functional tests such as perf, memory, bandwidth, stress, etc.

By categorizing tests and test suites, and prioritizing automated tests in a way to help identify important problems that might arise from continuous integration of new bits into the build we can control which test suites are ran and in what order.

We can also run a custom suite of tests based on the areas of code churn and upstream/downstream dependencies.

Finally, distribution of tests to run across multiple virtual machines (and real devices) is critical for time saving in any high-volume automated test runs. This is one of the gating factors for test suites of a few hundred automated tests to several thousands of automated tests per new build.

share|improve this answer
add comment

The book How Google Tests Software (chapter 2, "Test Execution" section) describes how Google categorizes tests by size: small, medium, large, or enormous. The size takes into account not only how long the test takes but also the extent of its dependencies. For example, small tests should run into under 100ms and use extensive mocking to minimize dependencies -- these are what you might consider to be unit tests. Medium tests run in less than a second and test interactions between a small set of modules.

Google uses shared infrastructure to run automated tests. The test runner puts on limit on the amount of time a test is allowed to run depending on its size. If the test tries to run longer than that, it is automatically killed. For example, small tests are killed after 1 minute, and medium tests are killed after 5 minutes.

I don't know of a general way to determine apriori how long a test will need to run. Taking a page from Google's tactics, perhaps the best you can do is to

  1. estimate the test's size and record it somewhere (e.g. in an annotation, external file, or database),
  2. Write/use a test runner that will use that information to kill tests that run too long for their size, and
  3. amend your estimates based on what your test runner observes
share|improve this answer
add comment

I know that in Java you can use TestNG framework in which you can group your tests. Eventually you can direct TestNG runner to execute tests only in that group. See this for more -

http://testng.org/doc/documentation-main.html#test-groups

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.