Automatically only run the fast tests?

Question

Christmas List:

• Manually I nark tests as being in category "long running". I'm lazy and don't want people to have to do this.

Are there any testing or CI frameworks out there that can run only tests that take less than 1s for example?

I know there's a down side that a test that should be quick takes an exceptionally long time which may not get highlighted - but I think CI servers should be able to highlight this too.

(JetBrains if your listening....)

Answer 1

If you are talking about unit tests I would consider moving the long running tests to a different test suite. If they are taking a long time they probably aren't unit tests and should not be treated as such. These tests should be moved to the location where you run integration tests.

If that's not what your looking for you could also look into running tests in parallel. Testng supports this and I've read that is this is possible with junit as well (although I've never seen it work in junit).

Answer 2

This is one reason why it is important to know how long each test takes, and to refactor if necessary. A lot of times folks will just continue to pile on functionality into a test, use unnecessary sleeps throughout their code instead of events or polling loops, or just add new (sometimes redundant tests) to an already over-bloated automated test suite.

In any high volume test automation used in continuous integration cycles it is important to manage those tests via categories/suites. Here is a brief overview of how we do it in my team.

• Pre-check-in suite - a subset of unit tests and a functional tests ran before each check-in (we capped the total run time of this suite to less than 15 minutes).
• Integration test suite - All unit tests and a subset of critical functional tests that would likely block self-hosting or integration into the main branch (this takes less than 2 hours). The purpose of this suite is to identify any critical issues as quickly as possible that are introduced due to code churn.
• Functional test suite - All remaining functional tests not run in the integration test suite (this can be further sub-categorized by priority)
• Non-functional test suite - All non-functional tests such as perf, memory, bandwidth, stress, etc.

By categorizing tests and test suites, and prioritizing automated tests in a way to help identify important problems that might arise from continuous integration of new bits into the build we can control which test suites are ran and in what order.

We can also run a custom suite of tests based on the areas of code churn and upstream/downstream dependencies.

Finally, distribution of tests to run across multiple virtual machines (and real devices) is critical for time saving in any high-volume automated test runs. This is one of the gating factors for test suites of a few hundred automated tests to several thousands of automated tests per new build.

Answer 3

The book How Google Tests Software (chapter 2, "Test Execution" section) describes how Google categorizes tests by size: small, medium, large, or enormous. The size takes into account not only how long the test takes but also the extent of its dependencies. For example, small tests should run into under 100ms and use extensive mocking to minimize dependencies -- these are what you might consider to be unit tests. Medium tests run in less than a second and test interactions between a small set of modules.

Google uses shared infrastructure to run automated tests. The test runner puts on limit on the amount of time a test is allowed to run depending on its size. If the test tries to run longer than that, it is automatically killed. For example, small tests are killed after 1 minute, and medium tests are killed after 5 minutes.

I don't know of a general way to determine apriori how long a test will need to run. Taking a page from Google's tactics, perhaps the best you can do is to

1. estimate the test's size and record it somewhere (e.g. in an annotation, external file, or database),
2. Write/use a test runner that will use that information to kill tests that run too long for their size, and
3. amend your estimates based on what your test runner observes

Answer 4

I know that in Java you can use TestNG framework in which you can group your tests. Eventually you can direct TestNG runner to execute tests only in that group. See this for more -

http://testng.org/doc/documentation-main.html#test-groups