This is one reason why it is important to know how long each test takes, and to refactor if necessary. A lot of times folks will just continue to pile on functionality into a test, use unnecessary sleeps throughout their code instead of events or polling loops, or just add new (sometimes redundant tests) to an already over-bloated automated test suite.
In any high volume test automation used in continuous integration cycles it is important to manage those tests via categories/suites. Here is a brief overview of how we do it in my team.
- Pre-check-in suite - a subset of unit tests and a functional tests ran before each check-in (we capped the total run time of this suite to less than 15 minutes).
- Integration test suite - All unit tests and a subset of critical functional tests that would likely block self-hosting or integration into the main branch (this takes less than 2 hours). The purpose of this suite is to identify any critical issues as quickly as possible that are introduced due to code churn.
- Functional test suite - All remaining functional tests not run in the integration test suite (this can be further sub-categorized by priority)
- Non-functional test suite - All non-functional tests such as perf, memory, bandwidth, stress, etc.
By categorizing tests and test suites, and prioritizing automated tests in a way to help identify important problems that might arise from continuous integration of new bits into the build we can control which test suites are ran and in what order.
We can also run a custom suite of tests based on the areas of code churn and upstream/downstream dependencies.
Finally, distribution of tests to run across multiple virtual machines (and real devices) is critical for time saving in any high-volume automated test runs. This is one of the gating factors for test suites of a few hundred automated tests to several thousands of automated tests per new build.