3

I am repeatedly seeing a pattern of us adding automated tests as we add functionality and the code base grows. Unfortunately this doesn't play out well as the "few minutes" test suite turns into 20-30 mins, then an hour, then 2-3 hours and of course development slows down dramatically. Not to mention morale.

What approach can we use to reduce the number of tests and reduce the run time? We'd like to keep the test running as close to the code being changed as possible (i.e. within minutes) if possible.

2

Here are some options:

  • Choose which tests to run based on what has changed. This probably requires some interaction between your version control system and your test framework. I believe the book "How Google Tests" claims Google runs tests this way. This may be hard to do and hard to maintain.
  • Partition your tests by degree of urgency, e.g. most urgent tests run immediately, next most urgent tests run within a half hour, and so on.
  • Run your tests in parallel across multiple machines.
1
  • The most urgent tests are probably the ones you'd associate with a smoke test, i.e. tests that show the most essential functions aren't broken. – user246 Nov 30 '16 at 22:25
2

For Python, coverage module can detect which test cover which lines of source code. Then on commit, version control can execute only the tests covering the changes. IIRC, someone is working (or solved) this problem, ask on http://lists.idyll.org/listinfo/testing-in-python mailing list for exact details.

Of course this is not 100% accurate, but it gives you 95% (WAG) assurance that changes are fine (and you can run full multi-hour set of tests later).

1

How do you execute your test cases? When do you execute them?

  • Execute your test cases during the nights when no one is in the office may help
  • Use Jenkins (for example) to schedule your test execution

Break down your tests

  • A big test can normally be broken down into smaller ones, for example, instead of verifying all elements on a web page in one test, we verify one element at a time by one test. You will have some flexibility to choose how comprehensively you execute your tests and you will have some flexibility to decide how long it takes. Doing this requires lots of code refactoring.

Centralized test ownership

  • I think everyone can expand test cases at the moment? Please do not give them this much freedom. By having a centralized test ownership, you will have more consistency in test size and coding standards. At least, be the dictator for a while, see if it works. If someone wants to expand anything, they have to sit down with you, discuss test architecture and etc.

I do not think you can reduce the number of test cases AND reduce test execution time at the same time.

  • Your software product is getting more complicated and bigger over time, hence you need more tests. But you want to reduce test execution time, the only way I can see is to introduce more test modularization, which will give you freedom over which tests to run which to not run, and test modularization will result in a larger number of smaller tests instead of a few bigger tests. My personal opinions.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.