What's the most efficient way to run lots of tests?

Question

Here's my current setup:

• Compile on teamcity
• Copy the dlls to 8 agents and run tests simaltaniousisly (via snapshot dependencies)
• Finally taper them in to a single 'publish' build configuration.

This way I can get through the tests in an hour, but am wondering is there a faster way? All these build machines have multi-core processors and my program should be thread safe, so is there a way to get the tests to run in parallel on one machine?

There's pnunit bundled with NUnit so TC should support it - has anyone had any experiences wiring it into TeamCity yet?

I'd also be keen to here answers from the Java side of the world.

(teamcity's a most excellent tool by the way and I don't work for JetBrains)

Answer 1

Personally I prefer the approach that you are currently implementing, i.e. scale out with a lot of test machines. The reasons I prefer it is:

• Machine level isolation of tests from each other. Even with the best intention, it is possible to get resource contention and tests interfering with each other, especially when web testing is involved (session state), and/or you have to deal with anything at the OS level, like OS error dialogues.

• Performance. I personally find that the main bottleneck with my tests is that the clients are CPU bound when launching applications or running the application as fast as it will go. How much you can leverage multi-cores is entirely dependent on the application being written to support it.

Answer 2

On thoughtworks tech radar I noticed this in the right direction:

https://github.com/test-load-balancer

I think it's probably early days for nunit / team city integration but it's the kind of thing I'm looking for.