Behat tests take over an hour to run

Question

We have a test suite of around 60 feature files, 135 scenarios, 1800 steps.

We currently use the free 100 builds on semaphoreCI which occasionally successfully runs all of the tests, but quite often it times out after an hour of execution. On a local windows machine this can take up to 4 hours.

I have increased our max build time on semaphoreCI to 2 hours, but surely I should be able to get these tests to run in less than an hour.

We're using phantomjs as the headless browser and we also use an actual MySQL database rather than any type of mock database.

The blank schema is created by the setup steps before we run behat and the seed data is populated using beforeScenario and beforeStep methods when it's appropriate.

What can I do to make our tests run quicker?

Answer 1

One good implementation would be using a selenium grid and try multiple test simultaneously. Next would be carefully debug and examine the places where it take lots of time (eg waitings, DB accessing ).If something found try to resolve that.

Another thing would be check the resource utilization of your windows PC ,common problem can be the Read Write speed of your hard disk. There for try adding a SSD.

Answer 2

1. Optimalization of the test code is a key. Shorten, remove or replace any fixed wait states. Try to find hints on technology, you have. For example, use CSS instead of XPath, etc.

2. The Paralelism could help. I do not know if this is feasible in Semaphore, but you can run multiple tests at once. Especially if the time is caused by the slow application, test runs usually on 1 or 2 CPU's. Ballance the number of parallelly running tests on CPU cores. Enable some monitoring of the PC resources, to get an idea, what is happening inside of the testing machine during the build/test process.

3. Do not underestimate the power of the hardware. On dedicated new i7 machine test could run much faster, than on old Core2Duo or some 1CPU server virtual machine.

4. If the application itself is slow, and not the test, it could help to optimize the application speed. Try to run some performance checks. You are the one, who write the code, so you should know it. If some 3rd party app/integration is slow, try to replace it by mocks, which could remove the wait states.

5. If nothing helps Prioritize Test Cases and run just the highest priority on Build time. Run the full set of the test as post-build process with different hardware.