1

We have a test suite of around 60 feature files, 135 scenarios, 1800 steps.

We currently use the free 100 builds on semaphoreCI which occasionally successfully runs all of the tests, but quite often it times out after an hour of execution. On a local windows machine this can take up to 4 hours.

I have increased our max build time on semaphoreCI to 2 hours, but surely I should be able to get these tests to run in less than an hour.

We're using phantomjs as the headless browser and we also use an actual MySQL database rather than any type of mock database.

The blank schema is created by the setup steps before we run behat and the seed data is populated using beforeScenario and beforeStep methods when it's appropriate.

What can I do to make our tests run quicker?

  • 2
    As ChathuD mentioned, the most apparent solution is to use parallelism, whether it is achieved by multiple PCs or by using Selenium grid or by refactoring your code. You can also review your test cases to remove test cases that are not really that important. – Yu Zhang Feb 16 '18 at 16:07
1

One good implementation would be using a selenium grid and try multiple test simultaneously. Next would be carefully debug and examine the places where it take lots of time (eg waitings, DB accessing ).If something found try to resolve that.

Another thing would be check the resource utilization of your windows PC ,common problem can be the Read Write speed of your hard disk. There for try adding a SSD.

1
  1. Optimalization of the test code is a key. Shorten, remove or replace any fixed wait states. Try to find hints on technology, you have. For example, use CSS instead of XPath, etc.

  2. The Paralelism could help. I do not know if this is feasible in Semaphore, but you can run multiple tests at once. Especially if the time is caused by the slow application, test runs usually on 1 or 2 CPU's. Ballance the number of parallelly running tests on CPU cores. Enable some monitoring of the PC resources, to get an idea, what is happening inside of the testing machine during the build/test process.

  3. Do not underestimate the power of the hardware. On dedicated new i7 machine test could run much faster, than on old Core2Duo or some 1CPU server virtual machine.

  4. If the application itself is slow, and not the test, it could help to optimize the application speed. Try to run some performance checks. You are the one, who write the code, so you should know it. If some 3rd party app/integration is slow, try to replace it by mocks, which could remove the wait states.

  5. If nothing helps Prioritize Test Cases and run just the highest priority on Build time. Run the full set of the test as post-build process with different hardware.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.