1

I'm part of a team building a relatively large platform that needs high code coverage in its testing.

  • With code coverage: 1 hour of execution
  • Without code coverage: 10min of execution

We are just at the beginning of the project; in the next few months with this test architecture I think we could get to 4-5 hours of execution with code coverage and 50 min without.

We want to run 100% of our tests to deploy, and sometimes we want to deploy our changes very quickly, so even 10 min execution is already a lot of time.

I don't want to particularly explain our test architecture, which is a lot about Symfony3, Phpunit, etc., because the question is the following:

How do big companies like Facebook and Google run their (compared to us) super massive tests bases quickly? Do you have some docs that are "must-reads"?

I already heard of distributed testing, etc. but it is a bit fuzzy.

  • 1
    "The problem is that we want the run 100% of our tests to deploy" - Why? I know one of the strategies in use with larger code bases is only run tests that will hit areas that are changed. – ernie Jul 11 '17 at 0:25
1

Some ways to help in approaching this scenario would be but not limited too:

  • Have the majority of your automated tests at the unit/integration level (much more lightweight than e2e GUI ones)
  • Incorporate e2e GUI tests as effeciently as possible, add them when there is real benefit of having them and run them independently in parallel to cut down execution time on that front.
  • Try to implement the testing pyramid from day 1 as effectively as possible (The Testing Pyramid
  • Respect the reality of the situation, if you are building a massive test suite, it is simply not possible to have it execute in "5" minutes
  • Manage your tests effectively into groups, making it easy to execute particular groups later incase you do not need to execute everything (you won't always need to run every single test every single time).
  • Spend time writing effecient code that runs as quickly as possible (definitely the case for the UI level automated tests moreso than anywhere)
  • Don't get too caught up with what google or facebook are doing, chances are its not very applicable to your needs
  • 70% unit test, 20% Integration and 10% end-to-end test is a great figure to aim for
1

IIRC, coverage.py has a module/plugin/runmode which can analyze the commit and run only tests which exercise changed code. Not sure if PHP has such runmode for coverage.

Of course it is not a silver bullet (nothing is), but it can help you to decrease test runtime. If you go that way, you may want (possibly on a separate build server) to run separate, full tests, even if they will run much slower, so they might not be able to keep up with every commit - run them continuously for all commits since the last run.

1

Some of the things you can do to improve on testing speed:

  • parallelize tests - some of the test runners that I've used before, like py.test and nosetests support running tests in parallel. In py.test world, there is the xdist plugin that allows to parallelize tests across CPUs and separate machines
  • prioritize and fail fast - make sure the critical tests are executed earlier and you get the feedback as early as possible
  • review your existing tests and make sure you understand the bottlenecks. See if you can detect unnecessarily slow tests
  • review how test data is prepared/generated and see if it can be improved. Make sure this step is not taking more time than actual tests.

How big companies like Fb, Google manages to quickly run their ( compaired to us ) super massive tests bases quickly ? Do you have some docs that I must read absolutely ?

Sometimes it does not make much sense replicating what huge tech giants do:

Follow-up read:

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.