8

This is a continuation of Speeding up end-to-end tests topic, but a more specific one.

Currently, it takes ~5 hours for our daily end-to-end testing run to complete. What can we do to improve on speed?

Details:

I understand that we have a lot of tests and end-to-end tests are generally slow, but I hope we can cut the execution speed.

Please let me know if any more details are needed.

Improve this question
7
  • 3
    Can you run multiple tests in parallel? BTW thanks for link to slow down Protractor. Interesting and very clever hack.
    – Peter M. - stands for Monica
    Commented Aug 4, 2017 at 14:42
  • @PeterMasiar right, this is one of the things we are currently thinking about. I think our plan allows 2 parallel sessions - we have some tests that cannot run in parallel, but we'll address that separately. Thanks!
    – alecxe
    Commented Aug 4, 2017 at 14:46
  • 1
    Do you have ~1024 End to end tests for a single application???
    – Vishal Aggarwal
    Commented Mar 30, 2018 at 1:30
  • I suspect a fundamental test smell here...
    – Vishal Aggarwal
    Commented Mar 30, 2018 at 1:31
  • @VishalAggarwal well, seriously more at this point. The app is mission critical for the company overall, but we do flip that testing pyramid upside down cause of the lack of proper unit and integration tests on the frontend side - so, yeah, it smells :)
    – alecxe
    Commented Mar 30, 2018 at 1:32

3 Answers 3

Reset to default
5

A few things to consider come to mind:

  1. Only run in Chrome
    Controversial huh? Let me explain more fully though. My question for multi-browser and indeed multi-device (responsive web sites) testing is always what kind of browser / device specific issues have we seen and do we expect to see? How many browser specific errors have occurred in the last year. That's the first question. frequently the answer is yes but the business demands that we support x,y,z, versions a,b and it means hundreds's of thousands of dollars a year... and before you know it boom everything takes 6 times longer for those combinations... So there is a second piece - blue/green releases. This is the process of being able to release two versions at once. The idea is that you release a small percentage of one version. You can start with 1% and then you monitor that version and the browser breakdown and if you see a browser specific issue you can roll back the new release to 0% and the old release to 100% and then investigate the issue. You'll only have affected a very small number of users, but you know that 100% coverage of every device ever doesn't ever happen either. The key point is not much money loss or customer inconvenience. The reason behind all of this is to speed up development and head towards waiting for minutes for full test runs rather than hours or days and thus enabling continuous integration and continuous delivery. This requires that there be a role for analysts who monitor the staged releases to know when there issues. Sometime you pick a quiet time of the day (depending on market) to do the release to avoid issues with user sessions, cookies, etc. May take a little fiddling here to get it right but I've seen it done.

  2. Session adaptation
    What this means is look into a way that you can get to a particular point in the UI by going directly to that page. Users may need to go through many pages but there are ways to set test sessions to just go to the desired page. With multi form page workflows that can be very significant. One example of this is ruby-rspec-capybara and the rack session gem https://github.com/railsware/rack_session_access

  3. Parallelism
    Not really sure from your answer if you are doing this but you definitely want to use the cheapo power of multiple virtual machines in the cloud to speed up tests. You'll need to make sure the tests are independent from each other which you should be doing, e.g. correct setup and teardown procedures for each test.

  4. Run headless
    There are a couple of flavors here. First is using a headless browser like phantomjs but we've found it only works for some tests, mostly non-js ones.
    The other (exciting) news here in 2017 is that google has produce a headless version of chrome, now available for download at https://developers.google.com/web/updates/2017/04/headless-chrome
    I haven't played with it yet.

  5. Tag browsers specific tests
    Tag tests to only run other browser tests for certain functions You may now from experience which pages or areas have historically have multi-browser issues. Based on this knowledge pick out a subset of tests that you want to run in the other browser(s), tag them appropriately and then setup the full suite to use those tags in runs.

  6. Get funding dollars
    Make the case to management for more funding for the vm's and other tools. So the tests take 5 hours. Lets say that means that 4 people have to wait a couple of times week for that. So that is 40 hours in lag time, in other words a full developers time. If a developer costs $70,000 a year, how much should you spend on vm's in comparison ? The fact they are different cost centers on different budgets, blah blah should not be an excuse for doing the right thing for the overall business finances.

Improve this answer
3
  • Very good breakdown - indeed we have multiple issues with re-doing things to get to a desired page, but due to the nature and complexity of our app, we will have to think of ways to reach the desired pages directly, not always straightforward in our case. Parallelism and prioritization is something I will definitely start with. Thanks so much!
    – alecxe
    Commented Aug 4, 2017 at 22:52
  • Yes figure out your approach to state. Sometimes it's easy, just set up URL params correctly, other times more complicated and other systems involved. That's when you start finding out that good testing is as much about configuration management as good it is good tests. There's books written on that - literally - barnesandnoble.com/p/…
    – Michael Durrant
    Commented Aug 4, 2017 at 23:55
  • +1 even though I disagree on the headless point
    – FDM
    Commented Nov 7, 2018 at 9:09
3

If you build your tests so that each test can run in isolation. Then you can scale your testruns by running them in parallel. Testing in isolation maybe a bit slower for the individual tests, but it is the only way to scale a larger test suite.

Also I would research which tests to can be removed or moved down the ladder of the test-pyramid. Most single slow end-2-end-tests could also be a couple of faster integration and or unit-tests.

Improve this answer
3

Use Docker-compose to run Protractor tests in parallel

I would suggest to make tests purely independent and run them in parallel in dockerised containers using Selenium Grid to achieve the desired speed with stability.

With docker-compose, this solution is scalable with given hardware capabilities.

https://stackoverflow.com/questions/36773167/how-do-you-set-up-selenium-grid-using-docker-on-windows

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.