4

The Story

We have a big end-to-end test codebase written in Protractor/WebDriverJS.

We are using jasmine-spec-reporter for the command-line test reports. It reports test execution time for every test spec:

About screen when logged in
    ✓ should show the license text after refresh (51 secs)

  About modal
    ✓ should show the correct title (0.396 sec)
    ✓ should display component versions in a correct format (0.782 sec)

...

The Problem

The problem is - we have multiple tests that are much slower than the others - performance is becoming critical when these kind of tests are used for "smoke" tests against production instances.

The Question

Is there an effective (preferably automated) way to identify the bottlenecks - the reasons why these tests are slow?

Ideally, it would be very convenient to see line-by-line breakdown reporting how long each line execution took. I am currently thinking of tweaking Protractor's "Control Flow" to log execution time of every single "command" coming through..

migrated from stackoverflow.com Jun 9 '17 at 16:15

This question came from our site for professional and enthusiast programmers.

  • You could get quite a lot of information from the 'Performance' tab of the Chrome Developer Console. Only thing is you will need to execute the steps manually. You already have the slow tests from the existing report. Synced up view of timeline, network details, function calls, event logs, go to specific code lines etc etc. Also can save the details and load it again. Pretty good read at this link -- developers.google.com/web/tools/chrome-devtools/… – Grasshopper Jun 9 '17 at 9:53
  • There are plenty of profiling tools on the web. Personally I'm using Very Sleepy. You could use this on the background processes. Is this the kind of tools you are looking for? – Dominique Jun 13 '17 at 12:30
  • @Dominique nice, have not heard of Very Sleepy. I'm looking for ways to profile Protractor/Selenium tests, to determine where are the potential slowdowns in the tests - which selenium command took what time, how much "beforeEach" or "afterEach" took - these kind of things. Thanks. – alecxe Jun 13 '17 at 13:13
  • @alecxe: in that case, you might need to perform multiple profiling tests: first you'll need to know which Selenium commands are impacted (in case profiling is needed for that, is this information nog logged somewhere?), and once you know this, you'll need to find out the relation between the Selenium commands and the functions of your application and profile your application afterwards. Now that I think about it, Very Sleepy can profile one application (even if it launches different threads) but it can't handle different applications being launched separately. – Dominique Jun 13 '17 at 13:55
2
+50

Some areas I would focus on for slow test bottlenecks:

  • Slow tests usually means dependency issues. With UI tests this can be hard if you use the application with a back-end connected database and a dynamically created and maintained DOM. Test performance here can sometimes be improved by using fixed HTML fragments and treating the tests more as unit tests.
  • http requests - one option for http requests is to use vcr - https://github.com/vcr/vcr
  • The type of tests. Happy path tests tend to be fast. Sad and Optional tests tend to be slow. Are your smoke tests using happy paths? I generally only use happy tests for smoke test suites.
  • Setup repetition. Do tests use the UI to set up conditions for the actual test? This often be done more directly and speedily through backend processes created specifically for testing.
  • Test independence. If tests are truly independent of each other with no shared resources such as database this open up the opportunity to run them in parallel leading to potentially massive improvements.
  • Wait strategy. Make sure you have a mature wait strategy with waits implicitly for elements to be present and appropriate waits for pages and assets to be loaded and rendered.
  • Use tools that help report on slow tests, for example rspec is frequently configured to show the slowest test for each run by using the --profile flag or add --profile to your .rspec file. This will track the top 10 slowest examples.
3
  1. From your test framework or from logs, first identify the slow ones.
  2. Debug each test (single stepping) to determine which area is slowest. Identify if it's client or host related, backend, frontend, or just exactly which step chews up the most time.
  3. Look for any exceptions during the test run and address properly.
  4. If you need to do this, run a communications trace to see the offending side.

It's the same process a developer uses when coding, start from the problem and work backwards.

1

The way we are solving this problem is to wrap each user action in custom functions where we are logging time difference against each action in customized html results table.

This helps as we analyse and isolate then on test and further on step level which tests and steps are taking most time.

Usually we find page loading/ navigations are main culprit so we are designing tests with least page reloads/navigations.

The solution is too customized to share here as specific to an project/framework however the general idea can be implemented in any framework.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.