How to profile end-to-end tests to find the slow ones?

Question

The Story

We have a big end-to-end test codebase written in Protractor/WebDriverJS.

We are using jasmine-spec-reporter for the command-line test reports. It reports test execution time for every test spec:

About screen when logged in
    ✓ should show the license text after refresh (51 secs)

  About modal
    ✓ should show the correct title (0.396 sec)
    ✓ should display component versions in a correct format (0.782 sec)

...

The Problem

The problem is - we have multiple tests that are much slower than the others - performance is becoming critical when these kind of tests are used for "smoke" tests against production instances.

The Question

Is there an effective (preferably automated) way to identify the bottlenecks - the reasons why these tests are slow?

Ideally, it would be very convenient to see line-by-line breakdown reporting how long each line execution took. I am currently thinking of tweaking Protractor's "Control Flow" to log execution time of every single "command" coming through..

Answer 1

Some areas I would focus on for slow test bottlenecks:

• Slow tests usually means dependency issues. With UI tests this can be hard if you use the application with a back-end connected database and a dynamically created and maintained DOM. Test performance here can sometimes be improved by using fixed HTML fragments and treating the tests more as unit tests.
• http requests - one option for http requests is to use vcr - https://github.com/vcr/vcr
• The type of tests. Happy path tests tend to be fast. Sad and Optional tests tend to be slow. Are your smoke tests using happy paths? I generally only use happy tests for smoke test suites.
• Setup repetition. Do tests use the UI to set up conditions for the actual test? This often be done more directly and speedily through backend processes created specifically for testing.
• Test independence. If tests are truly independent of each other with no shared resources such as database this open up the opportunity to run them in parallel leading to potentially massive improvements.
• Wait strategy. Make sure you have a mature wait strategy with waits implicitly for elements to be present and appropriate waits for pages and assets to be loaded and rendered.
• Use tools that help report on slow tests, for example rspec is frequently configured to show the slowest test for each run by using the --profile flag or add --profile to your .rspec file. This will track the top 10 slowest examples.

Answer 2

1. From your test framework or from logs, first identify the slow ones.
2. Debug each test (single stepping) to determine which area is slowest. Identify if it's client or host related, backend, frontend, or just exactly which step chews up the most time.
3. Look for any exceptions during the test run and address properly.
4. If you need to do this, run a communications trace to see the offending side.

It's the same process a developer uses when coding, start from the problem and work backwards.

Answer 3

The way we are solving this problem is to wrap each user action in custom functions where we are logging time difference against each action in customized html results table.

This helps as we analyse and isolate then on test and further on step level which tests and steps are taking most time.

Usually we find page loading/ navigations are main culprit so we are designing tests with least page reloads/navigations.

The solution is too customized to share here as specific to an project/framework however the general idea can be implemented in any framework.