Speeding up end-to-end tests

Question

When we execute the same set of end-to-end browser-automated tests on BrowserStack, we notice a dramatic slowdown - tests are much slower than if executed locally and there are occasional failures because of timing issues. We try to address the failures by adding explicit waits, which overall make the tests less readable and the test flow less understandable and clean.

What are the general guidelines to improve on end-to-end testing execution speed?

Our stack includes Protractor/WebDriverJS; we use grunt tasks to start a local build server and expose it to BrowserStack via ngrok.

Answer 1

I discovered the following points while speeding up my protractor(selenium) tests:

• Run tests in a headless browser(from version 59, chrome can be run
  headless natively)

• Run tests as direct connection(directConnect: true)

• Don't log off & reuse user session between tests with the same base state(Home Page) wherever possible.

• Make short & simple to the point tests with one and only one assertion only.

• Run tests in parallel(using shardTestFiles/maxInstances)

• Disable angular/CSS animations, it will help in faster page loads

  $('body').allowAnimations(false);

• remove all explicit waits & reduce implicit waits to a minimum and supplement it with conditional waits(using expected conditions)/fluent waits(with short polling intervals) carefully

Custom Fluent wait Code Example:

browser.manage().timeouts().implicitlyWait(0);
browser.wait(function () {
    browser.sleep(1000);
    return element(by.id('id1')).isDisplayed()
    .then(
        function (isDisplayed) { 
            return isDisplayed; 
        }, 
        function (error) { 
            return false 
        });
}, 20 * 1000);

• Remove redundant test steps in between tests(this is the biggest time saver)

• Don't use UI tests to create/setup test data(use APIs to seed data), use UI tests to validate end-user business scenarios only.

• Don't use shared(like with developers/business analysts/not even with manual testers)/or production replica(heavy) database .Use dedicated light QA automation database with carefully designed baseline data only.

• Restore/refresh database before runs to keep it clean & light.

• While designing test cases, eliminate unnecessary long UI steps and let the tests focus only on test validations with least page navigations.

Answer 2

From my point of view here are some general guidelines to improve end-to-end testing execution speed for GUI automation.

1- Build a solid Object Repository by using unique attributes/properties to define test objects.

2-Devide test script into small test functions that are simple (Do not re-implement complex business).

3- Define correct points where to add Sync functions (moving to new page, waiting to object appear/disappear , waiting for property change,.....etc).

4- Use Dynamic waiting not implicit, for the correct behavior / property of a test object.

5- Modify the Driver Time out (Selenium) to be 1 : 5 seconds no more, because each time we use FindElement() Function the Driver will wait this time until object loaded in DOM or throw exception.

[Conclusion] - Design your Test Script by a way to get benefit of tool speed and also Synchronized with Application performance.

Example of Selenium/C# Dynamic wait for object to be loaded in DOM and displayed on GUI.

Calling for bellow Function:

Public IwebElement TestObject;
TestObject.Exists(5); // where 5 is time out by seconds.

public static bool Exists(this IWebElement element,int timeOut)
    {
        //GlobalVars.Test.Browser is IwebDriver Object
         bool isExist = false;

        // check that element is not null;
        if (element == null)
        {
            return false;
        }

        // check the element existance.
        try
        {               

            GlobalVars.Test.Browser.Manage().Timeouts().ImplicitWait = TimeSpan.FromSeconds(0);
            WebDriverWait wait = new WebDriverWait(GlobalVars.Test.Browser, TimeSpan.FromSeconds((double)timeOut));
            wait.IgnoreExceptionTypes(typeof(StaleElementReferenceException));
            isExist = wait.Until<bool>((d) => { try { return element.Displayed; } catch (Exception ex) { return false; } });

            return isExist;
        }
        catch (Exception ex)
        {
            return false;
        }
        finally
        {
            GlobalVars.Test.Browser.Manage().Timeouts().ImplicitWait = TimeSpan.FromSeconds(GlobalVars.Test.BrowserTimeOut);
        }   

    }