Handling unstable remote selenium server connection

Question

We are currently running end-to-end tests on BrowserStack and tackle flakiness of tests on a daily basis. One of the recent problems was that rarely Protractor temporarily loses connection to the BrowserStack remote selenium server throwing errors like:

Failed: ETIMEDOUT connect ETIMEDOUT 203.51.112.2:80

Is there a way to not fail a test in this case and retry a selenium command? Or, if this is not easily achievable, at least retry a specific test?

Some details:

• jasmine timeout interval is set to 100000
• allScriptsTimeout is set to 50000

Answer 1

We log the Jasmine test fails into a text file with a custom reporter. Afterwards we rerun the failed tests again only. If these still fail we report these fails at failures to research.

jasmine.getEnv().addReporter({
      specDone: async (result) => {
        if (result.status === 'failed') {
          fs.appendFileSync(SeleniumBrowser.getFailedTestsFile(), `${result.description}\n`);
        }
        await this.quit();
      },
    });

We use a script to rerun the tests from the command line. We are using jest -T , but with Jasmine I think you can use command-line arguments like --specs='tests/**/*.js' --grep="TestNameFromFile"

I once read an article (which I can't find anymore) that the a re-run strategy of three was the perfect number. Sometimes your infrastructure or network just hangs. After three times you are sure it is the test that is failing.

I would record the flakiness of these tests though and schedule a flakiness research once in awhile, but you do not want developers to waste adhoc time to re-run or research once in a couple of run errors. Some flaky test do succeed when run alone, these are the tests you do want to fix, but random single incidents should be ignored. keep in mind it could be race-conditions so do check them, but not instant adhoc :)