Rerun failed (flaky) tests with Jest

Question

We are using Jest as our test runner for our JavaScript (React and Selenium) tests. We have one or two flaky tests (as TeamCity marks them flaky and we process them as technical-debt), although we will fix them, for the continuously-delivery process it would be very handy if we can retry the tests automatically once or twice.

Currently Jest does not support re-running of tests out-of-the-box.

• I have been thinking about parsing the output for failed tests and then passing them into a second run with the -t parameter.
• Alternatively we could try to fire the test again from the Reporter?

Example reporter:

jasmine.getEnv().addReporter({
  specDone: async (result) => {
    if (result.status === 'failed') this.takeScreenshot(result.description);
    // Rerun test here somehow?
  },
});

Hopefully some Jest users have already solved this.

So my question is: How can I retry/rerun failed tests with Jest easily automated?

Answer 1

We recently switched from mocha to jest and in the process came across a flake which appears to due to a bug in react. Given the difficulty we had in investigating this issue and that it should be fixed in react 16, we too decided to take the approach of simply retrying the failing tests (the flake rate was ~3% and only when running multiple workers, so the likelihood of it appearing a second time when running a single test should be zero).

Since there was no integrated option in jest to retry on fail, we had to work with what we had. We use jest-cli in gulp, which provides a result object passed to a callback upon completion of the test suite:

jest.runCLI(config, __dirname, (result) => { })

result contains a testResults object, which is an array of objects containing details on each test file including number of failing tests (within the file), error messages, and file path.

The flake we saw (debugID may not be empty) had a consistent error message which never appeared for any other test failure, so we could be confident that a flake had occurred. This knowledge, combined with the full path of the file in which the flake occurred, gave us enough information to retry flaky tests:

const FAILURE_MESSAGE = 'jest:modules found failing tests';
const FLAKE_ERROR_MESSAGES = ['debugID may not be empty'];

function retryIfOnlyFlakyTests(result, done) {
  // get only the failing tests from the testResults object
  const failingTests = result.testResults.filter((testResult) => {
    return testResult.numFailingTests > 0;
  });

  // select only the failing tests which have an error message that matches
  // what we know to be a flake
  const flakyFailingTests = failingTests.filter((testResult) => {
    return FLAKE_ERROR_MESSAGES.some((errorMessage) => {
      return testResult.failureMessage.includes(errorMessage);
    });
  });

  // if any non-flakes occurred, fail the test run
  if (flakyFailingTests.length !== failingTests.length) {
    done(FAILURE_MESSAGE);
    return;
  }

  // get the test file names of the flaky tests
  const failingTestPaths = flakyFailingTests.map((testResult) => {
    return testResult.testFilePath;
  });

  jest.runCLI({
    config: {
      ...CONFIG,
      testMatch: failingTestPaths
    }
  }, __dirname, (result) => {
    if (result.numFailedTests > 0) {
      done(FAILURE_MESSAGE);
    } else {
      done();
    }
  });
}

gulp.task('jest', (done) => {
  const jestFiles = getJestFiles();

  jest.runCLI({
    config: {
      testMatch: jestFiles
    }
  }, __dirname, (result) => {
    if (result.numFailedTests > 0) {
      retryIfOnlyFlakyTests(result, done);
    } else {
      done();
    }
  });
});