How to accelerate the release process by rethinking test automation

Question

Background: web security company which was acquired by a bigger one. It was a startup with only manual testing, test automation started after the acquisition with a contractor team with mostly UI testing with Selenium.

The current process is:

1. we have a quality gate, you have to run all of the tests in order to be able to merge your code to master. We have approximately 6000 unit, 300 integration, 1000 UI tests which would take about 70-110 minutes to execute.
2. We create a release branch by cherry-picking the code to be released, and we run the same tests again before deploying to RC.
3. On RC there is a separate test suite with about 550 UI tests.
4. On canary we have 20 more UI tests to run before we release to production.

Most of the problems are with the first step: The problem is that our test infrastructure are very much unstable under this heavy load and having 1000 Selenium tests doesn't help either. This would result that our quality gates are extremely flaky, resulting about 40% pass rate. It is possible that the devs are struggling merging their code for days as they have to re-run the whole suite 5-6 times. Now, unfortunately we cannot do big steps e.g. start refactoring the UI tests to integration tests at big scale because of other ongoing testing projects.

I had some ideas to restructure our testing efforts, but as I don't have experience in such these are just theoretical:

1. As I mentioned we have a lot of UI tests, and these were designed with covering features in depth instead of in breadth in mind. I think it would be great to have a suite with less tests and covering the features more broadly. We could execute this smaller suite before merging to master, and once the release package is finalized then running the full suite before promoting to RC.
2. Introduce an execution time limit for tests for the smaller suite. If a test is slow but necessary, then refactor, introduce shortcuts etc.
3. New tests should be written mostly to integration level.
4. Start implementing public APIs which is called by our frontends and move UI tests to API level where it is possible.

Does any of these points makes sense? What should be the order? How would you guys do it, where to start?

If someone faced similar issues any advice would be much appreciated.

Answer 1

"Perfection is achieved, not when there is nothing more to add, but when there is nothing left to take away." - Antoine de Saint-Exupery

In general, strive to make UI tests more horizontal(end to end , not deep) and lower level tests more vertical(in depth).

In my automation journey, I learned few hard lessons in UI tests as below:

• Have fewer UI tests, as straightforward and simple.
• Don't cover same verifications in multiple tests or on multiple test levels.
• Let every test pass and fail for one and only one reason , focused tests.
• Don't try to rebuild complex application logic in tests otherwise they are prone to same bugs as application itself.
• Strive to be broad , not deep.
• If the flakiness is in general for most of the tests , then test code quality is low.
• Review them regularly and don't hesitate to throw the ones which don't add value anymore. (Sunk Cost Theory )

My suggestion

---

Add a metrics to every test to gain positive points for each time it's found a bug and negative points for each flaky failure.Add new tests for missed bugs at appropriate level.

Calculate the relative points for each test and discard the lowest ones periodically, that should make your test suite lean and sleek over a period of time.

In your situation, I would start with removing the UI tests which add lowest value in the overall stack .

If a UI test has been failing intermittently in every other release for no reason and hardly ever found an actual application issue , that's an good candidate for removal.

Good read to measure flakiness