Background: web security company which was acquired by a bigger one. It was a startup with only manual testing, test automation started after the acquisition with a contractor team with mostly UI testing with Selenium.
The current process is:
- we have a quality gate, you have to run all of the tests in order to be able to merge your code to master. We have approximately 6000 unit, 300 integration, 1000 UI tests which would take about 70-110 minutes to execute.
- We create a release branch by cherry-picking the code to be released, and we run the same tests again before deploying to RC.
- On RC there is a separate test suite with about 550 UI tests.
- On canary we have 20 more UI tests to run before we release to production.
Most of the problems are with the first step: The problem is that our test infrastructure are very much unstable under this heavy load and having 1000 Selenium tests doesn't help either. This would result that our quality gates are extremely flaky, resulting about 40% pass rate. It is possible that the devs are struggling merging their code for days as they have to re-run the whole suite 5-6 times. Now, unfortunately we cannot do big steps e.g. start refactoring the UI tests to integration tests at big scale because of other ongoing testing projects.
I had some ideas to restructure our testing efforts, but as I don't have experience in such these are just theoretical:
- As I mentioned we have a lot of UI tests, and these were designed with covering features in depth instead of in breadth in mind. I think it would be great to have a suite with less tests and covering the features more broadly. We could execute this smaller suite before merging to master, and once the release package is finalized then running the full suite before promoting to RC.
- Introduce an execution time limit for tests for the smaller suite. If a test is slow but necessary, then refactor, introduce shortcuts etc.
- New tests should be written mostly to integration level.
- Start implementing public APIs which is called by our frontends and move UI tests to API level where it is possible.
Does any of these points makes sense? What should be the order? How would you guys do it, where to start?
If someone faced similar issues any advice would be much appreciated.