5

Background: web security company which was acquired by a bigger one. It was a startup with only manual testing, test automation started after the acquisition with a contractor team with mostly UI testing with Selenium.

The current process is:

  1. we have a quality gate, you have to run all of the tests in order to be able to merge your code to master. We have approximately 6000 unit, 300 integration, 1000 UI tests which would take about 70-110 minutes to execute.
  2. We create a release branch by cherry-picking the code to be released, and we run the same tests again before deploying to RC.
  3. On RC there is a separate test suite with about 550 UI tests.
  4. On canary we have 20 more UI tests to run before we release to production.

Most of the problems are with the first step: The problem is that our test infrastructure are very much unstable under this heavy load and having 1000 Selenium tests doesn't help either. This would result that our quality gates are extremely flaky, resulting about 40% pass rate. It is possible that the devs are struggling merging their code for days as they have to re-run the whole suite 5-6 times. Now, unfortunately we cannot do big steps e.g. start refactoring the UI tests to integration tests at big scale because of other ongoing testing projects.

I had some ideas to restructure our testing efforts, but as I don't have experience in such these are just theoretical:

  1. As I mentioned we have a lot of UI tests, and these were designed with covering features in depth instead of in breadth in mind. I think it would be great to have a suite with less tests and covering the features more broadly. We could execute this smaller suite before merging to master, and once the release package is finalized then running the full suite before promoting to RC.
  2. Introduce an execution time limit for tests for the smaller suite. If a test is slow but necessary, then refactor, introduce shortcuts etc.
  3. New tests should be written mostly to integration level.
  4. Start implementing public APIs which is called by our frontends and move UI tests to API level where it is possible.

Does any of these points makes sense? What should be the order? How would you guys do it, where to start?

If someone faced similar issues any advice would be much appreciated.

4
  • Are you using Continuous Deployment? Or is the whole process of creating a release branch and running the tests a manual process? If you don't have Continuous Deployment, is that a goal or factor? Dec 24, 2021 at 19:21
  • we are currently not doing CD, it is a manual process. but CD would be an end goal for sure
    – dtoman
    Dec 24, 2021 at 19:24
  • "you have to run all of the tests in order to be able to merge your code to master" Why? Is the development sloppy to the point that any change can affect any part or is this testing full of low-value work, not focused on the risks that each change does? Dec 25, 2021 at 13:27
  • indeed, the product is built as a monolith and any change can affect any part.
    – dtoman
    Dec 27, 2021 at 10:54

1 Answer 1

5

"Perfection is achieved, not when there is nothing more to add, but when there is nothing left to take away." - Antoine de Saint-Exupery

In general, strive to make UI tests more horizontal(end to end , not deep) and lower level tests more vertical(in depth).

In my automation journey, I learned few hard lessons in UI tests as below:

  • Have fewer UI tests, as straightforward and simple.
  • Don't cover same verifications in multiple tests or on multiple test levels.
  • Let every test pass and fail for one and only one reason , focused tests.
  • Don't try to rebuild complex application logic in tests otherwise they are prone to same bugs as application itself.
  • Strive to be broad , not deep.
  • If the flakiness is in general for most of the tests , then test code quality is low.
  • Review them regularly and don't hesitate to throw the ones which don't add value anymore. (Sunk Cost Theory )

My suggestion


Add a metrics to every test to gain positive points for each time it's found a bug and negative points for each flaky failure.Add new tests for missed bugs at appropriate level.

Calculate the relative points for each test and discard the lowest ones periodically, that should make your test suite lean and sleek over a period of time.

In your situation, I would start with removing the UI tests which add lowest value in the overall stack .

If a UI test has been failing intermittently in every other release for no reason and hardly ever found an actual application issue , that's an good candidate for removal.

Good read to measure flakiness

3
  • thanks for sharing your experience, i agree with your points. where should we start restructuring the automated testing, what gives the most value in your experience? also, for the metrics, do you use some automated mechanism?
    – dtoman
    Dec 28, 2021 at 21:20
  • You are welcome.Please accept the answer if helped. I would suggest to start with asking yourself some hard questions like what exactly is adding value in entire suite and what not and act accordingly. Dec 28, 2021 at 23:20
  • Also for the metrics part , my suggestion would be to use some simple method to just track the overall value added by each test ,maybe in an excel. I personally use simple tools like notepad, excel etc. It's not tools , the value added is important. Dec 28, 2021 at 23:22

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.