1

How do you ensure your passing tests(UI automated) actually pass?

As automation engineer I think we should question our passing tests in the same spirit as we dig in our failing tests to find out reasons to fail, at least periodically.

In long standing large automation suites, there are times where a passing test has not been questioned in long time as it was always 'passing' until a direct bug in that area surfaces.

Please share your approach used in actual large projects to test the tests? Preferably through automation.

2

1 Answer 1

2

Your automation is code, so why not treat it like code and test it?

While you develop the code exploratory test it and make sure to include tests to verify that the test can fail, test the internal logic of the test (use unit tests for example) and the test as part of a bigger test suite ("integration tests").

When you are satisfied with the basic functionality of the test turn those into an automated test suite, keeping only a bunch of the most important tests, that will run together with other regression cycles, or periodically.

Adding Static analysis of your tests might also help catch things like return SUCCESS as the first line of a long test.

Now you are asking about actual large project and the answer becomes more fuzzy,I don't remember seeing such tests and even the famous SQLite test suite doesn't contain one.

4
  • Thanks @Rsf for the info , however as you also mentioned I am more curious to know , how other QA/SDET professionals are 'actually' handling the testing of the tests in 'their' projects in an 'automated' way. For example , how are you really testing your tests in automated way in your project? May 1, 2023 at 0:53
  • It is actually very rare to see projects with tests for the tests, at least after the tests' initial development stage. There are projects that occasional run mutation tests, a method where parts of the code are actually changed to induce failures, to ensure the test suite can detect the changes. Mutation testing is slow, you need to build and deploy, and inefficient, it is not always clear what change triggers what error, so it is not a good candidate to be run often.
    – Rsf
    May 2, 2023 at 9:08
  • As long any changes are in UI automation code , that does not require build and deploy. May 2, 2023 at 12:08
  • 1
    I have seen this issue in multiple projects where there is an legacy code and automation code is being maintained for couple of years. In those cases , mostly people who originally wrote the code are gone and its being maintained by others .These maintainers more likely focus on failures considering sheer number of tests (in thousands) and hardly ever wonder about the passing tests which not always pass for true reasons over long time. May 2, 2023 at 12:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.