How do you ensure your passing tests(UI automated) actually pass?

As automation engineer I think we should question our passing tests in the same spirit as we dig in our failing tests to find out reasons to fail, at least periodically.

In long standing large automation suites, there are times where a passing test has not been questioned in long time as it was always 'passing' until a direct bug in that area surfaces.

Please share your approach used in actual large projects to test the tests? Preferably through automation.


2 Answers 2


Your automation is code, so why not treat it like code and test it?

While you develop the code exploratory test it and make sure to include tests to verify that the test can fail, test the internal logic of the test (use unit tests for example) and the test as part of a bigger test suite ("integration tests").

When you are satisfied with the basic functionality of the test turn those into an automated test suite, keeping only a bunch of the most important tests, that will run together with other regression cycles, or periodically.

Adding Static analysis of your tests might also help catch things like return SUCCESS as the first line of a long test.

Now you are asking about actual large project and the answer becomes more fuzzy,I don't remember seeing such tests and even the famous SQLite test suite doesn't contain one.

  • Thanks @Rsf for the info , however as you also mentioned I am more curious to know , how other QA/SDET professionals are 'actually' handling the testing of the tests in 'their' projects in an 'automated' way. For example , how are you really testing your tests in automated way in your project? May 1 at 0:53
  • It is actually very rare to see projects with tests for the tests, at least after the tests' initial development stage. There are projects that occasional run mutation tests, a method where parts of the code are actually changed to induce failures, to ensure the test suite can detect the changes. Mutation testing is slow, you need to build and deploy, and inefficient, it is not always clear what change triggers what error, so it is not a good candidate to be run often.
    – Rsf
    May 2 at 9:08
  • As long any changes are in UI automation code , that does not require build and deploy. May 2 at 12:08
  • 1
    I have seen this issue in multiple projects where there is an legacy code and automation code is being maintained for couple of years. In those cases , mostly people who originally wrote the code are gone and its being maintained by others .These maintainers more likely focus on failures considering sheer number of tests (in thousands) and hardly ever wonder about the passing tests which not always pass for true reasons over long time. May 2 at 12:10

Ensuring the reliability of our automation tests is crucial for building confidence in our test suite. To achieve this, we need to validate our passing tests regularly just like failing tests.

Here are some approaches I have used to test our tests in actual large projects:

  1. Test coverage analysis - We can use coverage analysis tools to identify the areas that our automation tests are covering. By analyzing the coverage, we can ensure that all the scenarios that need to be tested are actually getting tested. In addition, this can help us identify areas that are not getting tested and require more attention.

  2. Random data inputs - We can provide random data inputs to our passing tests to ensure that they can handle unexpected inputs gracefully. By doing this, we can ensure that our passing tests can handle edge cases and scenarios that we might have overlooked.

  3. Code reviews - We can review the code of our passing tests regularly to ensure that it follows best practices and standards. This will help us identify any code smells or anti-patterns that might have crept into our tests over time.

  4. Periodic manual testing - We can also perform periodic manual testing to validate our passing tests. This will help us ensure that our automation tests are not just checking for the expected output, but also ensuring that the application is functioning as expected.

  5. Re-running on different environments - We can execute our passing tests on different environments to ensure that they are not environment-specific. This will help us ensure that our tests are resilient to changes in the environment and are not dependent on a specific configuration.

Challenges that we may face while testing our tests could include:

  1. Time constraints - Testing our tests can be a time-consuming process, especially in large automation suites. We need to ensure that we have enough time allocated to testing our tests while maintaining a balance between testing and executing the tests.

  2. Lack of resources - Testing our tests requires additional resources such as time, people, and infrastructure. We need to ensure that we have enough resources available to perform these activities.

  3. Resistance from team members - Some team members might not see the value in testing the tests and might resist this effort. We need to communicate the importance of this activity and gain buy-in from all team members.

Testing our tests is a critical aspect of building reliable automation suites. By using the approaches mentioned above, we can ensure that our passing tests are robust and can handle edge cases and scenarios that we might have overlooked.

  • Thanks Milin for sharing, however my question is focused on 2 keywords- based on large actual projects experience and automated way. Apr 28 at 8:19

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.