How do you ensure your 'passing' tests(UI automated) actually pass?

Question

How do you ensure your passing tests(UI automated) actually pass?

As automation engineer I think we should question our passing tests in the same spirit as we dig in our failing tests to find out reasons to fail, at least periodically.

In long standing large automation suites, there are times where a passing test has not been questioned in long time as it was always 'passing' until a direct bug in that area surfaces.

Please share your approach used in actual large projects to test the tests? Preferably through automation.

Answer 1

Your automation is code, so why not treat it like code and test it?

While you develop the code exploratory test it and make sure to include tests to verify that the test can fail, test the internal logic of the test (use unit tests for example) and the test as part of a bigger test suite ("integration tests").

When you are satisfied with the basic functionality of the test turn those into an automated test suite, keeping only a bunch of the most important tests, that will run together with other regression cycles, or periodically.

Adding Static analysis of your tests might also help catch things like return SUCCESS as the first line of a long test.

Now you are asking about actual large project and the answer becomes more fuzzy,I don't remember seeing such tests and even the famous SQLite test suite doesn't contain one.

Answer 2

Ensuring the reliability of our automation tests is crucial for building confidence in our test suite. To achieve this, we need to validate our passing tests regularly just like failing tests.

Here are some approaches I have used to test our tests in actual large projects:

1. Test coverage analysis - We can use coverage analysis tools to identify the areas that our automation tests are covering. By analyzing the coverage, we can ensure that all the scenarios that need to be tested are actually getting tested. In addition, this can help us identify areas that are not getting tested and require more attention.

2. Random data inputs - We can provide random data inputs to our passing tests to ensure that they can handle unexpected inputs gracefully. By doing this, we can ensure that our passing tests can handle edge cases and scenarios that we might have overlooked.

3. Code reviews - We can review the code of our passing tests regularly to ensure that it follows best practices and standards. This will help us identify any code smells or anti-patterns that might have crept into our tests over time.

4. Periodic manual testing - We can also perform periodic manual testing to validate our passing tests. This will help us ensure that our automation tests are not just checking for the expected output, but also ensuring that the application is functioning as expected.

5. Re-running on different environments - We can execute our passing tests on different environments to ensure that they are not environment-specific. This will help us ensure that our tests are resilient to changes in the environment and are not dependent on a specific configuration.

Challenges that we may face while testing our tests could include:

1. Time constraints - Testing our tests can be a time-consuming process, especially in large automation suites. We need to ensure that we have enough time allocated to testing our tests while maintaining a balance between testing and executing the tests.

2. Lack of resources - Testing our tests requires additional resources such as time, people, and infrastructure. We need to ensure that we have enough resources available to perform these activities.

3. Resistance from team members - Some team members might not see the value in testing the tests and might resist this effort. We need to communicate the importance of this activity and gain buy-in from all team members.

Testing our tests is a critical aspect of building reliable automation suites. By using the approaches mentioned above, we can ensure that our passing tests are robust and can handle edge cases and scenarios that we might have overlooked.