As I get further into test automation, I have begun to wonder what QA activity should be directed towards verifying that automated tests do in fact do what they say on the tin?
- Require significant investment of time and resources to set up
- They claim to give the company confidence in the quality of its product
- They are programs in themselves, and can be quite complex (indeed, GUI testing can often be frustratingly difficult to get right)
- Test automation is often seen as a lesser activity, and assigned to more junior developers.
- Review of test automation cases can be weaker than review of production code.
Given that significant resources are put into creating automated tests, and that they serve a commercially valuable purpose, what review/ QA processes should, in your view, be directed at confirming that they work? Has anyone here found tests that appear to pass whilst not actually exercising the SUT in an epistemically valid way? Or even worse, tests that just end with "return true;"? Obviously we can't get into an infinite regression, where we write tests that test tests that test tests that test tests... etc., but is it reasonable to direct some quality assurance activity towards validating test code?
If you adopt page object model style testing, you may end up with complex object models that represent the SUT to your tests - this is another area that may require QA attention - the ancillary code that supports test execution. Test framework code should be rigorous. How do you ensure that rigor?
I understand that the answer to this will depend on the criticality of the SUT (e.g. dispensary systems require greater assurance that flash games), but what sort of questions should I be asking myself and others about the validity of our test code?