As I get further into test automation, I have begun to wonder what QA activity should be directed towards verifying that automated tests do in fact do what they say on the tin?

Automated Tests:

  1. Require significant investment of time and resources to set up
  2. They claim to give the company confidence in the quality of its product
  3. They are programs in themselves, and can be quite complex (indeed, GUI testing can often be frustratingly difficult to get right)
  4. Test automation is often seen as a lesser activity, and assigned to more junior developers.
  5. Review of test automation cases can be weaker than review of production code.

Given that significant resources are put into creating automated tests, and that they serve a commercially valuable purpose, what review/ QA processes should, in your view, be directed at confirming that they work? Has anyone here found tests that appear to pass whilst not actually exercising the SUT in an epistemically valid way? Or even worse, tests that just end with "return true;"? Obviously we can't get into an infinite regression, where we write tests that test tests that test tests that test tests... etc., but is it reasonable to direct some quality assurance activity towards validating test code?

If you adopt page object model style testing, you may end up with complex object models that represent the SUT to your tests - this is another area that may require QA attention - the ancillary code that supports test execution. Test framework code should be rigorous. How do you ensure that rigor?

I understand that the answer to this will depend on the criticality of the SUT (e.g. dispensary systems require greater assurance that flash games), but what sort of questions should I be asking myself and others about the validity of our test code?

1 Answer 1


Ideally, test automation should be subject to the same level of validation production code gets - which of course tends to lead to the "turtles all the way down" problem with test code that tests the test code which tests the test code which... ad infinitum.

What I've found as a viable alternative is a collection of practices and strategies that generally work to keep test automation in reasonable order (this is not an exclusive list - just what comes to mind at 7am):

  • Code reviews - either a developer or another test automator (the latter is better unless the developer is familiar with the automation code base) reviews any new code. This helps keep out the "return true" issues and also allows for suggestions to improve the code structure, mentions of helper libraries that already contain function a tester is trying to rebuild (this happens when the test codebase is large enough - I've done it then had to clean up and unify the functions), and so forth.
  • Result analysis - someone goes through every run and checks that any non-passing results are actual issues. This can be horribly time-consuming with badly written test automation (been there...) and acts as an incentive to clean up the automation code so it's easier to pinpoint the first difference from expectations. In my previous position, I inherited 10 year old automation code that had started life as record-playback using key presses to manipulate the application in test, and often the only way to figure out what had gone wrong was to replay the run (all 4+ hours of it...) and watch until something happened. I sincerely hope that particular script has been deprecated...
  • Maintain a summary of results over time - this can be really helpful when it comes to evaluating the reliability of your automation code. If one script gives a lot of false alarms, it's a good candidate for a refactor.
  • Allocate time to maintain the automation codebase - This is critical, and in my experience tends to be forgotten. When all your automators have to borrow time for maintenance on the automation codebase, it will grow more fragile over time, even if any additions include refactoring the areas affected by the new code. I'd recommend freeing someone to simply monitor how much time they spend on automation code maintenance for a few weeks to get an idea of how much time your code base needs, then using that figure as the basis of an "x hours per week" non-negotiable time allocation.
  • These are great practical tips, thanks! Is this all from your own experience, or are there any further resources you would suggest I look at? Commented Jan 6, 2015 at 17:54
  • 1
    These are all from my experience, but you could look through the automated-testing tag here, Joe Strazzere's blog allthingsquality.com and Alan Paige's blog Tooth of the Weasel (angryweasel.com/blog). Both are excellent resources and have a ton of links to other resources.
    – Kate Paulk
    Commented Jan 7, 2015 at 11:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.