Statistics about Success of Test Automation

Question

I'm looking for some statistical data on the success and failure of system test automation projects. I'm especially interested in the reasons of failed approaches. Can anyone help?

Answer 1

I'm afraid I can't offer references or statistics, but I'm pretty sure you'll find that failures of test automation projects (and the reasons for them) fall into the following broad categories:

• Heavy reliance on record/playback - this is notoriously fragile, but usually heavily sold by automation tool vendors. Automation projects failing because record/playback were used generally produce a whole lot of "automation" quickly, but are discontinued because after a few changes to the AUT, they're no longer functional.
• Unforeseen complexity - Most of the automation projects I've dealt with have taken longer than planned because critical AUT metadata was more difficult to find than reasonable expectations would have indicated.
• Shifting goalposts - any attempt to automate an unstable AUT - particularly one where the GUI hasn't settled - is likely to fail or at minimum consume much more time than intended. If the AUT changes rapidly, costs of maintaining even a well-designed automation framework become unsupportable.
• Resource crunch - there simply aren't enough test automation specialists to maintain the automation software and data and increase the automation code base.

I can't speak for anyone else, but the majority of the automation projects where I work did eventually complete and are actively maintained. The ones that have been deprecated are mostly for functions/features that are no longer supported, or have been replaced by a better iteration of the test automation application framework. (Where I work, we stretch the capacity of our automation tool (TestComplete) to its limits, and have built an object-oriented framework that runs the AUT and draws on a file-based database to determine the tests to run and a different file-based database for data. We use its record/playback features primarily to find ways to access the on-screen objects for programmatic manipulation. The result is that most of the maintenance I do happens on the older scripts that are pretty much glorified record/playback that hasn't been converted yet).

There are a number of inactive scripting projects, mostly ones where there hasn't been time available to finish them.

I know this isn't really what you want, but I hope it helps point you towards what you're after.

Answer 2

Think of the Test Automation as any other Software Project. The reasons of fail are the same.

Please consider the following article:

http://adante.ca/hardp/?p=57

Answer 3

It not clear what exactly you looking: automating testing framework itself or tool for specific framework?

I think testing framework Fitnesse have nice wiki system with pretty reports. But personally I prefer to use Selenium + JUnit. Failed approaches can be described in any way you want. Just need ask developer to implement prettyLog method. Method which handle and records all errors/exceptions in your way. For myself I split my tests into small function parts and if one part fail I easily can figure out what was wrong. If it necessary I can write custom message depend by caught error/exception.

Answer 4

I couldn't find any figures specifically for test automation projects. For software projects in general the CHAOS Report 2009 gives these figures.

Answer 5

If you are willing to dig, I suspect you will find some relevant statistics by seaching through Dr. Atif Memon's publications (and their respective references) at http://www.cs.umd.edu/~atif/publications.shtml.