I'm looking for some statistical data on the success and failure of system test automation projects. I'm especially interested in the reasons of failed approaches. Can anyone help?


I'm afraid I can't offer references or statistics, but I'm pretty sure you'll find that failures of test automation projects (and the reasons for them) fall into the following broad categories:

  • Heavy reliance on record/playback - this is notoriously fragile, but usually heavily sold by automation tool vendors. Automation projects failing because record/playback were used generally produce a whole lot of "automation" quickly, but are discontinued because after a few changes to the AUT, they're no longer functional.
  • Unforeseen complexity - Most of the automation projects I've dealt with have taken longer than planned because critical AUT metadata was more difficult to find than reasonable expectations would have indicated.
  • Shifting goalposts - any attempt to automate an unstable AUT - particularly one where the GUI hasn't settled - is likely to fail or at minimum consume much more time than intended. If the AUT changes rapidly, costs of maintaining even a well-designed automation framework become unsupportable.
  • Resource crunch - there simply aren't enough test automation specialists to maintain the automation software and data and increase the automation code base.

I can't speak for anyone else, but the majority of the automation projects where I work did eventually complete and are actively maintained. The ones that have been deprecated are mostly for functions/features that are no longer supported, or have been replaced by a better iteration of the test automation application framework. (Where I work, we stretch the capacity of our automation tool (TestComplete) to its limits, and have built an object-oriented framework that runs the AUT and draws on a file-based database to determine the tests to run and a different file-based database for data. We use its record/playback features primarily to find ways to access the on-screen objects for programmatic manipulation. The result is that most of the maintenance I do happens on the older scripts that are pretty much glorified record/playback that hasn't been converted yet).

There are a number of inactive scripting projects, mostly ones where there hasn't been time available to finish them.

I know this isn't really what you want, but I hope it helps point you towards what you're after.


Think of the Test Automation as any other Software Project. The reasons of fail are the same.

Please consider the following article:


  • 2
    While I suppose that is true at a high-enough level of abstraction, I think it is possible to identify more specific reasons for the failure of test automation projects. The trick will be finding more specific statistics. – user246 Jan 3 '12 at 23:18

It not clear what exactly you looking: automating testing framework itself or tool for specific framework?

I think testing framework Fitnesse have nice wiki system with pretty reports. But personally I prefer to use Selenium + JUnit. Failed approaches can be described in any way you want. Just need ask developer to implement prettyLog method. Method which handle and records all errors/exceptions in your way. For myself I split my tests into small function parts and if one part fail I easily can figure out what was wrong. If it necessary I can write custom message depend by caught error/exception.

  • I don't want to create reports of my own test automation. I'm looking for data from other people's projects from the past. – h0b0 Jan 3 '12 at 16:49

I couldn't find any figures specifically for test automation projects. For software projects in general the CHAOS Report 2009 gives these figures.

enter image description here

  • Thanks for this hint, but after some reading regarding the CHAOS Report I don't consider it a trustworthy source. – h0b0 Jan 4 '12 at 10:34

If you are willing to dig, I suspect you will find some relevant statistics by seaching through Dr. Atif Memon's publications (and their respective references) at http://www.cs.umd.edu/~atif/publications.shtml.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.