Take the 2-minute tour ×
Software Quality Assurance & Testing Stack Exchange is a question and answer site for software quality control experts, automation engineers, and software testers. It's 100% free, no registration required.

I'm looking for some statistical data on the success and failure of system test automation projects. I'm especially interested in the reasons of failed approaches. Can anyone help?

share|improve this question

5 Answers 5

I'm afraid I can't offer references or statistics, but I'm pretty sure you'll find that failures of test automation projects (and the reasons for them) fall into the following broad categories:

  • Heavy reliance on record/playback - this is notoriously fragile, but usually heavily sold by automation tool vendors. Automation projects failing because record/playback were used generally produce a whole lot of "automation" quickly, but are discontinued because after a few changes to the AUT, they're no longer functional.
  • Unforeseen complexity - Most of the automation projects I've dealt with have taken longer than planned because critical AUT metadata was more difficult to find than reasonable expectations would have indicated.
  • Shifting goalposts - any attempt to automate an unstable AUT - particularly one where the GUI hasn't settled - is likely to fail or at minimum consume much more time than intended. If the AUT changes rapidly, costs of maintaining even a well-designed automation framework become unsupportable.
  • Resource crunch - there simply aren't enough test automation specialists to maintain the automation software and data and increase the automation code base.

I can't speak for anyone else, but the majority of the automation projects where I work did eventually complete and are actively maintained. The ones that have been deprecated are mostly for functions/features that are no longer supported, or have been replaced by a better iteration of the test automation application framework. (Where I work, we stretch the capacity of our automation tool (TestComplete) to its limits, and have built an object-oriented framework that runs the AUT and draws on a file-based database to determine the tests to run and a different file-based database for data. We use its record/playback features primarily to find ways to access the on-screen objects for programmatic manipulation. The result is that most of the maintenance I do happens on the older scripts that are pretty much glorified record/playback that hasn't been converted yet).

There are a number of inactive scripting projects, mostly ones where there hasn't been time available to finish them.

I know this isn't really what you want, but I hope it helps point you towards what you're after.

share|improve this answer

Think of the Test Automation as any other Software Project. The reasons of fail are the same.

Please consider the following article:

http://adante.ca/hardp/?p=57

share|improve this answer
1  
While I suppose that is true at a high-enough level of abstraction, I think it is possible to identify more specific reasons for the failure of test automation projects. The trick will be finding more specific statistics. –  user246 Jan 3 '12 at 23:18

It not clear what exactly you looking: automating testing framework itself or tool for specific framework?

I think testing framework Fitnesse have nice wiki system with pretty reports. But personally I prefer to use Selenium + JUnit. Failed approaches can be described in any way you want. Just need ask developer to implement prettyLog method. Method which handle and records all errors/exceptions in your way. For myself I split my tests into small function parts and if one part fail I easily can figure out what was wrong. If it necessary I can write custom message depend by caught error/exception.

share|improve this answer
    
I don't want to create reports of my own test automation. I'm looking for data from other people's projects from the past. –  h0b0 Jan 3 '12 at 16:49

I couldn't find any figures specifically for test automation projects. For software projects in general the CHAOS Report 2009 gives these figures.

enter image description here

share|improve this answer
    
Thanks for this hint, but after some reading regarding the CHAOS Report I don't consider it a trustworthy source. –  h0b0 Jan 4 '12 at 10:34

If you are willing to dig, I suspect you will find some relevant statistics by seaching through Dr. Atif Memon's publications (and their respective references) at http://www.cs.umd.edu/~atif/publications.shtml.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.