Take the 2-minute tour ×
Software Quality Assurance & Testing Stack Exchange is a question and answer site for software quality control experts, automation engineers, and software testers. It's 100% free, no registration required.

Here's the basic setup - over a number of years the testing team at my workplace has developed a sophisticated (probably rather more complex than strictly necessary) object-oriented, data-driven framework for our company's software product (a suite of intercommunicating applications that operate transactionally - that is, more than just data entry).

An unfortunate combination of events has left the team with a manager who knows little about the product, and me as the automation expert. One new hire with a lot of experience in relatively static data-entry automation and the manager are convinced that the automation framework is "too complex" and the reason it takes too long to get automation running.

How do I convince them that despite the complexity the framework is a robust, reliable tool and their preferred solution (glorified record and playback with a host of duplicated functions because of the transactional nature of the system) is a maintenance nightmare in the making? (I maintain the things. I know which ones are easier to debug and maintain.)

Any advice here is welcome - my team is grossly overloaded with no sign of relief any time soon so I really want to avoid anything that could make things worse.

share|improve this question

6 Answers 6

up vote 10 down vote accepted
+50

Kate,

Great question.

Bret Pettichord's keynote presentation at this year's Selenium conference has a lot of great content that is directly on point. He talks in detail about why he got increasingly frustrated with the "playback and record" sales messages coming from large commercial tool vendors.

To view his presentation on YouTube, click here. Skip ahead to 9:48. It's there that he starts to talk about the insights he and other leading thinkers on test automation experts were sharing with one another in the mid 1990's. He also talks about how the "record and playback" features were added in the pre-cursor to SilkTest. Adding record and playback features was, in Bret's view, almost entirely because it was a good way to SELL tools (vs. a feature that actually worked.).

At 10:40, he tells the story of what led up to his decision to create a presentation titled "Capture / Replay is for Fools."

enter image description here

share|improve this answer
    
Bret's right on the mark. –  user246 May 13 '11 at 22:58
    
Thanks for that link - it's wonderful. –  Kate Paulk May 15 '11 at 19:22
    
That answer is well worth the 50 rep bounty... I can't think of any better way to answer it than to bring in such an expert that gives good, practical, insider look as to what record and playback was designed to do. Thanks, @Justin –  TristaanOgre May 16 '11 at 14:26
    
I'm marking this one as the accepted answer on the strength of that video - I'm going to be sending the video link to the whole team for review. Thanks everyone who responded - all the suggestions are good ones, and I'll probably end up using a mix of the responses to try to improve the situation I'm facing. –  Kate Paulk May 16 '11 at 16:14
    
As I mentioned elsewhere, there MAY be times when record and playback is the desireable method of creating an automated test. But if the goal is to create a test that is transportable, reusable, flexible, and adaptable to different conditions, record and playback only has benefit as a research tool towards creating the other tests. –  TristaanOgre May 18 '11 at 15:33

I think that communication and collaboration would be the best bet here. GUI automation is fragile, and requires constant maintenance when your AUT is undergoing constant change. Record 'n Play is even worse and would make maintainability a nightmare. Automation is also it's own development effort, and should be included in the time estimates for any additional work.

You should talk to your developers for ideas on improving the framework. While record 'n play is not recommended, perhaps there are other ways to structure your framework that would allow others to get up to speed quickly. If the developers are invested in the automated testing, then the framework is bound to improve.

You could also try what management is suggesting in a new framework/environment, so that your existing one can remain intact. This trial period could either prove your point, or end up working really well.

share|improve this answer

In your defense, if a major piece of the application under test is changed, will all of the recorded scripts need to be re-recorded and debugged again?

In their defense, is the automation framework so complex that a new QA/tester couldn't walk in and start making use of/modifying it inside of a few days? Like anything else, maintainability is a key feature.

Depending on the type of application and automation suite, could the record and playback use some of the complex features of the existing automation suite with a low amount of effort?

share|improve this answer
    
if a major piece is changed, yes, all the recorded scripts will need to be redone. Assuming familiarity with our software product (takes 3 - 6 months), I would have said that basic tests could be created with the object-oriented framework within a few days. We're using TestComplete, and some of the complex features should be accessible as test items, although so far it's proving more difficult than advertised to blend the code framework and the TestComplete keyword test framework. –  Kate Paulk May 12 '11 at 13:42

First, have you already talked to your manager about it? If not, it is possible that as the automation expert, and as someone with more experience with your product than the new hire, your manager may trust you enough to go with your choice of a preferred solution.

Second, by your own admission, your framework is probably more complex than necessary. If you want the framework to compete with simpler (albeit less sophisticated) technologies, you may need to find ways to make simpler tests easier to create. If you haven't already done so, I suggest sitting down with the new hire to understand why your framework is hard to use. Was the hire given the option to use your framework? Was there any documentation for it? Was someone available to help the new hire climb the learning curve for your framework?

Finally, maintenance costs show up after the tests have been around for a while, which means it will take time to demonstrate the flaws in the new hire's approach. Perhaps you should suggest to your manager that your team use both technologies through a trial period, keeping records of what the tests do and the amount of work required to maintain them. After the trial period, you should have some data to help the manager make an informed decision.

share|improve this answer
1  
@user246, the biggest problem that the team has had for many years is more testing to perform than time to perform it. Trying to get - or steal - time to refactor and clean up the framework is a losing bet. –  Kate Paulk May 12 '11 at 13:47
1  
That's a common problem with any software that's been around for a while. You didn't say whether your manager was in charge of just QA or QA and dev both. If just QA, does your manager have experience managing test automation projects? If not, they may be uncomfortable investing a lot of resources in an refactoring effort. –  user246 May 12 '11 at 19:55
    
Unfortunately, the manager is in charge of both, and has no QA-related experience that I know of. –  Kate Paulk May 15 '11 at 19:22

The problem in many cases, even where I am now, is that there is always new development, new features, new products being worked on so that any framework of tests, even record and playback (which is a framework of sorts) is going to require 1) maintenance in order to make sure existing automation can continue to run and provide assurance of existing functionality and 2) new test development (even record and playback) in order to build in the new tests needed for the new functionality. The problem is always to try and strike that balance between building new test capabilities quickly and efficiently (solving point 2) but making sure those new test capabilities are easiest to maintain (solving point 1).

So, how do you convince the managers? Really, it's going to take time and effort on your part to gather metrics of how long it takes to create new tests using both the old framework and the new methodology. That's one metric and, unfortunately, that's going to show that record and playback is going to show very well there. But what you'll need then is additional metrics measuring how long it takes to maintain a suite of record and playback tests versus maintaining libraried code units and data driven/table driven framework. When a feature changes, how long does it take to update all those record and playback tests for that feature versus, when a feature changes, how long does it take to modify the framework for that feature?

Ultimately, it's the bottom line of how much the manager is spending to do something versus the value of what is being done. Both record and playback automation and framework automation have, for the most part, the same value... assurance of regression of existing product feature. So, you biggest "bang for the buck" is to measure and evaluate the expenditure.

Now, along with that, rather than trying to preserve the existing framework, get practical. As you said, the existing framework has room for improvement. Perhaps some of what they are implementing for record and playback can be adapted to improve the framework. Are some of the record and playback tests being created candidates for incorporating into the framework? Are there parts of the framework that can be enhanced to make maintaining and using it easier? The management is not going to listen well to "this is wrong and bad" without something to back it up with "here's a better way of doing it".

Finally, back it up with some good research and knowledgeable papers. One that I find really helps alot is this article I found that describes, in detail, what framework testing is, why it's important, and even challenges with lists of do's and don'ts the specific problems of record and playback. That article has at the end of it an exhaustive set of references as well that can be cross referenced, researched, and so on.

Good luck!

share|improve this answer
    
FYI, I can't get the link to work. –  Ethel Evans May 12 '11 at 18:40
    
Try now... I had something bad in the URl. –  TristaanOgre May 12 '11 at 18:41
    
Thanks, it works now! –  Ethel Evans May 12 '11 at 18:43

It sounds like you manager does have legitimate concerns, and I think that the real solution is somewhere between the two. There are two points that I think you need to address:

Record and playback

Record and playback is essentially provided as a vehicle for automated tool sales. A record and playback approach will create a lot of the same code in multiple test cases.

The way I would structure a business case against record and playback is fairly simple, I would record and present metrics on how much time you need to allow to re-create the test cases when the application is made, vs. the code-only approach.

You could illustrate this visually with a quick side-by-side technical demo.

The complexity of your framework

If your manager thinks that it takes too long to get automation up and running, they are probably right. You should determine exactly the outcome that they are after and work out where you can recommend improvements to the framework so that you can create new test cases quicker. In my stack, (testing-stax.org) we built some generators to create a lot of that code for us, allowing us to get going very quickly, despite being coded.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.