Here's the basic setup - over a number of years the testing team at my workplace has developed a sophisticated (probably rather more complex than strictly necessary) object-oriented, data-driven framework for our company's software product (a suite of intercommunicating applications that operate transactionally - that is, more than just data entry).

An unfortunate combination of events has left the team with a manager who knows little about the product, and me as the automation expert. One new hire with a lot of experience in relatively static data-entry automation and the manager are convinced that the automation framework is "too complex" and the reason it takes too long to get automation running.

How do I convince them that despite the complexity the framework is a robust, reliable tool and their preferred solution (glorified record and playback with a host of duplicated functions because of the transactional nature of the system) is a maintenance nightmare in the making? (I maintain the things. I know which ones are easier to debug and maintain.)

Any advice here is welcome - my team is grossly overloaded with no sign of relief any time soon so I really want to avoid anything that could make things worse.

9 Answers 9


Bret Pettichord's keynote presentation at this year's Selenium conference has a lot of great content that is directly on point. He talks in detail about why he got increasingly frustrated with the "playback and record" sales messages coming from large commercial tool vendors.

To view his presentation on YouTube, click here. Skip ahead to 9:48. It's there that he starts to talk about the insights he and other leading thinkers on test automation experts were sharing with one another in the mid 1990's. He also talks about how the "record and playback" features were added in the pre-cursor to SilkTest. Adding record and playback features was, in Bret's view, almost entirely because it was a good way to SELL tools (vs. a feature that actually worked.).

At 10:40, he tells the story of what led up to his decision to create a presentation titled "Capture / Replay is for Fools."

enter image description here

  • Bret's right on the mark.
    – user246
    Commented May 13, 2011 at 22:58
  • 1
    That answer is well worth the 50 rep bounty... I can't think of any better way to answer it than to bring in such an expert that gives good, practical, insider look as to what record and playback was designed to do. Thanks, @Justin Commented May 16, 2011 at 14:26
  • I'm marking this one as the accepted answer on the strength of that video - I'm going to be sending the video link to the whole team for review. Thanks everyone who responded - all the suggestions are good ones, and I'll probably end up using a mix of the responses to try to improve the situation I'm facing.
    – Kate Paulk
    Commented May 16, 2011 at 16:14
  • As I mentioned elsewhere, there MAY be times when record and playback is the desireable method of creating an automated test. But if the goal is to create a test that is transportable, reusable, flexible, and adaptable to different conditions, record and playback only has benefit as a research tool towards creating the other tests. Commented May 18, 2011 at 15:33
  • It's really unfortunate. One of my dreams has been to create a robust record and playback that was smart enough to store what it records in an intelligently thought out pattern where it doesn't duplicate data and re-uses existing infrastructure as well as outputing the code in an abstracted manner that is maintainable. I'm sure it would never be 100%, but it could be a lot better than the crappy script that most record/playback features generate and might actually be useful.
    – Sam Woods
    Commented Feb 16, 2012 at 16:52

I think that communication and collaboration would be the best bet here. GUI automation is fragile, and requires constant maintenance when your AUT is undergoing constant change. Record 'n Play is even worse and would make maintainability a nightmare. Automation is also it's own development effort, and should be included in the time estimates for any additional work.

You should talk to your developers for ideas on improving the framework. While record 'n play is not recommended, perhaps there are other ways to structure your framework that would allow others to get up to speed quickly. If the developers are invested in the automated testing, then the framework is bound to improve.

You could also try what management is suggesting in a new framework/environment, so that your existing one can remain intact. This trial period could either prove your point, or end up working really well.


In your defense, if a major piece of the application under test is changed, will all of the recorded scripts need to be re-recorded and debugged again?

In their defense, is the automation framework so complex that a new QA/tester couldn't walk in and start making use of/modifying it inside of a few days? Like anything else, maintainability is a key feature.

Depending on the type of application and automation suite, could the record and playback use some of the complex features of the existing automation suite with a low amount of effort?

  • if a major piece is changed, yes, all the recorded scripts will need to be redone. Assuming familiarity with our software product (takes 3 - 6 months), I would have said that basic tests could be created with the object-oriented framework within a few days. We're using TestComplete, and some of the complex features should be accessible as test items, although so far it's proving more difficult than advertised to blend the code framework and the TestComplete keyword test framework.
    – Kate Paulk
    Commented May 12, 2011 at 13:42

First, have you already talked to your manager about it? If not, it is possible that as the automation expert, and as someone with more experience with your product than the new hire, your manager may trust you enough to go with your choice of a preferred solution.

Second, by your own admission, your framework is probably more complex than necessary. If you want the framework to compete with simpler (albeit less sophisticated) technologies, you may need to find ways to make simpler tests easier to create. If you haven't already done so, I suggest sitting down with the new hire to understand why your framework is hard to use. Was the hire given the option to use your framework? Was there any documentation for it? Was someone available to help the new hire climb the learning curve for your framework?

Finally, maintenance costs show up after the tests have been around for a while, which means it will take time to demonstrate the flaws in the new hire's approach. Perhaps you should suggest to your manager that your team use both technologies through a trial period, keeping records of what the tests do and the amount of work required to maintain them. After the trial period, you should have some data to help the manager make an informed decision.

  • 1
    @user246, the biggest problem that the team has had for many years is more testing to perform than time to perform it. Trying to get - or steal - time to refactor and clean up the framework is a losing bet.
    – Kate Paulk
    Commented May 12, 2011 at 13:47
  • 1
    That's a common problem with any software that's been around for a while. You didn't say whether your manager was in charge of just QA or QA and dev both. If just QA, does your manager have experience managing test automation projects? If not, they may be uncomfortable investing a lot of resources in an refactoring effort.
    – user246
    Commented May 12, 2011 at 19:55
  • Unfortunately, the manager is in charge of both, and has no QA-related experience that I know of.
    – Kate Paulk
    Commented May 15, 2011 at 19:22

The problem in many cases, even where I am now, is that there is always new development, new features, new products being worked on so that any framework of tests, even record and playback (which is a framework of sorts) is going to require 1) maintenance in order to make sure existing automation can continue to run and provide assurance of existing functionality and 2) new test development (even record and playback) in order to build in the new tests needed for the new functionality. The problem is always to try and strike that balance between building new test capabilities quickly and efficiently (solving point 2) but making sure those new test capabilities are easiest to maintain (solving point 1).

So, how do you convince the managers? Really, it's going to take time and effort on your part to gather metrics of how long it takes to create new tests using both the old framework and the new methodology. That's one metric and, unfortunately, that's going to show that record and playback is going to show very well there. But what you'll need then is additional metrics measuring how long it takes to maintain a suite of record and playback tests versus maintaining libraried code units and data driven/table driven framework. When a feature changes, how long does it take to update all those record and playback tests for that feature versus, when a feature changes, how long does it take to modify the framework for that feature?

Ultimately, it's the bottom line of how much the manager is spending to do something versus the value of what is being done. Both record and playback automation and framework automation have, for the most part, the same value... assurance of regression of existing product feature. So, you biggest "bang for the buck" is to measure and evaluate the expenditure.

Now, along with that, rather than trying to preserve the existing framework, get practical. As you said, the existing framework has room for improvement. Perhaps some of what they are implementing for record and playback can be adapted to improve the framework. Are some of the record and playback tests being created candidates for incorporating into the framework? Are there parts of the framework that can be enhanced to make maintaining and using it easier? The management is not going to listen well to "this is wrong and bad" without something to back it up with "here's a better way of doing it".

Finally, back it up with some good research and knowledgeable papers. One that I find really helps alot is this article I found that describes, in detail, what framework testing is, why it's important, and even challenges with lists of do's and don'ts the specific problems of record and playback. That article has at the end of it an exhaustive set of references as well that can be cross referenced, researched, and so on.

Good luck!

  • FYI, I can't get the link to work. Commented May 12, 2011 at 18:40
  • Try now... I had something bad in the URl. Commented May 12, 2011 at 18:41

It sounds like you manager does have legitimate concerns, and I think that the real solution is somewhere between the two. There are two points that I think you need to address:

Record and playback

Record and playback is essentially provided as a vehicle for automated tool sales. A record and playback approach will create a lot of the same code in multiple test cases.

The way I would structure a business case against record and playback is fairly simple, I would record and present metrics on how much time you need to allow to re-create the test cases when the application is made, vs. the code-only approach.

You could illustrate this visually with a quick side-by-side technical demo.

The complexity of your framework

If your manager thinks that it takes too long to get automation up and running, they are probably right. You should determine exactly the outcome that they are after and work out where you can recommend improvements to the framework so that you can create new test cases quicker. In my stack, (testing-stax.org) we built some generators to create a lot of that code for us, allowing us to get going very quickly, despite being coded.


While this question already has many great answers, I think times have changed a bit. In most of the cases, capture and replay indeed is a maintenance horror and a well-organized page object structure combined with hand-crafted test scripts is much easier to handle in the long term.

However, several approaches exist that leverage machine learning (ML) and other techniques to overcome the maintenance issue with capture and replay. For instance, Testim incorporates historical data to rank the locators for each element individually, which stabilizes the tests over time. That is, instead of using a single, hand-picked locator, the ML algorithm uses all of them and "heals" the script automatically. Another approach is done by retest (disclaimer: I'm a software engineer at retest). It captures the entire (visible) state of the system under test (SUT), which a) makes assertions needless and b) leaves vast amounts of data for component recognition (giving a "puzzle advantage").

Unfortunately, I'm not aware of similar open-source solutions, but I think it's just a matter of time until it's developed by the community for tools such as Selenium. The bottom line is that nowadays ML and other techniques can actually make capture and replay reliable. Nonetheless, it's rather the exception than the rule. Therefore, I would give the capture and replay tool a try (in isolation) and compare both solutions. If it's a traditional capture and replay tool, the test phase probably quickly shows that your OO solution is superior, especially in terms of maintenance.


It is difficult to convince the QA manager of the usefulness of the test automation framework if the

  • recorded scripts play back correctly; a lot of time, they don't

  • the number of recorded scripts is low

  • the recorded scripts do not require any custom changes

  • the application is rather stable

For the manager, the time for recording the scripts is the most important especially if those scripts do not break.

But start changing the application often, increase the number of scripts and customize the recorded scripts and soon the usefulness of the test automation framework is clear.

I had this discussion recently with a colleague that was asking the same question.

Why cant we just record and play the scripts?

I have detailed our conversation in this article.


This is an old question but the answer has indeed changed a great deal in the past 2-3 years. The recorders of old were a disaster. They existed before Selenium and there was even a Selenium recorder which was trashed. Selenium came about to rid ourselves of recorders. That was 15 years ago however.

A new generation of recording technology emerged in the last 2-3 years. Not open source but from reputable firms. testim, Tricentis, Appvance, Functionize. Probably a few others I am forgetting.

Some create scripts in a proprietary language and others in an open language with a full IDE (we use javascript). So an end-to-end test can be quickly generated by using the AUT.

And some systems have self healing (Appvance and testim at least) which reduces maintenance as accessors change.

We have measured improved productivity in teams from 3x to 10X over Selenium using Appvance Test Designer (not counting AI test generation, just recorder). I would expect other tools would see similar improvements as well.

As test teams must move faster to meet DevOps timelines, its a good time to take another look at the modern recording technologies and especially those with ML and self healing. Recording got a bad name. But today with time and budget constraints, it's time to rethink it.

disclaimer I work for appvance.ai and the above is my own experience

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.