How do I convince someone who's bought the Record-Playback line that object oriented framework is essential?

Question

Here's the basic setup - over a number of years the testing team at my workplace has developed a sophisticated (probably rather more complex than strictly necessary) object-oriented, data-driven framework for our company's software product (a suite of intercommunicating applications that operate transactionally - that is, more than just data entry).

An unfortunate combination of events has left the team with a manager who knows little about the product, and me as the automation expert. One new hire with a lot of experience in relatively static data-entry automation and the manager are convinced that the automation framework is "too complex" and the reason it takes too long to get automation running.

How do I convince them that despite the complexity the framework is a robust, reliable tool and their preferred solution (glorified record and playback with a host of duplicated functions because of the transactional nature of the system) is a maintenance nightmare in the making? (I maintain the things. I know which ones are easier to debug and maintain.)

Any advice here is welcome - my team is grossly overloaded with no sign of relief any time soon so I really want to avoid anything that could make things worse.

Answer 1

Bret Pettichord's keynote presentation at this year's Selenium conference has a lot of great content that is directly on point. He talks in detail about why he got increasingly frustrated with the "playback and record" sales messages coming from large commercial tool vendors.

To view his presentation on YouTube, click here. Skip ahead to 9:48. It's there that he starts to talk about the insights he and other leading thinkers on test automation experts were sharing with one another in the mid 1990's. He also talks about how the "record and playback" features were added in the pre-cursor to SilkTest. Adding record and playback features was, in Bret's view, almost entirely because it was a good way to SELL tools (vs. a feature that actually worked.).

At 10:40, he tells the story of what led up to his decision to create a presentation titled "Capture / Replay is for Fools."

Answer 2

I think that communication and collaboration would be the best bet here. GUI automation is fragile, and requires constant maintenance when your AUT is undergoing constant change. Record 'n Play is even worse and would make maintainability a nightmare. Automation is also it's own development effort, and should be included in the time estimates for any additional work.

You should talk to your developers for ideas on improving the framework. While record 'n play is not recommended, perhaps there are other ways to structure your framework that would allow others to get up to speed quickly. If the developers are invested in the automated testing, then the framework is bound to improve.

You could also try what management is suggesting in a new framework/environment, so that your existing one can remain intact. This trial period could either prove your point, or end up working really well.

Answer 3

In your defense, if a major piece of the application under test is changed, will all of the recorded scripts need to be re-recorded and debugged again?

In their defense, is the automation framework so complex that a new QA/tester couldn't walk in and start making use of/modifying it inside of a few days? Like anything else, maintainability is a key feature.

Depending on the type of application and automation suite, could the record and playback use some of the complex features of the existing automation suite with a low amount of effort?

Answer 4

First, have you already talked to your manager about it? If not, it is possible that as the automation expert, and as someone with more experience with your product than the new hire, your manager may trust you enough to go with your choice of a preferred solution.

Second, by your own admission, your framework is probably more complex than necessary. If you want the framework to compete with simpler (albeit less sophisticated) technologies, you may need to find ways to make simpler tests easier to create. If you haven't already done so, I suggest sitting down with the new hire to understand why your framework is hard to use. Was the hire given the option to use your framework? Was there any documentation for it? Was someone available to help the new hire climb the learning curve for your framework?

Finally, maintenance costs show up after the tests have been around for a while, which means it will take time to demonstrate the flaws in the new hire's approach. Perhaps you should suggest to your manager that your team use both technologies through a trial period, keeping records of what the tests do and the amount of work required to maintain them. After the trial period, you should have some data to help the manager make an informed decision.

Answer 5

The problem in many cases, even where I am now, is that there is always new development, new features, new products being worked on so that any framework of tests, even record and playback (which is a framework of sorts) is going to require 1) maintenance in order to make sure existing automation can continue to run and provide assurance of existing functionality and 2) new test development (even record and playback) in order to build in the new tests needed for the new functionality. The problem is always to try and strike that balance between building new test capabilities quickly and efficiently (solving point 2) but making sure those new test capabilities are easiest to maintain (solving point 1).

So, how do you convince the managers? Really, it's going to take time and effort on your part to gather metrics of how long it takes to create new tests using both the old framework and the new methodology. That's one metric and, unfortunately, that's going to show that record and playback is going to show very well there. But what you'll need then is additional metrics measuring how long it takes to maintain a suite of record and playback tests versus maintaining libraried code units and data driven/table driven framework. When a feature changes, how long does it take to update all those record and playback tests for that feature versus, when a feature changes, how long does it take to modify the framework for that feature?

Ultimately, it's the bottom line of how much the manager is spending to do something versus the value of what is being done. Both record and playback automation and framework automation have, for the most part, the same value... assurance of regression of existing product feature. So, you biggest "bang for the buck" is to measure and evaluate the expenditure.

Now, along with that, rather than trying to preserve the existing framework, get practical. As you said, the existing framework has room for improvement. Perhaps some of what they are implementing for record and playback can be adapted to improve the framework. Are some of the record and playback tests being created candidates for incorporating into the framework? Are there parts of the framework that can be enhanced to make maintaining and using it easier? The management is not going to listen well to "this is wrong and bad" without something to back it up with "here's a better way of doing it".

Finally, back it up with some good research and knowledgeable papers. One that I find really helps alot is this article I found that describes, in detail, what framework testing is, why it's important, and even challenges with lists of do's and don'ts the specific problems of record and playback. That article has at the end of it an exhaustive set of references as well that can be cross referenced, researched, and so on.

Good luck!

Answer 6

It sounds like you manager does have legitimate concerns, and I think that the real solution is somewhere between the two. There are two points that I think you need to address:

Record and playback

Record and playback is essentially provided as a vehicle for automated tool sales. A record and playback approach will create a lot of the same code in multiple test cases.

The way I would structure a business case against record and playback is fairly simple, I would record and present metrics on how much time you need to allow to re-create the test cases when the application is made, vs. the code-only approach.

You could illustrate this visually with a quick side-by-side technical demo.

The complexity of your framework

If your manager thinks that it takes too long to get automation up and running, they are probably right. You should determine exactly the outcome that they are after and work out where you can recommend improvements to the framework so that you can create new test cases quicker. In my stack, (testing-stax.org) we built some generators to create a lot of that code for us, allowing us to get going very quickly, despite being coded.

Answer 7

While this question already has many great answers, I think times have changed a bit. In most of the cases, capture and replay indeed is a maintenance horror and a well-organized page object structure combined with hand-crafted test scripts is much easier to handle in the long term.

However, several approaches exist that leverage machine learning (ML) and other techniques to overcome the maintenance issue with capture and replay. For instance, Testim incorporates historical data to rank the locators for each element individually, which stabilizes the tests over time. That is, instead of using a single, hand-picked locator, the ML algorithm uses all of them and "heals" the script automatically. Another approach is done by retest (disclaimer: I'm a software engineer at retest). It captures the entire (visible) state of the system under test (SUT), which a) makes assertions needless and b) leaves vast amounts of data for component recognition (giving a "puzzle advantage").

Unfortunately, I'm not aware of similar open-source solutions, but I think it's just a matter of time until it's developed by the community for tools such as Selenium. The bottom line is that nowadays ML and other techniques can actually make capture and replay reliable. Nonetheless, it's rather the exception than the rule. Therefore, I would give the capture and replay tool a try (in isolation) and compare both solutions. If it's a traditional capture and replay tool, the test phase probably quickly shows that your OO solution is superior, especially in terms of maintenance.

Answer 8

It is difficult to convince the QA manager of the usefulness of the test automation framework if the

• recorded scripts play back correctly; a lot of time, they don't

• the number of recorded scripts is low

• the recorded scripts do not require any custom changes

• the application is rather stable

For the manager, the time for recording the scripts is the most important especially if those scripts do not break.

But start changing the application often, increase the number of scripts and customize the recorded scripts and soon the usefulness of the test automation framework is clear.

I had this discussion recently with a colleague that was asking the same question.

Why cant we just record and play the scripts?

I have detailed our conversation in this article.

Answer 9

This is an old question but the answer has indeed changed a great deal in the past 2-3 years. The recorders of old were a disaster. They existed before Selenium and there was even a Selenium recorder which was trashed. Selenium came about to rid ourselves of recorders. That was 15 years ago however.

A new generation of recording technology emerged in the last 2-3 years. Not open source but from reputable firms. testim, Tricentis, Appvance, Functionize. Probably a few others I am forgetting.

Some create scripts in a proprietary language and others in an open language with a full IDE (we use javascript). So an end-to-end test can be quickly generated by using the AUT.

And some systems have self healing (Appvance and testim at least) which reduces maintenance as accessors change.

We have measured improved productivity in teams from 3x to 10X over Selenium using Appvance Test Designer (not counting AI test generation, just recorder). I would expect other tools would see similar improvements as well.

As test teams must move faster to meet DevOps timelines, its a good time to take another look at the modern recording technologies and especially those with ML and self healing. Recording got a bad name. But today with time and budget constraints, it's time to rethink it.

disclaimer I work for appvance.ai and the above is my own experience