What are the challenges around the integration of Manual and Automated Tests?

Question

This is more a request for your professional opinion than a question...

We are seeing how test automation is spreading to more and more organizations. This in itself is a good thing, but at the same time it brings creates big challenges to organizations that want to run automated tests but still rely on manual testing to cover an important part of their scenarios.

I wrote a post about what I learned from working with some customers and other QA Managers, asking for their insights on the subject; but I'd like to get inputs from other testing professionals who have run into similar or different issues.

What are the challenges you've encountered when integrating your manual and automated testing efforts?

Answer 1

The primary challenge that I've found when it comes to integrating automated tests with manual tests is the general lack of experience of folks writing GOOD automated tests. As noted in another post, there are serious problems with organizations buying into "record and playback" test automation. If an organization starts on this route and doesn't have anyone committed to creating a good automation structure or framework that is NOT based upon record and playback, this generally reduces the confidence in the automation as being unreliable, not reusable, and generally a waste of time.

Along with that challenge is the general perception that adding automated tests of any sort is a "silver bullet" that will cure all the quality ills of the software product. Automated testing is a tool that is used by software testers to help ensure the reliability of a product but just like you don't use a hammer to drive in a screw (well... you can, but it REALLY messes up what you're working on), there are certain things that automation is well suited for and certain things that it is not. Each organization, each product, has to make those evaluations.

Finally, if you have a good grasp on creating an automation framework and you realize that it's a tool, the problem that then faces organizations is having dedicated resource to the automation. When push comes to shove and deadlines start to pressure, as usual, testing time gets cut from projects. And the first piece of testing time to cut is automation. "Let's just get it tested and out the door. We'll catch up on the automation later." And, just like with any other part of the SDLC process, later never comes. There needs to be resource specifically set aside, both time and personnel, to create, maintain, update, and improve existing automation so that it will continue to be a good tool and a good asset to the organization.

Answer 2

When people talk about the maintenance cost of automated tests, the first thing that comes to mind is the cost of fixing things that break: APIs that change, UIs that are restructured, file formats that change, and so on. But you also need to keep the automation up to date with the feature it tests.

This isn't as easy as it sounds. It is possible that no one in your organization knows what specific test cases the automated test covers. With manual tests, you probably have some kind of documentation. The automation developer may have written something down when they first wrote the code. By default, the documentation will not stay up to date with the test.

The automation developer may remember what the test does if he/she still maintains it. On the other hand, if they're no longer on the project, there is a good chance that no one is keeping track of whether automated test is up to date with the feature it is supposed to test. Ultimately, the only way to figure it out may be to read and analyze the code.

With a product, if the feature set gets out of date, your users will tell you. With an automated test, the only output you really care about is PASS, FAIL, or ERROR. The test will pass regardless of whether it covers the new things the dev team added to the product in the last two releases.

A short time after it's written, an automated test becomes a leaky abstraction. People think of the automation as "The Thing That Tests Feature X" or worse, "The Test That Tells Me Whether Feature X Works", rather than "The Thing That Tests What Feature X Did As Of Release Y". Avoiding the leaky abstraction requires at the very least maintaining a list of what new things aren't in covered by the automated test. If you know that much, you may be able to bridge the gap with manual testing. Either that or you can ship bugs.

There are things automation developers can do to help with this problem. One strategy is to write things down, but as I mentioned earlier, documentation has to be maintained, just like comments in your code. Another strategy is to write your tests so that the test cases are separate from the test code. Sometimes this isn't possible or worth the trouble, but to the degree you can do it, it will be easier to discern what the automation does -- and what it doesn't do.

Answer 3

Since the test suite is always a combination of manual and automated tests, the first challenge is to decide which tests are to be automated. Only time consuming and repetitive tests need to be automated so that the effort can be focused on exploratory testing. Since regression testing is the single largest test exercise before the version launch, it is important to manage it well; it is a good idea to use spreadsheets. The challenges with maintaining your regression test suite through spreadsheets are - lack of access control to specific team members, problem of shared access among team members to keep the test suite updated, and the risk of duplication of critical path and the rest of the test suite. These can be solved by the use of cloud-based spreadsheets. I am a technical writer with OnPath Testing and I have written this article which discusses how to manage your regression test suite with spreadsheets.