Take the 2-minute tour ×
Software Quality Assurance & Testing Stack Exchange is a question and answer site for software quality control experts, automation engineers, and software testers. It's 100% free, no registration required.

This is more a request for your professional opinion than a question...

We are seeing how test automation is spreading to more and more organizations. This in itself is a good thing, but at the same time it brings creates big challenges to organizations that want to run automated tests but still rely on manual testing to cover an important part of their scenarios.

I wrote a post about what I learned from working with some customers and other QA Managers, asking for their insights on the subject; but I'd like to get inputs from other testing professionals who have run into similar or different issues.

What are the challenges you've encountered when integrating your manual and automated testing efforts?

share|improve this question
add comment

2 Answers 2

up vote 7 down vote accepted

The primary challenge that I've found when it comes to integrating automated tests with manual tests is the general lack of experience of folks writing GOOD automated tests. As noted in another post, there are serious problems with organizations buying into "record and playback" test automation. If an organization starts on this route and doesn't have anyone committed to creating a good automation structure or framework that is NOT based upon record and playback, this generally reduces the confidence in the automation as being unreliable, not reusable, and generally a waste of time.

Along with that challenge is the general perception that adding automated tests of any sort is a "silver bullet" that will cure all the quality ills of the software product. Automated testing is a tool that is used by software testers to help ensure the reliability of a product but just like you don't use a hammer to drive in a screw (well... you can, but it REALLY messes up what you're working on), there are certain things that automation is well suited for and certain things that it is not. Each organization, each product, has to make those evaluations.

Finally, if you have a good grasp on creating an automation framework and you realize that it's a tool, the problem that then faces organizations is having dedicated resource to the automation. When push comes to shove and deadlines start to pressure, as usual, testing time gets cut from projects. And the first piece of testing time to cut is automation. "Let's just get it tested and out the door. We'll catch up on the automation later." And, just like with any other part of the SDLC process, later never comes. There needs to be resource specifically set aside, both time and personnel, to create, maintain, update, and improve existing automation so that it will continue to be a good tool and a good asset to the organization.

share|improve this answer
3  
+1 I couldn't agree more. You need someone dedicated to creating and maintaining tests, while also having a firm grasp on 'what makes a good test'. As soon as you make automated testing something 'you do when you have spare time', it will get pushed aside. –  Todd Bumbarger May 18 '11 at 13:11
3  
Pretty much says everything I would have said :) Another problem I have dealt with before is the idea that automated scripts can be offshored and manual testing can be performed onshore. This generally doesn't work, the offshore team doesn't know what you are testing and you will spend more time supporting them, telling them how to make the automated tests do what you want and reviewing scripts than if you had written the automated tests yourself. There needs to be buy in for automation for it to work, it's not a case of hire some junior devs offshore and all will be good because it won't. –  Ardesco May 18 '11 at 13:12
    
Definitely "what they said" - if there's no-one dedicated to maintaining and building automated tests, the end result is that the tests become unusable and the concept of automated testing becomes tainted ground. The automation team needs to be good at automating, good at testing, and have a good understanding of the product they're testing or the tests are less useful than they could be. –  Kate Paulk May 18 '11 at 13:19
    
Let me add that there are times when record and playback is suitable for certain tasks for automated testing. If your intent is to create a quick bit of code to do something and then throw it away, record and playback is DEFINITELY the way to go. But automated tests that are intended to be reused need to take a lot more thought and planning and, while record and playback may give you some simple structures to work from, it should not be the final form of your automated test. –  TristaanOgre May 18 '11 at 15:17
add comment

When people talk about the maintenance cost of automated tests, the first thing that comes to mind is the cost of fixing things that break: APIs that change, UIs that are restructured, file formats that change, and so on. But you also need to keep the automation up to date with the feature it tests.

This isn't as easy as it sounds. It is possible that no one in your organization knows what specific test cases the automated test covers. With manual tests, you probably have some kind of documentation. The automation developer may have written something down when they first wrote the code. By default, the documentation will not stay up to date with the test.

The automation developer may remember what the test does if he/she still maintains it. On the other hand, if they're no longer on the project, there is a good chance that no one is keeping track of whether automated test is up to date with the feature it is supposed to test. Ultimately, the only way to figure it out may be to read and analyze the code.

With a product, if the feature set gets out of date, your users will tell you. With an automated test, the only output you really care about is PASS, FAIL, or ERROR. The test will pass regardless of whether it covers the new things the dev team added to the product in the last two releases.

A short time after it's written, an automated test becomes a leaky abstraction. People think of the automation as "The Thing That Tests Feature X" or worse, "The Test That Tells Me Whether Feature X Works", rather than "The Thing That Tests What Feature X Did As Of Release Y". Avoiding the leaky abstraction requires at the very least maintaining a list of what new things aren't in covered by the automated test. If you know that much, you may be able to bridge the gap with manual testing. Either that or you can ship bugs.

There are things automation developers can do to help with this problem. One strategy is to write things down, but as I mentioned earlier, documentation has to be maintained, just like comments in your code. Another strategy is to write your tests so that the test cases are separate from the test code. Sometimes this isn't possible or worth the trouble, but to the degree you can do it, it will be easier to discern what the automation does -- and what it doesn't do.

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.