Take the 2-minute tour ×
Software Quality Assurance & Testing Stack Exchange is a question and answer site for software quality control experts, automation engineers, and software testers. It's 100% free, no registration required.

While surfing today, I found this article on attitudes on automated software testing among academics and practitioners. Since I am currently an automated tester in industry, I agree with the practitioner side of the story: automation definitely can help with fault detection but isn't a guarantee, and there's no way all automation should be automated. Some academic testing researchers, apparently, believe that it is possible to have complete (100%) automated testing. (For the record, both groups believe that automation increases repeatability and saves tester effort).

Is this true? In my short experience, the idea that all testing can (or should be) automated makes little sense to me and has almost no utility in practice. Are there academics that follow this line of reasoning? Perhaps there's some theoretical value in 100% automated testing, but I don't know what it could be.

share|improve this question
    
Thanks for the responses everyone. It would be great to hear from anyone within academia with related knowledge as well :) –  joshin4colours Apr 7 '12 at 1:23

5 Answers 5

up vote 6 down vote accepted

Actually, there's a bit of a distinction here - namely between whether something CAN be automated and whether it SHOULD be automated. Just about anything software does can be automated, but whether it should be is an entirely different matter.

For instance, if you're going to test whether a printout matches the on-screen display, the manual method is, more or less, print it, walk to the printer, get the printout, hold it up against the screen and eyeball it.

You could automate this: you could build a mechanical catch into the printer to detect when a page is printed (or a robot to move the page somewhere visible, depending on printer model), use a webcam to capture an image of the printed image, then use either fuzzy image comparison or OCR to validate the data in the printout. (I don't recall who it was, but someone did actually do this as proof-of-concept).

It's pretty obvious which method here is easiest and has most benefits.

Academics can and often will lean toward the theoretical approach that anything generated by software can be automated. Practising testers will usually lean toward the practical approach of not automating if it's going to take too long and be too fragile (I won't go into the robotic finger to test a pin pad or a biometric fingerprint reader). A rule of thumb is that the more things that have to be strung together to perform the test, the more fragile any automation of it will be.

The short answer is that yes, it is possible to have 100% automated coverage, but there are very few situations where you would want to have 100% coverage (I have to admit it would be rather amusing to have an automated test robot trundling around the test lab operating turnstiles, pinpads, and other devices. I just wouldn't want to be the one maintaining it.).

share|improve this answer
    
+1 for giving a nice example of "try to automate everything" vs. "solving problems in the most efficient way". –  joshin4colours Apr 4 '12 at 17:56
    
Agree, and in addition to how much effort it is to automate it, there is also how much effort it is to execute the automation. If you automate everything, but it takes multiple days to complete the execution of that automation, or tons of resources then it may not make much sense either. –  Sam Woods Apr 5 '12 at 15:56

I am not sure it is productive to debate whether academics (all of them?) believe a certain thing that practitioners (all of them?) do not believe. (Edit: @joshin4colours subsequently clarified that he is only asking about some academics, not all of them.)

The article in question describes a literature review and an online survey. The literature survey attempts to discover what academics and practitioners believe by analyzing many papers by different authors about different subjects. Consider the amount of indirection that goes into that process, and the opportunities for error.

Consider too the opportunities for error that go into an online survey.

While the article is interesting, it is only one data point, the aggregate opinion of four researchers in Sweden. I would not assume that it represents the whole truth.

As a former grad student, you understand how an academic's priorities can be different from a practitioner's. It is not surprising how the two groups might reach different conclusions. Of course, the conclusions are only the distillation of many kinds of experience. More interesting than the conclusions might be a deeper understanding of those experiences. You might believe 100% automation is impossible, and I might not, but by talking with each other, we both benefit.

share|improve this answer
    
"I am not sure it is productive to debate whether academics (all of them?) believe a certain thing that practitioners (all of them?) do not believe." Well said! –  Joe Strazzere Apr 4 '12 at 11:05
    
Ok, fair point about saying whether "all" academics believe certain things. As for my experience, I have seen academics present models with potentially major issues in both their assumptions and analysis. Usually though, I understand why they did this from some standpoint. Here, I'm trying to see why an academic might even believe 100% automation is a good goal. –  joshin4colours Apr 4 '12 at 13:09
    
Academics never agree on anything so the answer is no. –  RicJHill Jan 8 '13 at 10:22

It's a known fact that not all testing can be automated.

If anybody disagrees stick a site with a CAPTCHA in front of them and tell them to automate some functionality that requires you to fill in the CAPTCHA correctly, then stand back and laugh.

Automation is good for regression, it's good for proving that known functional paths work correctly. It is not good for exploritory testing and it is not good for accessibility/usability testing.

Generally I would always advocate putting users in front of something to check the look and feel, automation cn't really tell you if the look and feel is correct.

share|improve this answer
1  
"If anybody disagrees stick a site with a CAPTCHA in front of them and tell them to automate some functionality that requires you to fill in the CAPTCHA correctly, then stand back and laugh." Bad example. I (and others) have automated the testing of a CAPTCHA feature. It's not hard to do if you build the feature with testing in mind - just like any other feature. –  Joe Strazzere Apr 4 '12 at 12:09
    
To automate the testing of a CAPTCHA you need to do one of the following: 1 - Turn it off; 2 - Fix the response to a static answer to enable you to enter it correctly; 3 - Supply the answer to the CAPTCHA to your test (e.g. hidden div with the correct answer). If you managed to automate by reading the picture the CAPTCHA is not fit for purpose and should have failed your testing. –  Ardesco Apr 4 '12 at 13:35
    
My point is that by default a CAPTCHA will kill automated testing. Yes you can work around the problems it presents by being clever and doing one of the options suggested above, but I'll guarantee I can kill your 100% automated dream with a CAPTCHA if I don't give you a way to bypass it. –  Ardesco Apr 4 '12 at 13:42
1  
But the same argument can be made about any feature. I guarantee I can kill your 100% automated dream with a feature I create if that's my goal. On the other hand, if I want to release a well-tested feature into production, I'll find a way to ensure it is testable. Sometimes that means providing a way to automate those tests, sometimes not. (I get what you are saying about a CAPTCHA, I just think that's a bad example for the point you are trying to make.) –  Joe Strazzere Apr 4 '12 at 14:43
    
The whole point of a CAPTCHA is to prevent automation, I fail to see how it's a bad example... –  Ardesco Apr 4 '12 at 15:11

Quoting the article- "The vision of automated testing aims at 100% automation" Vision, not a must.

At least the way I see it, achieving 100% automation would be a great thing. I do know however that with the current tools some tests can't be automated, but who knows what the future will bring.

As for the benefits they specifically say "We think that this is caused by publication bias regarding the benefits" meaning that success stories are being widely published while failures are kept in the dark (also known as the file drawer effect)

The quoted article "Software Testing Research: Achievements, Challenges, Dreams" (here) refers to 100% automation as a dream, or an ultimate goal.

share|improve this answer

It's not just academics - browse LinkedIn or the STC ( there's a recent discussion happening now ) and you'll find recruiters and testers saying that it is or thinking that it is a goal worth striving for

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.