Do Academic Researchers in Automated Software Testing Believe All Testing Can Be 100% Automated?

Question

While surfing today, I found this article on attitudes on automated software testing among academics and practitioners. Since I am currently an automated tester in industry, I agree with the practitioner side of the story: automation definitely can help with fault detection but isn't a guarantee, and there's no way all automation should be automated. Some academic testing researchers, apparently, believe that it is possible to have complete (100%) automated testing. (For the record, both groups believe that automation increases repeatability and saves tester effort).

Is this true? In my short experience, the idea that all testing can (or should be) automated makes little sense to me and has almost no utility in practice. Are there academics that follow this line of reasoning? Perhaps there's some theoretical value in 100% automated testing, but I don't know what it could be.

Answer 1

Actually, there's a bit of a distinction here - namely between whether something CAN be automated and whether it SHOULD be automated. Just about anything software does can be automated, but whether it should be is an entirely different matter.

For instance, if you're going to test whether a printout matches the on-screen display, the manual method is, more or less, print it, walk to the printer, get the printout, hold it up against the screen and eyeball it.

You could automate this: you could build a mechanical catch into the printer to detect when a page is printed (or a robot to move the page somewhere visible, depending on printer model), use a webcam to capture an image of the printed image, then use either fuzzy image comparison or OCR to validate the data in the printout. (I don't recall who it was, but someone did actually do this as proof-of-concept).

It's pretty obvious which method here is easiest and has most benefits.

Academics can and often will lean toward the theoretical approach that anything generated by software can be automated. Practising testers will usually lean toward the practical approach of not automating if it's going to take too long and be too fragile (I won't go into the robotic finger to test a pin pad or a biometric fingerprint reader). A rule of thumb is that the more things that have to be strung together to perform the test, the more fragile any automation of it will be.

The short answer is that yes, it is possible to have 100% automated coverage, but there are very few situations where you would want to have 100% coverage (I have to admit it would be rather amusing to have an automated test robot trundling around the test lab operating turnstiles, pinpads, and other devices. I just wouldn't want to be the one maintaining it.).

Answer 2

I am not sure it is productive to debate whether academics (all of them?) believe a certain thing that practitioners (all of them?) do not believe. (Edit: @joshin4colours subsequently clarified that he is only asking about some academics, not all of them.)

The article in question describes a literature review and an online survey. The literature survey attempts to discover what academics and practitioners believe by analyzing many papers by different authors about different subjects. Consider the amount of indirection that goes into that process, and the opportunities for error.

Consider too the opportunities for error that go into an online survey.

While the article is interesting, it is only one data point, the aggregate opinion of four researchers in Sweden. I would not assume that it represents the whole truth.

As a former grad student, you understand how an academic's priorities can be different from a practitioner's. It is not surprising how the two groups might reach different conclusions. Of course, the conclusions are only the distillation of many kinds of experience. More interesting than the conclusions might be a deeper understanding of those experiences. You might believe 100% automation is impossible, and I might not, but by talking with each other, we both benefit.

Answer 3

It's not just academics - browse LinkedIn or the STC ( there's a recent discussion happening now ) and you'll find recruiters and testers saying that it is or thinking that it is a goal worth striving for

Answer 4

Quoting the article- "The vision of automated testing aims at 100% automation" Vision, not a must.

At least the way I see it, achieving 100% automation would be a great thing. I do know however that with the current tools some tests can't be automated, but who knows what the future will bring.

As for the benefits they specifically say "We think that this is caused by publication bias regarding the benefits" meaning that success stories are being widely published while failures are kept in the dark (also known as the file drawer effect)

The quoted article "Software Testing Research: Achievements, Challenges, Dreams" (here) refers to 100% automation as a dream, or an ultimate goal.

Answer 5

It's a known fact that not all testing can be automated.

If anybody disagrees stick a site with a CAPTCHA in front of them and tell them to automate some functionality that requires you to fill in the CAPTCHA correctly, then stand back and laugh.

Automation is good for regression, it's good for proving that known functional paths work correctly. It is not good for exploritory testing and it is not good for accessibility/usability testing.

Generally I would always advocate putting users in front of something to check the look and feel, automation cn't really tell you if the look and feel is correct.