Can every test be done by automation?

Question

Can every test be done by automation?

Or is there anything that cannot be done by automation?

Answer 1

No.

You can't automate everything.

• You can't automate people's reactions (emotions) to your software.
• You can't automate things you don't think of. (eh? eh?)
• You can't automate users' thought patterns going from feature to feature.
• You can't run a "fun factor" metric (applies to games more than enterprise software, but still.)

And that's just in 2 minutes. I'm sure I could come up with dozens more.

Answer 2

How would you automate usability testing of app?

Answer 3

Automation is good for checking, verifying, validating. It is a tool that can be used as part of your testing. Most other tests require sapient thought and do not lend well towards automation. In the end, the automation can only check what you code it to in specific terms.

Answer 4

No. THere are several types of testing where automation is useless.

• Automated tests are pointless in exploratory testing. The main idea in such tests is to explore the functionality to find bugs, to go deviate from standard user stories.
• Security testing is mostly manual. While it is possible to use test scripts there (ex. testing an input field could contain strings with SQL or XPATH injection), the main point is to analyze program and crack it, which is hard to do using only automated tools.
• Usability testing cannot be performed with automated tests only. It is possible to use a tool gathers data about how the program is used. However, another techniques here are user survery, or interview, and user observartion, which cannot be done by running a test program.

Answer 5

I want to add one aspect to "automated tests are pointless in exploratory testing."

I can imagine some scenarios, where I would like to explore something, lets say how the program behaves when I executed funtionalities over and over for eight hours. Or certain workflows.

In that case manual exploring it would be too time consuming. So I might consider writing some automated tests helping me to follow up on that mission.

So as such, I would never say "automation" can't be helpful for ET, but as often (always?) it depends on the context behind.

Answer 6

You can automate every technical aspect of a project, plus some human behavior test if you like it, for example you can use ClickDensity to track the behavior of users on your website or online application (is that on-topic?)

http://www.clickdensity.com/

But as a rule of thumb, let's say that what is human behavior related is not testable.

Answer 7

I think attempting to automate everything without thinking about what you're actually going to test is a horrible (but unfortunately common) practice.

My standard line is that you should automate 100% of the tests that should be automated. Figuring out which tests to automate (or not to automate) is tha hard part. Testers frequently waste time attempting to automate something that should not be automated, while others waste teimt testing something manually that should be automated.

My personal litmus test is boredom. I automate the boring stuff and use my brain to test the things that interest me. There are, of course, exceptions, but this has worked well for me.

Answer 8

There are two answers, depending on how you interpret this question:

Yes, everything that can be done manually can be automated with the right combination of tools, and given the right environment.

However, actually creating/obtaining the tools, putting in the time to write/maintain the test, and verifying the results can be cost prohibitive, so...

No, you effectively cannot, and should not automate everything.

Answer 9

No, at the time of this writing. As has been related to me by professionals in the field, the best litmus for whether a problem cannot be automated is if it's AI-complete -- that is, if automating the task is at least as hard as solving the central artificial intelligence problem, making computers as intelligent as people (further reading).

Within this framing, consider several of the other answers to this question. Many of them hinge on problems that humans are very good at solving that computers are not, including large, multidimensional fuzzy matching and searching problems and problems bounded by EXPtime or EXPspace (such as a deterministic solver for the game Go, to derive the set of correct solutions an algorithm should come to). Humans are still imperfect at these problems, but their ability to make expert decisions and recognize patterns is still better than the cutting edge in AI research at the time of this writing.

Using this definition, you should be able to elegantly divide the problem space into automatable tasks and those that require some degree of manual intervention. From here, I would combine the former set (by analogy and design principles left outside the scope of this answer) and prune the latter set until you've arrived at a convincing and implementable set of tests for your system.

Good luck!

Answer 10

If possible, is it even desirable to automate 100% of your testing? If you've got an automated test for every path through the GUI with all supported clients/browsers on all supported platforms, I'd be curious about your process and strategy. Either your GUI has been static for a long time, your GUI is absurdly simple, or your test automation team are absolute rock stars. If none of those things are true then I'd have to wonder if you're devoting too much resources to automation. If not, then I would love to hear how you got an automation suite that good.