Take the 2-minute tour ×
Software Quality Assurance & Testing Stack Exchange is a question and answer site for software quality control experts, automation engineers, and software testers. It's 100% free, no registration required.

Can every test be done by automation?

Or is there anything that cannot be done by automation?

share|improve this question
    
please mark one answer as "right" if you are satisfied with it –  Tarun May 4 '11 at 16:03
    
@Tarun, sure, I just want to open a little bit more, since the topic is a bit wide and getting new answers until few hours ago. –  YOU May 4 '11 at 16:19
add comment

10 Answers

up vote 28 down vote accepted

No.

You can't automate everything.

  • You can't automate people's reactions (emotions) to your software.
  • You can't automate things you don't think of. (eh? eh?)
  • You can't automate users' thought patterns going from feature to feature.
  • You can't run a "fun factor" metric (applies to games more than enterprise software, but still.)

And that's just in 2 minutes. I'm sure I could come up with dozens more.

share|improve this answer
5  
Yeah. That's an easy one. –  Bruce May 4 '11 at 6:36
1  
Liked the simplicity and straightforwardness. –  Edgar Gonzalez May 5 '11 at 14:52
1  
Right on, the simplest answer is the best...+1 –  MichaelF May 5 '11 at 19:34
3  
This answer is only true until Google invents a server environment that simulates sending your application live, baking in client archetypes for users to study their emotional reactions and, of course, those users will think of everything (at least everything your real users will.) C'mon Google, we're counting on you! –  corsiKa May 5 '11 at 19:44
1  
@Marcus I'm inclined to disagree. Just because something can be measured doesn't mean it can be automated. It isn't feasible to have an MRI scanner ready to measure responses from testers (or should we call them patients?) and even if it was, you only get their initial reaction once, so you'd need an endless supply of subjects to 'test'. I would say that if it involves a human during the execution, it is not an automated test. What you describe, by definition, requires a human, which is why you can't automate it. –  corsiKa Aug 12 '13 at 1:27
show 4 more comments

I think attempting to automate everything without thinking about what you're actually going to test is a horrible (but unfortunately common) practice.

My standard line is that you should automate 100% of the tests that should be automated. Figuring out which tests to automate (or not to automate) is tha hard part. Testers frequently waste time attempting to automate something that should not be automated, while others waste teimt testing something manually that should be automated.

My personal litmus test is boredom. I automate the boring stuff and use my brain to test the things that interest me. There are, of course, exceptions, but this has worked well for me.

share|improve this answer
add comment

No. THere are several types of testing where automation is useless.

  • Automated tests are pointless in exploratory testing. The main idea in such tests is to explore the functionality to find bugs, to go deviate from standard user stories.
  • Security testing is mostly manual. While it is possible to use test scripts there (ex. testing an input field could contain strings with SQL or XPATH injection), the main point is to analyze program and crack it, which is hard to do using only automated tools.
  • Usability testing cannot be performed with automated tests only. It is possible to use a tool gathers data about how the program is used. However, another techniques here are user survery, or interview, and user observartion, which cannot be done by running a test program.
share|improve this answer
add comment

Automation is good for checking, verifying, validating. It is a tool that can be used as part of your testing. Most other tests require sapient thought and do not lend well towards automation. In the end, the automation can only check what you code it to in specific terms.

share|improve this answer
add comment

I want to add one aspect to "automated tests are pointless in exploratory testing."

I can imagine some scenarios, where I would like to explore something, lets say how the program behaves when I executed funtionalities over and over for eight hours. Or certain workflows.

In that case manual exploring it would be too time consuming. So I might consider writing some automated tests helping me to follow up on that mission.

So as such, I would never say "automation" can't be helpful for ET, but as often (always?) it depends on the context behind.

share|improve this answer
add comment

How would you automate usability testing of app?

share|improve this answer
add comment

There are two answers, depending on how you interpret this question:

Yes, everything that can be done manually can be automated with the right combination of tools, and given the right environment.

However, actually creating/obtaining the tools, putting in the time to write/maintain the test, and verifying the results can be cost prohibitive, so...

No, you effectively cannot, and should not automate everything.

share|improve this answer
add comment

No, at the time of this writing. As has been related to me by professionals in the field, the best litmus for whether a problem cannot be automated is if it's AI-complete -- that is, if automating the task is at least as hard as solving the central artificial intelligence problem, making computers as intelligent as people (further reading).

Within this framing, consider several of the other answers to this question. Many of them hinge on problems that humans are very good at solving that computers are not, including large, multidimensional fuzzy matching and searching problems and problems bounded by EXPtime or EXPspace (such as a deterministic solver for the game Go, to derive the set of correct solutions an algorithm should come to). Humans are still imperfect at these problems, but their ability to make expert decisions and recognize patterns is still better than the cutting edge in AI research at the time of this writing.

Using this definition, you should be able to elegantly divide the problem space into automatable tasks and those that require some degree of manual intervention. From here, I would combine the former set (by analogy and design principles left outside the scope of this answer) and prune the latter set until you've arrived at a convincing and implementable set of tests for your system.

Good luck!

share|improve this answer
add comment

You can automate every technical aspect of a project, plus some human behavior test if you like it, for example you can use ClickDensity to track the behavior of users on your website or online application (is that on-topic?)

http://www.clickdensity.com/

But as a rule of thumb, let's say that what is human behavior related is not testable.

share|improve this answer
add comment

If possible, is it even desirable to automate 100% of your testing? If you've got an automated test for every path through the GUI with all supported clients/browsers on all supported platforms, I'd be curious about your process and strategy. Either your GUI has been static for a long time, your GUI is absurdly simple, or your test automation team are absolute rock stars. If none of those things are true then I'd have to wonder if you're devoting too much resources to automation. If not, then I would love to hear how you got an automation suite that good.

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.