Can every test be done by automation?

Question

Can every test be done by automation?

Or is there anything that cannot be done by automation?

Answer 1

No.

You can't automate everything.

• You can't automate people's reactions (emotions) to your software.
• You can't automate things you don't think of. (eh? eh?)
• You can't automate users' thought patterns going from feature to feature.
• You can't run a "fun factor" metric (applies to games more than enterprise software, but still.)

And that's just in 2 minutes. I'm sure I could come up with dozens more.

Answer 2

I think attempting to automate everything without thinking about what you're actually going to test is a horrible (but unfortunately common) practice.

My standard line is that you should automate 100% of the tests that should be automated. Figuring out which tests to automate (or not to automate) is the hard part. Testers frequently waste time attempting to automate something that should not be automated, while others waste time testing something manually that should be automated.

My personal litmus test is boredom. I automate the boring stuff and use my brain to test the things that interest me. There are, of course, exceptions, but this has worked well for me.

Answer 3

No. There are several types of testing where automation is useless.

• Automated tests are pointless in exploratory testing. The main idea in such tests is to explore the functionality to find bugs, to go deviate from standard user stories.
• Security testing is mostly manual. While it is possible to use test scripts there (ex. testing an input field could contain strings with SQL or XPATH injection), the main point is to analyze program and crack it, which is hard to do using only automated tools.
• Usability testing cannot be performed with automated tests only. It is possible to use a tool gathers data about how the program is used. However, another techniques here are user survey, or interview, and user observation, which cannot be done by running a test program.

Answer 4

I want to add one aspect to "automated tests are pointless in exploratory testing."

I can imagine some scenarios, where I would like to explore something, lets say how the program behaves when I executed functionalities over and over for eight hours. Or certain workflows.

In that case manual exploring it would be too time consuming. So I might consider writing some automated tests helping me to follow up on that mission.

So as such, I would never say "automation" can't be helpful for ET, but as often (always?) it depends on the context behind.

Answer 5

Automation is good for checking, verifying, validating. It is a tool that can be used as part of your testing. Most other tests require sapient thought and do not lend well towards automation. In the end, the automation can only check what you code it to in specific terms.

Answer 6

How would you automate usability testing of app?

Answer 7

There are two answers, depending on how you interpret this question:

Yes, everything that can be done manually can be automated with the right combination of tools, and given the right environment.

However, actually creating/obtaining the tools, putting in the time to write/maintain the test, and verifying the results can be cost prohibitive, so...

No, you effectively cannot, and should not automate everything.

Answer 8

No, at the time of this writing. As has been related to me by professionals in the field, the best litmus for whether a problem cannot be automated is if it's AI-complete -- that is, if automating the task is at least as hard as solving the central artificial intelligence problem, making computers as intelligent as people (further reading).

Within this framing, consider several of the other answers to this question. Many of them hinge on problems that humans are very good at solving that computers are not, including large, multidimensional fuzzy matching and searching problems and problems bounded by EXPtime or EXPspace (such as a deterministic solver for the game Go, to derive the set of correct solutions an algorithm should come to). Humans are still imperfect at these problems, but their ability to make expert decisions and recognize patterns is still better than the cutting edge in AI research at the time of this writing.

Using this definition, you should be able to elegantly divide the problem space into automatable tasks and those that require some degree of manual intervention. From here, I would combine the former set (by analogy and design principles left outside the scope of this answer) and prune the latter set until you've arrived at a convincing and implementable set of tests for your system.

Good luck!

Answer 9

You can't automate:

• CAPTCHA
• Random numbers meeting a certain profile
• The different viewpoints that people from different cultures have

You can't practically automate (so in theory yes you could but not realistic in real world)

• All the combinations possible when procedures have many parameters
• Consistently running tests against all the different android devices and operating systems (thousands)

You don't want to automate

• UI automation for lots of different results from different data combinations (use unit testing)
• UI automation for a large number of cases/scenarios - the majority of tests should be unit tests
• Processes with a high risk of revealing confidential information. Keep a human in the loop
• Security bypasses
• Testing security without letting security know

Answer 10

You can automate every technical aspect of a project, plus some human behavior test if you like it, for example you can use ClickDensity to track the behavior of users on your website or online application (is that on-topic?)

http://www.clickdensity.com/

But as a rule of thumb, let's say that what is human behavior related is not testable.

Answer 11

There's a more fundamental answer to this question. The halting problem prevents us from being able to fully test any non-trivial software over all possible inputs. We can and should automate as much as possible in order to test more inputs more efficiently, but when doing so we need to take into account the fact that automated tests themselves are code and are also subject to the halting problem. It's turtles all the way down.

Because of all this, as tests become more complete, they become more suspect. One way to avoid a recursion trap (of writing tests to test the tests) is to recognize that the application code under test and the test suite are essentially testing each other. This means that the application itself needs to be able to track the effectiveness of the tests, which means more instrumentation built into the application to log and analyze both user inputs and user responses to application outputs. This architecture tends to show up more in mission-critical applications, the most obvious examples being telemetry systems and flight data recorders.

Another way to avoid recursion while keeping the application simpler is to have a third suite, a shim between the application and the test suite, which monitors both and provides its own analysis. Coverage test suites tend to fall in or near this architecture, for example.

So no, it's not possible to fully automate every test, but it's important to automate those we can. Because of the halting problem, software testing is a process of discovery, and the purpose of automation is to be able to quickly repeat what we know, so we have time to discover what we don't.

Answer 12

I agree with most of the people here. Not everything can be automated and by trying to do so you will move the focus away from actual testing and finding bugs.

There are many scenarios which cannot be automated or if automated may not be stable and won't provide the necessary value. Some of them are-

• Using automation for catching rendering issues in the application (look and feel) is a bad idea. There are few tools out there which does visual validation but it is really difficult to replace humans in this aspect. For example - I had a scenario where my mobile web page looked white on one Mobile phone, whereas it looked dark grey in another mobile phone. Yes, we can try to automate this but I think humans would be better at finding these subtle differences in look and feel of the application

• Using automation to figure out if an element location on a page is the same. Your automated tests will become really flaky if you start writing tests based on x,y coordinates of elements as the web page could be viewed in different browsers, devices and OS's and the coordinates are going to change based on the screen size.This means your automated tests are going to be really flaky.

• Using automation to test integrated systems which involve software, hardware, webservices, API's and cloud services all communicating in real time with each other would be a bad idea. For example - How are you going to write an automated test which tests all the end to end scenarios of fitness tracker like Fitbit? You can try as hard as possible to simulate real human movements etc but it is going to be a really difficult task to automate the entire process of a fitness tracker, rather we humans need to do actual exploratory testing on it.

And so on...

So as you can see there are many scenarios not worth automating. Automation is good in some scenarios and should be used complementary to manual testing. Also it is really important to understand the cost vs value of automating something.

-Raj

Answer 13

If possible, is it even desirable to automate 100% of your testing? If you've got an automated test for every path through the GUI with all supported clients/browsers on all supported platforms, I'd be curious about your process and strategy. Either your GUI has been static for a long time, your GUI is absurdly simple, or your test automation team are absolute rock stars. If none of those things are true then I'd have to wonder if you're devoting too much resources to automation. If not, then I would love to hear how you got an automation suite that good.

Answer 14

No, every test cannot be automated For example if you want to test a colour of a page in a web application, this is not possible with automation. Like this we can say that so many tests are there that cannot be automated like security tests, usability tests etc.

Answer 15

No you can't automate everything. Below are some of the cases where you cant use Automation.

1. Dynamically changing functionality
2. Test cases that require more Human interaction
3. CUI (command user interface) based test cases
4. We can not capture content of an Image
5. Look and Feel of Web Application.

Answer 16

Not yet, but Artificial Intelligence is getting better and better, so I expect in a couple of years that even some of the untestable options some of the answers describe can be tested with automation.

Still if your building a product I would aim to automate most of not all the testing (also known as checking). I would even argue to change the functionality to be testable with automation, or maybe not build it at all. Doing something manual for a while is fine, but every manual action in a process should be automatable if you want to be a real scale-up.