Manual testing is better than Automation testing. Is that true?

Question

With manual testing, you don't need to buy software automation tools or write scripts.

Automated tests are robotic and don’t necessarily act as a real would user. On the other hand, manual testing allows the developing program to be used as it would be upon launch. Any bugs that may pop up when a user handles the program in a certain way are more likely to be caught in manual testing.

Something that could change the course(Functionality, GUI, Etc..) of the project, you want to be able to work on it immediately. Automating tests take more time to set up, which doesn’t allow you to test ideas quickly and easily.

Answer 1

Imagine you are working in environment where new features come out rapidly and builds happen every few hours. Every new feature has a potential to break something existing in some part of the system.

You don't have time to manually do full regression testing every day, so it's a smart idea to invest in an automation suite that will perform regression testing while you do manual testing for that specific new feature/idea.

You could in theory get better results by employing more testers who will go through all regressions manually, and it could actually provide better 'testing', but it would be terribly inefficient and costly.

So, automated testing isn't 'better' than manual testing, it's simply way more efficient than manual testing.

Answer 2

I think the most appropriate answer to this is IT DEPENDS.

---

With manual testing you can always improvise and adjust your tests on run time and look into unexpected conditions and handle them well.

While in automation testing the script will do only what they are programmed to do. They will not handle unexpected conditions or any change in the AUT (Application Under Test).

Yes, you don't have to buy automation tools, but that's not just limited to manual testing. There are a lot of open-source tools out there which you can use to automate tests. So for automation testing as well you don't have to buy tools.

Yes, you have to write scripts and you have to be good at it. The scripts won't think on their own and improvise stuff. You will have to be good at coming up with correct requirements and logic and write the scripts for them.

---

You really need to understand what can be automated and what can't. You need to figure if it really is worth implementing automation for the AUT. If a project is a small project and wont have many maintenance and repetitive or consecutive cycles of development, then investing in automation will yield you no benefit because writing automation scripts require a lot of effort in terms of planning, skills, time and debugging. BUT if the project is big and is divided into several phases and has continuous release cycle, you might want to consider automation testing for regression. With each release your software will be updated with new features, bug fixes and updates. At this time you might want to do regression testing to make sure existing modules of the software have not been hampered with and the functionality is working fine. Here manual testing would become hectic.

Yes, developing automation scripts may take up a lot of time, but once developed they will save time on test execution. BUT you will also want to update your test scripts from time to time to check new conditions.

---

All this apart, if you really see, there is no such thing as automation testing. Yes YOU write some script which will execute the AUT and produce some results. BUT it won't happen on its own. YOU will be writing the script. YOU will be doing all the thinking and coming up with the logic for it. Once the script is ready YOU will be telling the computer or whatever device it is to execute the script and it will simply follow YOUR instructions. Once the script execution is completed, it is YOU who will analyze the results and decide if there was any problem in them or they were correct. Then YOU will get the list of issues from them and report them to the right person for fixing. So nothing happens on its own and hence its not automatic implying its actually manual testing!

Answer 3

The short version - It depends

The long version

Depending on the type of testing you are doing, some form of automation may be the best choice, it may be a helpful way to get yourself set up to perform the manual tests, or it may be worse than not testing at all. Or anywhere in between these extremes.

Computers are very good at doing something exactly the same way many times over and doing it faster than a human could. If the test task requires doing something many times over, doing it exactly the same way each time, and/or doing it a lot in a short space of time, automated tests are likely to be a better choice than manual tests.

People are very good at recognizing anomalies and picking up patterns. If the test task requires pattern-recognition, checking something that can't be easily encoded into a single correct oracle (such as whether a screen looks right), manual tests are likely to be a better choice than automated tests.

For everything in between, it comes down to return on investment. If you have a mature automation framework that makes adding new tests easy and modifying existing tests simple (because you don't have to change things in hundreds of places in the automation code), then you're likely to lean more towards automated tests where possible/reasonable.

In addition, there are tons of automation-assisted tasks that happen regularly, such as:

• running a query and checking the output against a saved known good file using one of the many comparison utilities (WinMerge is a nice free one).
• running a performance monitor tool to check for memory leaks, excessive memory use, and so forth.
• building a quick and dirty throwaway script to start and stop an application multiple times over to try to capture an intermittent shutdown error.
• building a throwaway script to generate a lot of data for use in a test.
• building setup scripts and teardown scripts to allow me to quickly generate a specific environment for testing something.

Some of the heuristics I use:

• 80/20 Rule - if it's part of the 20% of the application that gets 80% of the use, it's likely to be a target for automated regression.
• Release regression - if it's something that must be checked for each release, it's likely to be a target for automated regression.
• Critical path - if it would make the application or major parts of it useless should it break, it's likely to be a target for automated regression.
• ROI - if it takes a long time to do manually and is error-prone, AND it needs to be done regularly, it's likely to be a target for automated regression. An example here: at my previous workplace, tax calculations were critical to the customers. It took a team of 3 testers 5 days to run a partial tax regression test. Automating the full tax regression took several months, but once it was completed it could run every night and would catch any changes that broke the tax calculations. Within a year that automation effort was into positive ROI in terms of the time it saved the testers. In terms of regressions that didn't reach the customers, the ROI was a lot higher.
• Complexity - if there are a lot of moving parts (i.e. interacting applications and services) it probably isn't a good target for automation. The individual pieces may be automated, but the end-to-end flow probably won't be.
• External dependencies - if it relies on a third-party application or interface, it probably isn't a good target for automation. Credit providers fall into this category: not all of them leave their test environments open after an interface to them is certified, and any bugs in the environment are out of your control.
• Dependencies on hardware requiring physical manipulation - The simplest example here is checking a printout (which can be automated, but doing so is painstaking and horribly error-prone). If your software causes some physical change to hardware, then it probably isn't a good target for automation.
• "Hardness"/"Softness" - if it's something that has exactly one correct result, it may be a good target for automation (such as the correct outcome of tax calculations). If it's "soft", such as whether an image displays well or whether a screen layout looks good, it's probably not a good automation target.

This is by no means a comprehensive list, but it's a starting point.

Answer 4

I would like to make a strong case for automated testing through the following points:

• Coverage. Automated tests are much faster and give much better test coverage. What's not covered is potentially wrong.
• Reproducibility. Because timing and inputs are almost the same in different test runs the results are more consistent than with manual tests. Errors which occur are usually reproducible; not so with manual tests.
• Logging. Automated tests have automated logging and evaluation. The quality of the software is immediately visible.

Automated tests are indispensable for even slightly complex programs or devices. Manual testing is simply not able to cover a complex functionality with acceptable effort.

That said, manual tests must be performed as well; they often precede automated tests in the sense that the automation follows the user's actions. Exploratory tests are necessary supplement to the ever-identical automated tests.

But my personal experience as a tester is that test departments without a strong focus on automation don't get the job done.

Answer 5

Neither is better than the other. They aren't competing, really, but simply different approaches for a problem. Let's use an analogy.

Feet vs Car

Manual Testing (Feet)

Think of manual testing like walking or running. You need to walk places. I mean, even if just to get from your bed to your bathroom in the morning, you need to walk. With walking, it's easy to change your direction or destination. You can do a closer examination of things as you go by, including stopping to smell the roses. And all it takes to do it is your time.

But it's not the most efficient way to travel. It takes a long time to walk or run any significant distance. And doing so also takes a lot of energy. And with it, you're more prone to get lost along the way, especially on large trips. Additionally, if you walk the same route every day, you may not actually take the time to stop and smell the roses and you may miss when the wilt.

Automated Testing (Car)

On the other hand, automation is like a car. It's fast (vrooooom). You still have to drive it, but once you've got it down, it's great for getting you from point A to point B. And it's easy to follow the same route all the time. Plus, it has all these indicators so you can see exactly what might be going wrong.

But you can't stop and smell the roses. And you're stuck on a road, you can just drive around arbitrarily. Also, it's expensive. I mean, have you seen the price of a car? Plus the cost of gasoline. It can get spendy. There are ways to help mitigate that cost (ride-sharing or taxis, aka QA services), but it's still more expensive the walking. And it can only get you so far. You can't exactly drive up and down the aisles of your grocery store, you gotta walk for that.

Why not both?

This either/or approach to manual and automated is very limiting.

Manual testing is fantastic, but you can't rely solely on it for a large project. It's too much work and too much time, and no matter how fast you run, you'll never cover that distance. You need to augment it with automated testing (your car) so you can get where you need to go.

Automated testing is very powerful, but also fairly rigid and expensive. There are some things you can't test with it, and it is only setup to check for exactly what it is looking for, nothing else. You really need to do more than just automated testing and dig into things manually sometimes.

And all of this depends a lot on your project too. If you're going to your mailbox at the end of your driveway, you probably don't need to drive. Similarly, for small projects you may not need much, or any automation. But if you're trying to go across town or across the country, you definitely need automation to help you. Otherwise you're gonna be left in the dust.

Answer 6

Costs

With manual testing you also need to write scripts. If you do not make your testing efforts reproducible you will keep reinventing the wheel and maybe even forget functionality which is not obvious.

The best automated test tools are open-source and free of costs. I would really challenge if you want to buy-in to a commercial proprietary testing tool, but that is not in the scope of this question.

The learning curve of automation tools is much higher, this could be a possible cost factor to consider. The others you name are not really valid if you ask me.

Manual vs Automated testing

While writing automated functional (GUI) tests you often do manually execute the tests first to check how the application works. The Agile testing quadrants suggest you find a good balance between manual and automated testing.

I do think manual visual testing has enormous benefits to deliver better quality software. Exploratory testing and Usability testing should therefor always be preformed if possible. The ideal test pyramid does have a bit of manual testing in it, but only just a little bit. Modern software development is about automating everything, from development to deployment.

Still I would like to argue that automated testing is better. Automated testing is better for your project because:

1. It is a necessity to refactor the code to create a maintainable and extendable code-base.
2. You can release faster and get faster user feedback if you have good automated test coverage (65-90%). User feedback is much better then the feedback manual testing can ever give. (unless the application has life or death type of situations)
3. It is cheaper even on the short-term. Certainly if you do something like Test-Driven Development with YAGNI principles, as you will build only the code you really need.

Automated tests are there to safe-guard functional happy-paths in code or a product. Wonder if you want to verify all paths users probably will not execute, this could be they case when you do extensive manual testing. Don't waste testing time on things that will not happen. Do a risk analyses to think about what to test or not.

Also automated testing does not really exist. I like how James Bach puts it he calls it "Tool-Supported Testing". :)

Release cycle

More and more companies are doing insane fast release cycles to verify market-fit or to stay ahead of competitors. If you do only manual testing then your release cycle will be come slower and slower as your functionality grows. Therefor automating the core features of your product is a must now-a-days.

Don't fool yourself that manual testing is cheaper and quicker, it might feel that way if you do not have the skills to automate everything. Only the companies that automate everything will stand the test if time if you ask me. Manual testing is slower, prone to shortcuts under time pressure and more expensive from the start and certainly in the end.

Experimenting and quick learning, validating prodcuts from idea's can be done in prototypes. Have a look at http://sprintforenterprises.com/

Answer 7

Manual Testing for when you need human intelligence (Exploration Testing) to really try to screw with an interface and find weaknesses. Or when your software project is really small and doesn't warrant setting up a robot.

Automated Testing when you need big productivity of simple, repeatable procedures (Regression Testing) aka when you need heavy lifting. To give you some dimensions here - we run 10k+ End-to-End GUI tests on 70 machines in under 8 hours - created and maintained by 12 people.

Answer 8

Short answer

Manual testing is better than Automation testing. Is that true?

No. Neither is automatic testing better than manual testing. Both are tools for different purposes. They can, and should be used together.

Long answer

First of all, I applaud you for perfectly mimicking the arguments of some rather non-technical projectlead who just wants to get the job done quickly and cheaply and rush the product out as fast as possible (maintenance will be done by some other part of the company as well!).

It is a good excercise to think of arguments, all the way being aware that you're on a losing slope anyways. Great for job interviews, to play the advocatus diaboli. Let me answer in the same spirit.

Note: I think you are only knowing/talking about old-fashioned "automation software" that is very separate from the programming environment, and is just a "key stroke / mouse click recorder". There are other approaches which are much more integrated with the actual software development process (for example, https://cucumber.io )- that is what my following answers are about.

With manual testing, you don't need to buy software automation tools

True, but free tools exist.

or write scripts.

True, but repeated manual labour is much more expensive than doing a more complex job once only.

Automated tests are robotic and don’t necessarily act as a real would user.

True, and being robotic (i.e., reproducible, reliable, quick) is in favour of automated tests. Automated tests don't make humans dispensable.

On the other hand, manual testing allows the developing program to be used as it would be upon launch.

This can and should be done by automated tests as well, they are called "feature tests" or "integration tests", another buzzword is "behaviour driven development".

Any bugs that may pop up when a user handles the program in a certain way are more likely to be caught in manual testing.

True. That's why for any bug report, the first thing you do is create a "red" test to reproduce it (which is easy since you have a test ecology that supports this kind of things). This likely does not take longer than debugging the code and finding the bug with no test possibilities at hand. The process of fixing the bug makes the test green and you now also have made sure the bug will never come again.

Something that could change the course(Functionality, GUI, Etc..) of the project, you want to be able to work on it immediately. Automating tests take more time to set up, which doesn’t allow you to test ideas quickly and easily.

Having automated tests makes it so you can easily set up your data or whatever to get you to the point where you have to change something. This can easily be quicker than doing it by hand.

Answer 9

The two tests fulfill different roles in my view.

Manual testing is there to test how the software performs, how does it look, how does it feel, partially subjective and partially objective points which automation cannot cover.

automation however is useful for regression testing, and doing tests against the server or other functions of the software which return something a program can validate (like for example a data file, or a server status).

combined together they can help cover both bugs that the user will notice immediately, and also more subtle error which may not crash the app, but will certainly lead it to fall out of line in terms of the acceptance criteria. This of course is all my perspective based on experience.

Answer 10

I would like to add my 2 cents.

I believe both has its value. What's really important is the person who creates the test whether for manual or automated execution.

All tests should be created for a specific and achievable purpose. Good tests take time and effort to create. There's no point in creating and executing for the sake of it. Be it functional, non-functional e.t.c. There is no point in creating many tests just to impress the stake-holders.

There are many useful and effective test design techniques like equivalence partitioning, boundary value analysis, domain analysis (which I have never used) and pair wise. It would be ideal to create automated tests using these test design techniques. Don't forget, there's also the important test reporting and defect logging.

In the end, it's all about achieving high quality software.

Answer 11

It depends on when you should choose automation and when you choose manual You can test software through both automated and manual testing,

but which one you choose comes down to the associated costs and benefits of each on your particular project.

Automation Testing Vs Manual Testing

Runs scripts quickly and effectively but setup will take time

But in regression automation testing is more effectively run and produce results. Everything is done for you automatically.

Automation tools have some limitation can’t test for visual considerations like image color or font size but in manual you can do.

Answer 12

Manual and automated testing are very different.

Manual testing using idiots robotically following a script can be very poor compared to automated testing, which (if well written) is much more diligent. Humans performing repetitive actions over long periods of time are very bad at concentrating and spotting errors. Automated testing also allows far more testing to be performed, far faster, and so more frequently, e.g. as part of a build process.

Automated testing can also make far deeper inspections of data than humans can. For example, one system I worked on, automated tests routinely ran several hundred terabytes of data through the software under test, generating several hundred terabytes of output, and every byte was examined. This would just not be possible with manual testing.

However, manual testing (with intelligent testers with freedom to explore) is good for finding things you've not considered testing, or finding weird bugs. They're often better for testing for UX issues.

Automated testing is good for testing against precise specifications (e.g. API specs, document format specs, ...); manual testing is good where the spec isn't precise, and the tester needs to make judgement calls. This often occurs where the cost of writing a precise spec isn't worthwhile (e.g. UI is often not precisely spec'd, particularly on non-consumer software).

Answer 13

In my opinion both are desirable - but if I can only have one I'll take the automated testing, as long as I can add test cases to it. Manual testing of a non-trivial systems is extremely time-consuming and error-prone. Automated testing takes time to create but can be run quickly be run many times over. I have built systems which could not IMO have been manually tested alone and had a snowball's chance in hell of being correct. Push the button, get the green light - happy developer! :-)

Answer 14

Automated tests are robotic and don’t necessarily act as a real would user.

Not exactly true. You can use tools like jasmine that auto test in real browsers, or testing frameworks like jasmine and phantomJs that simulate webkit browsers.

Only thing automated testing can't really do is understand the context of interface design. In terms of web development your functionality could be working but the design could be complete skew-whiff. Even then I worked one place that had an inplace tool that would take a screenshot of pages during selenium tests on each commit then did image comparison to see if the image changed. Was very helpful for picking up when CSS broke other parts of a site you were not working on.

Answer 15

Some good answers here for what is a very subjective subject!

Are apples better than oranges?!! You might say that manual testing is better for exploratory testing, but even that is too simplistic - since the tester may lack a user interface, in which 'automation' becomes the way forward. You also need to look at what your definition of 'automated' is, since a lot of people think of only a single use (eg automating regression tests, in Selenium).

Automation can also create test data for manual testers for example or be used to sift through results from manual testers and produce reports...Man and machine in harmony :)

Answer 16

It is actually a very problematic question. I would argue that it is better only in some aspects. First of all, as you have said, it is cheaper and faster - in short term. If you have only a few days to test everything, there's hardly any time to automate something. But for longer periods of testing automation could bring benefits as long as automated tests are properly constructed.

And yes, most popular platforms such as Selenium cannot perform CSS testing/visual regression testing (unlike a manual tester). But there are many new QA automation tools developed by now, so one could pick an appropriate one for their project.