Software Quality Assurance & Testing Stack Exchange is a question and answer site for software quality control experts, automation engineers, and software testers. It only takes a minute to sign up.
Sign up to join this community
I have seen arguments that we should automate all our tests, and I have seen arguments that manual testing is necessary.
I don't know which one to believe. Is it even possible to automate all tests? When people say that all tests should be automated, do they mean the kind of tests where manual testers work through a detailed test script or do they mean the kind of tests where manual testers explore the application?
How do I decide which approach is correct?
IMHO, Anything which is monotonous & repeatable in testing can & should be automated.
Having said that, manual exploratory testing is irreplaceable and should be utilized exclusively for creative exploratory testing which is purely driven by tester's experience and intuition by using 'What if' questions to dig deeper beyond obvious test scenarios which takes skills and creativity.
The answer is "it depends".
Let's say testing is divided into two main categories; Functional and Exploratory:
Functional means "Prove that something works as per defined requirements", generally by following a test script:
Exploratory means "Try to break this", generally by the tester's own creativity and ingenuity. e.g.
Generally speaking, you should aim to automate the first kind of tests - normally after first performing them manually so you know they pass. But you can, with the right kind of software development flow, sometimes develop them before the code is even written. However in practice this is unfortunately rare.
One thing to be aware of is that sometimes it is hard to automate some functional tests.
The second kind (Exploratory) generally cannot be automated, because they rely on human intuition. However, there are exceptions - for example, data entry forms can be automated via "fuzz" tests which will try lots and lots of combinations of inputs to see if they can find some combination that causes a problem.
So when people say that all tests should be automated, yes they generally mean the Functional tests and not the Exploratory tests.
Another thing to consider is that Exploratory testing can often be an input into automated Functional testing; so for example, once you've found an edge case using manual testing, creating an automated regression test for that case can provide great value. In this was you've turned an unknown case into a defined, functional case. It's also a good time to talk to the Stakeholders and Developers to figure out if that new behaviour you've just discovered should be kept :)
The answer is again, it depends on things like
There is no point in manually performing simple functional tests thousands of times on a critical part of the application if a single day of coding can give you full automation on it (e.g a Login dialog box); on the other hand there is no point spending a month developing automation for a feature that is not important (like say a non-critical feature for an applications' "About" dialog box)
Hope that helps.
This is a pretty straight forward question. I think everyone will agree to this:
Is manual testing necessary ? --> A must
Can we replace everything with automation ? --> Mostly NO. When it comes to automation testing an application in a project, there are a lot of factors that is considered (e.g. timeline, feasibility, ROI, maintainability, future plans). In my experience, you have to be wise in deciding the extent of automation that you are planing in the project.
In summary, computers can only do test execution, and only a subset of it. Since testing encompasses more than execution, the answer is: No.
For more details and other factors, see my blog post on it:
http://thatsabug.com//blog/why_automation_will_not_save_you/
Manual Testing is the main purpose of testing it self, it's definitely necessary.
You can replace everything with automation, if you're working on a product that will be use by no one (which is I know doesn't exist).
Testing is BETTER with Automation, but full Automation?, NO.
I think the ratio will be different on each person and each project, for me it's 70% manual vs 30% automation.
Like you did right now,feedback, insight, perspective, wild idea, etc. And that's I believe something Automation can't give.
Even for a full automatic factory, they still involve human as safety fuse, right?
We can't make Manual Testing as Zero but we could minimize it. Things which are critical and repeatable must be automated.There has to be a starategy in place for converting Manual tasks to automated.
Not everything can be replaced by Automation testing and nor everything can be covered by manual testing, speaking int context of testing , they are in proportion. Automation testing help cover parts of manual testing which is repeated and can be used for stable builds with no major updates frequently. So, Yes.Manual testing is necessary.
One thing you can not test automatically is user-interface acceptance.
Maybe you can automate things like "does the OK button exist" and "does the correct thing happen if you generate a click-even on the OK button". But you can't catch bugs like the OK button being rendered in a size of 1x1 pixels, positioned outside of the viewable area, behind a different button, with the label "OK" in the wrong language, upside down and written in white font on white background.
A human tester would notice immediately that there is something wrong. But an automated test suit would need to be really advanced to detect these errors. And if you build such an advanced test suit, it would generate a ton of false positives. A human user might not even notice that the button is a pixel smaller than it used to be, as along as they can still find it. But an automatic test suit can't differ between notable and unnotable differences.
So no matter how much you automate your testing, you should always have a human tester in your deployment process as a final sanity check.
The key factor is not technical ability, but cost
Other answers have given good insights into the kinds of tests that can be readily automated, and those that can be performed better (on a technical/quality level) using Manual QA. However, I feel one key thing that is often missed is that the real decision is never "can we automate these tests", but "is it more cost effective to automate these tests".
When developing a test automation strategy, the goal is exactly the same as a manual testing strategy - to increase the product quality.
Similar to moving QA away from developers, to dedicated QA staff. Automation is not a move that directly increases quality by itself. Instead, it is simply a different way to achieve the same quality increase - which has different costs associated with it.]
Importantly, given enough time and resources - automation can be used to perform all testing on a project. However, the cost associated with most of this testing is far higher than hiring manual QA to test the same areas.
Note, that cost here does not exclusively refer to monetary cost; but also the time required and how that impacts the release schedule of your product
As such, in any situation when considering what "can" and "cannot" be automated - the real question that needs asked is; in what areas would using automation be more cost effective than using manual QA.
Key Considerations
When determining which areas may be appropriate for automation, some key criteria may be:
While development is ongoing, and features are being changed - automation will continue to require development to meet the changing requirements. In a single-release product, such as video games, the time spent developing most automation may never pay off; as testing finishes soon after development finishes. Manual testing has the advantage of flexibility - where humans can pick up any build and continue to check it. In a long-term project with only minor changes - automation costs can be recouped by running for years with only minimal maintainence, and will likely become cheaper than the equivilent manual testing.
In areas which are simple to test, but involve large amount of varied data - automation development costs may be low, with large payoffs. For example, testing that every one of 1000 configuration files loads without error - may be simple to develop as an automated test, but would have taken manual QA multiple days to check through. Likewise, localisation testing involves checking the same functionality for each language - if the automation can run in one language, it's likely to take no extra effort to run it in all others.
*For some products, there may be areas in which a failure will cause large knock-ons to future development and testing. For example, if the product takes 2 hours for manual QA to download but is untestable if it crashes on launch - automating this simple check may provide value in the reduction of lost-time for manual QA (every build your automation catches early, saves 2hours x number of testers, hours of wages).
For some products, the cost of not testing needs to be taken into account. While automation may be more expensive in the average case; it is sometimes necessary to compare its cost to the worst case. That is, if a manual test failed to catch a bug, which later makes your company liable to be sued - the cost of automation may be justified by the reduced risk, despite being more expensive than almost-equivilent manual checks.
Summary
There is no solid rule for what should and should not be automated. Each company pays different amounts for their manual QA, and for their automation developers - what makes financial sense in one company may not make sense in another, even with identical products.
As a final rule of thumb:
Automation can be considered to be an investment, which needs to be run repeatedly to pay-off against manual QA. That is, Automation scales starts expensive but scales well.
Manual QA is a flat-fee, which often starts cheaper than automation - but continues to be cost throughout the project. That is, manual QA starts cheap - but scales badly.
Once upon a time, I read and later wrote test plans which specified that certain tests must be done manually before each release.
The reasoning was that we had seen automated tests show all green even if the system was broken. When that happened we usually tried to fix the automated tests, but we accepted that no affordable test automation would give us the ease of mind of having a person say "I tested it on the stage system and it looks good" or "I logged in and it works, take the node into the load balancer again."
For instance, I have seen Selenium test suites where one test would log in, click the way to the profile page, and verify that the profile page opens, and another test would log in to create a session, navigate directly to the profile page, and then test it. Guess what? There was a new profile page with a different click path, the first test got changed, but the developers did not remove the old profile page or the second test. So the Selenium tests did not represent the customer journey any more. Yet they were all green.
There are other tests which should be automated if it is possible. Unit tests or API tests, big tests with a mind-numbing number of subtle variants. But automated tests merely prove that all assertions are green, which is necessary but not sufficient for a release.
