Why are manual tests in pen and paper style still popular?

Question

Background

In the previous company I worked for, we had manual tests since 2004 and until I left. In the beginning, these were very simple Word documents with 3 columns. The first column contained an icon, one for a step and one for verification. The second column contained an instruction to the tester. The third column was empty for the result. It looked like this:

Then I changed the company and found that the new company has a similar approach, but using three different file formats: Excel (very similar to above Word documents), plain text in Notepad++ and manual tests in MicroFocus QADirector, but only using the most basic Yes/No-style questions.

Problems of this approach

The problems I see with such a manual approach:

• the overall result of such manual tests must be determined by someone in a manual and error prone way (Word and TXT at least, Excel could depend, QADirector was the only format calculating it).
• missing metadata makes it almost impossible to select a subset of tests for retesting.
• testers unlike developers are not familiar with version control, so there is seldom a "backup" or base line available.

Question

It's now 2015 and there are lots of tools for automating tests. On the long run, automating tests saves effort and money.

Are manual tests in pen and paper style still popular or is it just a statistical accident that I worked in two companies doing testing like this?

If it is still popular, why? I see the problems, but what are advantages of such an approach?

Answer 1

Just because there are tools for automating testing on the market doesn't mean they work for everyone. See my question about Tool for automatic GUI testing of Gupta Windows Applications: we have two tools but we can't use either for full automatic testing because they can't reliably identify GUI elements. So we must do manual testing.

Furthermore, there are things in our system that must be done manually, like putting paper forms into the scanner to test the scanner and its result XML.

I would really like to see more automating testing in my company, but in addition to the technical problems (see above) there are also limited resources on the development side, so the manager can't assign resources to implement automating testing (like writing unit tests, improving the GUI for identification of elements, continuous integration). In addition, our company is part of government, which has software standards (because of licensing) and requirements to use a particular software--whether or not it fits our particular needs / wishes.

One more example: We manually found an error in our application that only occurs when two users on different machines do the exact same thing simultaneously (clicking the mouse at literally the same second). I don't know a way you could automate something like this. And even if you could, would it be worthwhile to put effort into automating such an uncommon error?

Answer 2

Manual testing, say with Word documents, indeed has some advantages, especially if you actually print the document:

• really anyone understands the concept: give someone a sheet of paper, tell him to follow the steps and place a checkmark if it's ok or a cross if not.

  As a result of that, you

  • save cost for a professional test tool
  • save training costs you'd have for the test tool
  • potentially hire untrained people, saving cost

• it scales well: just print more copies and you can perform tests in parallel

  As a result of that, you reduce time to market, earning more money.

• it does not need IT hardware

  As a result of that,

  • you save cost for IT hardware
  • you save cost for a IT administration and maintenance
  • the process it quite fault-tolerant

• anyone can evaluate the reports: give him the papers, tell him to check for crosses and build a report

  As a result of that, you

  • save cost for a professional reporting tool
  • save training costs you'd have for the reporting tool
  • potentially hire untrained people, saving cost

Even if you don't print the Word document, it still leaves you with a reduced skill set for the testers: basic Office knowledge.

Answer 3

Performing test plans on printed papers should not be considered as a common practice for several reasons:

• physical documents could get lost
• you should have a safe space for storing them
• it's difficult to look up for an old test
• environmental staff
• and many more

However there are at least two cases for performing tests in printed paper:

1. When you are executing the commissioning of a software and the final result has to be signed for a formal acceptance
2. When you are performing an inspection and you are physically not allowed to use a laptop or a mobile device

Answer 4

Manual tests in pen and paper are still popular because:

• almost no setup or configuration is required
• they can be started immediately at any time.
• no programming knowledge is required
• programmers are in very short supply and and are very expensive in most places
• almost anyone can do them including entry-level or remote overseas folks
• real humans are still involved and they are better at spotting unplanned things that are wrong
• they are flexible. You can quickly change when you don't have to maintain a testing infrastructure
• they are fast in the short term and you don't get stuck with test suites that run for hours.
• the results are recorded in a format that provides structure as guidance to testing
• there is no "it works on your machine but not mine, configuration issues" which burn time and $
• automated UI testing can result in brittle or fragile tests that don't add value but do take time & $

Basically it makes the testing cheap, easy to do and widely available.

Please note - I am not defending manual testing! I think automated testing IS the way to go. I am just looking to answer the exact question posed of why it is still popular.