What is the sense of having a monkey tester execute your test script?

Question

In our organization project managers are responsible for writing manual tests scripts that are to be executed later by testers. However, we found hard to find the right level of details. The right means giving valuable feedback about product.

We tried the following approaches:

Option 1: Project manager writes tests with high-level perspective, e.g. "Create a user and verify it in DB", instead of "Go to view A, click B and enter C. Expect, the user has been added to the table D". The result of those tests was somehow frustrating to both project managers and testers. As test scripts were ambigous to testers who did not know the specification of product well, so was the feedback given to project managers. Additionally, in some cases, this has shown that project manager did not attempt to write details of expected output, because he did not know the application to build well.

Option 2: Project manager writes detailed tests for the testers. This has the advantage the project manager must understand the application under controls well. He uses all his creativity to invent scenarios that have certain coverage over application functionality and risk. On other hand, there remains little room for creativity of testers. In fact, we got feedback from testers they were frustrated of doing "monkey work". So, why bother testers, if project manager wrote the tests?

Option 3: A Compromise of above. Project manager writes a subset of tests. He sits with a tester for an hour or two to introduce the application and assist during performing initial tests. They share some common sense about the application expected behaviour in general. There still remains some space for tester creativity (exploration, variations of test scenarios given), but testers stop asking questions like "Is it correct to work like this?". The meeting (one or more, if necessary) gives immediate feedback to the project manager, if the tests are little detailed or expected output for some actions has not been defined. The added value is that the tester receives training about the application. This will be useful, when he will serve as application support for customers later (yes, we use testers for both in our organization). The disadvantage is that those meetings cost the time of both persons.

There is one more approach, I would be delighted to try, if our testers had more time dedicated for our project.

Option 4: Autonomous tester team. Testers are handed specification and write tests themselves based on the specs. This is ideal situation, because they have time to understand the application domain, while still use their experience to create useful tests.

How do you write tests for testers if you do not have budget for Option 4 and still want to do better than Options 1, 2 and 3?

Answer 1

Your question seems to raise two issues: which test process is the most effective, and which test process is the most satisfying (or the least frustrating) to your testers? You may believe those issues are so interrelated as to be inseparable, but your management may have a different perspective. You said Option 1 did not give the testers enough information to determine whether the software was behaving correctly, and Option 3 was time-consuming. You said Option 2 was frustrating (or perhaps even insulting) to testers, and the way in which you titled your question reinforces that sentiment. However, you did not say whether Option 2 is effective. If Option 2 is effective -- if it finds enough bugs -- then perhaps your organization needs testers who more content with monkey testing than your current testers.

Given your time constraints, if Option 2 is not effective, I would be hard-pressed to improve upon Option 3. In some organizations, testers learn about the application by attending meetings with the rest of the project team. However, if Option 3's meetings take too long, I suspect the testers do not have time for project meetings either.

You did not mention how your development team tests their work. If none of the four options is possible and/or effective, you may need to push more testing work back onto the developers. In my experience, this is a weak solution -- many developers do not want to test, especially not on a regular basis -- but it may help in the short term until you can hire more testers or give your current testers more time to do their jobs.

Answer 2

Any of the listed approaches will work with the right people on the right positions.
However you need to understand, that:

1. providing ~100% product coverage with the detailed test cases is a full time position (or even 2 or more), if done right. And assigning this task to PM is a waste of money.
2. overqualified people soon will become unhappy doing monkey job.
3. automated tests suite is a standalone project with its own bugs/debugs/releases/roadmap, which requires:
   a. requirements (which mostly are detailed test cases).
   b. developer/operator, who will run the suite and file/fix/hotfix/hack-real-time-workarounds issues found for a while.
   c. coding skills, to be able to use/fix/improve this suite the next day after initial author quit.

---

So, if you have 1 busy PM and a bunch of mid level testers, consider this:

1. Project manager gives priorities to the features with the short description.
2. Split product modules between the testers, and let each tester cover his/her own modules with tests.
3. When the regression tests time comes, make testers test others' modules. Rotate assignments every time.

This will generate some:

1. Free time for project manager for his direct responsibilities.
2. Creative and happy time for testers, as well as ownership pride boost.
3. Feedback and knowledge transfers between the testers, resulting in documents quality growth.

Answer 3

Sounds like you are working in crazy land...

My first question is how expensive are your testers? How can it be cheaper for the PM to sit down and write all the tests, hasn't (s)he got better things to be doing (like managing the project)?

It sounds like you need somebody to sit between the PM and the testers (e.g. a test manager) who can collaborate closely with the PM to ensure that they have transfered all of thier knowledge over to the testers. The test manager can also liase with the dev team to find out what they have implemented (it probably won't be the same as what the PM envisioned) and ideally to users to find out exactly what they wanted (again probably different to what the devs and PM have done).

Some positives (probably not all):

• A test manager should be there for the testers all the time whereas the PM is going to have limited time due to the nature of thier role.

• The test manager can give the PM regular updates as to how testing is progressing, and should pick up on the general feeling about the status of the product as they work with the testers all the time (this can be invaluable to the PM who may not get the true feeling of how good/bad the product is from the limited exposure they would normally get to the test team).

Some negatives:

• If you get a bad test manager with poor communication skills, or somebody who want to horde information to themselves and not pass it on you are going to have real problems (but then this person shouldn't be a test manager in the first place).

The testers should be writing scripts for the regression pack (or even better automating them and getting them running on a CI system), and performing as much exploritory testing as possible (because thats where you find the real bugs).