How to develop and review test cases?

Question

This is how our test case development process looks like. A tester writes a test case in a spreadsheet, with each test step and expected result. Then, another tester reviews the test case and approves it if its ok. The test writer then executes the test when needed.

The problem with the above approach is that sometimes, the test reviewers approve tests even when the test steps are missing some crucial information. This omission was discovered only when a third person was asked to execute some tests.

So, I was thinking of changing the process by making the reviewer execute the test. I believe that this might make the reviewer be more thorough because he has to execute the test, i.e. he cannot afford to miss important details. As an aside, the new way might bring out lack of documentation. When, reviewers find that system documentation is needed in order to run a test, then they will ask for it and mention it to our QA manager (during scrum meetings).

Does the new approach make sense ? If not, then are there any other approaches that we could try ?

Answer 1

Making reviewer to execute the test could catch some omission but it would not be 100%, ever.

Manual test are executed not by computers, but sapient, thinking humans. Humans can think, learn, see patterns. Computers, not so much (they can, but it requires deep learning).

Difference is the unsaid assumptions, based on the experience of the human (which is different for every person). Something obvious to one person is far from obvious for another.

Even if I write directions how to execute steps for myself, when I try to execute them a year later, I often find I forgot to mention some "obvious" steps. So I add them to my instructions, and year later, sometimes it is again not obvious even to me, and I might need more investigating.

And opposite is also true: sometimes too much details might be annoying. Like in those "for dummies" books, when they repeatedly explain some trivial details again and again, even if I learned it 20 pages ago, as if I was really a dummy who cannot remember a thing.

Best test cases has right amount details for the audience who will be using the script but not more.

Answer 2

This reminds me one of Agile Manifesto:

Individuals and interactions over processes and tools

If the missing one are "crucial information" or "important details", seems your approach will be a patch at the result but not at the root cause. When the important details are missing from testing ( or even in developing phase ), this is an issue that need to be addressed with team:

• Do staffs really understand story, its benefits and impacts to the system?
• Do they realize this information from the first place?
• Are these information/detail clear communicated during the requirement phase and documented in a story management tool you use?
• etc - any other question to identify the issue root cause.

If an answer is no, so you know the area that need to be fixed rather than the additional-cost-review-process. You can get feedback from team and keep improving it.

If answers to these questions is yes, they do understand the story. and the detail is clear communicated and documented. I believe this kind of issue should be small.

However I still see benefits of reviewing for test approach, test technique, test coverage with corner cases or functional/non-functional/security test to gain more confidence in product not the important detail of a story.

Hope it helps,

Answer 3

Does the new approach make sense?

Maybe, but how do you know for sure? Measure the number of testcase that where incomplete for a while. Change the process and measure again. Is it better now? This might be a bit time consuming, but your team is probably already doing something similar for defect leakage.

Better is to hold a systems thinking session. Thinking about how your current systems influence the number of incomplete test-cases, finding loops that have a positive effect and try to make those better. Define some actions and experiment. Monitor your process/system and keep improving it. This means adding and removing steps in your process/system.

Be careful of adding more gates to your process. Do you really want to make your process more heavy and time consuming? The goal is not better test-cases, but less defects in production, maybe there are totally different means to reach your goal.

My advice: Experiment, evaluate, adapt continuously. Start trying. Use a PDCA cycle.

Answer 4

Moving one work-product from one person to another one just for validation is not what you can call "agile". If you wish to get the maximum value for your product, then align your process toward this goal.

So instead of having one person write the test cases and another one validate them, make them work together to define what needs to be tested and how. This way you'll move most of the issues detected during the validation/execution to this single phase which will eliminate all the back and forth and the related documentation.

Then have one of them write the tests cases matching these tests requirements and have him run them at least once. There's no real value in validating the test cases since they can be adjusted during the execution and mainly because you'll get a feedback if something is missing. So I would skip the validation if the way the tests are executed has no impact on people's safety. Instead, monitor the issues and give the necessary training if the level becomes unacceptable.

Bottom line is you'll get more value by having people working together toward a common goal compared to simply ask them to execute a single task with a vertical mindset.