Decision to separate or combine Test Procedure and Test Case documentation

Question

Background: Combining separate Test Procedure and Test Case documentation has advantages, but so does leaving them separate. IEEE std. 829 mentions system level of complexity as a contributing factor to the decision to combine. From experience, it seems the tradeoff in terms of complexity is whether including test procedures makes the suite too cumbersome to navigate. Reused test procedures that are lengthy may slow down the tester when they could instead repeat a procedure from memory. On the other hand, switching between documents can be slow.

Has anyone reached a more concrete point of view on the tradeoffs of these approaches and a method for deciding between them?

Answer 1

I have wrestled with this problem a number of times.

For instance, I have had a number of occasions where I needed to hand each member of a group of testers a stack of paper to describe the steps they had to take.

1. The first time a tester needs a detailed procedure it has to be there for them to read immediately. Similarly, when the procedures are very similar but differ in important details, the procedure needs to be right there.
2. Subsequent times, I don't fold the procedures into the test script, but simply refer to it by a distinguished name.

I have written a number of different tools to handle this problem, each with its own set of quirks and annoyances, for me and the people doing the testing.

My current vision of a test documentation package involves generating fairly sophisticated HTML. For instance, it might use disclosure triangles or pop-ups or pop-overs to allow the tester to view the details as often as they need in order to become familiar. The basic test documentation looks short and straightforward, with most of the detail hidden until needed.

In a similar vein, the web page with the test procedure in it would be an active form where items could be checked off as they are completed, and details written down when needed, and anomalies or bugs recorded at the exact point where they were noticed in the script. This requires some non-trivial back end software. I have looked into using a survey generator/capture tool for this kind of support of manual testing.

Answer 2

Since you have said it yourself that having separate documents may add to tester's time, wouldn't it make sense to have a checklist instead of such vast and mostly not so easy to read documents. The checklist can be used for testing and whatever information is gathered during testing can be properly formatted in a document so that the intended stakeholder can easily read and understand it!