How to effectively document exploratory testing sessions for future re-use without slowing down?

Question

Background: On using exploratory testing heavily complimenting automated testing , where every tester documents his/her observations in individual personal ways in fast timebox sessions which is working out well in finding crucial bugs.

Question:How to effectively document exploratory testing sessions observations for future re- use by other team members in the form of formal test cases?

Management Perspective: The idea is in long term, a big subset of those test cases can be good candidates for automation as well.

Concern: One big concern received from QA group is if they start documenting these sessions output in formal test cases , they will loose the advantage of speed because documenting each test case step by step takes time. Sometime more time than the testing session itself.

Answer 1

I would suggest picking up either a free browser plug-in or some other free tool that can run in the background while your testers run their exploratory sessions.

I would treat the output of the tool this way:

• If no problems are found discard the log - The nature of exploratory testing requires intuition, guesswork, and a fair amount of serendipity. If a session finds no problems, and your automated checks find no problems, you have reasonable evidence that the part of the system in test being explored is fit for use.
• If there are problems use the log to generate reproduction steps - Testers may not remember exactly what they did when the problem happened, so the log will help to isolate and minimize reproduction steps.
• If anything logged looks like it might be an automation candidate, keep it - All that's needed is to save the session in a dedicated location for potential automation candidates, possibly with a note stating which part of the session should be considered for automation.

A lightweight system like this won't place a huge burden on the test team, may help with refining and reporting bugs, and may - if proposed the right way - be enough to satisfy your management team.

The alternative I use is to document test scenarios rather than test cases - that is, I have a list of broad scenarios that I document. They might be as simple as "valid login with full permission", "valid login with manager permission", "valid login with cashier permission", "invalid username", "invalid password". They can also be more complicated, like "manager creates an order for new product type, with tax set C applied and per-month invoice payments", or even "Manager-created orders vs cashier-created orders" - it depends on the level of detail I need at the time I perform the tests.

Answer 2

Concern: One big concern received from QA group is if they start documenting these sessions output in formal test cases , they will loose the advantage of speed because documenting each test case step by step takes time. Sometime more time than the testing session itself.

Support via tool: We got similar situation in our case. Hence we searched for a tool which fits our needs regarding exploratory testing. So we didn't decide for a tool like HP ALM, we decided for a tool called QASymhpony because it fits our needs, when executing test cases it directly created test cases. So there was no need to create test cases - because this is done during test execution (sure we had to make some adaptions but it was running smoothly). So I think exploratory testing depends also on the test tool which you have.

Mob testing / pair testing: Furthermore you can do some kind of mob testing which is nearly same than pair testing. We got in contact with our Product Owner and/or developer and tested together on application. So we just didn't documented anything because the tool was already documenting the test steps. If you don't have the tool you can easily write notes and pin them up the board which we also did for collecting ideas how we can proceed with testing. And in case that we found defects or bugs THEN we created test cases just to make sure that we don't repeat the same defect. So you can reduce your relevant test cases. More about pair testing you can find in Lisa Crispins page, whichs is very helpful and delivers some input! Paaring for learning

Invite business department to reduce test cases: Furthermore we invited the business department for testing. We were very astonished how they tested. Because we tested mainly based from the user story and the business department checked other things from the customers perspective (somehow like this: Catalogue examples of exploratory testing. So I think inviting business department for testing (or even potential customers) can be a good advice for creating test cases. Of course we didn't created for each person from business department test cases, only in case that a bug has been detected. This helped us to reduce the amount of test cases (which we actually wanted to create).

Answer 3

Pick some samples for training but don't add more process for all

I recommend not documenting the details of all exploratory testing sessions for future re-use because trying to script, what is essentially a voyage of exploration, is not in the nature of exploratory testing. The main exception to that is to get examples for training and education.

Instead I would focus on teaching good exploratory testing approaches and techniques and skills, including training as mentioned above. For example, make sure your exploratory testing charter is well maintained and well published.

Always bear in mind the Agile Manifesto, specifically:

• Individuals and interactions over processes and tools
• Working software over comprehensive documentation

It's harder than it seems cos we all love the (real) advantages of process

Answer 4

Either way, it's gonna slow down testers.

However, I think you can work in pairs, one tester doing exploratory testing and the other helping out and documenting. They can swap roles once in a while. I guess this would not slow them down as much because one tester would be mainly focused on exploratory testing just like now.

There's also a question of how much you want to document the exploratory sessions. Perhaps you still want to allow future testers to use their own e.g. values/inputs. If so, the documentation can stay lean, which might not slow the testers down much even if they work alone.