metrics in exploratory testing

Question

I want to track and measure session based exploratory testing effectively. What are the main metrics used in session based exploratory testing to measure efforts, charters, bugs, opportunities and coverage?

Answer 1

According to James Bach on his SBTM page, the metrics you may gather from session reports are quoted below. Of course, sessions give you simple stats. You will need manage these into a meaningful context. Personally I have used a dashboard in Excel (manually updated when sessions were completed) but if you can automatically extract the metrics and import them in a test management tool, more power to you.

The session metrics are the primary means to express the status of the exploratory test process. They contain the following elements:

• Number of sessions completed
• Number of problems found
• Function areas covered
• Percentage of session time spent setting up for testing
• Percentage of session time spent testing
• Percentage of session time spent investigating problems

Edit: of course you also have metrics at the level of charters. Especially their link to requirements (for traceability and total coverage).

Answer 2

Be careful what you establish as metrics. What you measure is what you will get. People are incredibly good at gaming system to satisfy your metric, any metric.

This is not a testing problem but management problem - underlying problem you are trying to solve is that you don't know how to measure performance of testers. You may want to research workplace exchange about how to establish useful metrics to measure performance, and what are the pitfalls.

Say, you want to measure number of problems found? Good, I will report as many problems as I can, even most trivial ones, without trying to find out if they are dupes of existing bugs, and spend no time describing the problem to make it repeatable.

Exactly this happened once in company where I worked, and it was a disaster and incredible time sink. Because low-paid "testers" had no incentive to avoid dupes, higher-paid managers had to find them and discard them. Experiment abandoned after just few weeks.