Can exploratory testing replace executing manual functional tests?

Question

After a feature is delivered, we have recently started to perform exploratory testing on the feature before we execute manual functional tests. We find 95% of the defects during exploratory testing so is it worth while spending the time to write a full suite of manual functional tests if hardly any defects are found from them?

Answer 1

Do you think you would find the remaining 5% of bugs that are currently found through a suite of manual tests if you spent that time doing exploratory testing instead? Would managers have the confidence in test coverage that they need to make decisions without those functional tests?

I think the biggest risk of not having a manual functional test suite would be a reduction in coverage, so that things might get missed. However, a "full suite" of manual regression tests might not be necessary to ensure that coverage. Automated tests are often considered an ideal alternative, since they simplify regression testing and save testing tome long-term. They can really free up your exploratory manual testers so they can focus on the most productive forms of testing.

You could also try guided exploratory testing, where testers are given a checklist instead of fully scripted manual tests, and then the testers just make sure they cover the checklist as they perform exploratory testing, in addition to anything else they think is worth trying. This technique is sometimes called "Guided exploratory testing". This might be a great alternative for your team, since it sounds like you have some very skilled exploratory testers on hand.

Answer 2

"Is it worth while spending the time to write a full suite of manual functional tests if hardly any defects are found from them?"

That depends - is the purpose of writing them to find defects? If so, then it sounds like you would be better off putting your energy into spending more time testing. If you're considering dropping them, then it doesn't sound like you have any contractual or political obligations to produce them. Manual functional test suites tend to be expensive to produce and expensive to maintain, so unless you have really good reasons for producing that documentation there are probably better ways to spend your money/time.

So what's stopping you?

Some common concerns might be:

1) How do we look at how well we're covering the system with our tests? Isn't there a risk that we'll all end up running the same tests? Or maybe we'll all run the same sort of tests and nobody will think of testing security, or installation?

If this is a concern for you, then one way of approaching this would be to write a set of exploratory test charters instead of test scripts. This article by Michael Kelly gives a good explanation of managing testing with charters, part of a series.

I've also been really enjoying Darren McMillan's blog posts about how he's using mindmapping to plan his testing and record test sessions. I haven't used mindmapping seriously yet for ET myself, but the mindmaps I see other exploratory testers using are just such a lovely clear way of presenting what was covered, and it's much easier to see gaps that remain to be tested.

2) How do we prove what we did?

You don't have to write stuff down before you test, to prove that you tested it. You can write stuff down while you test, and then you can still respond to what you're finding. (Actually, making good notes while you test is a lot more likely to reflect what you really tested than making them before - most scripted testers I've observed frequently went "off-script", so the scripts weren't a good guide to what was tested.)

Here are some good resources/ideas on recording ET sessions:

• http://www.quicktestingtips.com/tips/2009/11/template-for-session-notes/
• Alternatives to Session Tester for recording exploratory test session results

And here's a great list of general resources about exploratory testing.

Answer 3

I will predicate my answer on the assumption that you believe you are leveraging exploratory testing as well as you can; in other words, that you do not believe you can discover more than 95% of your defects via exploratory testing. I think the gist of your question is, "Is it worth the additional work of having a full suite of manual tests in order to uncover the other 5% of defects."

You and your team need to decide whether skipping the additional testing is worth the risk of letting a bug slip through. When considering the risk, think about the kinds of bugs that might slip through, the frequency of those bugs (i.e. will the circumstances be rare or common?), the severity (when the bug occurs, how detrimental will it be?), and how forgiving you think your customers will be (e.g. high-tech early adopters may be more forgiving than online banking customers).

Aside from that, you need to consider how your customers will perceive your testing process. In some industries, your testing process is an private, internal matter. In other industries, having a well-documented process that is strictly followed is a necessary condition to achieving a certification.

Finally, how you answer those questions may change over time as your product, organization, and customer base matures. It may be that today your exploratory testing is enough, but in two years you need to invest the extra effort in a manual test suite.

Answer 4

In using "exploratory" testing we've found that it's VERY dependant on the testers experience and understanding of the system. If the tester has a broad knowledge of how the software should work and what should be displayed then they'll be able to catch all those little bugs. The manual process may provide better coverage but might miss an element that should be displayed on an existing report.

Exploratory testing is very good for bringing in external users. Less step by step with the freedom to roam throughout the application. With this being said exploratory isn't the best style when data integrity is involved... just my two bits. Happy testing all!

Answer 5

What do you do with your scripted manual tests? If they are just executed by testers once or twice then tossed you probably can change your process & reduce/eliminate the extra paperwork.

But if any one else is using them, then consider the impact outside of QA before elimating them.
Do you have requirements so that the legal department is expecting them to be there? Or do you do test plan reviews that coders, analysts or others expect & use to help them identify gaps or other issues? Do some tests get rolled into long term regression meaning someone else may have to pick up writing those? (And I'm sure there are other things I'm not thinking of.)

Assuming you find & address any downstream impact of changing the process, I'd say less paperwork to write & maintain is usually better.

Answer 6

We find 95% of the defects during exploratory testing so isit worth while spending the time to write a full suit of manual functional tests if hardly any defects are found from them???

The fact that you find 95% of the bugs you currently find using one approach is interesting. But the real questions are:

• are you being effective?

• Are you finding all the important bugs with this approach?

• Are you finding all the important bugs at all?

• What do you miss by focusing most/all of your efforts on this one approach?

If I find 100 bugs in total, and I found 95 of them using exploratory testing, but another 100 bugs that I didn't find are subsequently reported after I go into Production, then I have probably failed.

If on the other hand, no bugs are reported after I go into Production, I'd dig in and consider the value of finding the 5 bugs using non-exploratory methods. I might determine that a bit more exploratory testing would have found them, or that the only way I could have found them was using a different approach.

Answer 7

Exploratory and Manual testing are not mutually exclusive, you probably mean scripted regression vs. exploratory.

Answer 8

Exploratory tests can help to plan and write an effective functional test suite. When we are required to test a new feature it is always better to explore it before planning or automating it. Exploratory testing can generate new testing ideas/path and can reveal uncovered areas for testing as we explore the software.

As Ethel Evans said with automation we can better track the test coverage.

Moreover automation is cheaper than manual testing. Hence exploratory testing should be used to complement the running of functional test suite and not replace it.

Answer 9

In this case, you can regularly go through the manual functional test suite, let's say every other release cycle or once a month, and deprecate the not-so critical Test Cases. Keeping the manual functional test suite well groomed, you can make sure that they only test the critical scenarios. This will save you time and make sure that the code does not go in production with the bugs in critical components that should have been very obvious find had you run the functional test.

Based on how regularly you groom your test suite, it should take minimal time relatively.