16

After a feature is delivered, we have recently started to perform exploratory testing on the feature before we execute manual functional tests. We find 95% of the defects during exploratory testing so is it worth while spending the time to write a full suite of manual functional tests if hardly any defects are found from them?

Improve this question
2
  • 8
    Did the people doing the manual testing actually find ALL of the same bugs that were found during exploratory testing plus 5% more? If not then I'd suggest you try another experiment next time, swap the order, do the manual functional tests first, then do the exploratory and see how many bugs you find during exploratory that were NOT found during the manual functional tests. I say this because the way the question is phrased makes it sound like the manual process resulted in finding all the same bugs that exploratory did and then some, which is not what I would have expected.
    – Chuck van der Linden
    Commented Jun 23, 2011 at 4:34
  • 3
    +1 for Chuck's suggestion, which I love. I am a HUGE proponent of running experiments and capturing data. Doing so helps people learn a lot. In the absence of actual data, opinions prevail. The opinions that prevail often belong to those with the loudest voices and/or are HIPPO's (the Highest Paid Person's Opinion). Experiment. A Google search for W. Edwards Deming and PDSA would pull up some interesting articles and blog posts on this topic.
    – Justin
    Commented Jul 19, 2011 at 14:36

10 Answers 10

Reset to default
20

Do you think you would find the remaining 5% of bugs that are currently found by a suite of manual tests if you spent that time doing exploratory testing instead? Would managers have the confidence in test coverage that they need to make decisions without those functional tests?

I think the biggest risk of not having a manual functional test suite would be a reduction in coverage so that things might get missed. However, a "full suite" of manual regression tests might not be necessary to ensure that coverage. Automated tests are often considered an ideal alternative since they simplify regression testing and save testing time long-term. They can really free up your exploratory manual testers so they can focus on the most productive forms of testing.

You could also try guided exploratory testing, where testers are given a checklist instead of fully scripted manual tests, and then the testers just make sure they cover the checklist as they perform exploratory testing, in addition to anything else they think is worth trying. This technique is sometimes called "Guided exploratory testing". This might be a great alternative for your team since it sounds like you have some very skilled exploratory testers on hand.

Improve this answer
4
  • liked the idea of "guided exploratory testing"
    – Aruna
    Commented Jun 21, 2011 at 22:16
  • 2
    Having well established standards of behavior are great to aid in guided exploratory testing. Some of our users have been using the same system for 17 years now, and they don't appreciate it when someone makes a new screen that's different from all the rest. One could argue that this is not a bug, but simply a violation of the element of least surprise, but regardless of what it's classified as, it's something that if not caught during testing will get reported as a bug. (Even if it gets shot down, it's going to waste some analysts and manager's time.)
    – corsiKa ♦
    Commented Jun 21, 2011 at 22:51
  • 1
    @glowcoder I'd tend to classify stuff like that as a 'usability' issue (inconsistent UI elements that violates the paradigms that you've used elsewhere make the product harder to use, and not appear uniform and consistent to the user. )
    – Chuck van der Linden
    Commented Jun 23, 2011 at 4:17
  • 1
    I like Ethel's answer a lot here.. The only thing I'd add to it is have you tried to figure out why your exploratory testing missed the 5% that it did, and for that matter did it find issues your scripted tests would NOT have found? I'd be looking at the stuff that was missed, and asking what you can do to improve your exploratory process such that you would have found those missing bugs the next time around.
    – Chuck van der Linden
    Commented Jun 23, 2011 at 4:30
9

"Is it worth while spending the time to write a full suite of manual functional tests if hardly any defects are found from them?"

That depends - is the purpose of writing them to find defects? If so, then it sounds like you would be better off putting your energy into spending more time testing. If you're considering dropping them, then it doesn't sound like you have any contractual or political obligations to produce them. Manual functional test suites tend to be expensive to produce and expensive to maintain, so unless you have really good reasons for producing that documentation there are probably better ways to spend your money/time.

So what's stopping you?

Some common concerns might be:

1) How do we look at how well we're covering the system with our tests? Isn't there a risk that we'll all end up running the same tests? Or maybe we'll all run the same sort of tests and nobody will think of testing security, or installation?

If this is a concern for you, then one way of approaching this would be to write a set of exploratory test charters instead of test scripts. This article by Michael Kelly gives a good explanation of managing testing with charters, part of a series.

I've also been really enjoying Darren McMillan's blog posts about how he's using mindmapping to plan his testing and record test sessions. I haven't used mindmapping seriously yet for ET myself, but the mindmaps I see other exploratory testers using are just such a lovely clear way of presenting what was covered, and it's much easier to see gaps that remain to be tested.

2) How do we prove what we did?

You don't have to write stuff down before you test, to prove that you tested it. You can write stuff down while you test, and then you can still respond to what you're finding. (Actually, making good notes while you test is a lot more likely to reflect what you really tested than making them before - most scripted testers I've observed frequently went "off-script", so the scripts weren't a good guide to what was tested.)

Here are some good resources/ideas on recording ET sessions:

And here's a great list of general resources about exploratory testing.

Improve this answer
0
9

My answer assumes you believe you are leveraging exploratory testing as well as you can; in other words, that you do not believe you can discover more than 95% of your defects via exploratory testing. I think the gist of your question is, "Is it worth the additional work of having a full suite of manual tests in order to uncover the other 5% of defects."

You and your team need to decide whether skipping the additional testing is worth the risk of letting a bug slip through. When considering the risk, think about the kinds of bugs that might slip through, the frequency of those bugs (i.e. will the circumstances be rare or common?), the severity (when the bug occurs, how detrimental will it be?), and how forgiving you think your customers will be (e.g. high-tech early adopters may be more forgiving than online banking customers).

Aside from that, you need to consider how your customers will perceive your testing process. In some industries, your testing process is a private, internal matter. In other industries, having a well-documented process that is strictly followed is a necessary condition towards achieving a certification.

Finally, how you answer those questions may change over time as your product, organization, and customer base matures. It may be that today your exploratory testing is enough, but in two years you need to invest the extra effort in a manual test suite.

Improve this answer
4
  • I'm not sure how your answer relates to the question - how does additional testing increase the risk of letting a bug slip through?
    – testerab
    Commented Jun 22, 2011 at 0:29
  • Thanks -- I meant to say, "...whether skipping the additional testing is worth the risk...". I hope that clarifies how my answer relates to the question.
    – user246
    Commented Jun 22, 2011 at 2:21
  • Ah, ok - yep, that makes more sense to me now. I'd disagree that you're skipping testing though by replacing creating test scripts ahead of time by structured, responsive, exploratory test sessions. I'd like to know a bit more about what you think exploratory testing is - it may be that what you're thinking of when you say "ET" is very very different to what I'm thinking of.
    – testerab
    Commented Jun 22, 2011 at 17:58
  • 1
    The OP asked whether it was worth the trouble to have a full suite of manual tests if they have already done exploratory testing. The OP also said 5% of their defects did not come from exploratory testing. My answer was, essentially, "What do you lose if you don't discover that 5%." No doubt there are more effective and less effective ways to perform exploratory tests, but that wasn't what the OP was asking. Sorry if it sounded as if I was impugning exploratory testing; that was not my intent.
    – user246
    Commented Jun 22, 2011 at 18:05
4

We find 95% of the defects during exploratory testing so isit worth while spending the time to write a full suit of manual functional tests if hardly any defects are found from them???

The fact that you find 95% of the bugs you currently find using one approach is interesting. But the real questions are:

  • are you being effective?

  • Are you finding all the important bugs with this approach?

  • Are you finding all the important bugs at all?

  • What do you miss by focusing most/all of your efforts on this one approach?

If I find 100 bugs in total, and I found 95 of them using exploratory testing, but another 100 bugs that I didn't find are subsequently reported after I go into Production, then I have probably failed.

If on the other hand, no bugs are reported after I go into Production, I'd dig in and consider the value of finding the 5 bugs using non-exploratory methods. I might determine that a bit more exploratory testing would have found them, or that the only way I could have found them was using a different approach.

Improve this answer
4

What do you do with your scripted manual tests? If they are just executed by testers once or twice then tossed you probably can change your process & reduce/eliminate the extra paperwork.

But if any one else is using them, then consider the impact outside of QA before elimating them.
Do you have requirements so that the legal department is expecting them to be there? Or do you do test plan reviews that coders, analysts or others expect & use to help them identify gaps or other issues? Do some tests get rolled into long term regression meaning someone else may have to pick up writing those? (And I'm sure there are other things I'm not thinking of.)

Assuming you find & address any downstream impact of changing the process, I'd say less paperwork to write & maintain is usually better.

Improve this answer
2
  • 1
    Good point on considering other users of the documentation.
    – testerab
    Commented Jun 22, 2011 at 18:01
  • 1
    Yes, good comment. I've worked in industries where the test cases are actually a deliverable to our customer. Sometimes you have to write the tests anyway. In this context, I've used exploratory testing to find the problems early & efficiently, then ran the formal/written test cases for "score".
    – jruberto
    Commented Apr 23, 2014 at 14:50
4

In using "exploratory" testing we've found that it's VERY dependant on the testers experience and understanding of the system. If the tester has a broad knowledge of how the software should work and what should be displayed then they'll be able to catch all those little bugs. The manual process may provide better coverage but might miss an element that should be displayed on an existing report.

Exploratory testing is very good for bringing in external users. Less step by step with the freedom to roam throughout the application. With this being said exploratory isn't the best style when data integrity is involved... just my two bits. Happy testing all!

Improve this answer
3

Exploratory tests can help to plan and write an effective functional test suite. When we are required to test a new feature it is always better to explore it before planning or automating it. Exploratory testing can generate new testing ideas/path and can reveal uncovered areas for testing as we explore the software.

As Ethel Evans said with automation we can better track the test coverage.

Moreover automation is cheaper than manual testing. Hence exploratory testing should be used to complement the running of functional test suite and not replace it.

Improve this answer
4
  • 2
    Heh. Unfortunately it's perfectly possible to make automation MORE expensive than manual testing, if you're determined enough.
    – testerab
    Commented Jun 22, 2011 at 18:00
  • Sorry I couldn't understand your point, automation usually has lower costs associated with it compared to manual. It saves us cost in terms of time and human intervention, fatigue etc. Please explain how it could be costlier
    – Aruna
    Commented Jun 22, 2011 at 19:23
  • Well, if your automation has lower costs, you're probably doing it right! Sorry, should have been clearer - I think it's worth pointing out that in order to save money and time, automation needs to be done well. If done very badly, it can be more expensive. (E.g. I have seen an expensive test tool where over more than a year, only 8 tests had been written. Think of the licence fees! I could have run those tests in a day, manually.)
    – testerab
    Commented Jun 22, 2011 at 20:05
  • For functional testing, creating well done (non brittle, low maintenance automation is generally more expensive than executing the same test manually, often my several multiples. However, once it is created, it's very cheap to run, so run it often if you have it. IOW not cheap to create, but very cheap to use. (the exception being combinatorial esting, where even creating disposable quick and dirty automation beats doing the same thing with small variations thousands of times)
    – Chuck van der Linden
    Commented Jun 23, 2011 at 4:25
2

In this case, you can regularly go through the manual functional test suite, let's say every other release cycle or once a month, and deprecate the not-so critical Test Cases. Keeping the manual functional test suite well groomed, you can make sure that they only test the critical scenarios. This will save you time and make sure that the code does not go in production with the bugs in critical components that should have been very obvious find had you run the functional test.

Based on how regularly you groom your test suite, it should take minimal time relatively.

Improve this answer
2

Manual and exploratory are two different things but together can make huge difference to the testing. Both things comes into picture when time is the factor, so if you have time then scripted manual testing is good option but when time is less then exploratory testing is obvious a good choice.

Improve this answer
2
  • 1
    Did you intend to say that if there's time testers should use scripted testing instead of exploratory testing? I'd have to disagree if that was what you meant.
    – Kate Paulk
    Commented Apr 23, 2014 at 17:07
  • Can you clarify what you are intending to say?
    – Dan Snell
    Commented Apr 29, 2014 at 18:06
1

Exploratory and Manual testing are not mutually exclusive, you probably mean scripted regression vs. exploratory.

Improve this answer
1
  • You have a good point, perhaps it's worth expanding on it a bit? Especially if the author of the question doesn't understand?
    – Chris Kenst
    Commented Jul 9, 2012 at 22:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.