10

We've noticed that free-style exploratory testing is oftentimes a better source of new previously undiscovered issues. But, from time to time, I see our testers spending hours in a row in "free" testing without a specific idea in mind.

How can we train and improve our QA team exploratory testing skills?


Some of the ideas I had in mind:

  • set up regular workshops demonstrating how were some of the bugs discovered during previous exploratory testing sessions, what led to them being discovered
  • maintain a shared page/notebook of the potential exploratory testing ideas
  • pair-testing in a "free" style manner
  • 1
    Exploratory testing is not about hanging around without specific focus. For instance, you split your testing around certain chapters, e.g. in the coming hour you focus on testing localization-related stuff. Sure, you can follow spontanenous ideas, like you look for localization-related issues but if you discover invalid input problems, you can check them as well. But you start with certain topic in mind. – dzieciou Jul 11 '17 at 9:04
  • Modified title to try and define 'better' a little. Please feel free to change more @alecxe – Michael Durrant Jul 18 '17 at 11:00
  • testers spending hours in a row in "free" testing without a specific idea in mind - question - are they finding bugs in that process ? – Michael Durrant Jul 18 '17 at 11:01
  • I would also make the 'shared page/notebook' an 'electronic shared page/notes one as answers about physical pen and paper bring up old issues, no need to repeat them for this. – Michael Durrant Jul 18 '17 at 15:14
  • @MichaelDurrant at these moments, bugs are very rarely found..it's quite an unproductive time. Thanks so much for the edit and the ideas! – alecxe Jul 18 '17 at 18:59
9
+50

30 Good Practices to improve Exploratory Testing

  • Use a bug tracking system
  • Use boundary testing of values
  • Consider using testing personas
  • Use happy, sad and optional paths
  • Become skilled at reading server logs
  • Learn about usability and accessibility
  • Learn to use emulators and simulators
  • Learn more about the customers needs
  • Be present in business process meetings
  • Maintain an easy to use library for devices
  • Ask immediate product questions in person
  • Understand the company metrics for quality
  • Develop good relations w/ product managers
  • Learn the business domain and the key factors
  • Learn about visual, auditory and tactile disabilities
  • Understand the database design and implementation
  • Learn about both internal and external business processes
  • Use the value proposition to determine what bugs to report
  • Get documentation of application workflows and functionality
  • Learn to use local virtualization, e.g. Parallels, VirtualBox, etc.
  • Categorize bug severity in a way that is relevant to the business
  • Learn relevant knowledge for localization and internationalization
  • Learn and use remote services such as browserstack and saucelabs
  • Become skilled in using browser tools such as console and inspectors
  • Use automation for repetitive tasks and humans to spot other changes
  • Replicate real user environments for network speed, sunlight, heat, etc.
  • Ensure the system used for exploratory testing closely mimics production
  • Use company data to determine what devices and browsers customers use
  • Use a well integrated 'drop to cloud' system* for screenshots and links to them
  • Gain a wide variety of experiences in life to help think about different approaches

For Agile Folks: enter image description here

* CloudApp for Mac is a good example

  • and of course my signature curve of bullet points ;) – Michael Durrant Jul 18 '17 at 11:44
  • Reminds me of the Black Mesa symbol - a bit foreboding. – doobop Jul 18 '17 at 13:29
  • I do the curve for fun, but to be clear, every single bullet point has actually been key at some point in my most recent position :) – Michael Durrant Jul 18 '17 at 15:10
  • While a lot of these things definitely make sense, I think they're not directly related to exploratory testing; it's more like a salmagundi of tips & tricks for (manual) testing. Moreover, some points are very product specific. For instance, you don't have to be skilled in browser tools when you're SUT is a native desktop app. Nonetheless, nice layout! :-) – beatngu13 Jul 18 '17 at 22:07
  • They've all been relevant to me. Note that I have skipped any mention of Automation, Selenium, Page Objects, Programming, Performance Testing, Unit testing, etc. Experience in the ones I did list in my answer helped me in my exploratory testing - which I do consider to be the same as manual testing, but it's true they can be different with manual testing according to fixed criteria / script and Exploratory being more free-stye. I think my answer is most applicable to the free-form exploratory type as in the question. Some help directly, others more indirectly. – Michael Durrant Jul 18 '17 at 22:38
6

Personally, I really like exploratory testing based on the following three ingredients:

Timebox

  • The amount of time a tester gets
  • Makes planning possible
  • Team decides per sprint/week/day etc.
  • Allows lean manual testing for others (e.g. developers or domain experts), too
  • Testers are extremely focused (higher chance of finding bugs)

Example: usually between 30 – 60 minutes.

Protocol

  • So we know what has been tested
  • Helps to continuously improve manual testing skills
  • What worked in the past, what not?
  • Go back in time and see why we didn't find a particular bug
  • Reuse powerful test cases or automate them
  • Helps to onboard new team members

Example:

Scope: XYZ creation wizard
Tour: landmark
Timebox: 60 minutes

Test cases:

1) open wizard - object selection dialog opens - C
2) go through all steps in order (1, 2, 3, 4) - arrived at final step - C
3) jump around on navigation bar: 3, 5, 2 - crash - B

Opportunities: (C)orrect, (B)ug, (?) open question

Testing Tour

  • Describe how the SUT should be tested
  • Developed at Microsoft, used by many others
  • Distinct and catchy names to improve communication
  • Allows testers to come up with new test cases on the fly
  • Would be too open without tours (less focus, less control)
  • Become quickly part of the team vocabulary

Example:

  • Landmark: Most important features in different orders
  • Anti-social: Always do the opposite
  • Garbage collector: Go street by street, house by house
  • Supermodel: UI only

Note that this are just loose guidelines. For instance, some teams may like Git-versioned Markdown files for their protocols, whereas others may prefer Excel sheets in a private cloud. Feel free to adapt the whole process to your team's/department's/company's needs, while keeping the basic idea in mind.

For further reading, I can recommend the following literature:

4

set up regular workshops demonstrating how were some of the bugs discovered during previous exploratory testing sessions, what led to them being discovered

  • Yeah, definitely a good idea. I happen to have been thinking of something similar. Please read the bullet point below:
  • It will be especially useful to set up a workshop that teaches/demonstrates how to visualize an object in variables. In an abstract, testing is all about altering one variable at a time and observe the outcome; if we could visualize a subject under test into its composing variables then we could even start tweaking one variable at a time. This practice would come close to a systematic exploratory testing. The reason I am saying "close to" is we will never able to systematically test every single variable in all possible ways.

maintain a shared page/notebook of the potential exploratory testing ideas

  • I personally think this is not a good idea. From my personal experience, IT companies try to keep hard copies of testing ideas but no one would read them. The reasons are shown as below:

  • Bad hand writing/grammars, surprisingly, not everyone writes down their ideas carefully enough so that others can read them in the future. Most of times, people's notes are so disorganized that they cannot even read their own notes in two weeks.

  • People have different ways of building logics; it is hard to copy/paste someone's logics from reading notes.

  • Of course, idea notes can be reviewed to make sure that they are passing on ideas clearly but again, having someone review/edit notes cost resources. It is not a price every IT company wants to pay.

  • I speculate that having a short story of how an interesting bug is found/how a bug is found via an unexpected way and post those stories around the office and replace them regularly may be a good idea. When people are walking down a corridor, a story catches their eyes and a light bulb may get lit up.

pair-testing in a "free" style manner

  • Yeah, this is definitely a good idea. It is not just limited to pair-testing, a whole group of people can be involved.

Put people on different projects that require different knowledge base

  • Different knowledge base, different background offers different perspectives.

Send testers to sit with customers

  • Testers may not use a piece of software as a customer would.
3

Understanding the Technology.

If they test a web application make sure they understand how JS, HTMl and CSS works. Maybe let them do some projects. You can't expect them to test for XSS vulnerabilities if they don't know how to inject JS. If you've got an idea of how the underlying tech works you can choose your input to be as evil as possible, if you know the backend uses excessive amounts of python throw in an emoji in your username and make sure they've set the right encoding, if it uses java with it's native utf-8 support this might be less of a problem that needs to be tested.

In my team I make a point of having them extend our intranet QA service to schedule tests, organize lists and so on on a regular basis. This way they get a feeling of what can go wrong and can transfer their problems to problems our product might have.

1

To become better at exploratory testing , tester have to know the system they are working , They should have vast knowledge about the system . And some good level of IQ . Also , experiece is a fact , as much experience one tester is , much good bug finding will result.

  • -1. This doesn't provide any guidance at actually becoming better at exploratory testing, other than "get more experience". While experience and familiarity with the SUT will obviously help, the other answers offer more practical help in improving in the short term. – c32hedge Jul 20 '17 at 17:41

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.