Good practices for finding valid issues in exploratory testing?

Question

We've noticed that free-style exploratory testing is oftentimes a better source of new previously undiscovered issues. But, from time to time, I see our testers spending hours in a row in "free" testing without a specific idea in mind.

How can we train and improve our QA team exploratory testing skills?

---

Some of the ideas I had in mind:

• set up regular workshops demonstrating how were some of the bugs discovered during previous exploratory testing sessions, what led to them being discovered
• maintain a shared page/notebook of the potential exploratory testing ideas
• pair-testing in a "free" style manner

Answer 1

30 Good Practices to improve Exploratory Testing

• Use a bug tracking system
• Use boundary testing of values
• Consider using testing personas
• Use happy, sad and optional paths
• Become skilled at reading server logs
• Learn about usability and accessibility
• Learn to use emulators and simulators
• Learn more about the customers needs
• Be present in business process meetings
• Maintain an easy to use library for devices
• Ask immediate product questions in person
• Understand the company metrics for quality
• Develop good relations w/ product managers
• Learn the business domain and the key factors
• Learn about visual, auditory and tactile disabilities
• Understand the database design and implementation
• Learn about both internal and external business processes
• Use the value proposition to determine what bugs to report
• Get documentation of application workflows and functionality
• Learn to use local virtualization, e.g. Parallels, VirtualBox, etc.
• Categorize bug severity in a way that is relevant to the business
• Learn relevant knowledge for localization and internationalization
• Learn and use remote services such as browserstack and saucelabs
• Become skilled in using browser tools such as console and inspectors
• Use automation for repetitive tasks and humans to spot other changes
• Replicate real user environments for network speed, sunlight, heat, etc.
• Ensure the system used for exploratory testing closely mimics production
• Use company data to determine what devices and browsers customers use
• Use a well integrated 'drop to cloud' system* for screenshots and links to them
• Gain a wide variety of experiences in life to help think about different approaches

For Agile Folks:

* CloudApp for Mac is a good example

Answer 2

Personally, I really like exploratory testing based on the following three ingredients:

Timebox

• The amount of time a tester gets
• Makes planning possible
• Team decides per sprint/week/day etc.
• Allows lean manual testing for others (e.g. developers or domain experts), too
• Testers are extremely focused (higher chance of finding bugs)

Example: usually between 30 – 60 minutes.

Protocol

• So we know what has been tested
• Helps to continuously improve manual testing skills
• What worked in the past, what not?
• Go back in time and see why we didn't find a particular bug
• Reuse powerful test cases or automate them
• Helps to onboard new team members

Example:

Scope: XYZ creation wizard
Tour: landmark
Timebox: 60 minutes

Test cases:

1) open wizard - object selection dialog opens - C
2) go through all steps in order (1, 2, 3, 4) - arrived at final step - C
3) jump around on navigation bar: 3, 5, 2 - crash - B

Opportunities: (C)orrect, (B)ug, (?) open question

Testing Tour

• Describe how the SUT should be tested
• Developed at Microsoft, used by many others
• Distinct and catchy names to improve communication
• Allows testers to come up with new test cases on the fly
• Would be too open without tours (less focus, less control)
• Become quickly part of the team vocabulary

Example:

• Landmark: Most important features in different orders
• Anti-social: Always do the opposite
• Garbage collector: Go street by street, house by house
• Supermodel: UI only

---

Note that this are just loose guidelines. For instance, some teams may like Git-versioned Markdown files for their protocols, whereas others may prefer Excel sheets in a private cloud. Feel free to adapt the whole process to your team's/department's/company's needs, while keeping the basic idea in mind.

For further reading, I can recommend the following literature:

• James A. Whittaker: Exploratory Software Testing: Tips, Tricks, Tours, and Techniques to Guide Test Design
• Cem Kaner, Jack Falk, Hung Q. Nguyen: Testing Computer Software
• Cem Kaner, James Bach, Bret Pettichord: Lessons Learned in Software Testing: A Context-Driven Approach

Answer 3

set up regular workshops demonstrating how were some of the bugs discovered during previous exploratory testing sessions, what led to them being discovered

• Yeah, definitely a good idea. I happen to have been thinking of something similar. Please read the bullet point below:
• It will be especially useful to set up a workshop that teaches/demonstrates how to visualize an object in variables. In an abstract, testing is all about altering one variable at a time and observe the outcome; if we could visualize a subject under test into its composing variables then we could even start tweaking one variable at a time. This practice would come close to a systematic exploratory testing. The reason I am saying "close to" is we will never able to systematically test every single variable in all possible ways.

maintain a shared page/notebook of the potential exploratory testing ideas

• I personally think this is not a good idea. From my personal experience, IT companies try to keep hard copies of testing ideas but no one would read them. The reasons are shown as below:

• Bad hand writing/grammars, surprisingly, not everyone writes down their ideas carefully enough so that others can read them in the future. Most of times, people's notes are so disorganized that they cannot even read their own notes in two weeks.

• People have different ways of building logics; it is hard to copy/paste someone's logics from reading notes.

• Of course, idea notes can be reviewed to make sure that they are passing on ideas clearly but again, having someone review/edit notes cost resources. It is not a price every IT company wants to pay.

• I speculate that having a short story of how an interesting bug is found/how a bug is found via an unexpected way and post those stories around the office and replace them regularly may be a good idea. When people are walking down a corridor, a story catches their eyes and a light bulb may get lit up.

pair-testing in a "free" style manner

• Yeah, this is definitely a good idea. It is not just limited to pair-testing, a whole group of people can be involved.

Put people on different projects that require different knowledge base

• Different knowledge base, different background offers different perspectives.

Send testers to sit with customers

• Testers may not use a piece of software as a customer would.

Answer 4

Understanding the Technology.

If they test a web application make sure they understand how JS, HTMl and CSS works. Maybe let them do some projects. You can't expect them to test for XSS vulnerabilities if they don't know how to inject JS. If you've got an idea of how the underlying tech works you can choose your input to be as evil as possible, if you know the backend uses excessive amounts of python throw in an emoji in your username and make sure they've set the right encoding, if it uses java with it's native utf-8 support this might be less of a problem that needs to be tested.

In my team I make a point of having them extend our intranet QA service to schedule tests, organize lists and so on on a regular basis. This way they get a feeling of what can go wrong and can transfer their problems to problems our product might have.

Answer 5

To become better at exploratory testing , tester have to know the system they are working , They should have vast knowledge about the system . And some good level of IQ . Also , experiece is a fact , as much experience one tester is , much good bug finding will result.