How to combine exploratory testing with test automation creatively?

Question

Background :There is an definite value in automated test execution of scripted tests to find regression defects.

On the other hand there is an tremendous value in exploratory testing and going beyond any scripted tests by designing & verifying tests 'on the fly'.

Thought process:Both provide value in different forms in a mutually exclusive way however I wonder if there is something we can even get more powerful by combining them further in few more creative ways.

Example: I tried introducing 'randomness' in long user journey tests by randomly selecting a sub step from couple of available alternative options. Ex. Payment method- in long scenarios we have chosen payment randomly by either credit/Debit/by points/ by coupon.

This gave us sometimes unique defects /strange behavior from application which we generally don't see.

Any further creative ideas?

Answer 1

You've brought a bit of the human factor (randomness) to automated tests. What if you try to bring some flavor of automation to your exploratory tests?

In general, I think the problem stems from thinking that test automation is only about automating tests, or to be more precise, automating whole regression tests. No, think rather of using automated tools that you choose just as a carpenter does when building a wardrobe. Or a doctor chooses when diagnosing a patient. Or a data scientist does when looking for certain trends in data. There's a great interview with James Bach about this mindset change.

You can always automate part of the process and leave rest to exploration and what humans are better at. For example, people are better at critical thinking while computers are better at repetitive boring tasks:

• to check if your application looks consistent you can automate the process of navigation along all links in your application and automatically take screenshots of every page your process landed on. Then leave evaluation of screens consistency to a human.
• to check if your HTML emails render properly in different email clients like Outlook, etc. automate the process of showing and taking screenshots of your emails in different email clients. Then leave evaluation to a human. This is what, for instance, Litmus does.
• to find incorrectly processed data by your backend system, you can automate the process of extracting data from logs (parser tool), group similar data together (Excel tool) and scan manually through data in your Excel to find, for instance, truncated sentences.

This is what worked for me. For more examples on using tools see "A Context-Driven Approach to Automation in Testing" by James Bach and Michael Bolton, especially the section "Third: Explore the many ways to use tools!".

Answer 2

In our project we used both methods.

Automation testing: We used the automation for repeating test cases. Especially for Smoke Tests and for normal Regression test suites. E.g. a user story has been created, then we adapted the user story in our test automation procedure. This means that our test automation suite has been enlarged dramatically but that was ok. But you have to make sure, that you always have the updated test suites / cases.

Exploratory testing: In case of e.g. layout testing or testing in different languages we used exploratory testing. Here we invited people who are e.g. native speaker to do some kind of "user journey test". Means that a application should be tested in combination with mobile app versions. For example let's say we want to do a purchase order at amazon. And I want to get paypack points, so I take an item (e.g. playstation) put it in shopping cart and in order to get paypack points I use my mobile phone and continue testing on my real device.

So in short we can say that we used exploratory testing mainly for:

• testing different layouts
• testing in different languages (here we invited native speakers e.g students we got about 30 languages to test)
• with the usage of real devices we also used exploratory testing
• user story testing with different heuristics& data attacks

So regarding the last point, we created automatic test cases for user stories AND we tested user stories with different heuristics. Because exploratory testing is about "Simultaneously-designing and executing test to learn about the system, using your insigths from the last experiment to inform the next." (see link: Exploratory testing definition

So in our case exploratory testing was absolutely required. Once we found out that the exploratory testing was working smooth we created automatic test cases (if required).

Here are some inputs for exploratory testing links:

Satisfice Link Exploratory testing

Quality trees Software Heuristics

I would use both combinations to receive a better test coverage. Because as already said, automatic testing is just testing step by step from scripts.

Answer 3

You can't. You can add random values, use monkey test that will randomly click some stuff, but in the end it is not exploratory testing. For a proper exploratory test you need the tester to know the System Under Test and go beyond what is already defined in test cases. Often this will require some specific setup (user profiles, test data, date, environment, etc...) and knowledge.

Answer 4

Essentially, automation testing is a repetitive steps from test cases by scripts. It looks like that you're teaching computer for manual test steps then command it "run, run, run" in a same things without thinking.

exploratory testing is discovery of freedom base on personal experiences with "thinking" activity.

For me, best combination of 2 testing types above is that you should develop test scripts for repetitive things and do exploratory testing while waiting for result from test scripts.