Agile Manual Testing: For the Tests that you CAN'T Automate?

Question

So it seems not all tests can be automated, and for small teams the effort in creating or "Automating" manual tests might be more work than worth it in some cases.

What then? Im trying to diverge away from the old "Step Based: Expected Result" test plans, which are really specific and when requirements change....so do those. So I've taken a look at Session Based/Exploratory testing.

This seems like probably the right way to do it? Is there something else that's better for Agile for a small team that doesn't have the manpower to automate everything?

Or is the old way actually better? If you are trying to look for full coverage it seems theirs a gap between automated testing and then the manual type of exploratory testing (Which...maybe I'm wrong, but it feels almost like smoke testing but you don't really have a specific test "plan" for it).

Answer 1

If you are trying to look for full coverage it seems theirs a gap between automated testing and then the manual type of exploratory testing (Which...maybe I'm wrong, but it feels almost like smoke testing but you don't really have a specific test "plan" for it).

Okay, from this description, I think the problem may be that you are limiting your definition of exploratory testing to just completely freestyle unplanned exploration. (I may be wrong, that's just what I'm reading in to the above).

Most people sitting down for a SBTM session will plan their coverage - just not in a heavyweight document, but in test charters. This lets you get a high level view of what you are covering, and also, because at the end of the session you also add new charters that you've realised are needed, it allows you to be flexible to incorporate newly discovered or changing requirements.

Here's an example of a charter (from this blogpost Rigorous Exploratory Testing):

“Exercise the Publish feature in various ways to find any instances where a valid publish request does not complete successfully or where the user does not receive any feedback about the actions the Publish feature took on their behalf.”

For further reading, I'd recommend Elizabeth Hendrickson's awesome book, "Explore It!". It's a fantastic fit for an Agile team (unsurprisingly, considering the author).

If time is the issue, then I would also recommend that it can help to break down the automate/manual distinction a little, and instead think about "tests we want to keep" vs "tests we just need to run once". It can be very quick to write throwaway scripts that automate part of what you're wanting to do, but then do the rest manually - e.g. capturing results that you can evaluate yourself later, or getting through a common set of steps repeatedly to the point where you want to start varying things. (This may sound obvious, but an awful lot of teams don't think of doing this).

Answer 2

In Agile QA I focus on the testing quadrants

I focus on my developers doing a good job in Q1
I work to write good automated UI tests in selenium and capybara for Q2
Exploratory testing is essential and is represented in Q3
Performance and load in Q4

For exploratory testing (Q3) this is where I cover those 'hard to cover',. 'not worth automating', 'visual', 'judgement involved', 'domain knowledge required' forms of exploratory testing.

The quadrant helps me keep track of all the relevant areas to consider in testing.

I would also refer you to https://en.wikipedia.org/wiki/Exploratory_testing which includes

To further explain, comparison can be made of freestyle exploratory testing to its antithesis scripted testing. In the latter activity test cases are designed in advance. This includes both the individual steps and the expected results. These tests are later performed by a tester who compares the actual result with the expected. When performing exploratory testing, expectations are open. Some results may be predicted and expected; others may not. The tester configures, operates, observes, and evaluates the product and its behaviour, critically investigating the result, and reporting information that seems likely to be a bug (which threatens the value of the product to some person) or an issue (which threatens the quality of the testing effort).

Answer 3

I would suggest possibly assigning some "free play" time with a minimum number of participants, specific classes of player or levels of experience and with a number of objectives and/or story points from the core functionality and from the latest couple of sprints to try to achieve within the time period.

This could be followed by a structured feedback including:

• Could core points be achieved within time limits
• Could recent story points be achieved
• Usability of interface
• Accuracy and applicability of naming, tips, etc.,
• Responsiveness

and some less structured:

• Problems noted
• General Impressions
• Suggested Improvements
• Comments

Possibly this could be followed with a "Can you break it" session with questions of could you - how?

Usually you would look to have at minimum an experienced user, an inexperienced but technically aware use possibly a member of another team or a teenager, a totally inexperienced user if training is not normally provided consider a member of clerical or non-technical managerial staff for this and a typical customer end user.

You can consider allowing anonymous feedback but if so the user should indicate level of experience and background which for the short term will render it less anonymous in the short term but will allow weighting of the results with customer representative feedback on usability being given a very high weighting.

Version control your results/feedback with a reference to as your source code version information.

Also, consider using a screen recorder and possibly a video camera for supplementary information, the first so that if it turns out that clicking on the pixel at [12,42] crashes the app or does something strange you can reproduce the problem. The second will let you see when the user is having problems knowing what to do next even if they say that they had no problems.

Most important plan, structure, ensure that you can reproduce and aim to compare apples with apples not with mangoes.

Answer 4

ANY test CAN be automated, given enough resources (time and manpower) - but for many tests such effort does not make sense. If user interface or functionality is under constant churn, you are better off do just manual tests, because developing automated test (programs) takes more time, so may not be able to keep up with development (unless you have as many testers as developers).