Agile: Most important “types” of test cases?

Question

So if you were to look at wikipedias list of the different types of test cases you'll see a huge list. In an ideal world QA would have an infinite number of hours to create test plans for all the different types.

Of course we don't live in an ideal world, so a lot of times we have to compromise and create what is viable and important given the time allotted.

That being said, what "types" of test cases generally have the largest impact?

To me it seems like Acceptance Testing is of course a huge one for anything a customer is going to be getting. Unit testing is important, but generally done by developers. Plus any type of "steel thread" testing, but sometimes people clump those up with Acceptance testing.

So what else is left. I suppose this question could be seen as an "opinion" based question. But I think it has an actual answer. Im just trying to gauge if you were to prepare a minimum viable set of test plans, which types generally make the most impact.

Answer 1

It does depend quite a lot on the user story, but there are some general guidelines, with more or less the following order of priority:

Essential Tests

• Acceptance Tests - By definition, the user story can't be accepted if it doesn't pass acceptance testing. The exact form the acceptance tests take will depend on the user story: for back end focused user stories, acceptance tests could well be performed by the tester or by another developer, where front-focused user stories are more likely to see acceptance tests performed by the product owner or customer.
• Developer and Unit Tests - Whether automated or not, these tests will happen before the user story is made available to the test specialists.
• Steel thread Functional Tests - Functional testing of steel thread for the user story will happen. It may or may not be automated depending on the nature of the user story, the maturity of the team, and the time available.

Other Tests

• Other Functional Tests - If there is time available (and realistically, even with an agile methodology there is likely to be pressure to complete features within a time frame the team does not find comfortable - I've yet to work in a situation where there is no time pressure), functional testing outside the steel thread will happen. This covers a lot of ground, particularly since even at the user story level if there isn't a really good software architect on the team there's likely to be important functionality missed in user stories (especially when user stories are customer-driven and the application serves multiple customers).
• Regression Tests - In my experience regression may or may not happen during a sprint, but it will definitely happen prior to release. If tester automation is being written, it will usually fall into the regression test bucket. My experience with automated regression is that if time is not allocated to develop, maintain and analyze automated regression, it will not be done.
• Security/Penetration Tests - I'll perform functional tests with an eye towards security and penetration issues, but this is a very specialized field, and many testers (including me) lack the skills to properly test an application for security issues. As a result, security and penetration testing is often an afterthought.
• Performance Tests - The issues that apply to security testing also apply to performance testing. Teams with good architectural designers will build for good performance and scalability, but the expertise needed to perform good load and performance tests means that these are often afterthoughts and may not happen.
• Usability Tests - Theoretically, usability should be designed in from the start, but without a usability specialists, that doesn't always happen. I've often seen usability issues reported by customers after release.
• Compliance Tests - Whether ensuring the software meets the appropriate regulatory standards is a must-test, a should-test, or a will-test-if-there's-time depends on the software: in some industries, such as health care in the USA, it's a must-test. Casual games, not so much because that segment isn't nearly as heavily regulated.

The list could go on forever at this point, but you get the idea. My experience is that some teams will make some of the test types I've classed as "other" essential. I've never encountered any industry that treats all of them as essential.

Answer 2

Yes, it is an "opinion" based question but I think that there is not a standard answer because in the Agile world the time is divided into sprints and each sprint contains one or more user stories. The point is that the US of a sprint sometimes are more focused on the back end of the product and sometimes are focused on the front end so the effort of testers changes from time to time and should properly be tuned.

My personal idea is that you should avoid creating automated UI testing at the very beginning because when you present the UI to stakeholders often you receive requests of changes (even if minor requests) and the consequence is that you have to re-do part of the work (move the buttons from here to there, I don't want a combo I prefer a flat list, etc.). So I'd mainly focus on APIs automation (it depends on your software and tools you use) because of the minor risk of changes and manual test plans oriented to satisfy the Acceptance Criteria.

Answer 3

I would like to propose that you consider what is needed to support an opinion that the quality of what is being delivered is "Good Enough".

In as much as we try to develop systems in general with the end user in mind, asking this question helps you to design the type of data you need to collect.

I have found that the generic answers that we would typically get can't really be answered. Questions like "How's the testing going?", "How much testing is left to do?", "Why does it take so long to test a website for God's sake???" aren't entirely helpful.

I try to help the person posing the question to better understand what it is they want. How much testing is left to do can become, "Have we covered all of the features in the software?", "How many tests are left to run?", "How much regression testing do we need to do before we can ship?", "Have all of the bugs that were fixed for the release been closed as a result of testing?"

Answering those type of questions means we cannot get away with saying things like "Testing is going well", "We're nearly finished", "Well if they didn't deliver the code so late we'd be able to get home on time for my Son's birthday party and I wouldn't be here waiting to find out if the build will even work.....".

Instead we can say, "Of the 10 stories, 4 have met their acceptance criteria, although we have to execute a performance test run on those stories, we expect it to take 2 working days to have the results. The 6 other stories have not met their acceptance criteria and we are currently reviewing that with the Product Owner. We have identified 20 additional tests across those 6 stories that should be executed based on our understanding of unspecified integration requirements that arose as the we tested the stories. All of this data is present in Jira, attached to each Story as a Testing sub-task."

AS I said at the start, it is more important to know how you want to tell the story and how the Product Owner wants to hear it. That will determine the data you need to capture.

At the very least your data needs are simply:

1. Story
2. Acceptance Criteria
3. Test Executed
4. Test Type
5. Status of Test
6. Bugs Found

After that you start to add elements that allow you to extend your reporting:

1. Product Feature
2. Risk
3. Priority
4. Log Data
5. Environment
6. ..... You get the picture :)

When you can break down the various activities that are carried out while testing the product, you will see what are the most important type of testing you need to execute.

Please note: I have studiously avoided referring to test cases, scenarios, scripts etc. The reason is simple. Knowing what you need to collect, when you collect it and how you collate it will dictate the form of the testing documentation you need.

Apologies if I meandered a bit.

Answer 4

Acceptance tests and Unit tests are critical to testing on Agile teams, but you can't forget to continue to consistently test non-functional requirements (Usability, Performance, etc.) and whatever additional testing needs to be done to ensure there is enough coverage at all levels (subjective per team) to confidently send the completed code to the business partners for Validation/Acceptance Testing.