Criteria for deciding what is more important to test

Question

Which criteria do you usually use to decide what is more important to test and what is less important? I mostly have two parameters: what's the likelihood that the user will encounter the scenario under test, and what is the impact to the user if the scenario breaks. I will tend to test those scenarios with the highest occurrence rate and with the highest impact. Sometimes I would also add to the mix the probability that there is a bug in the scenario. So I would usually put more testing resources to areas that were not tested enough, have a more complex functionality or have a higher occurrence of bugs.

So which criteria do you apply to decide what to focus your testing efforts on?

Answer 1

You want to save your team time and pain, and you do that by focusing on potential issues that are going to be much less costly if we catch them sooner rather than later. I usually start by asking myself "what feature(s), if broken will…”

• block me from the majority testing,
• be particularly difficult/time-consuming for the developer, or
• require fixes that are potentially wide-reaching and will therefore result in me needing to re-test other scenarios in my test plan.

The faster I get those bugs back to my developer, the faster he or she can commit a fix. The earlier in the process I do that, the less likely it will be that I’ll need to re-test things that the new commit might have impacted.

1. Test the most basic happy path surrounding NEW code, for 10 minutes. If you aren't able to perform the most basic functions of a new feature, you need to get the code back in the hands of your developer as soon as possible. Chances are, if something is broken here, it could be a longer fix.
2. Test functionality that impacts a large percentage of the code base as a whole. This is where we find dependencies that we didn't think could possibly be impacted by our changes. It’s much better to find these early, because they could indicate a need for deeper code reviews, and also wider regression testing and an adjustment to your test plan.

3. Test the critical functionality, and the most commonly used areas. What does your app NEED to be able to do? What do most people use it for? What are the potential blockers that are the only reason some users use your site or app?

4. Test the potential areas of high risk. Data loss. Load/Performance issues. This should be part of a conversation you have as a team - what are the biggest risks involved in this sprint? What are the worst-case scenarios? This is also the time where it’s most beneficial to test in a stage environment that replicates production as closely as possible.

5. Then normal regression testing on down to edge cases and thinking of creative ways to break stuff (my favorite part).

Answer 2

Other "must test" areas

• Regulatory requirements - Industry/Government/etc musts.

• Safety requirements - personal/mechanical/etc safety requirements

• Data protection

Answer 3

(Of course) it is subjective, here are two key parameters that I employ....

1. So, how important (business wise) is this feature/area to the end user, w.r.t. the current deliverable (release,patch update,sprint etc) ? What "matters" to the end user ? Which existing features they use most/more ? Which new features are they most likely to use more/most?

2. So, when was it last tested ? How much do we know about the feature/area as of today ? What has changed since then (time, code, test environment,change in user expectations, change in business expectations ) ? What risk does that change carry ? How does the risk impact 1 above ?

Answer 4

Here is a simple answer: the most important area to test will be where the bugs are. That sounds flippant, but it is true.

Now compare that to your statement in the description about testing the most commonly-used areas of the application. Instead, what you should be testing are the boundaries and gaps in the application. You should be taking functions that are well documented and setting them up the wrong way (Did that crash the application?). You should be testing as an impatient, nervous, or manually-disadvantaged user (with a hand motor control problem): click around the screen where it should not respond, multiple click buttons, click buttons very fast, click buttons very slowly, click them out of order. Reenter the screen multiple times, send multiple repeated control characters to a screen.

You can't do that with all application functions, but you can do it on a regular basis, especially with new functionality.

Good luck!

Answer 5

Probability * impact = risk

That's the way to go in theory.

Practically it's not so easy to correctly, accurately, and consistently across tests to quantify impact and probability. Each of the stakeholders might have a different view of the probability or impact, and the data itself might not be obvious especially when you are dealing with a never released product.

Also remember that talking about scenarios is what we usually do, but failures are the ones creating an impact, and failures might be related to more than one scenario, require a series of scenarios or a high impact one can hide in a low probability scenario making the risk high but unknown to you (unknown unknowns etc.)

Answer 6

I'd say this:

1. Ask for what the other areas affected by the change/fix/feature
2. Check the history of testing those areas to see where is the biggest probability to have serious issues and most issues
3. Start with the most critical area that can be affected.
4. Now on testing the actual change/feature/fix make a list with the most common - to - least common flows the user will use the feature. Start testing those from BOTTOM to TOP. Usually what we consider least common will be what the crazy users :) will actually do and break it.

Other than this, just build your own criteria based on the product, needs, history and team.

Answer 7

Please check my article share on this website where I've shared my views on what test cases to automate. Hope that might be somewhat useful to this question. http://qainsights.com/focus-automation-tool-place/