Testing matrix: occurence/impact/probability of bug

Question

I am a test manager, and in the 3 years in the job I learned that when deciding whether to write a particular test or not, it is beneficial to look at the following:

1. What is the occurrence of the scenario? The more frequent the scenario that more important is to have a test that covers it.
2. What is the impact to the user if the scenario breaks? Things like crashes or data loss have high impact and should be tested. Things like misaligned buttons or sub-optimal translations, even if occur in frequent scenarios, have a low impact and usually should not be tested.
3. What is the probability that a bug will be found in this scenario? If the functionality was tested multiple times already and few changes were made since you probably don't need to write a test. If bugs are common in this functionality or if the underlying software infrastructure is problematic, then you probably should write a test.

So my current methodology is to write tests with the highest scenario frequency, impact and probability of finding a bug, and then moving downward in this 3D matrix until no time is left for testing.

Before I add this to the curriculum for new employees, would you care to comment on it? Maybe I missed something important, or maybe I am looking at this incorrectly.

Note: Here I am talking about manual tests that are only run in the final verification cycles.

Answer 1

Your approach seems to be correct. Although, one thing I conclude from it is that you probably work a lot on time frame based projects. I mean there is always a time limit on the task. So it is always hood to have a priority based approach.

To decide the priority you may first look for things like -

1. What is the software intended to do?

2. Who would be possible users of the software?

3. Does the software handle sensitive and critical user data. For eg. Credit card info. In such case your priority may be security.

4. What frequency of users the software is suppose to handle? In this case your priority may be performance.

5. If it is an art gallery than the priority will be the GUI.

6. ... Think of as many scenarios as you can.

Now decide what is your priority. Is it security, performance, ui or functionality, etc. And which modules of the software would be used more frequently than others.

Then depending on the priority go from top to bottom based on the availability of time!

Hope that helps... :)

Answer 2

Basically your methodology for choosing tests is correct- start with common use cases, highest impact, buggy areas BUT then there is reality.

First of all you have to multiply the values for probability impact and bugginess, a rarely used feature but with disastrous impact is something you want to test.

Now let's see how those three metrics behave in reality.

When your product is new, or when you are testing a generic product (let's say a software library or the software for a new chip) you don't know what would be the most used features and use cases, and even if you do know you can "kill" a less used feature if you neglect to test it properly.

Impact is an important metric, but the only impact you can predict is a simple "feature not working", something that is usually with lower severity. Usually the deadly and costly bugs are hard to predict and except for some simple cases it's even hard to tell where to find or look for them.

Like you said you can asses how buggy something will be, but this won't work on new software.

Finally don't forget module dependency- a lot of times a bug manifests itself in a "simple" module, but the root cause is somewhere deeper and more complex.

Take for example the software for a USB stack. There are some must-test-first things, but then you have a long long list of equally medium priority features. Most of the features have complex code, and are inter dependent.

Answer 3

My typical Order of Operation for testing:

1. Functionality Validation

Ensuring that the change functions correctly is typically the first order of the testing. Ensuring that the basic things. Make certain that x does y when z happens.

2. Data level validation/Penetration testing

Ensuring that the impact of the change on the data level is inserting correctly and verification that all new data inserts do not expose a security vulnerability. This means SQLI and such in order to attempt to compromise your data's security. Also ensuring that your data is accurate both on the front end and in storage.

3. Usability Testing

Luckily, I don't deal with GUIs too often, I have mainly focused on back ends but I suppose this is where testers would verify that buttons are aligned and such. I would personally put a higher preference on Performance Testing and having your manual testers working with the performance testing team in order to ensure full functionality under stress but during this stage the performance testers will likely be writing and verifying their scripts.

4. Performance testing

With a background in development I will typically review the source code at this point and look for bottlenecks within the system. This can also be achieved by manually triggering events or asking your loadtesting or automation teams to kick off some scripts. Manual testing in conjunction with a load on the system will expose a lot of different issues that you otherwise may not see. A great loadtest that I perform with all new insertions is what happens with a trickle load (A long, conistent load). If the system is designed to clear a queue before it ends than by designing a trickle load it will essentially make the system constantly run. This could cause a dramatically higher load on the database than may be necessary and cause other issues within the system.

---

As for test writing, I will typically write as many tests as I possibly can. Some would be put into a regression suite as a 'Nice to have tested' area while others are organized neatly, depending on their priority. I will take the above order of operation and rank my tests based on where they would go within the above, and then how high of a priority for the above scenarios.

My typical rule of thumb is data-based, so after initial smoke testing ensuring that data is accurate and safe will be the highest priority. Second priority would be ensuring that the amount of code coverage is as close to 100% as possible. Third Priority would be usability and performance testing.

As for your methods, I do like them. They are very well planned out it seems but I do have a question for you.....

How do you know the impact and probability of a defect prior to testing it? I have seen seemingly minor changes have massively large impacting defects that could have caused data leaks and security vulnerabilities. Had these not been tested and fixed, that company could have been Home Depot or one of the several other companies on the news recently.

Answer 4

when deciding whether to write a particular test or not

It sounds like you are talking about Test Design. Test Design isn't simply about creating tests, its about applying your test strategy to choosing specific test techniques, generating test ideas (test cases), support data - basically the totality of your decisions.

For newbies test design can be about generating new and interesting test ideas and for experienced testers it can be about limiting them.

Comments: Your bullet points make it sound like you mainly use scenario testing and maybe functional testing as your test techniques. Is this the case? But what about the others like:

• Claims
• Domain (boundary / equivalence class)
• User
• Stress (load)
• Risk
• Flow
• Automatic (not just regression)

So my current methodology is to write tests with the highest scenario frequency, impact and probability of finding a bug, and then moving downward in this 3D matrix until no time is left for testing.

This sounds like you are using some type of risk based testing which is good.

Things like misaligned buttons or sub-optimal translations, even if occur in frequent scenarios, have a low impact and usually should not be tested.

This could be true now but might not always be the case. It could change based on your information objective or your mission. I'm testing something right now where the Interface and Usability are a high-priority and should be tested.

Here I am talking about manual tests that are only run in the final verification cycles.

Are you telling your testers not to automate? Are you saying manual testing as a comparison to automated regression? I've found the most useful types of automation are not used for regression.

I'm not sure what you mean by "final verification"- but as is the case with many questions on these forums it's can be hard to comment on the exact nature of your work without knowing more about it. =)

In Summary:

There are a lot of decisions that go into deciding whether to write a particular test, or not like the amount of time you have, experience with the product, skills in particular test techniques, the ability to conceptualize all of the things that could go wrong, etc. One of the tools that helps me with these decisions (helps me recognize the possible things to do) is James Bach's Heuristic Test Strategy Model.

However simply applying the exact same strategy (order of operation / order to test techniques) to every single situation is probably not the best idea - unless every situation is the same and/or identical.

To answer your question "when deciding whether to write a particular test or not, it is beneficial to look at... " - everything.