How do you know when to stop testing?

Question

Background: While manually testing an application in scenarios, where an application has complex functionality features, supporting multiple platforms(Web/mobile/Web APIs)/versions(OS Versions) where most of the features can interact with other features in multiple ways which give rise to a significant number of scenarios due to permutations.

Problem statement: After validating the happy paths as per the requirements, where functionality can be further tested in thousands of scenarios if each and every combination of features is tested and if we keep testing more and more with 'what if' scenarios and keep finding more issues; then as a tester, how one gets to know/decide when to stop testing?

Answer 1

How do you know when to stop testing?

There are a number of factors. They are not exclusive and the degree to which they apply will vary for each situation for each organization.

• Time - This is clearly a common constraint for most organizations. It can also be a guide for testing efficiently and finding and fixing the most critical errors first.

• Human Resources - For either manual testing or automation you can only do as much as you have people to actually do it in terms of testing or writing automation code

• Financial Resources - Manual and automated testing requires both people and technology and they both cost money.

• Risk based - If you test for critical, standard and low level issues then when you are not finding any more critical (or critical and standard) issues it is up to the business as to whether it is worth it in terms of money and people to do more to address the low level issues.

• User base for a given situation - If an issues affects most users it is often high priority, depending on the actual issue. If user only affects a very small number of users, for example only those on 10 year old browsers, it may not be worth fixing the issues as the cost is high and it may also affect how the other 99% are serviced, e.g. inefficient and unwieldy code may need to be used and then maintained for all future changes even for users of newer technology.

• Most commonly used combinations - You may have an almost infinite combination of data and options that users can use, but in practice there is usually a limited set of commonly used options that can cover most user cases.

• It depends on how you monitor production - If you do AB testing, Blue Green graduated releases (e.g. try with 1% of consumers first) and good metrics on your workflow conversion rates (e.g. from form page to form page) and are immediately aware of any issues and can work to correct them, then less testing may be required up-front.

• It depends on how you develop and release - If you develop and release in a quick and agile fashion and can roll back a release easily and quickly, this can reduce the amount of testing you need to perform. Release early and often and learn from users requires this sort of approach.

• Experience - You quickly learn with most application as to what many of the breaking points are and you get a 'gut' feeling of what might break and how and the effect. This must be combined with other factors in order not to be surprised by new things that haven't happened or been possible before.

• Product Stage - If your product is still in R&D the biggest value may come from rapid change and development with minimal testing. If the product is in full production and will be used by a million users tomorrow different levels and standards of testing may be required.

Overall: When the business is comfortable. You should ensure that the business knows that testing can never usually be considered 'done'. There are always more devices and data combinations and edge cases that could be tested. They need to know that this is always true, that it is impractical and unrealistic to have found 'all' bugs' before production deployments and that when it does happen it should mostly be used as a learning opportunity as to where and what to test more.

Answer 2

Think of what is relevant for you from following and some more-

• Risk, is this a rover going to Venus or a mobile app that can be easily fixed?
• Resources, time, cost, personnel
• Standards, laws and other liabilities. For example medical and aviation equipment
• Marketing, are you committed to a date? are bugs acceptable?
• Competition, what do your competitors do?

Answer 3

Stop testing when the people you inform no longer have any decisions to make that would benefit from the information you’re producing.

Answer 4

You know that you can stop testing after you finish those items:

1. When you finished all test cases you had on the test plan
2. All high priority bugs are fixed.
3. Rate of new bugs introducing has dropped below pre-defined minimum(for that industry)
4. Deadline is approached

This is according to the ISTQB certificate