What should Testers do if they are not able to find good defects in the product?

Question

I work as manual/black-box tester. My responsibility primarily includes functional testing of a web application and to some extent database testing using sql server, based on the functional specification. It is coded in .Net technologies. There isn't much scope of performance, security or automation testing for my role/position as company has hired automation/performance experts to do that.

The problem is that even after trying out quite a few negative testing scenarios and using complex test data , i am still not able to find any critical defects in the product, except some minor UI/cosmetic bugs.

A part of this can be due to the fact that the application itself is very old and very mature, stable product. In-fact, many testers have worked on that and serious problems have already been found/fixed. It is indeed challenging and difficult to find a critical defect in the process flows.

I just keep on writing test cases/scripts and execute them, but then I don't have anything to show to the management. Sometimes it makes me think that I am not providing any value to the product owners, as they hired me to identify crucial, important problems.

The product that I test, is a financial application and has a lot of file processing for financial data.

What to do in these situations, when you don't find any serious problems even after testing the product long enough. Is a testers worth at job directly proportional to the Number of defects found?

Does it make you an ineffective tester, if you just show test execution metrics to the management, but haven't found any real vulnerabilities in the product?

Answer 1

Testing no longer means testing

Confused? We can imagine! The purpose of testing used to be fairly clear–“Testing is the process of executing a program with the intent of finding errors”. This changes when adopting agile and lean development.

read more...

I think the testing manifesto has it at the right end. Focus on preventing defects over finding defects. If defects do reach the production do a root-cause analyses and prevent similar defects from happening in the future.

Reads:

• https://less.works/less/technical-excellence/thinking-about-testing.html
• https://moreagiletestingmanifesto.wordpress.com/2015/08/02/agile-testing-manifesto/
• http://www.jamesshore.com/Agile-Book/root_cause_analysis.html
• http://moderntesting.org

Answer 2

I'd like to address the reporting aspect of the question. You say...

I just keep on writing excellent test cases and executing them, but then I don't have anything to show to the management. Sometimes it makes me think that I am not providing any value to the product

Even in the perfect world where the software leaves the developers' hands 100% bug free and feature complete (as a developer: "HAHAHAHAHAHAHAHAHAHAHA"), this is simply not true. Yes, when you find a mission critical bug at the 11'th hour, that's awesome, and you're the hero. But when you execute all excellent test cases and discover no defects you've produced something of equal or greater value. This is Quality Assurance.

In the company I work for "Tester" is a dirty word. Instead we have the "QA" team. This is because the business doesn't really benefit from testing and fixing bugs. The business doesn't even really benefit from having bug free code. No, the business needs to know that it has bug free code. I have worked in environments where this need was not reached, and the business did not have confidence in its product. In these environments everything falls apart.

• You can't estimate the work involved in a new feature with confidence. Because a bug could be hiding just under the surface of a critical component.
• You can't test a change set efficiently. Because without a baseline of confidence every test must be a full regression test.
• You can't sell your product to serious buyers, because when they ask "can it do XYZ?" your engineer says "well, in theory."

What I'm getting at is that the front page of whatever report you deliver to management at the end of each development cycle should not be about where the software fails:

**Defects Found**
BUG-1: $0.01 of every transaction is transferred to Joe A.'s offshore account.
BUG-2: Rounding error when withdrawing from an ATM on Tuesday.

Instead, the deliverable that really matters is what you've proven the software can do. And can do correctly. The open defects are a part of this report, yes. But they are not what really matters. What really matters is:

**Features Verified**
Can open a new account           [PASSED]
Can close an existing account    [PASSED]
Can issue a transaction          [FAILED, BUG-1]
Can withdraw from a bank         [PASSED]
Can withdraw from a ATM          [FAILED, BUG-2]

And then, after the bugs are addressed. You release with a QA Report that looks like this:

**Features Verified**
Can open a new account           [PASSED]
Can close an existing account    [PASSED]
Can issue a transaction          [PASSED]
Can withdraw from a bank         [PASSED]
Can withdraw from a ATM          [BACKLOGGED]

And that's the document that management should want to see.

Answer 3

My testing motto has always been this. "The purpose of testing is to reduce the risk of implementation." I have always found that when a testing organization is graded simply on the number of bugs that are found, it is very inaccurate. If you find a large number of bugs, is that because you are that awesome or your developers are that bad. Or in your situation, if testing does not find bugs, is that because testing is bad (not in your case) or your developers are that good (sounds like your case).

When you need to worry is if you are not finding any issues during your testing process, but there are issues found in production. As long as the end-users are not finding defects that your testing team did not find, then it sounds like you are doing a good job.

The other thing you can do at this point is to augment your testing. Have you tried looking for places where the application can be improved?

Answer 4

In addition to the other answers, it would seem important to me that you find out IF you are actually letting bugs slip through. The way you write this would make it seem the software is up and running in production, so surely if there are bugs they would be reported. Then you can check if you should have been able to find them or not.

I would also encourage you to look into automated testing anyway even if it's technically not part of your job, many manual tests can be automated eventually and it would be a matter of time before someone else finds out. Manually testing something that can be scripted is low end work. I get that you are also designing the tests yourself, but I think you can improve and show your value to the company by slowly moving away from manually executing them and making that an automated process. It depends on the company of course, but most companies value people that want to improve themselves and make their job easier (and cheaper/faster).

Answer 5

You can enhance the product.

Adding to all the answers,

There are two things you already know. Tell yourself in such situations that,

1. No product is bug-free.
2. Testing is a never-ending process.

There maybe a few reasons due to which you are in such a situation. The product you are working on, may be a matured, developed-for-a-long-time project. A lot of QA/Testers may have tested the product already. So you may not find a lot of bugs in the product. Never worry, There is always scope for enhancement. Think yourself as a "New-customer" and document the troubles you are facing in using the application. Also understand the work flow and try to optimize, for example try to cut-down a few tasks which are unnecessarily drinking the system performance, something like that. These kind of tasks may apply to a quality assurance tester/engineer very much.

Answer 6

My testing motto is that it is always better that the QA find a bug, instead of the customer \ user finds one.
It might be that the product \ specific version \ specific feature you are testing, doesn't have bugs. If so, it is very good.
As long as you continue to get "pass" when you execute your tests, you continue to write new scenarios for the tests and you try to make your test package cover the entire product - i don't see any problem.

Answer 7

Your coders may just be good. Testing is done to find bugs before the customer finds them and you have to go through a find the bug, create a patch, distribute a patch cycle. If the customers are not complaining about bugs, then the testing scenarios are correct. If the customers find bugs that should have been detected in the Test/QA stage, then your testing is faulty.

If there are NO bugs, then you can't find them. You cannot prove a negative, EVER. If you don't find any bugs don't feel discouraged you are doing your job.

Are you also testing for FAILURES?

What happens if someone attempts to enter a string into an integer? Testing for overflows?

Answer 8

Two suggestions:

1. Could you expand your scope to Usability Testing?
2. You could ask the programmers to put in a Litmus Test, i.e., seed it with one known bug, but perhaps subtle, that they KNOW is in it. (I do this all the time to test the tester). You could even turn it into a game, they could put in a more subtle bug (hard to detect or hard to characterize). This makes it more fun for you and confirms for management that you ARE finding bugs.

Answer 9

To directly answer the question, no. Metrics are not the end all be all of measuring the value of a tester. I would argue that identifying user flows that are not defined in the spec can be more important than a discovering a bug.

In my experience no one is going to prevent you from doing things outside of your job title (performance, automation) as long as you also perform your expected duties. Showing that you are trying to improve your skill set looks good and can lead to identifying new classes of bugs.

Answer 10

Well. I don't see any issue with you not finding any critical bugs as you have already provided the product background and the fact that it has been rigorously tested by many different QAs.

So, another thing that I want to highlight here is that if you are looking at this application for quite some time now, the chances of finding bugs also go down because as a human you slowly stop getting excited and enthusiastic when you are seeing the same thing every day.

It is better that for some time, just try to plan something around coverage of the functional scenarios. Are the test cases covering all the functional aspects of testing? Try to look at the RTM (Requirement Trace-ability matrix) and see if it covering everything. If RTM doesn't exist, then try to prepare one.

See, there is this bitter fact. If you are not finding bugs, it is good as long as no (or only a few) issues are reported after production deployment. So, try to come up with ideas to ensure the test coverage and strengthen your position and confidence in the quality of the application.

Answer 11

Question should be how to find good defects in the product, What are the best practices to maximize the chances to find valid defects?

Each individual has their own distinctive style for testing, Read This

You will find this very useful and you can learn so many things from this article.

Answer 12

Testing is fundamentally an insurance activity.

It's primary job is not to find more and more defects but to ensure there are no critical defects which may be found at later stages and will impact business and cost of correction/fixing will be higher.