Why would an external testing team find more bugs then an internal testing team?

Question

I work on a project as an internal testing team, our team put 100% effort to all formal testing procedure like complete testing process & at a certain level of time we don't find bugs in application, but when we pass build to external testing team they found a lot of bug in the same build.
One thing I don't understand that we know product requirement better than them but even after that they found more quality bug then us, can you please explain this why is it so?

Answer 1

Some of the reasons:

• fresh eyes
• more time for exploratory testing
• existence or non existence of a testing plan
• working without boundaries, without knowing the product requirements
• working without executing any text cycle, more exploratory
• no time constraint pressure that might be present in the SDLC between sprints
• they are focused on finding bugs, you re focused on fulfilling the requirements

Answer 2

It is inevitable that you are not testing 100% of the product. You are using your formal process which might test things always in exactly the same way. All bugs found this way will eventually be fixed, and your formal process only tests parts of the code that have been made bug free.

Your external testers don't have any such restrictions. They test whatever they like. They tests parts of your code that haven't been tested. You have a test that says "if the user enters X then the output is Y". As an external tester I'd try "what happens if I don't enter X but ABC?£££839 ?".

There may be a subconscious thing going on: That your team wants the tests to succeed (because the developers or your management hate you if you find too many bugs. Deadline missed because bugs were found, and you get the blame). And the external team wants to prove how clever they are by finding bugs. That difference in attitude can make a lot of difference.

Answer 3

Somethings come to mind which could be the reason would be:

• Your internal team members aren't overly motivated, for X reasons
  perhaps they are unhappy in their jobs and do not put as much effort as you think into the testing.

• You have relatively new members internally, perhaps they aren't as
  familiar with the system and thus will struggle to find a lot of the issues.

• Its not so much a case of internal vs. external in the sense that its just a fresh pair of eyes if you like, perhaps the external team have better indepth test cases than you do and what you are executing is lackluster in discovering a lot of the issues.

• They are covering more than you are, likely due to resources issues, perhaps you are testing mostly the new features and they are doing a regression suite across the entire project.

• Lack of training on the internal team in what is actually a defect, perhaps some things are spotted but left because the tester thinks they are intentional or "product funtionality".

• Your environment could be relatively toxic, perhaps dealing with non approachable devs/PM's etc some testers are in fear of logging the defects as it A) ends with it not being fixed anyway or B) confrontation.

• Inefficiencies and/or bad estimations mean you are struggling to actually test what you set out to cover originally.

The list could go on forever, but it's almost hard to really pinpoint without more detailed information on your end. Who is responsible for the testing effort in your organisation, sounds like you need more root cause analysis in order to discover why you tend to miss a lot of these issues and modify your processes accordingly.

Answer 4

You should take a few of the bugs found by the outside testers and ask your team how/why those weren't found by the internal team. Be clear that you aren't looking for fault, only trying to understand why they found bugs afterwards - for the purpose of learning & improving.

Answer 5

Problems that internal testing teams face:

Continuously working on the same application is a monotonous activity. I have many examples, where people don't even refer to the test cases that have been written; because they are so sure that they know the application in and out. Many times, internal testing teams are under too much pressure to meet the deadlines. This results in poorly created test cases and poor reviews. This is the reason, that the teams are used to writing test cases in a certain way, covering some common scenarios. Generally, reviewer is also from the same team, and being a part of the team for a long time, his brain is also tuned to accept only these scenarios. So, gradually, the whole team loses the ability to think out of box.

What does the external testing teams do different:

There is always more pressure on external testing teams. If they miss an issue, that is mostly likely, will go to production too. So, there thought process is more aligned with the business use of application. Even this is visible from the way external testers create test cases. They try to cover high level scenarios and also try to be on top of them. So, they will pick a scenario and test whatever they can (trying all the permutation and combinations); within the scope of that scenario. On the contrary, internal testers have in-depth test cases covering even the text for each error message. Because of so much detail, they are not on top of the test scenarios. At any point in time, an external tester knows "how many scenarios for a particular functionality have been executed?" whereas internal testers are just solely relying on test cases. Apart from this, external testers get a relatively stable product to test. Thanks to the internal testing team. So, they can focus on what they are targeting to test. There are hardly any distractions because of which they can lost focus. So, whatever they find is generally of great value for the team.

What internal testing teams should do?

Better planning (as you don't have too much time to experiment during the execution phase)

Generally, development phase activities take longer that planned and consume time of testing phase, squeezing the testing window, mounting more pressure on internal testing teams.

So, Create scenario based test cases. Before looking at test case coverage, ensure that all the test scenarios are covered.

Ensure that test case reviews are effective.

Get the test cases reviwed externally from another team. Because if you are missing too many issues then that means that you need to do away with the peer review policy because the person reviewing the test cases has the same mindset as of the test case writer. So, it is important to invite people who are looking at the application with a fresh eye.

Update the automation/manual regression suite regularly:

If you are not doing it periodically then application bugs are immune to this suite. So, you have caught whatever falls in the way of your test suites but nothing other than that. So, a six month or more older test suite which has not been updated is generally has limited capability to catch bugs.

keep your test suites updated for all the minor or major changes that have been introduced in the application.

Even if no changes have been made to the application, still try to include more and more scenarios in your test suite to increase the application coverage

After every release, do RCA of bugs and try to identify areas of improvement.

Do RCA of bugs. Try to find out major areas of improvement for testing team. Try to work on the weaknesses identified in order to be a stronger QA team after every release cycle.

Answer 6

I believe this comes down to a personal pet peeve of mine. It has become a fairly common thing for the testers to be integrated into the development teams, which, in itself, is a good thing. It shortens the feedback paths and accelerates the development. On the other hand the testers are much closer to the development process and are in danger to adopt certain biases. Let's assume a new feature gets implemented and validation of said feature is accomplished by using a complicated regex. The tester will be objected to discussions within the team and he will, willingly or not, develop a bias against the input validation, something the devs probably tested thoroughly themselves. An external tester would test the input validation but since he wouldn't suspect it to break he wouldn't spend as much time on it. It's a concept called believe bias and sadly enough there is nothing you can do against it (at least I came up with nothing that would solve the problem yet).

I'm not saying it's the only factor but it is an often overlooked factor.

Answer 7

I agree with gnasher729 and Aalok.

Also the additional point I would like to mention is about the Test environment. Please look into it whether both the teams are testing in the exact and more authentic environment.