Should bugs found by exploratory test methods be taken against QA?

Question

Should "unplanned" test cases be taken against QA's metrics in the organisation?

For context: there is an established process during the development life cycle where the development team and QA get together and discuss tests to be performed and the development approach based on the work item. During this meeting, QA and development members create a set of "expected" tests and the developer codes against them. Many of these are "happy path" tests with the occasional exploratory/error handling scenarios.

What tends to happen during testing week however, is that the planned tests generally get pass marks but QA still is able to find many bugs of the exploratory nature. I believe that this is normal as exploratory testing exists to help uncover defects, but is it in any way QA's fault that there are bugs found via unplanned methods?

Management believes that QA should be able to identify as many scenarios as possible during the initial meeting, but there is also a limit to how much one can plan ahead when only reviewing requirements and actually using the system. This I feel also encourages over reliance on the QA team for quality and puts blame on QA for doing exploratory testing, when in fact it should be a shared responsibility within the team.

Thoughts?

Answer 1

I think the first big problem here is the attempt to assign blame. In accident reports, the FAA doesn't start with who to blame, they start with what went wrong and what were the causal factors. Assigning blame causes all types of unintended consequences.

In the worst case, if you blame/penalize QA for finding bugs that were not foreseen during the design phase, you will encourage QA to not find them during testing. NOTE: this means that it is the customer that will find those bugs. The idea of counting unplanned bugs is worse than penalizing developers for KLOC drops.

To explain KLOC drops, understand that IBM tracked developer stats by counting lines of code written (a KLOC is a thousand lines of code). The numbers only really have meaning for new, greenfield projects. Once code is in place, changing existing code to be better doesn't show in the metric. Changing the code to be simpler/easier to understand can actually reduce your numbers. It is one of the problems that Microsoft had with IBM on OS/2 (Microsoft didn't measure KLOC, IBM did). So Microsoft would want to fix/improve the code and IBM employees would resist because changes that didn't increase the KLOC count were not valued.

In this blame game are the developers penalized for each bug that QA finds? If not, then you have an inconsistent view of blame and responsibility.

Another thing to remember about metrics is to count what matters, not what you can measure. Too much of what people do with metrics is to count what they can measure. Not all metrics are valuable (some even reduce value).

Answer 2

There can be a several reasons for test cases for bugs found later during exploratory testing not being identified during the project planning phase.

Couple of them may be,

1. Testers are humans. To err is human. Humans make mistakes. IF not wouldn't we be god instead?

2. From what you have written, you do the planning along with the whole team and not in isolation. If so, why didn't anyone else catch the case early on? What exactly were the Business Analysts doing? And the Project Manager, developer, etc? If it's only about blaming, they should all be blamed for missing it.

Why is it a concern that the cases weren't identified and listed prior to development?

The project hasn't gone live yet. So infact, it should be a good thing that the issues were caught then. What if you had ignored exploratory tests and the bugs would have leaked into production?

Is the concern that the managers hadn't considered a buffer time to unplanned issues, roadblocks, disasters and hence didn't give an well thought through ETA for project delivery?

If everything goes well in a projects, are the testers alone to get all credit for job well done in your team? If not, wouldn't the same apply for any issues found in the project before or after production release?

Answer 3

First, the previous answers are good and bring up different view points to consider.

I agree that management shouldn't be finding ways to blame QA for missing bugs or finding bugs in unexpected, unplanned ways. That's a huge benefit to the team and a good skill to have when doing testing.

My take on this is why is there only "happy path" test cases being considered when doing test planning? To me, this is a bare minimum to what QA should do. Why not plan for negative test cases during planning or "sad panda" paths/scenarios? This can certainly be done based on the given requirements of a feature/story.

If the requirements in the ticket only mention positive aspects of what's to be expected, you can ask about different negative aspects or different states a user can expect.

As an example, if you have a requirement that states "a user should be able to create a password with a specific character set." Then you can ask about error conditions or error messages a user should get if they don't follow that password guideline. You shouldn't have to wait until testing is in progress to say "I found a bug. When I input non valid characters, I don't see an error message when I expect one." You can and should know this ahead of time!

While you can't expect to know of all test cases ahead of time, you should be able to use the right testing techniques (like boundary value analysis, equivalence classes, state management, etc), prior application knowledge, and prior usage of similar apps to uncover more tests cases.

The well known tester, Michael Bolton, has a heuristic called FEW HICCUPPS that can be used to guide test case creation. The acronym stands for:

Familiarity - how familiar are you with your product or similar products?

Explainability - can you explain or understand the behavior of the product?

World - is the project/feature consistent with your prior observations and general view (familiarity) of similar software products?

History - are the requirements consistent with prior versions of the software?

Image - are the requirements consistent with the branding, marketing, reputation of the company?

Comparable Products - similar to familiarity and knowledge of similar software products.

Claims - are the requirements consistent with prior expectations?

Users Desires - are the requirements consistent with inline with what the users or stakeholders want?

Product - are the requirements consistent with or integrate well with other features of the product?

Purpose - similar to user desires.

Statutes and Standards - are the requirements consistent with inline with any laws, standards, regulations?

The details and definitions of each of these can be long, so I don't want to go into too much detail of them. I highly encourage reading the link.

Answer 4

I think your tittle can also read "should we blame QA for finding bugs?", to which I hope no one replies yes, although I'm not so certain about the management in your organisation.

You got it right, not all test ideas or bugs can be expected when reading documentation, requirements, or other documents. You simply have to interact with the system to come up with other test ideas that might expose other bugs.

If anyone thinks all test ideas could be figured out in advance, perhaps they can try testing for a month as a way of learning about the work. Otherwise, your only hope is to keep educating them about how testing is performed.

Answer 5

Identifying problems should never count against anyone, as it leads to a culture of defensiveness, cover-ups and people attempting to game the metrics.

In theory you could track the ratio of bugs found during planned and unplanned testing, as it could indicate that you are not spending enough time on risk assessment.

However, if the developers know what the planned tests are before they release the code to testing, the developers should check that their code will pass all the tests. The planned tests should find no bugs, and all the bugs will be found by unplanned testing.

If the developers do their job properly, it will then look as if your planning session was terrible and didn't anticipate any of the bugs, when in fact it identified lots of potential bugs that were fixed before the code reached the testers.

Personally, I think the thing to watch for with exploratory testing is that it can cause scope creep, as testers think of new requirements (or find problems with the original requirements) and raise them as bugs. If the requirements change, you need to revisit the project plan.

Answer 6

Testing is always partially exploratory, and for any reasonably complex software, testing can never completely exhaust all the possibilities. Meaning, you will always find more issues and think of more scenarios as your knowledge about the application grows. Requiring all the scenarios upfront is just not realistic.

Answer 7

"One can predict the ways in which a software should work but cannot predict the ways in which it will not work."

Testing is inherently an exploratory activity.

If we deprive or even blame an activity for its core characteristic(s) then basically we are turning it into an empty ritual.

Exploratory Testing

Software testing itself essentially is exploratory testing as has been called in the above link by the great testing pioneers in our field .

Any QA should NOT be blamed for finding additional "unplanned" bugs through exploratory testing as long it makes sense for the business. A tester cannot know in advance in which ways bugs will be manifested in the behavior of an application although positive or as sometimes called "Happy Path" scenarios can and should be identified in the initial planning meeting.

So in a nutshell, positive scenarios can be identified in the initial planning meeting whereas negative scenarios cannot and need to be identified later during actual testing sessions using exploratory approach.

However the extent and duration to which exploratory testing will be used will differ on the nature and business context of the given application as an gaming application might be tested differently significantly rather than a application supporting medical treatment.

Answer 8

I agree with you regarding QA should find as many problems as they can especially with exploratory tests.

Assuming your management implements a policy in wich your testers are not allowed to deviate from your predefined behaviour. With this policy in place you just created a very expensive automatic testing suite. My point is: test cases you can anticipate should be handled by automatic systems.

Another point is: problems and paths you can anticipate likely won't translate to bugs, well since you anticipated them.

So basically the situation is: your testers test cases, your developers coded against, everyone already anticipated and wich can be (probably) done fully automated.

Also I heard an urban legend about a company aggressively disallowing any contact between the developers and testers in order to impede the ability of the developers to code against the testers cases. They had separate canteens, were not allowed to speak to each other and were penalized for breaking those rules. All communication went over bugreports and related official channels.

tl;dr: My recommendation is to implement a workflow that focuses on exploratory tests. Any problems you find there can be implemented as tests in an automated system.

Answer 9

The "blame QA for bugs" mentality should be stamped out in management circles. The primary point of QA is to find problems which means they should be actively praised and sometimes even rewarded for doing so. Each bug they find is one that customers don't (assuming that it gets fixed). That not only improves sales but also reduces maintenance costs drastically.

Regarding unplanned testing, how can QA plan for how developers choose to implement designs? At best, they can write tests against requirements and designs and what additional features developers reveal during discussions about their work. QA should be testing in ways unexpected by developers to help in finding and squashing inadvertent mistakes in code.

Ideally, all regression testing (and the bulk of requirements-based testing) should be handled by automated testing. This would free QA to go bug-hunting for the unexpected but important problems.

As noted above, usefully complex software will ALWAYS have a super-abundance of logic errors. It's impossible to prevent this. QA will ALWAYS have more work to do than they can cost-effectively complete. Why discourage them from at least trying to find the problems?

Answer 10

Well, ideally the designed tests should cover all error cases.

Exhaustive tests are very costly, and in the end the management must decide what confidence level they want to achieve, and how much money they want to spend on it, aiming at a sweet spot depending on the application. This money should obviously be used most efficiently.

Exploratory tests, especially when not automated, are not very efficient due to their random nature. It would be much better to identify and test possibilities for errors in a systematic fashion.

Lots of errors found only in exploratory tests make it likely that lots of undiscovered bugs are still in the software; the ultimate exploratory tester is the end user who will not be happy.

Therefore, defects only found "by chance" (i.e., by exploratory tests) should be analyzed to find out whether a better test design would have found them earlier.

Whether anybody is to blame is a different question, and even then it is unclear whether they should be blamed. But of course: If the reason bugs are not found is the incompetence of the testers it's high time to make them competent, or to replace them.

Because your question is a bit ambiguous, let me make clear: QA is only to be blamed for errors they didn't find, but should have.

Answer 11

No.

But I can see why management are making the terrible mistake of thinking the answer is yes. There are three cases:

1. A bug is caught by a planned test
2. A bug is caught by QA exploratory testing, or other means
3. A bug is not caught

While case 1 might be better than case 2, both are much, much better than case 3.

But the attitude that testing should be all planned in advance is fundamentally flawed. It is not possible to cover every possibility in testing (and anyone who claims otherwise should go and read about the Pentium FDIV bug which was triggered by a tiny subset of inputs to a single instruction - much easier to test than the millions of instructions needed by even a small program).

Having all the tests defined and visible to the developers is like giving students their exam questions before they start their course. One of the jobs of developers is to try to write code which correctly covers all cases, and one of the purposes of testing is to try to determine to what extent they have succeeded. In manufacturing tangible objects, testing frequently involves random sampling to verify the process is satisfactory. Software testing should include an element of this - if your QA stage starts finding too many bugs, you can then deduce that there are probably many more that have not been found and you have a big problem. It's much better to know you have a problem than to fool yourself.

Answer 12

Sounds like you have a tough gig there. Unfortunately your situation isn't uncommon and even in mature organizations, you will hear phrases like, 'What you didn't test that!'

There is so much good advice already, however I think there is something missed.

Your issues do in part sound as if it's a top down issue, with your management needing to take a more constructive proactive role in quality management. Management needs to communicate that quality is the responsibility of the whole organization, starting with Product Owners and Project Managers, all the way to Release Management.

However, exactly this 'happy path' testing arrangement you have with development needs to change. This 'practice is effectively creating a sort of insecticide paradox and contributing to lazy development practices. Development needs to deliver the specification to you and then you decide how it should be tested. Yes, by all means talk with development, query every single part of the spec so you know it and how it should all function, but ultimately you write the tests and you incorporate the 'exploratory' elements into it.

Customer and usage knowledge with some risk assessment will help you here, to find the RIGHT PATH and every other side alley that you need to check or shine a torch down.

Actually, the meetings where you talk with development, are if it's a regular thing, a great way to start building up profiles of your developers. Knowing who are the lazy coders vs the diligent ones also helps with planning ad hoc and exploratory testing. I've used it for many years and it works well, just don't let them know your doing it (they can get a bit funny about the idea).

Please don't take this the wrong way, but it sounds like your pretty new to testing in the way you describe the testing and organizational practices. Do yourself a favor and read the ISTQB Foundation Testing Syllabus. It will ramp up your skills and knowledge so much, in such a short time and I think, give you the confidence to deal with the developers and answer the management when they come blaming you.

https://www.istqb.org/downloads/syllabi/foundation-level-syllabus.html

As a Tester or QA you are the end users last line of defense and you have to own it.

There will always be defects, but if you can show that you have done due diligence to try and identify them, then it is up management to change bad processes and development practices.

TLDR; Management needs to own quality from top down. That also means you need to own the quality. You need to steer testing and you need to ramp up your skills (See link). You'll be OK but you need to fight your corner.

Answer 13

In my opinion, no product is perfect. Especially, when there are endless devices available in the market running on different operating systems or device configurations, the test process develops so many limits naturally.

Since exploratory testing is more of an approach that follows not so common methods of testing a product. The results obtained must be considered to improve the upcoming versions of the given application.

Exploratory testing helps QA teams to identify the exceptional flaws and therefore it should never be considered as a reason to blame QA or raise conflicts between the development companies and QA service providers.

Answer 14

In principle, there are of course different approaches. Explorative testing, however, always means testing for unexpected behaviour. As I understand the initial question, there is a misunderstanding on the part of the management as to what exploratory testing actually is.

Exploratory testing is particularly suitable when requirements and specifications are incomplete or when there is not enough time for testing. The approach can also be used to check whether previous tests have detected the most important errors. (see http://www.kaner.com/pdfs/QAIExploring.pdf)

Exploratory testing –

• Is not random testing but it is ad-hoc testing with a purpose of find bugs
• Is structured and rigorous
• Is cognitively (thinking) structured as compared to the procedural structure of scripted testing. This structure comes from Charter, time boxing etc.
• Is highly teachable and manageable
• It is not a technique but it is an approach. What actions you perform next is governed by what you are doing currently

Challenges of Exploratory Testing: There are many challenges of exploratory testing and those are explained below:

• Learning to use the application or software system is a challenge
• Replication of failure is difficult
• Determining whether tools need to be used can be challenging
• Determine the best test cases to execute can be difficult
• Reporting of the test results is a challenge as the report doesn’t have planned scripts or cases to compare with the actual result or outcome
• Documentation of all events during execution is difficult to record
• Don’t know when to stop the testing as exploratory testing has definite test cases to execute.

When use exploratory testing? Exploratory testing can be used extensively when

• The testing team has experienced testers
• Early iteration is required
• There is a critical application
• New testers entered into the team

This results in new tests after each run, which can then be tested in a further run.

It is not in the least about the test having to recognise everything immediately, whether it is a positive or negative test, but about recognising misbehaviour that is not apparent in a normal test run.

"Exploratory testing is an approach to software testing that is described as simultaneous learning, test design and test execution. Cem Kaner, who coined the term in 1993, defines exploratory testing as "a style of software testing that emphasises the personal freedom and responsibility of the individual tester to continually optimise the quality of his or her work by using test-related learning, test design, test execution, and interpretation of test results as mutually supportive activities that occur in parallel throughout the project."

http://www.kaner.com/pdfs/QAIExploring.pdf

As the software is tested, the tester learns things that, together with experience and creativity, produce new tests. Exploratory testing is often mistakenly thought of as a black-box testing technique. Correctly, it is a testing approach that can be applied to any testing technique at any stage of the development process. The key is neither the test technique nor the object to be tested or reviewed. The key is the tester's cognitive engagement and the tester's ownership of the meaningful use of his or her time for testing."

http://www.kaner.com/pdfs/QAIExploring.pdf

The most important advantage of exploratory testing over traditional testing is that less preparation is required, important bugs are found quickly and it is more intellectually stimulating than running scripted tests.

Another major advantage is that testers can react to results of previous tests, and modify their current tests appropriately. They do not have to complete a series of script tests, but can focus on or explore new requirements in a targeted way. This also speeds up error detection.

Another advantage is that most errors in a programme or part of a programme are discovered after a short time with exploratory testing. Programmes that pass certain tests tend to continue to pass the same tests and are more likely to pass other tests or scenarios that have not yet been explored."

A disadvantage of exploratory testing is that the tests invented and executed on the fly cannot be verified in advance, preventing misunderstandings about requirements or errors in test cases. Furthermore, it can be difficult to show exactly which tests have been performed. The traceability of the errors found may suffer.

It is unlikely that exploratory tests will be run in exactly the same way when they are run again. This can be an advantage when it is important to find new bugs, but also a disadvantage when it is more important to repeat certain details of the earlier tests. This can be controlled with specific techniques (e.g. recording the tests) by preparing automated tests.

Used Informations:

1. http://www.kaner.com/pdfs/QAIExploring.pdf
2. https://www.guru99.com/exploratory-testing.html