18

Should "unplanned" test cases be taken against QA's metrics in the organisation?

For context: there is an established process during the development life cycle where the development team and QA get together and discuss tests to be performed and the development approach based on the work item. During this meeting, QA and development members create a set of "expected" tests and the developer codes against them. Many of these are "happy path" tests with the occasional exploratory/error handling scenarios.

What tends to happen during testing week however, is that the planned tests generally get pass marks but QA still is able to find many bugs of the exploratory nature. I believe that this is normal as exploratory testing exists to help uncover defects, but is it in any way QA's fault that there are bugs found via unplanned methods?

Management believes that QA should be able to identify as many scenarios as possible during the initial meeting, but there is also a limit to how much one can plan ahead when only reviewing requirements and actually using the system. This I feel also encourages over reliance on the QA team for quality and puts blame on QA for doing exploratory testing, when in fact it should be a shared responsibility within the team.

Thoughts?

6
  • In a way yes: QA should have read the requirements and ensured that all requirements are tested. Obviously this presupposes good requirements. If nobody in your organisation knows how to write good requirements you are not going to succeed.
    – RedSonja
    Dec 2 '21 at 13:36
  • Do not allow people to do random unplanned tests. Ensure that your tests cover all possibilities. Use a proper management tool to keep track.
    – RedSonja
    Dec 2 '21 at 13:39
  • 2
    @RedSonja Can you tell me the difference between random unplanned tests and exploratory tests?
    – corsiKa
    Dec 3 '21 at 22:01
  • 2
    @RedSonja how is it possible to ever know about all possibilities?
    – Lee Jensen
    Dec 3 '21 at 22:41
  • 1
    Predictions such as all possibilities are very hard to make accurately, especially when they concern the future.This was an essential learning of Agile and why people, communications and constant learning are at the heart of quality. Dec 5 '21 at 16:49

14 Answers 14

39

I think the first big problem here is the attempt to assign blame. In accident reports, the FAA doesn't start with who to blame, they start with what went wrong and what were the causal factors. Assigning blame causes all types of unintended consequences.

In the worst case, if you blame/penalize QA for finding bugs that were not foreseen during the design phase, you will encourage QA to not find them during testing. NOTE: this means that it is the customer that will find those bugs. The idea of counting unplanned bugs is worse than penalizing developers for KLOC drops.

To explain KLOC drops, understand that IBM tracked developer stats by counting lines of code written (a KLOC is a thousand lines of code). The numbers only really have meaning for new, greenfield projects. Once code is in place, changing existing code to be better doesn't show in the metric. Changing the code to be simpler/easier to understand can actually reduce your numbers. It is one of the problems that Microsoft had with IBM on OS/2 (Microsoft didn't measure KLOC, IBM did). So Microsoft would want to fix/improve the code and IBM employees would resist because changes that didn't increase the KLOC count were not valued.

In this blame game are the developers penalized for each bug that QA finds? If not, then you have an inconsistent view of blame and responsibility.

Another thing to remember about metrics is to count what matters, not what you can measure. Too much of what people do with metrics is to count what they can measure. Not all metrics are valuable (some even reduce value).

5
  • 4
    "Microsoft would want to improve the code and IBM employees would resist because changes that didn't increase the KLOC count were not valued." - Thank you, I did not know this. All these years I've been blaming Microsoft for that hot steaming mess. Dec 2 '21 at 20:09
  • 2
    Adding to the ‘count what matters’, a lot of people also tend to misuse metrics . Alongside the KLOC example, one of my favorite examples of this is people who try to use daily changes in Docker pulls as counted by Docker Hub to count the number of active users of a container, which is absolutely not what that metrics means for multiple reasons. Dec 2 '21 at 22:34
  • 1
    @Walter Adding to your "count what matters" point... I was in a place in the mid 00s where software development quality was "measured" by how many version control commits were needed, on the theory that you'd only commit code after you thought it worked. The "commit early and often" concept had completely passed them by. The result was that if an engineer was away, no-one could check the code, because the only copy of the latest code was sat on their PC. It was painful explaining me committing almost daily was right and I wasn't going to stop best practise for some arbitrary bullshit.
    – Graham
    Dec 3 '21 at 13:08
  • 2
    "counting unplanned bugs" - is there such a things as a planned bug? I hope not... Dec 3 '21 at 15:59
  • 1
    @GregBurghardt Unfortunately, the unscrupulous practice of planned bugs still exists today. Bugs are intentionally introduced (or not fixed) so that fees can be charged (to clients or end users) to fix them. It's an ugly practice. Dec 3 '21 at 17:05
13

There can be a several reasons for test cases for bugs found later during exploratory testing not being identified during the project planning phase.

Couple of them may be,

  1. Testers are humans. To err is human. Humans make mistakes. IF not wouldn't we be god instead?

  2. From what you have written, you do the planning along with the whole team and not in isolation. If so, why didn't anyone else catch the case early on? What exactly were the Business Analysts doing? And the Project Manager, developer, etc? If it's only about blaming, they should all be blamed for missing it.

Why is it a concern that the cases weren't identified and listed prior to development?

The project hasn't gone live yet. So infact, it should be a good thing that the issues were caught then. What if you had ignored exploratory tests and the bugs would have leaked into production?

Is the concern that the managers hadn't considered a buffer time to unplanned issues, roadblocks, disasters and hence didn't give an well thought through ETA for project delivery?

If everything goes well in a projects, are the testers alone to get all credit for job well done in your team? If not, wouldn't the same apply for any issues found in the project before or after production release?

10

First, the previous answers are good and bring up different view points to consider.

I agree that management shouldn't be finding ways to blame QA for missing bugs or finding bugs in unexpected, unplanned ways. That's a huge benefit to the team and a good skill to have when doing testing.

My take on this is why is there only "happy path" test cases being considered when doing test planning? To me, this is a bare minimum to what QA should do. Why not plan for negative test cases during planning or "sad panda" paths/scenarios? This can certainly be done based on the given requirements of a feature/story.

If the requirements in the ticket only mention positive aspects of what's to be expected, you can ask about different negative aspects or different states a user can expect.

As an example, if you have a requirement that states "a user should be able to create a password with a specific character set." Then you can ask about error conditions or error messages a user should get if they don't follow that password guideline. You shouldn't have to wait until testing is in progress to say "I found a bug. When I input non valid characters, I don't see an error message when I expect one." You can and should know this ahead of time!

While you can't expect to know of all test cases ahead of time, you should be able to use the right testing techniques (like boundary value analysis, equivalence classes, state management, etc), prior application knowledge, and prior usage of similar apps to uncover more tests cases.

The well known tester, Michael Bolton, has a heuristic called FEW HICCUPPS that can be used to guide test case creation. The acronym stands for:

Familiarity - how familiar are you with your product or similar products?

Explainability - can you explain or understand the behavior of the product?

World - is the project/feature consistent with your prior observations and general view (familiarity) of similar software products?

History - are the requirements consistent with prior versions of the software?

Image - are the requirements consistent with the branding, marketing, reputation of the company?

Comparable Products - similar to familiarity and knowledge of similar software products.

Claims - are the requirements consistent with prior expectations?

Users Desires - are the requirements consistent with inline with what the users or stakeholders want?

Product - are the requirements consistent with or integrate well with other features of the product?

Purpose - similar to user desires.

Statutes and Standards - are the requirements consistent with inline with any laws, standards, regulations?

The details and definitions of each of these can be long, so I don't want to go into too much detail of them. I highly encourage reading the link.

1
  • 1
    You're missing the second P in Hiccupps
    – Tim
    Dec 2 '21 at 0:18
7

I think your tittle can also read "should we blame QA for finding bugs?", to which I hope no one replies yes, although I'm not so certain about the management in your organisation.

You got it right, not all test ideas or bugs can be expected when reading documentation, requirements, or other documents. You simply have to interact with the system to come up with other test ideas that might expose other bugs.

If anyone thinks all test ideas could be figured out in advance, perhaps they can try testing for a month as a way of learning about the work. Otherwise, your only hope is to keep educating them about how testing is performed.

1
  • 1
    Tittle... giggle... Nobody wins in the blame game. Fix it and move on. If management is fixated on keeping score, then there are too many managers IMO.
    – ToastMan
    Dec 14 '21 at 15:22
5

Identifying problems should never count against anyone, as it leads to a culture of defensiveness, cover-ups and people attempting to game the metrics.

In theory you could track the ratio of bugs found during planned and unplanned testing, as it could indicate that you are not spending enough time on risk assessment.

However, if the developers know what the planned tests are before they release the code to testing, the developers should check that their code will pass all the tests. The planned tests should find no bugs, and all the bugs will be found by unplanned testing.

If the developers do their job properly, it will then look as if your planning session was terrible and didn't anticipate any of the bugs, when in fact it identified lots of potential bugs that were fixed before the code reached the testers.

Personally, I think the thing to watch for with exploratory testing is that it can cause scope creep, as testers think of new requirements (or find problems with the original requirements) and raise them as bugs. If the requirements change, you need to revisit the project plan.

2
  • 1
    Before the developers even start touching keys, the requirements have to be validated and placed under version control. If your requirements are clean, they can be used to generate tests as well as code. If your organisation is not managing its requirements they are bound to fail.
    – RedSonja
    Dec 2 '21 at 13:09
  • 1
    Tester programs should start with the speech "All non-trivial code contains bugs. Our job is to find them before the customer does."
    – RedSonja
    Dec 2 '21 at 13:10
3

Testing is always partially exploratory, and for any reasonably complex software, testing can never completely exhaust all the possibilities. Meaning, you will always find more issues and think of more scenarios as your knowledge about the application grows. Requiring all the scenarios upfront is just not realistic.

7
  • It is realistic. You and the customer have to agree when the scenarios and thus the requirements are done. Otherwise your project will keep growing. New requirements (scenarios) have to be agreed and should cost the customer more money, because they will cost more time.
    – RedSonja
    Dec 2 '21 at 13:32
  • Part of the testing is learning about the app and clarifying the requirements. If all scenarios are given up front that means there is no learning/exploration/clarification required or possible.
    – Mate Mrše
    Dec 2 '21 at 13:39
  • Are you a research team or a software company? A serious company will not contemplate just adding new requirements at no cost to the customer. Imagine trying to build physical objects like cars that way. New requirements can always be added, but not at no cost.
    – RedSonja
    Dec 2 '21 at 13:42
  • I'm not talking about new requirements, just a better understanding of the existing ones.
    – Mate Mrše
    Dec 2 '21 at 14:28
  • Requirements should be clear and understandable and testable. Writing good requirements is non-trivial. It takes time and training. Whether you are building cars or software.
    – RedSonja
    Dec 2 '21 at 14:33
2

"One can predict the ways in which a software should work but cannot predict the ways in which it will not work."

Testing is inherently an exploratory activity.

If we deprive or even blame an activity for its core characteristic(s) then basically we are turning it into an empty ritual.

Exploratory Testing

Software testing itself essentially is exploratory testing as has been called in the above link by the great testing pioneers in our field .

Any QA should NOT be blamed for finding additional "unplanned" bugs through exploratory testing as long it makes sense for the business. A tester cannot know in advance in which ways bugs will be manifested in the behavior of an application although positive or as sometimes called "Happy Path" scenarios can and should be identified in the initial planning meeting.

So in a nutshell, positive scenarios can be identified in the initial planning meeting whereas negative scenarios cannot and need to be identified later during actual testing sessions using exploratory approach.

However the extent and duration to which exploratory testing will be used will differ on the nature and business context of the given application as an gaming application might be tested differently significantly rather than a application supporting medical treatment.

1

I agree with you regarding QA should find as many problems as they can especially with exploratory tests.

Assuming your management implements a policy in wich your testers are not allowed to deviate from your predefined behaviour. With this policy in place you just created a very expensive automatic testing suite. My point is: test cases you can anticipate should be handled by automatic systems.

Another point is: problems and paths you can anticipate likely won't translate to bugs, well since you anticipated them.

So basically the situation is: your testers test cases, your developers coded against, everyone already anticipated and wich can be (probably) done fully automated.

Also I heard an urban legend about a company aggressively disallowing any contact between the developers and testers in order to impede the ability of the developers to code against the testers cases. They had separate canteens, were not allowed to speak to each other and were penalized for breaking those rules. All communication went over bugreports and related official channels.

tl;dr: My recommendation is to implement a workflow that focuses on exploratory tests. Any problems you find there can be implemented as tests in an automated system.

1

The "blame QA for bugs" mentality should be stamped out in management circles. The primary point of QA is to find problems which means they should be actively praised and sometimes even rewarded for doing so. Each bug they find is one that customers don't (assuming that it gets fixed). That not only improves sales but also reduces maintenance costs drastically.

Regarding unplanned testing, how can QA plan for how developers choose to implement designs? At best, they can write tests against requirements and designs and what additional features developers reveal during discussions about their work. QA should be testing in ways unexpected by developers to help in finding and squashing inadvertent mistakes in code.

Ideally, all regression testing (and the bulk of requirements-based testing) should be handled by automated testing. This would free QA to go bug-hunting for the unexpected but important problems.

As noted above, usefully complex software will ALWAYS have a super-abundance of logic errors. It's impossible to prevent this. QA will ALWAYS have more work to do than they can cost-effectively complete. Why discourage them from at least trying to find the problems?

1

Well, ideally the designed tests should cover all error cases.

Exhaustive tests are very costly, and in the end the management must decide what confidence level they want to achieve, and how much money they want to spend on it, aiming at a sweet spot depending on the application. This money should obviously be used most efficiently.

Exploratory tests, especially when not automated, are not very efficient due to their random nature. It would be much better to identify and test possibilities for errors in a systematic fashion.

Lots of errors found only in exploratory tests make it likely that lots of undiscovered bugs are still in the software; the ultimate exploratory tester is the end user who will not be happy.

Therefore, defects only found "by chance" (i.e., by exploratory tests) should be analyzed to find out whether a better test design would have found them earlier.

Whether anybody is to blame is a different question, and even then it is unclear whether they should be blamed. But of course: If the reason bugs are not found is the incompetence of the testers it's high time to make them competent, or to replace them.

Because your question is a bit ambiguous, let me make clear: QA is only to be blamed for errors they didn't find, but should have.

1

No.

But I can see why management are making the terrible mistake of thinking the answer is yes. There are three cases:

  1. A bug is caught by a planned test
  2. A bug is caught by QA exploratory testing, or other means
  3. A bug is not caught

While case 1 might be better than case 2, both are much, much better than case 3.

But the attitude that testing should be all planned in advance is fundamentally flawed. It is not possible to cover every possibility in testing (and anyone who claims otherwise should go and read about the Pentium FDIV bug which was triggered by a tiny subset of inputs to a single instruction - much easier to test than the millions of instructions needed by even a small program).

Having all the tests defined and visible to the developers is like giving students their exam questions before they start their course. One of the jobs of developers is to try to write code which correctly covers all cases, and one of the purposes of testing is to try to determine to what extent they have succeeded. In manufacturing tangible objects, testing frequently involves random sampling to verify the process is satisfactory. Software testing should include an element of this - if your QA stage starts finding too many bugs, you can then deduce that there are probably many more that have not been found and you have a big problem. It's much better to know you have a problem than to fool yourself.

0

Sounds like you have a tough gig there. Unfortunately your situation isn't uncommon and even in mature organizations, you will hear phrases like, 'What you didn't test that!'

There is so much good advice already, however I think there is something missed.

Your issues do in part sound as if it's a top down issue, with your management needing to take a more constructive proactive role in quality management. Management needs to communicate that quality is the responsibility of the whole organization, starting with Product Owners and Project Managers, all the way to Release Management.

However, exactly this 'happy path' testing arrangement you have with development needs to change. This 'practice is effectively creating a sort of insecticide paradox and contributing to lazy development practices. Development needs to deliver the specification to you and then you decide how it should be tested. Yes, by all means talk with development, query every single part of the spec so you know it and how it should all function, but ultimately you write the tests and you incorporate the 'exploratory' elements into it.

Customer and usage knowledge with some risk assessment will help you here, to find the RIGHT PATH and every other side alley that you need to check or shine a torch down.

Actually, the meetings where you talk with development, are if it's a regular thing, a great way to start building up profiles of your developers. Knowing who are the lazy coders vs the diligent ones also helps with planning ad hoc and exploratory testing. I've used it for many years and it works well, just don't let them know your doing it (they can get a bit funny about the idea).

Please don't take this the wrong way, but it sounds like your pretty new to testing in the way you describe the testing and organizational practices. Do yourself a favor and read the ISTQB Foundation Testing Syllabus. It will ramp up your skills and knowledge so much, in such a short time and I think, give you the confidence to deal with the developers and answer the management when they come blaming you.

https://www.istqb.org/downloads/syllabi/foundation-level-syllabus.html

As a Tester or QA you are the end users last line of defense and you have to own it.

There will always be defects, but if you can show that you have done due diligence to try and identify them, then it is up management to change bad processes and development practices.

TLDR; Management needs to own quality from top down. That also means you need to own the quality. You need to steer testing and you need to ramp up your skills (See link). You'll be OK but you need to fight your corner.

0

In my opinion, no product is perfect. Especially, when there are endless devices available in the market running on different operating systems or device configurations, the test process develops so many limits naturally.

Since exploratory testing is more of an approach that follows not so common methods of testing a product. The results obtained must be considered to improve the upcoming versions of the given application.

Exploratory testing helps QA teams to identify the exceptional flaws and therefore it should never be considered as a reason to blame QA or raise conflicts between the development companies and QA service providers.

0

In principle, there are of course different approaches. Explorative testing, however, always means testing for unexpected behaviour. As I understand the initial question, there is a misunderstanding on the part of the management as to what exploratory testing actually is.

Exploratory testing is particularly suitable when requirements and specifications are incomplete or when there is not enough time for testing. The approach can also be used to check whether previous tests have detected the most important errors. (see http://www.kaner.com/pdfs/QAIExploring.pdf)

Exploratory testing –

  • Is not random testing but it is ad-hoc testing with a purpose of find bugs
  • Is structured and rigorous
  • Is cognitively (thinking) structured as compared to the procedural structure of scripted testing. This structure comes from Charter, time boxing etc.
  • Is highly teachable and manageable
  • It is not a technique but it is an approach. What actions you perform next is governed by what you are doing currently

Challenges of Exploratory Testing: There are many challenges of exploratory testing and those are explained below:

  • Learning to use the application or software system is a challenge
  • Replication of failure is difficult
  • Determining whether tools need to be used can be challenging
  • Determine the best test cases to execute can be difficult
  • Reporting of the test results is a challenge as the report doesn’t have planned scripts or cases to compare with the actual result or outcome
  • Documentation of all events during execution is difficult to record
  • Don’t know when to stop the testing as exploratory testing has definite test cases to execute.

When use exploratory testing? Exploratory testing can be used extensively when

  • The testing team has experienced testers
  • Early iteration is required
  • There is a critical application
  • New testers entered into the team

This results in new tests after each run, which can then be tested in a further run.

It is not in the least about the test having to recognise everything immediately, whether it is a positive or negative test, but about recognising misbehaviour that is not apparent in a normal test run.

"Exploratory testing is an approach to software testing that is described as simultaneous learning, test design and test execution. Cem Kaner, who coined the term in 1993, defines exploratory testing as "a style of software testing that emphasises the personal freedom and responsibility of the individual tester to continually optimise the quality of his or her work by using test-related learning, test design, test execution, and interpretation of test results as mutually supportive activities that occur in parallel throughout the project."

http://www.kaner.com/pdfs/QAIExploring.pdf

As the software is tested, the tester learns things that, together with experience and creativity, produce new tests. Exploratory testing is often mistakenly thought of as a black-box testing technique. Correctly, it is a testing approach that can be applied to any testing technique at any stage of the development process. The key is neither the test technique nor the object to be tested or reviewed. The key is the tester's cognitive engagement and the tester's ownership of the meaningful use of his or her time for testing."

http://www.kaner.com/pdfs/QAIExploring.pdf

The most important advantage of exploratory testing over traditional testing is that less preparation is required, important bugs are found quickly and it is more intellectually stimulating than running scripted tests.

Another major advantage is that testers can react to results of previous tests, and modify their current tests appropriately. They do not have to complete a series of script tests, but can focus on or explore new requirements in a targeted way. This also speeds up error detection.

Another advantage is that most errors in a programme or part of a programme are discovered after a short time with exploratory testing. Programmes that pass certain tests tend to continue to pass the same tests and are more likely to pass other tests or scenarios that have not yet been explored."

A disadvantage of exploratory testing is that the tests invented and executed on the fly cannot be verified in advance, preventing misunderstandings about requirements or errors in test cases. Furthermore, it can be difficult to show exactly which tests have been performed. The traceability of the errors found may suffer.

It is unlikely that exploratory tests will be run in exactly the same way when they are run again. This can be an advantage when it is important to find new bugs, but also a disadvantage when it is more important to repeat certain details of the earlier tests. This can be controlled with specific techniques (e.g. recording the tests) by preparing automated tests.

Used Informations:

  1. http://www.kaner.com/pdfs/QAIExploring.pdf
  2. https://www.guru99.com/exploratory-testing.html

enter image description here enter image description here

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.