17

We have a very huge product. Although we had all kinds of tests (units test, integration tests and automation tests), there are still lots of bugs and the customer complains a lot about it.

My manager has asked me to figure out why we are failing to provide a good product.

Please notice, here I am talking about product defects. Thus, even though process and project are the other aspects of quality, I am focused here only on the product.

As a potential solution, I suggested to collect a random sample of 50 tickets (bugs) and study them carefully, so that we figure out why we did not detect them during the development phase.

But, what else shall we do to debug our QA process?

Improve this question
6
  • 1
    Could you describe in more detail how is PDCA implemented in your context? Without it, your context is pretty much hidden and the question is "Too broad".
    – João Farias
    Oct 20, 2019 at 23:55
  • Who writes the tests?
    – jcaron
    Oct 21, 2019 at 9:41
  • 3
    "figure out why we are failing to provide a good product" is a question with only one answer--process. If you aren't allowed to fix the process I don't see how you are on the path to any sort of success.
    – Bill K
    Oct 21, 2019 at 23:55
  • 1
    Based on what data , manager decided , "we failed to provide a good product", that's a good starting point.
    – Vishal Aggarwal
    Oct 22, 2019 at 9:39
  • Are you involving your customer in this process? What have you asked them about so far?
    – corsiKa ♦
    Oct 22, 2019 at 23:57

10 Answers 10

Reset to default
16

I would start by collecting cleaning up the data - don't pick 50 random bugs, but start by classifying them, manually or semi-automatically, maybe using keywords or information from the bug's logs - bug's age, environment, software module, etc. A bug can end up belonging to more than one category.

Move on and look for correlations in the data. Again this might be a semi-automated process using ad hoc scripts or Excel sheets.

And now comes the fun part. Pick a few representative bugs from each category or correlation, and dig down to find the underlying problems.

"Problems" and not "a problem", because it's best to ask why more than once. Five times is usually a good choice.

Improve this answer
2
  • 7
    It could probably be also helpful to go through the list of already fixed bugs and classify the reasons why those bugs appeared -- eg developer misinterpreted specs or code impacted other code which was not taken into account, etc. Of course there could be multiple reasons for a single bug, however, they should be classifiable by origin, source, aspect etc. You can rely only do it after the bug is fixed after all
    – Gnudiff
    Oct 21, 2019 at 21:54
  • Be sure to do this with the product owner / business or you will move away from them in terms of value to them.
    – Michael Durrant
    Nov 1, 2019 at 8:24
9

Ask more questions.

Strictly speaking, to have higher quality in product, have

  • Higher standards
  • More testing
  • Better feedback

However, I would determine the maturity of your development processes.

Ask questions such as:

  • How closely are developers and testers together to share understanding?
  • How do you do three amigos when a change arises?
  • What shape is your testing pyramid?
  • What code coverage do you currently have?
  • What is your current integration feedback process?
  • What are your test data preparation procedures?
  • Is the backlog growing or shrinking?

These all affect product quality. They are certainly about process and project, but those things affect product quality.

Note also that your title currently says Quality Assurance Process, but details says product only.

You can formalize your (good) idea of analyzing some tickets by doing

  • Root Cause Analysis
  • 5 whys
  • Postmortems

to dig into the real reasons for poor product quality including the all important questions of 'how to we define and measure quality'. Be careful of stats like 'number of bugs' which can be gamed. Consider looking to customer experience, revenue and also efficiency of internal operations such as how quickly to fix a bug once the reason and solution are known, for example Mean Time To Recover.

Finally, look for more business involvement in writing good tests and make sure to measure tests in terms of the business value they add. Never use number of tests as a sign of success. That's like paying for 'lines of code' because over time, less is more (less tests, more business value from speed).

Improve this answer
3
3

Generically speaking, creating a software product (pretty much like anything else) has the following major steps. There might be small differences from company to company, but the main idea remains.

  • behavioral requirements - they describe mainly what the user sees and how the user interacts with the product;
  • architecture and design - specifies how the product must be built;
  • actual coding;
  • unit tests (white box);
  • integration tests (mixed white box and black box);
  • validation tests (black box).

What you need to do every time when you receive a complaint is to assess where problem was created, in the above steps. You must find the earliest step.

This statistic will tell you (the company) where you need to improve.

collect a random sample of 50 tickets (bugs) and study them carefully

This is a good start, but it can kick back. It might not give you the most correct answers, since the sample is too small. There is no hurt in extending the procedure to all tickets, for the entire future. If you do not get a very valuable outcome from this, the method will be dumped - and with it, your only chance to improve.

... and study them carefully?

Actually, why study only some problems carefully? All problems should be analyzed carefully, so a proper solution can be implemented for each of them. Without good analysis, the wrong solution can happen, which will do more harm than good.

I am focused here only on Product

Maybe you want to say "I am focused here only on product development". A product just is. Good or bad. Finished or not finished. With bugs or without bugs. You need to change the way you do the development in order to have a better product.

Improve this answer
4
  • This is waterfall. That is not a negative thing but you should be aware of that. Agile (the main development style in the industry today), also has the Behavior step first (BDD) with tests that fail but then has unit tests (that fail) then are made to pass, then the BDD tests themselves can pass. The architecture and design step is largely replaced by emergent design from within BDD/TDD and relying on the latest technologies that continue to abstract away the underlying layers.
    – Michael Durrant
    Nov 1, 2019 at 8:22
  • Not only waterfall. V-cycle also. Actually, all processes follow the same major steps. What is different is the sequence of the steps / activities, the timing, and the adopted names. Also, Agile is the main style only in some industry(ies), but not all.
    – virolino
    Nov 1, 2019 at 8:33
  • Yes some industries and not all. My impression from recent conferences about non-agile topics, is that Agile seems atr the 80/90% level. Once I heard it is used in the army - the very definition of a command and control organization - I realized how pervasive it has become. These are my impressions.
    – Michael Durrant
    Nov 1, 2019 at 8:51
  • My understanding is that Agile can be used successfully where the product / service cannot kill people directly. If safety is an issue (cars, airplanes), then (at least) some activities will have to follow different processes, Agile will not help as much. Also, please take these with some additional "salt" (not provided :), as I am not up-to-date with the statistics of which models are used where.
    – virolino
    Nov 1, 2019 at 9:00
1

Lots of good answers. I've been in a similar situation and it took process changes in order to overcome the product issues.

Product quality issues are a reflection of the process used to develop the software.

For me, my QA philosophy is what I call "Quality First Development." This entails:

  • Deliver Excellence: Users should not be finding bugs.
  • Collaboration: Quality is a cross-team responsibility.
  • Find Clarity: Reduce risk and uncertainty by testing early and testing often, which will improve confidence.
  • Consider Perspective: Focusing on user experience, usability, and empathy improves quality. Be an advocate for the customer.
  • Efficiency: Automate for functional efficiency and long term success.
  • Be curious: “Why” and “What if” are the best questions to ask.

General questions to ask and consider:

When does testing start?

If it's only after the development is completed, you are waiting too long. Bugs, defects, glitches found earlier in the process are easier and cheaper to fix. I have found that once the stories and requirements are written, review them with the PM, Lead Dev, QA to ensure they are written clearly and without ambiguity. One reason bugs happen is a misinterpretation of the requirements between dev and QA, so if you can solve ambiguity earlier, less chance of bugs occurring.

This same goes for once the design mockups are finished. Review the design against the requirements to ensure they match. I can usually start test planning, test case design after the requirements and designs are finished and before dev starts. If you are doing testing automation, you can start creating automated tests before dev starts.

Are you grooming the bugs/defects weekly or on a routine basis?

This can help set bug priority, severity, as well as, including bugs for your development sprint. Bug grooming includes all bugs found by QA, Dev, and users/customers. With every bug found, you need to find the root cause analysis. This is where white box testing and knowledge of the code base can help.

I also like the advice given by the Ministry of Testing on fixing bugs.

Are the bugs following a bug template?

Every bug ticket written should be done the same way and include as much information as possible. This also helps speed up bug grooming and helps speed up root cause analysis.

When bugs do happen, find the root cause. Dig into the code, use 5-Why's. Ultimately, you want to understand where in the process the bug occurred, why it occurred, and how it could have been prevented.

What does your test automation pyramid look like? Are unit tests being done? Are integration testing being done? Are there UI, end-to-end testing being done? If there's no automation, it's time to start.

Are the devs using linting tools and following the same rule set in the linters?

Linters are an often overlooked static analysis tool, but they help all the developers play by the same rules, using the same coding standards. They can also help bubble up bugs early.

What about team attitude toward bugs? Is QA "blamed" for not finding bugs that ended up in production? If so, this "blame and shame" needs to stop as it lowers morale of the team. Bugs are going to happen. We are all humans and that's part of the creation process. How you deal with them matters.

In my experience, when I've helped setup these changes on a dev team, the amount of bugs going to production and being reported by users/customers decreased significantly, which also meant customer/user satisfaction went up.

Improve this answer
1

So now product is live right? So take the defect leakage ratio. Number of defects in post production/client or stake holders found vs defects found by qa team. There you can analyse how much you missed and find the root cause of the same. Have a pareto chart for the same. Find 80:20 principle. Then work on those 80% issues and get corrective measures.

Improve this answer
0

A random sample is giving you bad data, because on the one hand there are defects which generate a lot of low-impact low-effort tickets and on the other there are defects which generate just one ticket with a huge impact and require a huge effort to solve. If you just take a random sample, then your selection will be skewed towards the first, but what you actually want is the latter.

So the first step should be to classify tickets and create a list of defects which are the causes for these tickets. Then classify these defects from least impactful to most impactful.

When you have the most impactful defects, then you can start to look at your QA processes and try to find out which steps should have found these defects, why they failed to find them and how the processes could be changed in order to find such defects in the future.

Improve this answer
0

Depending on how frequently your process changes, I may not want to look at historical data or would want to limit how far back I go for historical data. Instead, I'd set up a method for looking at new issues and performing causal analysis on them. Prioritize based on when the issues were found - issues found in production or post-deployment are the top of the list for analysis, and then work upstream from there.

There are different techniques that you can apply to understand why an issue was injected, why it was detect where it was instead of earlier in the process, or to highlight all of the contributing factors that led to it.

Improve this answer
0

In my opinion you should keep the focus on the main problem in hand, which is that the client is not happy. In other words, there is a deviation on how the client expects your product to work, and how it is developed to work (the product team's point of view). Given that your product is huge, there is good chance that the client requirements got gradually misinterpreted down the line of product development process. I would suggest you to identify these deviations from the bug list you have. Another point of destruction can be that the functionalities(or requirements) that have conflicts between them. That leads creation of a buggy application model itself. Something like Alloy may be helpful in that case. The following links from software testing solutions might also be helpful,

https://blog.qasource.com/resources/comparing-a-test-plan-vs-test-strategy-for-software-qa https://blog.qasource.com/resources/enhance-your-software-quality-assurance-and-testing-in-5-steps https://blog.qasource.com/resources/compare-software-testing-methodologies-for-your-needs https://blog.qasource.com/resources/plan-your-software-testing-life-cycle-for-total-coverage

I am employed by QASource.

Improve this answer
1
  • 2
    Please disclose your affiliation with QA Source. With so many links from your employer's site, people are likely to flag your answer as spam. You can read sqa.stackexchange.com/help/promotion for more information on how to avoid being mistaken for a spammer.
    – Kate Paulk ♦
    Oct 22, 2019 at 11:48
0

Seems to me you already found your company's weakness, it's right there in the way your phrase your question: broken (if even existing) processes.

When all's said and done, a product and the product's quality is a sum of processes that led to its creation.

In software, those processes may include:

  • Correctly gathering and understanding the client's wants and needs
  • Correctly translating client's needs to actionable requirements (ones that specify the action(s) to be taken, that lead to expected results)
  • Correctly coding the requirement into a software product (and also, taking ownership of your code)
  • Continuously monitor the product's state and adherence to the requirements
  • Testing with the client in mind; releasing to the clients' in a methodical fashion with ample documentation and support.
  • Rinse and repeat.

Your boss may not like this answer, but without upping your entire company's processes, you will never be able to produce more quality products.
It a very simple case of cause and effect.

And yeah, if, in spite of all the testing you have in place, there are still many issues that escaped all those tests and are discovered by the clients... I'm guessing you have no processes at all (or, you do, and they're all broken and done incorrectly).

And one last, unsolicited piece of advice: if your manager doesn't want to up his game and improve on the entire company's processes? I'd start looking elsewhere.

With so many issues, most of them found in production and a reluctant manager, it's a matter of time before a scapegoat will be looked for, and guess who that will be?

Improve this answer
0

Impacted Areas of Product This is what i would suggest apply 80 20 Rule, pick up the prod issues and try to understand the impacted areas , these will be your most impacted and critical areas. Any kind of existing metrics will help you to do this.

Test Data Analyse if possible the data used in prod and compare it with the ones you are using , there might be lot of gaps between lower environment testing and actual prod scenarios

Automation If not already available try to get the critical areas regression cases automated and run them on schedule or on deployment basis this will help you to fail fast and functional resources can perform more exploratory test.

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.