14

Preface: I know that the question may be a bit broad, so I hope it doesn't get closed. However, I feel this is a topic that a lot of QA engineers and managers struggle with and I don't think there's only one good way to do something about (like a lot of other topics in our field).

As the question says it, how do you determine that your project's quality has increased over time? What are you using and how do you measure project quality?

Personally, I don't think the number of test cases and the number of bugs are a good metric in this case, especially in an Agile (Scrum and Kanban in my case) working organisation. We develop X features across all teams during a 2-weeks-sprint and they get deployed as soon as the testing is completed (not at the end of the sprint). While the bugs coming from support may be relevant as a starting point, the ideal scenarios would be to not get there.

So, how do you assess your project's quality now? How do you know to say about your project that it's in "good quality" shape or in a "bad quality" one?

What do you do, proactively or retroactive, to get, in time, to the desired level of quality and perhaps avoid business loss?

  • there is the question of what 'quality' is? software's like hotdogs: they can taste good but you don't want to know how it's made. Quality from a user's perspective is different from maintainability or from 'good' in the eyes of a purist, etc. You can tell a project is more mature, stable, etc as you need less and less engineers on it, but it's not a general statement either – Thomas Oct 11 at 11:50
  • As any clearly defined problem is half solved; similarly we need to clearly define "Quality" first in our contexts before chasing it. – Vishal Aggarwal Oct 17 at 16:30
  • Do you mean Product? Measuring, improving quality is usually applied to a product or process. First there has to be a measured baseline. – charles ross Oct 23 at 2:01
18

A good starting point is to define what quality means in your context. And then find out ways how to measure it. It seems you want to measure something without saying what that something is.

Having said that, we can hardly tell you that here, but perhaps we can give you some starting points that you might use in discussions with your teams and other people who have something to do with the product(s) you're creating. That starting point could be this pyramid:

enter image description here

(mentioned in this book)

The product should fulfill all these levels. What it means in practice is something you should define with your team, your customers and clients, in your context. But at least you have some guidelines to start with.

I don't think the number of test cases and number of bugs are a good metric in this case.

Just a number alone doesn't tell you much. Although their might be some industries where even the number alone might be important, but even then, there should be more context given along with these numbers to get a more complete picture about what we're dealing with.

What do you do, proactively or retroactive, to get, in time, to the desired level of quality and perhaps avoid business loss?

Focusing on high priority features, on biggest risks, extensive checking and exploratory testing. Talking to clients/customers if possible/other members of the team often, letting them use/test the new features before production, being interested in what they have to say and following up on that.

| improve this answer | |
6

For me it is about measurable faster delivery of valuable features, while reducing downtime of functionality. I think good products and teams go faster and faster overtime. This might be different than most people have experienced, typically software projects go slower and slower overtime. Which I would say is due to "bad quality" in Functional, Structural and Process quality aspects.

From a recent study the state of devops 2019 report states that measuring and improving the follow key metrics improves the net results (e.g. money, profit) of these companies.

  • Lead time
  • Deployment Frequency
  • Change Fail
  • Time to Restore
  • Availability

The first four metrics that capture the effectiveness of the development and delivery process can be summarized in terms of throughput and stability. We measure the throughput of the software delivery process using lead time of code changes from check-in to release along with deployment frequency. Stability is measured using time to restore— the time it takes from detecting a userimpacting incident to having it remediated— and change fail rate, a measure of the quality of the release process.

See page 16 in https://services.google.com/fh/files/misc/state-of-devops-2019.pdf

enter image description here

Measure where you stand and now try to become an Elite performer :)

| improve this answer | |
2

These are the metrics our company uses in priority order:

  1. Number of support cases per number of active users
  2. Number of errors generated and logged by our application

This covers a number of underlying aspects of software development:

  1. How user friendly is the product? If it's not then you get more support requests and or less users.
  2. The more errors logged, the more bugs have been missed during development/deployment.
  3. How good is our in product documentation? If it's not good we get more support requests. This is hard to test, but is part of the quality. We have a feedback loop between our support team, documentation writers and testers.
  4. Test coverage is a great metric but you can have great test coverage but still lots of support cases. In the eyes of the customer, your product isn't quality if they need to contact support hence our main metric.
| improve this answer | |
New contributor
anichols is a new contributor to this site. Take care in asking for clarification, commenting, and answering. Check out our Code of Conduct.
0

Putting the usual metrics aside such as the trend for open and closed defects and the number of issues reported for each incoming build or test cycles we run, I would say time to delivery and post-deployment issues reported would be a good point to start. If the time to delivery is reduced and so are the defects from post-release, it will eventually mean that there's less remedial action and the overall performance and health of the project are towards a good trajectory. However, this also indirectly validates the metrics that we are using right now in one way or the other.

Thank you for your answer, Niels. This seems very interesting. We are currently using a test management tool named Kualitee and using the usual metrics such as the issues trend analysis and test cycle comparisons to review project health.

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.