How do you determine that your project's quality has increased over time?

Question

Preface: I know that the question may be a bit broad, so I hope it doesn't get closed. However, I feel this is a topic that a lot of QA engineers and managers struggle with and I don't think there's only one good way to do something about (like a lot of other topics in our field).

As the question says it, how do you determine that your project's quality has increased over time? What are you using and how do you measure project quality?

Personally, I don't think the number of test cases and the number of bugs are a good metric in this case, especially in an Agile (Scrum and Kanban in my case) working organisation. We develop X features across all teams during a 2-weeks-sprint and they get deployed as soon as the testing is completed (not at the end of the sprint). While the bugs coming from support may be relevant as a starting point, the ideal scenarios would be to not get there.

So, how do you assess your project's quality now? How do you know to say about your project that it's in "good quality" shape or in a "bad quality" one?

What do you do, proactively or retroactive, to get, in time, to the desired level of quality and perhaps avoid business loss?

Answer 1

A good starting point is to define what quality means in your context. And then find out ways how to measure it. It seems you want to measure something without saying what that something is.

Having said that, we can hardly tell you that here, but perhaps we can give you some starting points that you might use in discussions with your teams and other people who have something to do with the product(s) you're creating. That starting point could be this pyramid:

(mentioned in this book)

The product should fulfill all these levels. What it means in practice is something you should define with your team, your customers and clients, in your context. But at least you have some guidelines to start with.

I don't think the number of test cases and number of bugs are a good metric in this case.

Just a number alone doesn't tell you much. Although their might be some industries where even the number alone might be important, but even then, there should be more context given along with these numbers to get a more complete picture about what we're dealing with.

What do you do, proactively or retroactive, to get, in time, to the desired level of quality and perhaps avoid business loss?

Focusing on high priority features, on biggest risks, extensive checking and exploratory testing. Talking to clients/customers if possible/other members of the team often, letting them use/test the new features before production, being interested in what they have to say and following up on that.

Answer 2

For me it is about measurable faster delivery of valuable features, while reducing downtime of functionality. I think good products and teams go faster and faster overtime. This might be different than most people have experienced, typically software projects go slower and slower overtime. Which I would say is due to "bad quality" in Functional, Structural and Process quality aspects.

From a recent study the state of devops 2019 report states that measuring and improving the follow key metrics improves the net results (e.g. money, profit) of these companies.

• Lead time
• Deployment Frequency
• Change Fail
• Time to Restore
• Availability

The first four metrics that capture the effectiveness of the development and delivery process can be summarized in terms of throughput and stability. We measure the throughput of the software delivery process using lead time of code changes from check-in to release along with deployment frequency. Stability is measured using time to restore— the time it takes from detecting a userimpacting incident to having it remediated— and change fail rate, a measure of the quality of the release process.

See page 16 in https://services.google.com/fh/files/misc/state-of-devops-2019.pdf

Measure where you stand and now try to become an Elite performer :)

Answer 3

These are the metrics our company uses in priority order:

1. Number of support cases per number of active users
2. Number of errors generated and logged by our application

This covers a number of underlying aspects of software development:

1. How user friendly is the product? If it's not then you get more support requests and or less users.
2. The more errors logged, the more bugs have been missed during development/deployment.
3. How good is our in product documentation? If it's not good we get more support requests. This is hard to test, but is part of the quality. We have a feedback loop between our support team, documentation writers and testers.
4. Test coverage is a great metric but you can have great test coverage but still lots of support cases. In the eyes of the customer, your product isn't quality if they need to contact support hence our main metric.

Answer 4

Putting the usual metrics aside such as the trend for open and closed defects and the number of issues reported for each incoming build or test cycles we run, I would say time to delivery and post-deployment issues reported would be a good point to start. If the time to delivery is reduced and so are the defects from post-release, it will eventually mean that there's less remedial action and the overall performance and health of the project are towards a good trajectory. However, this also indirectly validates the metrics that we are using right now in one way or the other.

Thank you for your answer, Niels. This seems very interesting. We are currently using a test management tool named Kualitee and using the usual metrics such as the issues trend analysis and test cycle comparisons to review project health.

Answer 5

Determining the quality of project can get tricky at times when you have not defined any benchmarks to measure the quality. However, if you need to upgrade the project quality and keep a check over all the scaling needs, then your team may aim at developing rapid feedback reports on all the test activities pursued.

Besides, the idea of quality may vary depending on the project or functionality under test as the test process applied to that particular functionality or features needs to be upgraded from time to time. The approach may or may not include regression testing which is a mandate for all the features you integrate into an already existing feature or platform.