4

For non-functional requirements as performance, one can find simple metrics to measure the software quality. If you want to know, for example, if your system's performance got better or worse after some code changes, you can compare the time taken to perform certain actions (ideally all actions) before and after these changes.

For functional requirements, however, whether the system got better or worse since the previous version is a more subjective issue. A simple comparison of the number of failed test cases across versions may bring misleading results, as this does not account for the importance of the test case nor the impact of the problem on the user experience. One could even try to give appropriate weights to the test cases according to their importance, but I think the rules to assign the weights would themselves be subjective.

What kind of metrics can be used as an indicative of how well a piece of software meets its functional requirements?

1
  • Judging from the description, is this the question you really want to ask: "What kind of metrics can be used to indicate how well a post-1.0 piece of software meets its functional requirements?"
    – user246
    May 12, 2011 at 15:34

6 Answers 6

3

When I define my test cases, (Normally in Team Foundation Server) I cross reference them to the requirements. Then, as they pass or fail I use that as a way to determine if a requirement has been delivered or not.

Once all the test cases have passed to, it is the testers, not the developers who set the requirement from 'Ready for test' to 'Built'.

That is how I get the two key measures there that you refer to in your question.

The trend of the requirements being set to "built" which we follow by UAT and setting them to "Verified by customer" is the indication of how well it meets the requirements.

The test results trend over time, gives the long term, version-to-version view of how the software maintains that quality over time.

1

I don't think there's a single, magic, metric to give you that. You can measure some common metrics and combine them somehow to determine the quality of the software. For example- number of covered requirements like mentioned above (very misleading by itself since we don't know how good the coverage is), number of low-med.-high severity bugs found in this version and whether they are all closed (not sure how to interpret the results though) etc.

1

This is a great question, and it's an issue that every product company faces. The answer is very subjective -- it depends on your product, your organization, and your customer base. Although it is subjective, there are trends you should keep in mind when trying to evaluate whether a product meets its functional requirements.

As time passes, not only does your product evolve, but your customer base evolves too. Your first customers are more likely to be early adopters. As your business grows, you will probably take on larger, more risk-averse customers. Your functional requirements will need to evolve with your customer base. Bugs and ease-of-use features that were low-priority to your early adopter customers may be high-priority to your newer customers.

If you host your own product, you may also have operational needs that evolve as well, e.g. scalability requirements and diagnostic/trouble-shooting capabilities.

So how does this mean for QA metrics? You need to revisit your bug list on a regular basis to ensure that priorities evolve with your customer needs. You also need to judge any product metrics you collect, e.g. capacity measurements, responsiveness, or speed, in the context of your customer needs.

Of course, you never have enough information to know with full certainty what the functional requirements need to be. Someone will take their best guess, and when you're wrong, your customers will tell you.

Bottom line: everything changes, including functional requirements. Whatever you decided constitutes high-quality in the last release needs to be re-evaluated for this release.

0

As for usability you can measure time spent on given scenarios or run a survey among users to asses ease of use, readability etc.

0

There are any number of metrics one can use to measure the quality of a product and how well it meets requirements. Let's work backwards, though. What are you really interested in knowing? Customer happiness? Willingness of the customer to buy the product again? To recommend to a friend? To generate a positive profit for the company? Once we know that we can work backwards to find a metric that reflects the things that really matter to us.

Maybe the functional requirements are completely met, but the requirements suck. Is the product successful? The customer is the one who can answer that.

Customer satisfaction surveys are an ages-old metric used by marketing teams. A metric I have used is Twitter-mining. I set up alerts for hashtags of #CompanyName, #ThisAppSucks and #ThsAppRocks and watched the trends following a release.

I've worked with products that have Thumb Voting built into the app, reporting anonymously the customer satisfaction. This can be driven to a company dashboard showing live customer sentiment.

0

What kind of metrics can be used as an indicative of how well a piece of software meets its functional requirements?

There are a myriad of measures... and the International recognized standard for these is the ISO/IEC/IEEE 250xx series (also known as the SQuaRE series)

ISO/IEC 25010 defines the characteristic of functional suitability as:

... the degree to which a product or system provides functions that meet stated and implied needs when used under specified conditions.

This characteristic is composed of the following sub-characteristics:

  • Functional completeness - Degree to which the set of functions covers all the specified tasks and user objectives.
  • Functional correctness - Degree to which a product or system provides the correct results with the needed degree of precision.
  • Functional appropriateness - Degree to which the functions facilitate the accomplishment of specified tasks and objectives.

So how do you measure these?

  • Functional completeness is measured by requirements coverage, driven by requirements traceability... are all requirements (functional and non-functional) mapped to source code (and arguably, is all source code mapped to a requirement!)
  • Functional correctness is then measured by your requirements test coverage - are all requirements mapped to test cases, and do those test cases provide the correct results with sufficient precision
  • Functional appropriateness is a measure of how well the system does what it should do

Specific measures are identified in companion standard ISO/IEC 25023

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.