Are metrics something important?

Question

When I worked for a CMMI level 5 company, they used a predictive approach for almost all and for that same reason metrics were very important, they were the inputs for next projects. But now, I think it is quite different (IMO). So, my question is:

Are metrics in testing still important? If the answer is yes, which ones and why?

Update: CMMI, at least five years ago, suggests a waterfall SDLC (predictive approach) in which a lot of different metrics (for all the development phases) were collected and added to a kind of corporate database for two reasons: control and to predict (estimate) effort, defects, LoC and others for the next projects.

Currently, most of us use an adaptative approach (agile) and those databases are less common. However, there are several metrics that are generated by the testing team and most of them are used just to report to managers, are not used for other projects and really few decisions are made based on them.

I´m asking because currently I don´t see a significant value on metrics (no just for testing). However, I would like to know what the testing community think about metrics. Are they important? why?

Answer 1

Like most of the others who have responded, I have used a few metrics in the past, when there was something I wanted to learn (via counts or numbers or trends). Most often, I collected some metrics, attempted to learn what I was seeking, then stopped collecting.

I have not found a universally-useful metric - one that I always feel merits the time and money needed to collect it, and doesn't have adverse side-effects.

As with most experiments, collecting metrics is based on several assumptions/hypotheses - that you are able to efficiently and effectively collect a number/count/trend that correctly represents something you want to know. For example (and I'm not saying that this is a good idea), if you want to know when your code might be of sufficient quality to ship, you might choose to collect bug counts and watch the trend of open bugs. (Hint: Don't do that! http://www.allthingsquality.com/2010/04/misuse-and-abuse-of-bug-counts.html )

And as with all experiments, you must challenge those assumptions. How do you know that your metric really represents the bigger question you are trying to answer?

And as with all experiments, you must carefully watch for side-effects and unintended consequences. Read "Measuring and Managing Performance in Organizations" by Robert D. Austin for an in-depth treatment on that topic.

My sense is that a Metrics Program (with a capital M) as implemented company-wide in many companies, is a misguided attempt to publish numeric, easy-to-digest summaries of complex issues. At best they are useless, at worst they are completely dysfunctional (as in some of the examples raised by Austin).

On the other hand, contained, short-duration metrics (with a lower-case m), can be useful if you are careful.

I like to tread lightly here. I prefer talking to people over being handed a metric. I prefer working with the team as a means to gain understanding, over a metrics-laden dashboard.

Answer 2

Metrics are a tool, not a goal on themselves.

You might use metrics for different purposes, and for those purposes metrics might be important.

I use metrics from time to time when I want to measure something or prove a point, and stop using them when the need is no longer there.

For example counting the number of bugs opened by clients vs. number of bugs found internally

is meaningless as a one time effort but when you track it you can see trends.

But on the other hand I calculated several one time metrics for a test review meeting with my upper management.

Answer 3

IF used correctly I find metrics very important and a great tool to emphasize that the product is on the right (or wrong) track.

BUT the wrong metrics are propoby more harmful than good metrics are helpful so be very careful when choosing metrics and what you are presenting to stakeholders / testers / managers.

I also find that the trend chart of a certain metric is often of more intrest than the key number it self.

Example: I tend to like we call 'defect work index' and here I as test manager rand each bug 1-4 depending how important it is to fix (I always use an even number else most seem always to be in the middle). The most important get 4 points, then 2, 1 and ½. Here the total number isn't of any intrest since it doesn't really say anything. But the trend chart is... If it's rising unusally fast; why? or decreasing; why? Might be focusing too much on bug-fixes and too little on new development. It all depends where you are in the project of how important this is. And I can't stress this enough: Don't set any goal of the index since it's the trend which is the intresting one.

There are of course unlimited different metrics, all of intrest in one way or another. Whats good in one project might be totally wrong in the next.

Answer 4

IMHO, the single biggest problem with software metrics is that people seem to love to measure simply for the sake of collecting data. But, some problems with mindless metrics include:

• Wasteful data. We sometimes waste time collecting data that is not used to measure things we want to improve or provide insight into answers to actionable questions
• Data bones. abusing the data as tea leaves/oracle bones (see Meaningful Measures. In other words we collect data and then pick over the bones of data asking ourselves what we can conclude from these measures.
• Random metrics. Assuming we are measuring to improve some aspect of our process, a good measurement should look for trends over time. One-time snap-shot measures rarely produce a picture of reality. One time measures are good for random reactions, while trending data helps guiding actions towards constant improvement.

Obviously, I personally am not a big fan of collecting data for the sake of collecting data. I am a big fan of Dr. Victor Basili's Goal, Question, Metric paradigm.

For example, one thing I am looking at right now with my team is reducing maintenance costs of automated tests throwing false positives (Goal - reduce false positives of intermittently failing tests). First, we analyzed our existing data to understand the different reasons why tests intermittently fail and throw false positives (e.g. service outage, synchronization, etc.). Next, we decided which things we could take action on and implemented some actions to improve our test designs we think will lower the incidents of false positives occurring in our automation runs and improve reliability of our tests. (Question - what can we improve/how can we improve?) And finally, we implemented measures to track to see if those actions are actually having a positive impact. (Measure - are our actions having a positive impact, and are we measuring the right things.)

Answer 5

Metrics totally depend upon the industry which you operate.If you are testing for a banking or a lifescience industry then metrics are very important since these industries are very strict and rigorous about standards.As for the other industries I don't think they matter much,but if you take testing as a whole if u can find your way to reproduce a bug that you found and make the development team or any other who is involved to reproduce it then I don't think metrics matter much.

Testing greatly varies from a service based and product based company.If it is a service based company which offers testing as a service then these metrics play a bigger role.But if it is a product based company that does testing of its own products then they probably wont worry much about metrics and stuff.

My personal answer to this question is that if metrics take too much of your time and you spend more time on the preparation/documentation of these metrics than actually testing of software then you should probably look at alternate solutions.You can possibly try the maximum to automate metrics preparation and collection.IMHO I don't think these metrics play a direct impact on test quality,it greatly depends upon the testers who do the actual testing.

Answer 6

Yes provided you gather the right metrics. I would like to share my experience how metrics were manipulated for different reasons

Metrics to Project QA Improvements - Lot of metrics were gathered to project QA improvements in terms of QA Processes and test cases. Examples - Test Cases Per Feature, Bugs per Test Case - Test Cases Executed to Bugs Logged

My Opinion - These metrics were used to justify how adding more QA members improved test coverage, more test execution and more bugs. This is not useful metric that is needed for REAL improvements

Metrics for Management - Similar to Earlier QA Improvements these metrics were drawn to justify critical QA defects when compared to DEV - QA - UAT - Prod bugs. How QA Has Performed compared to other environments.

My Opinion - These metrics provides good details on how bugs passed through different phases. Overall quality of product can be arrived. This alone is not sufficient

Meaningful Metrics - We need few critical items to begin with (Bugs logged against functionality, code defects, requirements related). These need to be retrospected and should continuously feed in to next cycles.

Metrics are only a guideline, Results depend on how we leverage it, Entering Actual Data (Not manipulating to show numbers), Accepting the facts and continuously working on improvements mind-set matters

Answer 7

Metrics are important to measure the progress of your project and to keep you and your team motivated. As you try to work towards your goals, wouldn't you want to know how far you have come and on which areas to focus for achieving your goals? For projects in software, they are vital for management to know the health of the project, and this data-driven approach helps the team to focus on those aspects of the project which need more attention.