Take the 2-minute tour ×
Software Quality Assurance & Testing Stack Exchange is a question and answer site for software quality control experts, automation engineers, and software testers. It's 100% free, no registration required.

When I worked for a CMMI level 5 company, they used a predictive approach for almost all and for that same reason metrics were very important, they were the inputs for next projects. But now, I think it is quite different (IMO). So, my question is:

Are metrics in testing still important? If the answer is yes, which ones and why?

Update: CMMI, at least five years ago, suggests a waterfall SDLC (predictive approach) in which a lot of different metrics (for all the development phases) were collected and added to a kind of corporate database for two reasons: control and to predict (estimate) effort, defects, LoC and others for the next projects.

Currently, most of us use an adaptative approach (agile) and those databases are less common. However, there are several metrics that are generated by the testing team and most of them are used just to report to managers, are not used for other projects and really few decisions are made based on them.

I´m asking because currently I don´t see a significant value on metrics (no just for testing). However, I would like to know what the testing community think about metrics. Are they important? why?

share|improve this question
2  
Re: "But now, I think it is quite different". By now you mean another company? Different in what a way? Can you give an example of the difference? Prediction of what? Development task estimates? Test focus? Can you be more specific about that? –  dzieciou Nov 19 '12 at 21:22
1  
Also, I guess you ask such a question for a reason. Do you have any suspect they are no longer important? You see no difference between using and not using them? –  dzieciou Nov 19 '12 at 21:25
    
CMMI is actually a model for continuous process improvement which is tangential to a development lifecycle model. –  Bj Rollison Nov 20 '12 at 18:54

6 Answers 6

up vote 2 down vote accepted

Like most of the others who have responded, I have used a few metrics in the past, when there was something I wanted to learn (via counts or numbers or trends). Most often, I collected some metrics, attempted to learn what I was seeking, then stopped collecting.

I have not found a universally-useful metric - one that I always feel merits the time and money needed to collect it, and doesn't have adverse side-effects.

As with most experiments, collecting metrics is based on several assumptions/hypotheses - that you are able to efficiently and effectively collect a number/count/trend that correctly represents something you want to know. For example (and I'm not saying that this is a good idea), if you want to know when your code might be of sufficient quality to ship, you might choose to collect bug counts and watch the trend of open bugs. (Hint: Don't do that! http://www.allthingsquality.com/2010/04/misuse-and-abuse-of-bug-counts.html )

And as with all experiments, you must challenge those assumptions. How do you know that your metric really represents the bigger question you are trying to answer?

And as with all experiments, you must carefully watch for side-effects and unintended consequences. Read "Measuring and Managing Performance in Organizations" by Robert D. Austin for an in-depth treatment on that topic.

My sense is that a Metrics Program (with a capital M) as implemented company-wide in many companies, is a misguided attempt to publish numeric, easy-to-digest summaries of complex issues. At best they are useless, at worst they are completely dysfunctional (as in some of the examples raised by Austin).

On the other hand, contained, short-duration metrics (with a lower-case m), can be useful if you are careful.

I like to tread lightly here. I prefer talking to people over being handed a metric. I prefer working with the team as a means to gain understanding, over a metrics-laden dashboard.

share|improve this answer

Metrics are a tool, not a goal on themselves.

You might use metrics for different purposes, and for those purposes metrics might be important.

I use metrics from time to time when I want to measure something or prove a point, and stop using them when the need is no longer there.

For example counting the number of bugs opened by clients vs. number of bugs found internally

is meaningless as a one time effort but when you track it you can see trends.

But on the other hand I calculated several one time metrics for a test review meeting with my upper management.

share|improve this answer
    
"I use metrics from time to time when I want to measure something or prove a point, and stop using them when the need is no longer there." I think this is wise! –  Joe Strazzere Nov 20 '12 at 13:17

Metrics totally depend upon the industry which you operate.If you are testing for a banking or a lifescience industry then metrics are very important since these industries are very strict and rigorous about standards.As for the other industries I don't think they matter much,but if you take testing as a whole if u can find your way to reproduce a bug that you found and make the development team or any other who is involved to reproduce it then I don't think metrics matter much.

Testing greatly varies from a service based and product based company.If it is a service based company which offers testing as a service then these metrics play a bigger role.But if it is a product based company that does testing of its own products then they probably wont worry much about metrics and stuff.

My personal answer to this question is that if metrics take too much of your time and you spend more time on the preparation/documentation of these metrics than actually testing of software then you should probably look at alternate solutions.You can possibly try the maximum to automate metrics preparation and collection.IMHO I don't think these metrics play a direct impact on test quality,it greatly depends upon the testers who do the actual testing.

share|improve this answer
    
So you don't use code coverage, test coverage metrics at all? –  dzieciou Nov 20 '12 at 6:28
    
I never meant it that way.I am saying that it should be used in a way that it actually matters to the project.For short releases,the regression suite would matter much more than the metrics themselves.I am not saying that metrics are not needed at all,but there are other more important things. –  Madusudanan Nov 20 '12 at 6:31
    
I agree that having beautiful metrics without code developed and tested gives you nothing. But even (or maybe especially when) you have limited time, deciding were to focus you effort on regression suite can be done better if you have simple metric to guide you like code coverage. Running tests and getting report what lacks still lack tests does not cost that much, especially when you have CI. –  dzieciou Nov 20 '12 at 6:44
    
That's correct.But I am not speaking about metrics on test planning,test schedule,test scope here,they are absolutely required.As far I have done testing in my career real bugs from ad-hoc testing and negative testing.These are the areas where developers miss out.But again metrics are viewed differently from a normal tester than a QA manager,if a metric does not matter much,its better to drop it out than just do it for the sake of doing it. –  Madusudanan Nov 20 '12 at 6:55

IF used correctly I find metrics very important and a great tool to emphasize that the product is on the right (or wrong) track.

BUT the wrong metrics are propoby more harmful than good metrics are helpful so be very careful when choosing metrics and what you are presenting to stakeholders / testers / managers.

I also find that the trend chart of a certain metric is often of more intrest than the key number it self.

Example: I tend to like we call 'defect work index' and here I as test manager rand each bug 1-4 depending how important it is to fix (I always use an even number else most seem always to be in the middle). The most important get 4 points, then 2, 1 and ½. Here the total number isn't of any intrest since it doesn't really say anything. But the trend chart is... If it's rising unusally fast; why? or decreasing; why? Might be focusing too much on bug-fixes and too little on new development. It all depends where you are in the project of how important this is. And I can't stress this enough: Don't set any goal of the index since it's the trend which is the intresting one.

There are of course unlimited different metrics, all of intrest in one way or another. Whats good in one project might be totally wrong in the next.

share|improve this answer

IMHO, the single biggest problem with software metrics is that people seem to love to measure simply for the sake of collecting data. But, some problems with mindless metrics include:

  • Wasteful data. We sometimes waste time collecting data that is not used to measure things we want to improve or provide insight into answers to actionable questions
  • Data bones. abusing the data as tea leaves/oracle bones (see Meaningful Measures. In other words we collect data and then pick over the bones of data asking ourselves what we can conclude from these measures.
  • Random metrics. Assuming we are measuring to improve some aspect of our process, a good measurement should look for trends over time. One-time snap-shot measures rarely produce a picture of reality. One time measures are good for random reactions, while trending data helps guiding actions towards constant improvement.

Obviously, I personally am not a big fan of collecting data for the sake of collecting data. I am a big fan of Dr. Victor Basili's Goal, Question, Metric paradigm.

For example, one thing I am looking at right now with my team is reducing maintenance costs of automated tests throwing false positives (Goal - reduce false positives of intermittently failing tests). First, we analyzed our existing data to understand the different reasons why tests intermittently fail and throw false positives (e.g. service outage, synchronization, etc.). Next, we decided which things we could take action on and implemented some actions to improve our test designs we think will lower the incidents of false positives occurring in our automation runs and improve reliability of our tests. (Question - what can we improve/how can we improve?) And finally, we implemented measures to track to see if those actions are actually having a positive impact. (Measure - are our actions having a positive impact, and are we measuring the right things.)

share|improve this answer

Yes provided you gather the right metrics. I would like to share my experience how metrics were manipulated for different reasons

Metrics to Project QA Improvements - Lot of metrics were gathered to project QA improvements in terms of QA Processes and test cases. Examples - Test Cases Per Feature, Bugs per Test Case - Test Cases Executed to Bugs Logged

My Opinion - These metrics were used to justify how adding more QA members improved test coverage, more test execution and more bugs. This is not useful metric that is needed for REAL improvements

Metrics for Management - Similar to Earlier QA Improvements these metrics were drawn to justify critical QA defects when compared to DEV - QA - UAT - Prod bugs. How QA Has Performed compared to other environments.

My Opinion - These metrics provides good details on how bugs passed through different phases. Overall quality of product can be arrived. This alone is not sufficient

Meaningful Metrics - We need few critical items to begin with (Bugs logged against functionality, code defects, requirements related). These need to be retrospected and should continuously feed in to next cycles.

Metrics are only a guideline, Results depend on how we leverage it, Entering Actual Data (Not manipulating to show numbers), Accepting the facts and continuously working on improvements mind-set matters

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.