What are good metrics for a test team to use to start measuring their workload and ability to meet external deadlines?

Question

The tester team where I work has a demanding customer base (we do business-to-business software, some of it ultimately public-facing, some business-facing), an application that's gone beyond anyone's ability to test all the configurations, much less how all those config flags affect all the features, major releases that usually include at least 3 major features and a raft of minor features as well as dozens of bug fixes. No-one in the company has any idea how to tell if a release is too full - which means that no-one can tell if it's possible for the test team to meet the deadlines (usually imposed by our customers' needs).

I'm looking for a good starter set of numbers (as everyone knows, metrics that resolve to nice, need numbers are the way to communicate important things up the command chain!) that we can use to get some kind of handle on how many features in a release is too many so we can start pushing back before we get to two weeks before release day with a bunch of features still in active development and enough bugs to sink the Titanic waiting for a coder to start on them.

Unfortunately, I'm going to need hard data for this, since most of our scheduling is customer driven, and rarely has any give in it - but I don't want to trap myself by providing the wrong kind of data to the management team.

(If anyone has some good suggestions for similar kinds of measurements for coders, I suspect our development team would like that - they're under the same kind of stress we are, and just as overworked).

Okay, that's a question gone epic...

Answer 1

There are a number of techniques that need to be combined for this sort of thing.

Tradeoff approach

The tradeoff approach (aka tradeoff triangle) needs to be agreed and documented up front, with the key stakeholders. This is great way to discuss and decide something that they normally won't budge on.

The trade-off triangle conceptualizes the idea that resources, schedule, and features are three interconnected elements of any project, and that constraining or enhancing one or more of these elements requires trade-offs. An element not defined in the triangle is quality; however, it is assumed to be a constant that is clearly articulated in a quality bar by the project team at project inception.

For example, if the organization wants to incorporate a new set of features into the solution, the team will have to make a trade-off somewhere, whether in the schedule, the resources, or other features.

Given fixed ____________, we will choose a ___________ and adjust ___________ as necessary.

Given fixed schedule, we will choose how many resources and adjust the number of delivered features as necessary.

You can't test everything

Computer software is inherently complex, and even the simple act of entering a six character user name into a text box. Limiting ourselves to ASCII input still requires 127 x 127 x 127 x 127 x 127 x 127 = 4,195,872,914,689 test cases. If each test case takes us only 1 second to manually run, it will take 7.9 million years to test the user name field before we then moved onto the password textbox.

So if “complete” testing is out of the question, where should you start and how much testing you should do test before you can be confident that you can ship?

Determining what to test within a fixed schedule

Transfer your test cases them into a spreadsheet, adding columns for estimated execution time, and numerical ratings say from 1-5, (where higher is better) for the: priority (relative importance) of the test, the visibility of the area to users, the quality of the existing code etc.

The columns are then added together (in a column called “importance rating” to provide an overall score for the test. If you are testing a new release of an existing product you will probably want to add additional columns that identify the relative quality of the existing teat area.

With your spreadsheet in hand, you should circulate it and solicit input around the priority of the tests. You may even find it easier to submit the spreadsheet with a column for each area, say Testing, Development, Product Management and asking each area to fill in their own set of numbers. This will allow you to apply a weighting to each of the areas, say increasing the PM vote to 150% and decreasing the development vote to say 75%.

This is what it can look like, assuming you only had 10 minutes of testing.

With the ratings complete you can then add a column to the spreadsheet to sum the execution times. Then, sort the spreadsheet by the importance rating column, work out how long your schedule gives you to test and then draw a line across the sheet when the estimated execution time equals the time that you have available, and that is your level of test coverage.

Tracking progress

Once all that expectaiton has been managed, and you are in-progress I then use the following metrics to track where we are

1. Active bug trend - If I had to chose only one metric, this would be it. I use this to manange expectations around the ship date and which bugs will be fixed in the current releease.

2. Feature burn down - If testing is happening, and the software looks good but new features aren't being added then testing is giving a false positve. This chart will tell me that.

3. Test exectuion progress - I use this to know how well we are getting work done.

4. Requirement completion - One key rule I use is that testers say when a feature is done, not the developers. This metric tells me how the product is progressing towards tested, shippable, actual completion.

5. Assigned work per person - I use this to know how well I am sharing work allocation between team members

6. Bug find rates - I use this to know how effective we are, I am looking for a flatish line here, well above zero every day.

With all thise metrics, I am communicating daily how we are progressing against the communicated plan and adjusting priorities as required.

Answer 2

The number of features in a release is the most flexible piece, honestly. Quality can NEVER suffer, that's a given. With heavy customer demand, most likely release dates can't shift. So, you are limited (time-boxed) based upon how many man hours are available until the release goes out. So, quality is boxed, time is boxed, the only thing left is to limit features.

The best measurement to determine, then, if a release is too full is to relate features, bugs, and other development activities to time. Researching past stuff works to some extent but each feature, each bug, each situation is slightly different so, while checking the past to see what trends are helps, the best way to make that relationship is to look towards the future, towards what hours are available. So, here's a few things to do to start making those relationships.

1. For new features and/or products, hopefully you're already putting up estimates for "Hours Remaining" for all necessary tasks for deploying the new feature. You cannot forget some of the "soft" things like release note reviews, automated test runs and result reviews, etc., that may not be part of the direct development process but are impacted by the new feature.
2. One thing that I don't think gets done enough is to do the same for any new bugs fixes that are added to a release. Those are where, in my experience, the problem happens most. A bug rarely gets a "hours remaining" estimate for anything other than the development time. There needs to be similar tasking for testing, documentation, regression, etc., for every bug fix because those bugs take up man hours, too. If you have no measurement of man-hours for every bug, you'll very quickly run out of time.
3. Undocumented and unexpected "features" need to be estimated. Again, in my experience, developers, when they are in a section of code, when they see something that could get improved, they usually go in and do the tweaks and improvements to them. While this is good, if you don't have sufficient unit test coverage or regression coverage for all those areas of code (and by the sound of the size of your app, I'm betting such is the case), those little refactor jobs need to be tossed into the hours remaining.
4. General maintenance tasks need to be documented in the hours remaining as well. There's branching the source base, creating back ups of files, updating or upgrading tools, all the "paper work" that goes along with any development or testing task, those ancillary documentation tasks that aren't really attached to any task such as SCRUM meetings, etc. All this sucks up time.

I'm sure this list isn't complete, but the general idea is that everything that goes into creating a release of software needs to be accompanied by an estimate of "hours remaining" and then the associated burn-down charts that go along with it. That chart then should be compared to the actual number of hours available until release and, if the hours remaining exceeds hours to release, then you've hit your "too full" level.

It may happen that you'll get "too full", not because of adding more features, but because "hours remaining" estimates get revised upwards at which point then someone needs to re-prioritize and determine what gets pushed off to the next round of development.

Since release dates can't shift, and the number hours in a day per person until the day of release will never change (until someone invents time travel), comparing those two values will be your best determining factor as to whether or not the release is "too full".

Answer 3

Please check article

Software Quality Metrics:The complete picture - http://blog.infostretch.com/?p=56

Metrics for

• Test Case Execution Time - Per Build/ Per Module
• Defects Metrics (Defects Identified in QA Phase Vs Defects Identified after QA (UAT/Prod)
• Resource based metrics - Effort Spent / Percentage of work completed