What is the best way to measure the productivity of software testing in agile teams?
The short version: If you're trying to measure productivity on an individual basis you're doing software development wrong.
The longer version: Productivity, especially on an individual level, is something that can only apply to a situation where what's being produced is fundamentally similar. No two software projects are the same, and with Agile development, no two sprints are the same. Often they're not similar enough to make valid comparisons.
Consider these situations, all of which I've encountered in my career (in the same company, and sometimes in the same sprint):
- Developer A writes a lot of code quickly and tends to throw it 'over the wall' to testers. As a result, the code usually - but not always - has a lot of defects and rework.
- Developer B writes code more slowly, includes a lot of unit testing, and usually works with testers from the start. The code rarely has defects, but when there are defects they tend to be the kind that are difficult to identify and equally difficult to fix and retest.
- Project C is a small, clearly-defined addition to existing functionality. Both the developer and tester are intimately familiar with the application and this specific functionality.
- Project D is large, complex, and not clearly defined. Nobody on the team has expertise in the project domain.
- Tester E is dedicated to the project being sprinted on.
- Tester F is juggling three different projects and maintaining automated regression.
- Sprint G is dedicated to a single project
- Sprint H includes work from multiple projects
- Project I has no defined schedule
- Project J has a contracted delivery date and customer-defined required feature set.
How can anyone measure individual productivity - or even team productivity - in circumstances this varied? This doesn't even touch the tester's level of skill or knowledge of the domain.
As a tester with over ten years experience, I have times when - despite being the only tester in my organization - I am unable to do any testing because my other responsibilities have a higher priority.
In my opinion, the closest to a valid productivity measurement that anyone can get is team velocity, and this is only useful as a broad predictive measure ("Team X averages 50 points per sprint. This is a 250 point project, so if team X works on it, the project will not be completed in fewer than 5 sprints.")
Software testing should be a core part of the iteration cycle, better to measure the teams productivity as a whole. Its a team effort and coding and testing are not separate partial tasks you can measure.
Productivity in Agile is measured in how much valuable working software is delivered. To quote the manifesto:
Working software is the primary measure of progress.
There are some metrics that can you can use to see if the teams testing efforts are on par:
- Number of delivered defects vs number of delivered features
- Code coverage of all automated tests
- Velocity over time should increase
Productivity is a tricky word and I'm not sure it's something that you could measure easily from a complex activity such as testing.
A colleague wrote a blog here which talks about some valid issues with this.
I've been on projects where our productivity was measured by how many automated tests we delivered. Clearly this is an incredibly poor metric - but I've seen it done even to this day.
Once you accept that delivering automated tests is similar to the act of development in general, you can start to use comparisons to developers. Measuring the productivity of development is also extremely hard, though! Rely too much on pure delivery metrics and all notion of quality goes right out the window, which ends up bogging you down in the long term.
Here are some ways you can start to measure the quality of testing work.. but inevitably some of these are tradeoffs and you are going to take a holistic view of things. IMO delivering a few good quality tests is better than a crapton of bad ones.
- Quality and maintainability of testing code
- Quality of test analysis (e.g. breadth of scenarios covered, sensible requirements parsing)
- Feature coverage
- Defect discovery rate (of course, you might just have good quality code from developers and not find many!)
- Time between code commit and the testing team spotting the defect
- Input to product decisions, retros, etc
- Speed of acceptance test delivery