4

My team (40+ QA Engineers) executes functional, integration and automation testing services for a large customer. We are not developing products, only test them. We test various applications built by variuos development teams (which are teams on customer side or other outsourcing vendors) and testers need to ajust their work with the methodologies and processes implemented in each of that teams (so it varies from true scrum approach to some kind of waterfall..). We have projects with Test Cases such complex that passing one integration TC through all front-to-back systems may take 1 day. We have projects with large number of UI-related regression test cases where tester can pass 100+ in one day

Having all this differences is it a documented way to define "Software Testing as a Service" operation model and to develop metrics that can measure team success and improvement in a long term run?

3 Answers 3

4

As a service, testing's product is information about the dev's product. What you might be able to measure, then, is timeliness and quality of information about the product. With so many different project types, you might not be able to compare between projects or testers easily, but maybe you can get some trends over time. Here are some thoughts about specifics, drawn largely from metrics I've seen in the past when working with a "service" mentality:

  • How fast can you get testing results back to the developers after a build? This could include multiple times, such as the time to initial results (pass / fail), first-pass analysis (test issue, code issue, needs analysis, etc.), and final analysis (which could include more thorough bug reduction). You might also want to consider if you are getting the most interesting (e.g., the riskiest) results out sooner.

  • How thoroughly are failures investigated? Are developers just given a stack trace, logs, and test case number? Are testers able to reproduce the bug and reduce it to the simplest possible set of steps that generates a consistent reproduction of the issue? Note that there are trade-offs here, and some customers might prefer to pay you less and investigate their own issues as soon as they are detected - but long-term, bug-reducing and debugging skills are great for testers to build.

  • How many bugs are detected by your testing on the first pass, and how many are found later through other means? How many are found by the testers, and how many by developers or - worse - users? Automatically detected bugs are great signs. Bugs found in exploratory testing are good signs if they should have been detected automatically but were still caught in-house on the first build where the bug existed; and great signs if they are "weird" bugs. Bugs found outside the team or several builds later should be reduced as much as possible (but not punished . . . you don't want people to become afraid of reporting 'stale' bugs).

  • How many of your automated tests are running at any given time, and how many are broken, blocked, or otherwise unusable? When tests are broken for reasons other than being blocked by bugs, how long does it take to fix them? This can help identify brittle tests or infrastructure issues that are technical debt, so you can fix them and quit 'paying interest' on them.

  • How accurate are your time estimates? How frequently does testing run long, and why does this happen? How can you recognize risk factors for extended testing cycles and point them out to the client sooner?

  • How easy is it for the client to get information to and from testers? Can they gather information 'on demand' from a web portal, or do they need to wait for a report to be emailed to them? How do your clients feel about the transparency of the testing process?

  • Is it easy for clients to control the amount of testing and the areas being tested to control costs and time investments while mitigating risks?

  • When you get a bad build, how quickly can you accurately report its non-testability to the client?

  • What do clients say on satisfaction surveys?

3

Offhand, I do not know of any relevant articles or sites, and I doubt I am better at using Google than you are. At the risk of answering a different question, I offer some observations from my own experience.

It is difficult to measure a QA team's success independent of the rest of the project team; their roles are too interdependent.

One could quantify a "software testing as a service" team's success in financial terms (e.g. profit per team member, profit per project, revenue per project), and those measurements are important to someone in your company, but your question does not appear to be about finances.

There are metrics you might collect about your QA team that do not necessarily relate to testing. For example, you might measure how satisfied your QA engineers are with their jobs, or how much turnover your team experiences. The degree to which these metrics are important depends upon your organization.

Another possible measurement is to what degree your customers are referenceable. You could measure this by asking your customer, "Are you willing to write a positive reference for us?" You might also try to measure factors that, in your previous experience, have influenced whether a customer would be referenceable. You already know what some of things are, and your best practices reflect that knowledge, e.g. how you write test plans, how you report bugs, and how you communicate with the customer. As with all things, these factors will change over time, so you must be diligent about reviewing your metrics in light of recent experience.

2

I've always thought of software QA as enabling change. I.e. making changes less risky and thus enabling quicker turnarounds.

However there are clearly pleantly of QA practices that build up brittle tests so rather than enabling change, they prevent changes (for example large quantities of screen record / replay tests).

A lot of testing needs to be aimed at the requirements of the client. For example SQL injection testing may not be required for internal software. As to making things rock solid in areas where if there was a bug it wouldn't matter.

So risk weighted testing. Maybe we can't automate the testing, but I bet there's a few dull bits that could be speed up? semi-automated testing...

The most amount of confidence in the shortest amount of time. Now how to metric that?

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.