Hot answers tagged

31

KPIs can be dangerous - it's very easy to measure the wrong thing or worse, reward the wrong thing. The general rule is that people will do more of the things that get rewarded. Some of the things you should consider: If you use the number of bugs raised by the team, you will see bugs raised for things like misplaced pixels. Modifying this to use the ...


21

The best KPI's for testing are ones that you wouldn't expect. Can the team ship with a boring level of predictability ? Are customers happy ? Is the product selling ? Are there very few critical issues found in production ? Are budgets and schedules being met ? These co-incidenty are 'whole of team' KPI's which testing plays a key part of.


18

Tying cash rewards to metrics like bug counts is always a very risky business, one that is almost certain to introduce dysfunction. I never want people on my QA team to have to worry about their bonus when I just want them to be doing the right thing. Imagine you are the tester working with a single developer (who happens to be your best friend) on a ...


10

When someone asks me to do something and I have questions about what they want, I go back and ask them. That's the most direct way to get an answer. As a manager, I would want these fields: description, i.e. what's the issue and why it's a problem amount of risk if not fixed pervasiveness level of effort to fix, including testing effort associated bug ...


10

No It absolutely is not valuable to measure bug count per hour of dev time. It is especially bad to measure bug count per hour for individual devs. Devs on more complex or difficult code will generally have more bugs/hour than devs working on cleaner, more straightforward code. Similarly, devs will produce more bugs/hour at the end of a long workday than ...


8

I agree with Joe's assertion that metrics can be badly misused and counterproductive. That said, error seeding can be a useful way to answer the right question. After we develop a test plan, we generally assume two things: The test plan is unlikely to uncover every error in the system. (If the system is large and complex, we can assume some undiscovered ...


8

I haven't been part of a project where fault seeding was used. I have, however been part of projects where new metrics were introduced - which is essentially what you are proposing here. If my experience is any predictor, you should expect that people will get better at specifically whatever it is you measure, at the expense of everything else. Here, you ...


8

There is no definitive guide without narrowing the context dramatically. What you are searching for has no universally-accepted set of metrics. If you search for web quality metrics, you'll find millions of hits full of personal opinions, or what matters most to selected individuals. If you are trying to promote your technology, the right thing would be to ...


8

It depends, and there are no industry standards. Seriously. Any metric can be gamed (and will be, if you use it for assessment). I'm not aware of any standard approaches, not least because the teams are - or should be - evaluating themselves regularly and looking for ways to improve their own processes (if they aren't then they're probably using SCRUM-but......


7

In addition to what everyone else has said, metrics used for performance bonuses are a seriously bad idea in the software world. Metrics are helpful as a descriptive tool, but that's about it in my view. Here's some of the reasons why: every software project is different. That means that even if all the same things are measured, comparisons won't be ...


5

Can you prevent code with known bugs from going out? If you can't actually control the thing being measured, it's a bad metric. What if the right thing for the business is to release the code, with bugs, to get it out faster so users can start benefitting? Sometimes having some software is better than none (but not always!). If the metric could ever ...


5

At my current workplace, we don't distinguish between different ways that bugs are discovered. If a developer expects the test team to test the bug fix, they log it. If they don't expect the test team to test it, they don't log it. They understand there actions have consequences, and so they make that decision carefully. We never penalize anyone for ...


5

In all honesty, my experience is that there are no best practices anywhere, even in something like code duplication. There are only principles that work to guide practice, and practices that are good in some circumstances. You don't mention whether you're working with automation code or application code: depending on the automation method and tooling, it's ...


5

In addition to user246's suggestions, I'd also recommend looking at these areas: presence/absence of unit tests effectiveness of unit tests (that is, do the unit tests cover more than the code paths - they should be covering all the potential logical boundaries as well) presence/absence of higher-level automated tests (including system, API, GUI, ...


5

By the success of the company. Buy-in for QA will need to come from the top rather then the justification being looked for in data. You will, over time, be able to point to things like some major bugs prevented from reaching customers performance issues anticipated and planned for unusual bugs discovered for certain conditions more new customers more ...


4

I have personally found that the more you can reduce the overhead of developers fixing bugs before the code hits the main source branch the better off you are. I generally use a rule that as soon as a bug will be seen by or could effect someone else then it must be logged. This allows testers and developers to pair together as part of a pre-checkin review ...


4

You may find it useful to search for "bug taxonomy" or "failure mode catalog". This paper, "Bug taxonomies: Use them to generate better tests" provides a great overview of taxonomies, discusses how you can use them to brainstorm better test ideas, and provides useful practical tips on how to use existing bug taxonomies or how to go about creating a bug ...


4

The deliverable is information about the system. The purpose is to help people make better decisions, based on information about the system.


4

Coverage is always coverage related to some model. This often gets skipped over, which leads to much confusion "you said you had 100% coverage so how come there's a bug?!" When you're looking at unit tests, then it's possible to use code coverage as an indicator (there are tools that can measure what percentage of the lines in your code are exercised when ...


4

Agile teams are cross-functional teams. They estimate the work from design to delivery often on a story based level. This includes the testing work, since testing should be part of the definition of done. Story points are the relative size of the complexity of the task at hand. Read more about estimating with relative sizes in this blog. I as a tester part ...


4

Test Efficiency is a measure of the relevance of the bugs being reported. A low efficiency would imply that the test team are reporting many bugs that aren't worth fixing. This is pretty limited, and simplistic. I've had better success with a pie chart that shows the "resolution" for all of the resolved bugs. This shows the resolution categories like "...


4

Black box approach in and of itself is specifically designed to come from the user perspective backwards. The difference between black box and white box testing is knowledge of the underlying code and components. Therefore when you are prepping for black box testing you should be coming from the user perspective who utilize the application. This is any ...


3

Classifications will never be finite & will be specific to what and how you're testing. Like the list of 'tags' on the various SO sites. If you're trying to deal with case/issue/bug management. The best classification is the priority of the issues. Severity is also interesting, but can confuse a developer in what they need to do next. Also note.. 10....


3

I'm a workmanlike Tester who's had the good fortune to work with some excellent people over the years. When I hire people, there are some key characteristics I look for: Eye for detail This is the obvious one. Testers need to be able to see the little details that others miss. You know those Facebook questions? The ones that go: Spot the mitsake : 1 ...


3

% of Rejected Defects - makes sure your QA really understands what he is doing. Time to test a feature - how long does it take the QA to actually test a feature % of Escaped defects - how many defects were found by customers post a release Usability grade of your system I'd also count how many enhancements requests your QA is opening - making sure they also ...


3

Metrics are a tool, not a goal on themselves. You might use metrics for different purposes, and for those purposes metrics might be important. I use metrics from time to time when I want to measure something or prove a point, and stop using them when the need is no longer there. For example counting the number of bugs opened by clients vs. number of bugs ...


3

Like most of the others who have responded, I have used a few metrics in the past, when there was something I wanted to learn (via counts or numbers or trends). Most often, I collected some metrics, attempted to learn what I was seeking, then stopped collecting. I have not found a universally-useful metric - one that I always feel merits the time and money ...


3

The problem with known performance metrics is that it changes behaviour. Not necessarily in the way you'd imagine. When the plague was going on in Europe, rewards were set up for people catching/killing rats and presenting the bodies. Pay per carcass, nice and simple right? Until people worked out they could breed rats and make more money that way... ...


3

Metrics are useful, but they should be supplemented with subjective judgement. It's useful to have something measurable to use as the basis for management decisions such as promotions, raises, and bonuses. Of course any metric can be manipulated to defeat its purpose, so a competent manager will use their own subjective judgement too. The bonus formula ...


3

I would suggest you look at their Device Fragmentation Scope of testing, (white box / black box) test aspects, App store testing, versioning, Security testing, memory usage and usage of best practices (mind maps, automated tools, carriers etc..) Tools used / Automation Efforts Previous projects in similar domain / expertise Meaningful Measure of Number of ...


Only top voted, non community-wiki answers of a minimum length are eligible