Tag Info

Hot answers tagged

20

KPIs can be dangerous - it's very easy to measure the wrong thing or worse, reward the wrong thing. The general rule is that people will do more of the things that get rewarded. Some of the things you should consider: If you use the number of bugs raised by the team, you will see bugs raised for things like misplaced pixels. Modifying this to use the ...


17

Tying cash rewards to metrics like bug counts is always a very risky business, one that is almost certain to introduce dysfunction. I never want people on my QA team to have to worry about their bonus when I just want them to be doing the right thing. Imagine you are the tester working with a single developer (who happens to be your best friend) on a ...


12

The best KPI's for testing are ones that you wouldn't expect. Can the team ship with a boring level of predictability ? Are customers happy ? Is the product selling ? Are there very few critical issues found in production ? Are budgets and schedules being met ? These co-incidenty are 'whole of team' KPI's which testing plays a key part of.


10

The easiest way I can think of is to base it on how many hours the qa people are saving by not having to regression test all the scenarios covered by the automated test. If the response is they wouldn't regression test it, the answer to that that they should. The ideal scenario is that the entire application should be tested from one release to the next, ...


10

The first thing you need to do is isolate whatever could be messing with your system. Make it as independent as possible. The fewer background processes, extra hardware and software, and even network traffic you can get, the better. Sometimes it's even a good idea to run these on a batch job that runs at night at a time when you know there's nothing else ...


8

When someone asks me to do something and I have questions about what they want, I go back and ask them. That's the most direct way to get an answer. As a manager, I would want these fields: description, i.e. what's the issue and why it's a problem amount of risk if not fixed pervasiveness level of effort to fix, including testing effort associated bug ...


7

As well as the usual "what they said" to joshin and user246, I'd add this: there's no guarantee that the number of times a bug gets reopened is going to give you any useful information. Here's a few reasons why: There are several different bugs involved, but they all have the same symptoms. Here, someone could have reopened the bug thinking it was the same ...


7

The problem I can see with counting defects, is that all defects are not equal. You might move a release deadline when you find a single bug - a showstopper resulting in complete data loss for the customer, but you'd be unlikely to do so for even a few dozen cosmetic bugs. On the other hand - you can't just discount cosmetic bugs either. What if that ...


7

There is no definitive guide without narrowing the context dramatically. What you are searching for has no universally-accepted set of metrics. If you search for web quality metrics, you'll find millions of hits full of personal opinions, or what matters most to selected individuals. If you are trying to promote your technology, the right thing would be to ...


7

It depends, and there are no industry standards. Seriously. Any metric can be gamed (and will be, if you use it for assessment). I'm not aware of any standard approaches, not least because the teams are - or should be - evaluating themselves regularly and looking for ways to improve their own processes (if they aren't then they're probably using ...


6

I agree with Joe's assertion that metrics can be badly misused and counterproductive. That said, error seeding can be a useful way to answer the right question. After we develop a test plan, we generally assume two things: The test plan is unlikely to uncover every error in the system. (If the system is large and complex, we can assume some undiscovered ...


6

I haven't been part of a project where fault seeding was used. I have, however been part of projects where new metrics were introduced - which is essentially what you are proposing here. If my experience is any predictor, you should expect that people will get better at specifically whatever it is you measure, at the expense of everything else. Here, you ...


6

I'm sure that the automated tests have helped and there should definitely be a ROI present. The question is, since there are several different types, what kind of ROI are you getting for the automated tests you have. Time savings - This one should be more quantifiable than other types of ROI. Time savings comparing manual test execution to automated ...


6

I can very quickly think of three benefits that should be somewhat measurable. (1) Freeing up testers from the monotony of repeating regression tests and allow them to focus on new development and areas that need their attention. Add up the time they are not spending on regression & show how that goes to new projects. (2) Accelerated testing window. ...


6

This is going to sound crazy, but... it depends. What you want from a tester who will be doing a whole lot of routine, pre-defined tests (one hopes this isn't the case, but sometimes it is necessary) manually isn't what you want from a tester who's going to need to handle a highly dynamic system. Assuming you're looking for someone who will be good with ...


6

In addition to what everyone else has said, metrics used for performance bonuses are a seriously bad idea in the software world. Metrics are helpful as a descriptive tool, but that's about it in my view. Here's some of the reasons why: every software project is different. That means that even if all the same things are measured, comparisons won't be ...


5

It depends on what you consider useful. If you start tracking bugs with some metrics (number of times reopened, number of bugs per developer, etc), you might inadvertently start creating incentives for particular behaviours. If you start tracking bugs per developer, you might find dramatic changes the number and severity of bugs reported, for example. ...


5

There are a number of techniques that need to be combined for this sort of thing. Tradeoff approach The tradeoff approach (aka tradeoff triangle) needs to be agreed and documented up front, with the key stakeholders. This is great way to discuss and decide something that they normally won't budge on. The trade-off triangle conceptualizes the idea that ...


5

At my current workplace, we don't distinguish between different ways that bugs are discovered. If a developer expects the test team to test the bug fix, they log it. If they don't expect the test team to test it, they don't log it. They understand there actions have consequences, and so they make that decision carefully. We never penalize anyone for ...


5

Can you prevent code with known bugs from going out? If you can't actually control the thing being measured, it's a bad metric. What if the right thing for the business is to release the code, with bugs, to get it out faster so users can start benefitting? Sometimes having some software is better than none (but not always!). If the metric could ever ...


4

For me it's easy. Happy Customers (ie low production support) = High Software Quality Unhappy Customers (ie lots of production support) = Low Software Quality Also, talking to the team. As one of the other posts stated, not all bugs are equal in magnitude. Encourage your testers to speak up when the quality is going down the drain. They should know better ...


4

The deliverable is information about the system. The purpose is to help people make better decisions, based on information about the system.


4

I have personally found that the more you can reduce the overhead of developers fixing bugs before the code hits the main source branch the better off you are. I generally use a rule that as soon as a bug will be seen by or could effect someone else then it must be logged. This allows testers and developers to pair together as part of a pre-checkin review ...


4

In all honesty, my experience is that there are no best practices anywhere, even in something like code duplication. There are only principles that work to guide practice, and practices that are good in some circumstances. You don't mention whether you're working with automation code or application code: depending on the automation method and tooling, it's ...


3

Just as you would for any other role in your software development company, you decide what you want the new hire to be able to do, then seek people qualified to fulfill that role. If you already know that you want to hire a "QA person", then you must have some sort of idea about what this QA person will do? There are no magic traits/questions here, just as ...


3

It's hard to put a price on catching a bug before it goes out the door to a customer. It may be a small problem that can be corrected in the next release, or it may be the showstopper that keeps you at the office frantically trying to fix it so that you can get it to the customer as soon as possible. If you are relying on your automation to check ...


3

I can't tell whether the OP is a tester or a developer. I'll assume they're a tester. It is difficult to measure the rate of return on the investment on automated tests because the relative cost and the return is speculative. Here is my personal experience. In some cases, an automated test may be the only way to reproduce a problem or to verify that a ...


3

There are no "best" practices in my opinion. As others on the internet stated before, there might be "good" or "good enough" practices. But. You have to see it in context. One good practice which fits perfectly for your company, product, people, etc.. (~context), might be TOTALLY wrong in my company, product, people, etc. :-) Also on the note of ...


3

You may find it useful to search for "bug taxonomy" or "failure mode catalog". This paper, "Bug taxonomies: Use them to generate better tests" provides a great overview of taxonomies, discusses how you can use them to brainstorm better test ideas, and provides useful practical tips on how to use existing bug taxonomies or how to go about creating a bug ...


3

Classifications will never be finite & will be specific to what and how you're testing. Like the list of 'tags' on the various SO sites. If you're trying to deal with case/issue/bug management. The best classification is the priority of the issues. Severity is also interesting, but can confuse a developer in what they need to do next. Also note.. ...



Only top voted, non community-wiki answers of a minimum length are eligible