Tag Info

KPIs can be dangerous - it's very easy to measure the wrong thing or worse, reward the wrong thing. The general rule is that people will do more of the things that get rewarded. Some of the things you should consider: If you use the number of bugs raised by the team, you will see bugs raised for things like misplaced pixels. Modifying this to use the ...

The first thing you need to do is isolate whatever could be messing with your system. Make it as independent as possible. The fewer background processes, extra hardware and software, and even network traffic you can get, the better. Sometimes it's even a good idea to run these on a batch job that runs at night at a time when you know there's nothing else ...

The best KPI's for testing are ones that you wouldn't expect. Can the team ship with a boring level of predictability ? Are customers happy ? Is the product selling ? Are there very few critical issues found in production ? Are budgets and schedules being met ? These co-incidenty are 'whole of team' KPI's which testing plays a key part of.

The easiest way I can think of is to base it on how many hours the qa people are saving by not having to regression test all the scenarios covered by the automated test. If the response is they wouldn't regression test it, the answer to that that they should. The ideal scenario is that the entire application should be tested from one release to the next, ...

The problem I can see with counting defects, is that all defects are not equal. You might move a release deadline when you find a single bug - a showstopper resulting in complete data loss for the customer, but you'd be unlikely to do so for even a few dozen cosmetic bugs. On the other hand - you can't just discount cosmetic bugs either. What if that ...

As well as the usual "what they said" to joshin and user246, I'd add this: there's no guarantee that the number of times a bug gets reopened is going to give you any useful information. Here's a few reasons why: There are several different bugs involved, but they all have the same symptoms. Here, someone could have reopened the bug thinking it was the same ...

I haven't been part of a project where fault seeding was used. I have, however been part of projects where new metrics were introduced - which is essentially what you are proposing here. If my experience is any predictor, you should expect that people will get better at specifically whatever it is you measure, at the expense of everything else. Here, you ...

There is no definitive guide without narrowing the context dramatically. What you are searching for has no universally-accepted set of metrics. If you search for web quality metrics, you'll find millions of hits full of personal opinions, or what matters most to selected individuals. If you are trying to promote your technology, the right thing would be to ...

I agree with Joe's assertion that metrics can be badly misused and counterproductive. That said, error seeding can be a useful way to answer the right question. After we develop a test plan, we generally assume two things: The test plan is unlikely to uncover every error in the system. (If the system is large and complex, we can assume some undiscovered ...

It depends on what you consider useful. If you start tracking bugs with some metrics (number of times reopened, number of bugs per developer, etc), you might inadvertently start creating incentives for particular behaviours. If you start tracking bugs per developer, you might find dramatic changes the number and severity of bugs reported, for example. ...

There are a number of techniques that need to be combined for this sort of thing. Tradeoff approach The tradeoff approach (aka tradeoff triangle) needs to be agreed and documented up front, with the key stakeholders. This is great way to discuss and decide something that they normally won't budge on. The trade-off triangle conceptualizes the idea that ...

At my current workplace, we don't distinguish between different ways that bugs are discovered. If a developer expects the test team to test the bug fix, they log it. If they don't expect the test team to test it, they don't log it. They understand there actions have consequences, and so they make that decision carefully. We never penalize anyone for ...

I can very quickly think of three benefits that should be somewhat measurable. (1) Freeing up testers from the monotony of repeating regression tests and allow them to focus on new development and areas that need their attention. Add up the time they are not spending on regression & show how that goes to new projects. (2) Accelerated testing window. ...

I'm sure that the automated tests have helped and there should definitely be a ROI present. The question is, since there are several different types, what kind of ROI are you getting for the automated tests you have. Time savings - This one should be more quantifiable than other types of ROI. Time savings comparing manual test execution to automated ...

I have personally found that the more you can reduce the overhead of developers fixing bugs before the code hits the main source branch the better off you are. I generally use a rule that as soon as a bug will be seen by or could effect someone else then it must be logged. This allows testers and developers to pair together as part of a pre-checkin review ...

Classifications will never be finite & will be specific to what and how you're testing. Like the list of 'tags' on the various SO sites. If you're trying to deal with case/issue/bug management. The best classification is the priority of the issues. Severity is also interesting, but can confuse a developer in what they need to do next. Also note.. ...

You may find it useful to search for "bug taxonomy" or "failure mode catalog". This paper, "Bug taxonomies: Use them to generate better tests" provides a great overview of taxonomies, discusses how you can use them to brainstorm better test ideas, and provides useful practical tips on how to use existing bug taxonomies or how to go about creating a bug ...

For me it's easy. Happy Customers (ie low production support) = High Software Quality Unhappy Customers (ie lots of production support) = Low Software Quality Also, talking to the team. As one of the other posts stated, not all bugs are equal in magnitude. Encourage your testers to speak up when the quality is going down the drain. They should know better ...

This is going to sound crazy, but... it depends. What you want from a tester who will be doing a whole lot of routine, pre-defined tests (one hopes this isn't the case, but sometimes it is necessary) manually isn't what you want from a tester who's going to need to handle a highly dynamic system. Assuming you're looking for someone who will be good with ...

There are no "best" practices in my opinion. As others on the internet stated before, there might be "good" or "good enough" practices. But. You have to see it in context. One good practice which fits perfectly for your company, product, people, etc.. (~context), might be TOTALLY wrong in my company, product, people, etc. :-) Also on the note of ...

It's hard to put a price on catching a bug before it goes out the door to a customer. It may be a small problem that can be corrected in the next release, or it may be the showstopper that keeps you at the office frantically trying to fix it so that you can get it to the customer as soon as possible. If you are relying on your automation to check ...

I can't tell whether the OP is a tester or a developer. I'll assume they're a tester. It is difficult to measure the rate of return on the investment on automated tests because the relative cost and the return is speculative. Here is my personal experience. In some cases, an automated test may be the only way to reproduce a problem or to verify that a ...

First question: why do you feel that you need to quantify this? Are you being asked to provide metrics? Would a qualitative evaluation be more valuable here? It also sounds to me like you're focusing at the wrong level - i.e. on too small a group: it's not so much about making the testers more effective, or the developers more effective, but about making ...

The number of features in a release is the most flexible piece, honestly. Quality can NEVER suffer, that's a given. With heavy customer demand, most likely release dates can't shift. So, you are limited (time-boxed) based upon how many man hours are available until the release goes out. So, quality is boxed, time is boxed, the only thing left is to limit ...

If you have an average visit length of 5 minutes, a single user could create 60 min / 5 min = 12 visits per hour. To get to the target number of 1000 visits/visitors per hour, you need 1000 visits / 12 visits per user = 84 concurrent user. In total you will have 1000 visits and 3000 page views. Just as a side note: 5 min visit length and only 3 page views ...

Just as you would for any other role in your software development company, you decide what you want the new hire to be able to do, then seek people qualified to fulfill that role. If you already know that you want to hire a "QA person", then you must have some sort of idea about what this QA person will do? There are no magic traits/questions here, just as ...

When I define my test cases, (Normally in Team Foundation Server) I cross reference them to the requirements. Then, as they pass or fail I use that as a way to determine if a requirement has been delivered or not. Once all the test cases have passed to, it is the testers, not the developers who set the requirement from 'Ready for test' to 'Built'. That is ...

Metrics are a tool, not a goal on themselves. You might use metrics for different purposes, and for those purposes metrics might be important. I use metrics from time to time when I want to measure something or prove a point, and stop using them when the need is no longer there. For example counting the number of bugs opened by clients vs. number of bugs ...

Like most of the others who have responded, I have used a few metrics in the past, when there was something I wanted to learn (via counts or numbers or trends). Most often, I collected some metrics, attempted to learn what I was seeking, then stopped collecting. I have not found a universally-useful metric - one that I always feel merits the time and money ...

In my opinion, there is no "best" classification. I'd follow testerab's links and do some more research - the context in which you're working will give you a good idea of what will work best for you and your situation. For instance, where I work, bugs are classified by a combination of the following: which module of the system in test they occur in; ...