Bug hunts and possible alternatives?

Question

Recently, our QA department has been running bug hunts. Once a sprint all the teams do 10 min demos then they spend the whole day submitting bugs for a feature getting ready for release. They've been fairly successful. We find lots of bugs before they hit production, it makes the higher ups really happy to see results, and the winning QA members get lunch.

However, I've noticed a very hostile tone encroaching from the teams themselves. We have 4 scrum teams and the bug hunts have been pitting the QA from each team against each other. This starting to cause a headache for myself and the other QA leads. Testers trying to game the system, complaints about other teams cheating, and generally less than happy office chatter about the hunts. Everyone seems to really enjoy the hunts, but everyone has a reason as to why the other teams found issues shouldn't qualify or they just have general paranoia about what the other team is doing.

Has anyone else had to deal with conflicts arising from bug hunts before and if so, how have you gotten around them?

TL;DR Our scrum teams hunt each others features for bugs. Originally this went well, now QA spends as much time, if not more bickering over validity of found issues. Anyone have an alternative approach?

Answer 1

it makes the higher ups really happy to see results, and the winning QA members get lunch.

Since you have set the system up so that there are "winners" and "losers" based on bug count, it's not much of a surprise that testers are trying to find a way to win. You are focusing on the wrong goal. You have basically told them that the goal is not improved quality - the goal is to have the highest bug count.

Instead, consider that quality is everyone's job. There should be no winners, and no losers. Everyone should benefit from your testing.

Stop declaring winners immediately. Stop singling out individuals or teams. Reward everyone for a job well done, and for an improved product - not for achieving a higher bug count.

This might help: https://strazzere.blogspot.com/2010/04/misuse-and-abuse-of-bug-counts.html

And this might be a warning: http://www.dilbert.com/fast/1995-11-13/

Answer 2

Bug hunts should be fun and productive for the entire team, and a little professional competition can go a long way to improve moral of the entire team and help them grow.

This is a rare case that I slightly disagree with Joe. I also like the idea of a point system.

But would expand to include:

• Everyone is involved in the bug hunt; not just testers. Having developers, PMs, business analysts, and anyone else on the product team engaged in the bug hunt makes everyone have skin in the "quality" game.
• Have "seasonal" bug hunts. For example, have 1 bug hunt focused on security, another focused on globalization, etc., and of course have 1 or 2 "open" season bug hunts. This spreads the "wealth" around because rarely is 1 single person/team going to 'win' various categories.
• Recognize more then 1 contributor/team by expanding the categories. E.g. most reproducible functional bugs, highest severity issue, most creative bug, etc.
• Reward everyone on the team - pizza or other treats during the bug bash, team morale event soon after the event and the bugs are triaged (e.g. team lunch, etc)
• After the bugs are triaged upper mgt should send mail to the team outlining high level results of bug bash, talk about value prop for customer and team, and thank everyone for participating and continuing to 'self-host.'
• Use bug hunts as learning tools. Have the person with the most bugs/most severe bug, etc explain or share some techniques or approaches they used to help them find those issues.

Answer 3

I'm with Joe on this one. When there's a reward only for the "best" - no matter how that "best" is determined, there's an immediate temptation to game the system and an incentive to undermine teamwork.

I'd look towards a more group-oriented arrangement, one that doesn't necessarily reward the number of bugs found (the last thing you want is people submitting one bug report per typo - it's wasting their time and developer time) but instead rewards the entire team - with a separate reward for developers who are fixing these problems. Possibly have a whole swath of more "fun" awards (certificate and party-hat style) for categories like "strangest bug", "bug with the most complex reproduction", "the why would anyone do THAT? bug" - whatever your imagination can devise (you're leading a test team - I'm sure you can come up with some good ones and make the rewards humorous and appropriate) - to be awarded at the lunch. I'd include some generic awards, as well, like "the plugger" who worked on it, and maybe found some interesting bugs, but didn't find many, or "the lucky so and so" who tested in the cleanest code in the system and couldn't find much no matter what he/she did (A fun prize for the development team involved in that one wouldn't go astray, either), "most imaginative test approach" (no, dropping watermelons on your keyboard isn't really what we expect of users, but we'll give you credit for imagination) and so forth.

If your company treats the bug hunt as a team building exercise rather than a competition between teams, they're more likely to have a happier QA team - and it helps your testers to operate as a larger cohesive team working in smaller scrum teams.

Answer 4

Couple of Alternatives to check

• This issue would occur if everyone tries to test the whole system. Initally to start with you can assign a component to each team. They can reports defects in that areas. After 30 mins, this component can be moved to another team. Now if the earlier team has missed defects would be known when next team logs defect

• Maturity of testers and the team matters. I am sure only one or two team members actions might lead to conflicts. Best thing is to educate them or put those testers in the same team

Answer 5

The bug hunt is a great idea. However, if it has turned into a competition, you must calibrate the rules to encourage the behavior you want.

Here is an example. Many years ago, I worked for a company that did not initially use a bug tracking system. When we finally adopted one, I tried to be diligent about logging defects against my own code whenever I found a problem. I was already in the habit of tracking my bugs in a text file, so logging them in a bug tracking system was an easy adjustment. I learned later that my behavior was in the minority; most developers avoided logging defects against their own code and asked testers to report problems directly to them. Apparently these developers assumed that they would be judged by the number of bugs logged against them, and so they gamed the system in ways that defeated some of the advantages of having a bug tracking system in the first place.

Management could have fixed the problem by telling everyone that developers were not judged by their bug count. Or perhaps they could declare that every check-in should have an associated bug number. There are many ways to address this issue, each with its own trade-offs, and I will not try to list all of them here. My point is that changing the rules -- changing the incentives -- could have redirected some bad behavior into better behavior.

Another example is staring you in the face: Stack Exchange was a experiment into using a point system to encourage behavior that would create a certain kind of site.

I would think your technical staff -- both QA and developers -- would enjoy designing a point system that furthers the goals of the bug hunt and encourages the kind of behavior all of you want for your organization. Why not ask some of them what they think?

Answer 6

In our case, we faced the similar problem while appreciating an individual against others in a team created issues. Instead of productivity going up, it actually came down. What we did, credited points for every defects as per the severity level and later converted points as "Team Prize". With the amount collected, the team decided how to use that prized money ethically :)

We didn't find it necessary to highlight who raised highest number of defects to management but at the same time, team was small and everyone was aware of their efforts / output and every time when we had this exercise the goal was to accumulate as many points as possible. Later management capped the upper limit though.