How to prevent cheating by manual testers?

Question

I'm the guy who is motivated to bring some quality assurance into our team. The problem is, that our developers very often hate testing and even if they have a test protocol, not all of them are really testing. Very often they are asking their colleagues instead of testing themselves where the bugs were and practices like this. They are very often not very interested to do a good job.

This is of course a general problem and they could got fired for such an attitude. But in reality things aren't as simple as this.

My question now is if there are clever tactics which prevent them from cheating. I would let some random errors occur for each tester to prevent it, but as they are the developers it's nearly impossible to prevent searching the source code for these errors. Another idea would be to offer a reward for the person who found the most bugs. Or don't let multiple people test the same things (but this reduces the quality of the tests).

What are your clever suggestions to "force" the developers to really test and give them no possibility to cheat? Funny tactics are of course welcome too...

In my situation the application is written in PHP and the developers are the testers too (same persons).

Answer 1

As user246 says, tricks to force developers to test can always be gamed: you're much better off finding out why they don't like testing and what the actual problem is then building a culture of testing and quality from that.

You're working in PHP - there are unit test frameworks available for PHP that your devs can use. If they have no idea how much trouble testing can save them, that's where you get to evangelize. I'd suggest for starters working up a smallish project with some unit tests, and then refactoring it - use it as a demonstration of how well-designed unit testing can help save work: if you know you'll find any change that breaks existing features the moment you compile, you also know you can safely refactor and extend your system.

Some other things that can help:

• When they gripe about something not working, or talk about games they play (there's almost for certain at least one gamer in the team), see if you can feel out the way they respond to bugs in software they use.
• Google "Maybe they should have tested more". I guarantee you, your developers do NOT want to find their application on that blog!
• Be tactful. If you're obnoxious about it, you'll get resistance and sabotage. That's just human nature - people don't like feeling that someone is railroading them somewhere they don't want to go.
• Make sure that you're creating unit tests and testing your work. If you're modeling the things you want the rest of your team to do, that's always a good start. From that, you can drop the occasional seed comment in, such as mentioning offhand how that last enhancement was so easy because you had all those tests to tell you as soon as you broke something (don't overdo it - mention it once then leave it for a while: let them come to you).

In general, you'll have a much better chance of building a culture of quality if your team wants to build it and thinks it's their idea. In the meantime, try not to get too frustrated: your team most likely sees "testing" as something any trained monkey can do and something that takes away from their main purpose.

Answer 2

Good luck with that. Your developers are probably bright enough to subvert any gimmick or incentive you put in place. Instead, you need to face the problem head-on: testing needs to be a part of your organization's culture. That's not to say that every developer needs to love testing, or needs to prefer testing over development. Rather, developers need to know and believe that finding and eliminating bugs is just as important as every other aspect of their job. If they do not believe that finding bugs is important, their product will fail and their careers will fail too. It is that simple.

If you agree with that, then the question becomes, "How do I promote a culture of testing?" You need to show developers how the testing process impacts their success. Show them what's happening to the bug backlog. Give them some examples of how your product is at risk of failure because of its quality. Measure how much time is spent in testing, then talk about how much more work your team could do if you could finish testing faster by having everyone chip in.

You might also ask them why they aren't interested in doing a good job in testing -- not in an accusing way, but in an inquisitive, open way. It is possible that they like (or at least tolerate) some kinds of testing and not others. Or they might dislike testing because they don't really know how to test. You won't know unless you ask.

Answer 3

Don't ask them to test. Ask them for the information you need. Help them understand what information you need, and what makes it valuable for you.

1. Tell them what decisions you and others need to make that would benefit from good information about the system.
2. Tell them what kinds of information would help you and others make better decisions.
3. Ask for that information. For evidence. For help.
4. Pay attention to the quality of the information -- whether you're getting information at all, whether you understand the information, and whether it's helping you and others make better decisions.
5. Adjust what you ask for as you learn more about what they are able to deliver, and as your information needs change.

Testing is not what you want. Testing is just how they produce what you want. What you want, the thing you're really after, is information. So ask for that.

Answer 4

not all of them are really testing

How do you know this? Just because they are asking other programmers where the problems are? While this isn't a holistic strategy it's not a bad way to find well understood problems.

They are very often not very interested to do a good job.

How do you know this? Perhaps you haven't communicated what a "good job" testing means? It's entirely possible you and your developers don't see eye to eye on what you mean by testing. Perhaps they see their approach as being good? Does the organization have a common understanding of what testing means? What quality means? What you consider a bug / issue, etc?

Trying to apply some new formal testing in an organization that's never had it, without a skilled, dedicated resource requires some strategy and influence. Chances are your developers don't know what testing is and think unit tests are as far as they need to go.

My advice: Think about what you want to see with regard to a testing strategy, then figure out what areas your developers can reasonable do given their current set of knowledge and skills. Then work on expanding that current set of knowledge and skill until they can gradually fit your test strategy.

Another idea would be to offer a reward for the person who found the most bugs.

An approach like this will likely lead to gaming the system where large numbers of bugs are filled but the most important, critical bugs that take days to isolate and report will be left in the system. You are basically asking for low hanging fruit - really shallow testing.

Or don't let multiple people test the same things (but this reduces the quality of the tests).

How do you know the quality of their tests are any good? Do you review their tests and testing strategy with them? If not this might help as well.

Answer 5

Testing is a motivational issue. For most programmers it can become a unbearable threadmill. I have heard that a irrelevant positive reward, like a free beverage on finding an error, can help. It is not so much the reward, as well as the game mode.

In general look for means to diminish testing. Otherwise good developers will leave, and such. Switching to TDD, Test Driven Development, can be such an incremental solution. It needs organized code and data though, but that adds to the software quality. One writes small unit tests, which serve as regression tests later on, when code is changed.

Typically a data model, listeners for changes, actions and so one is done. Specific parts are emulated, like the selection of an HTML select box.

The advantage is fast develop results and less fluff.

Answer 6

Hire some professional testers Ask them to secure test evidence (screenshots, logs, Bug IDs etc.) and let it be known they will be audited on a regular basis.

Mark.