yeah, I'm picking that up. I'm finding it frustrating, because automated testing can and should be thought of as being - at least potentially - so much more. Anyways, I think I've found a new pet peeve :) Time to go laugh at myself!

The statement, "If you run a test with the same input, it should have the same output" breaks down for other types of testing besides fuzzing - stability testing often is "run until it breaks", testing features with inherent randomness can't have this feature (e.g., I'm testing a random sampling feature right now), etc. Someone distinguished between "testing" and "checking" once, but I can't find the articles - but I thought that was a useful distinction.

Thanks, Bruce, I thought that was the distinction you were aiming at. I appreciate your patience in clarifying these terms for me. I think the concepts got conflated because testers so naturally added in semi-random data with their random testing (since it's a fairly small set of changes to the fuzzing tool to have it run semi-random data as well).

The one change I would make is to make step 4 be step 1. Do the reviews on the plans first. This allows the developer to build in testability features and helps the tester avoid writing useless tests or missing key areas for testing. A review after coding might also be good, but I think a pre-review is much more important. @glowcoder, with the modification I mentioned, we do this on our (small, Agile) team, and I saw this at Microsoft as well (large, branch-and-merge). Both hire management with experience as senior developers, which seems key.

Thanks, I wasn't getting the idea that the bug 'isn't a bug' if it's fixed so quickly that it doesn't really spend any time in the code. Do you have more information / good links to read on 'test data generators'? I'm trying to figure out if the people I've worked with have used the term 'fuzzing' loosely when they were doing what you are talking about at least some of the time, or if there is a greater distinction that I'm just not familiar with. When I've seen people 'fuzz' in the past, they did have x% of the tests be category A, y% category B, etc., where A & B are defined by a regex.

I second Tangurena's comment. Drilldown reports are very handy for displaying this kind of information.

This is what I was thinking! I've never used it either, but all pairs or something similar sounds promising.

Maybe something like a Design Pattern archive for developers, but with testing techniques? I could see that being useful. It seems analogous to this site dofactory.com/Default.aspx

It's getting a little outdated IMO, but for an absolute beginner I still don't know of anything better.

+1 for team wiki. I love, love, love this solution. I've seen every one of these solutions, but like the wiki best (followed by source control).