Dustin,
Great question. The diminishing marginal returns you saw are frequently encountered. The following chart, while not the easiest to decipher, explains a lot of why diminishing marginal returns occur:
The chart, taken from an article called "Combinatorial Software Testing" published in IEEE Computer that I co-wrote with 3 PhD's, explains that the significant majority of defects can be triggered by testing each value you can think of at least once. Several thorough studies were done to answer the straightforward and important question, "how many test inputs would be required to trigger defects in production today?" Four very different Systems Under Test were examined (including defects in medical devices - in red - and the Mozilla browser - in green). Each defect in each SUT was categorized... to trigger this defect, we only need one test input; in order to trigger this one, this specific PAIR of test inputs need to appear in the same test case, etc. Approximately 85% of the defects in production could be triggered by just one or two test inputs. Exactly one defect in the four studies required six specific test inputs to be triggered. None required 7 or more.
What are the implications of these findings?
They are huge. According to these findings:
The biggest bang for your buck will come from testing every test input in at least one test case.
- That sounds pretty similar to what you were doing in your initial set of tests.
The next biggest bang for your buck will come from testing every PAIR of test inputs together in at least one test case.
- Your initial set of tests also included a great many pairs of values; they would have to unless your tests each consisted of just a single test input.
The next biggest bang for your buck will come from testing every single possible combination of three test inputs together in at least one test case.
If you continue from there to cover all combinations of 4 test inputs together, and/or all combinations of 5 test inputs together, and/or all combinations of 6 test inputs together, in many cases, you may execute hundreds of thousands of combinations of test inputs without seeing a single defect.
As you moved from your early tests (consisting of a relatively small number of tests) to your later tests (consisting of tens of thousands of tests), what additional new coverage were you achieving?
From my analysis of dozens of similar projects with findings similar to yours, I strongly suspect that the tens of thousands of tests you ran in the "second round" of your testing, you were adding very few combinations involving "as yet untested PAIRS of test inputs" (which would have given you a reasonable chance of finding an as yet undetected bug). Rather, the net new coverage added in each test would often be several "as yet untested specific combinations involving, say, 4 or more test inputs tested together for the first time in this test" (which, statistically, we can see are extraordinarily unlikely to trigger defects).
The final implication from these findings is that if you want to maximize the efficiency of finding defects in a situation like yours:
First, execute all the test inputs at least once.
Next, execute a set of tests that will cover all pairs of test input values (e.g., execute pairwise tests - AKA allpairs tests / 2-way tests), then consider executing the smallest possible set of tests that will ensure coverage of all possible combinations involving 3-way combinations. If you find very few defects when executing your 3-way tests, you may seriously consider stopping your testing without executing higher strength test sets.
