What are the downsides of this technique that involves randomness?

Question

I am writing test cases in JavaScript at the moment. One typical scenario I have come across is:

1. Read a text string from a web browser (this string is dependent on a block of json code)
2. Compare this text string read from step 1 against a hard-coded reference, which is the same expected string
3. If the string read from step 1 is identical to the hard-coded string, it is a pass; it is a failure otherwise.

I am personally against this approach, as for each test execution, we literally bash subject under test against the same wall. There is little value of doing it. What I have in mind is:

1. Randomize the string by injecting different json each time
2. Record the expected string by reading json
3. Read the string from a browser window
4. If the string read from step 3 is identical to the string recorded from step 2, it is a pass otherwise it is not.

Overtime, the second approach will test a larger spectrum of strings instead of a static string.

But, as rule of thumb, there has to be a trade off somewhere. Can anyone see the downsides of the second approach, comparing against the first one?

Answer 1

Generally I avoid randomness in tests.

I like to test a particular known state. I like to test by example. I like to have tests that are very easy re-runnable and produce the same result every time (deterministic). Thus when app codes that uses it breaks we can fix the app code and the test returns to green.

So for these reasons I nearly always avoid randomness in the data that I test with, even if (not your case) the app has randomness.

I use the basic cases, e.g. for a 5 digit number I always consider having tests for 5 numbers(valid), < 00001, > 99999, alphabetic, blank. Assuming that all the numbers between 00001 and 99999 have the same rules I only test one of them.

I'm ok with only having 4 ways to head bang against the wall and I want to make sure the result (headache) is the same for each of them, every time I supply the same data.

Answer 2

I agree mostly with Michael's answer. However, in certain scenarios, randomness not only becomes useful but necessary as well. As recently to re-calculate a price(floating number), I needed to randomize few input values so that application is forced to re-calculate price on every test run which will not be possible with static price value.

Also on randomizing numbers in decimal points up to the large range, I found an issue where the application is incorrectly rounding off the price(after complex algorithm calculation) after certain decimal points which is not acceptable.

This issue was undiscovered over a long period with static values.