3

I am writing test cases in JavaScript at the moment. One typical scenario I have come across is:

  1. Read a text string from a web browser (this string is dependent on a block of json code)
  2. Compare this text string read from step 1 against a hard-coded reference, which is the same expected string
  3. If the string read from step 1 is identical to the hard-coded string, it is a pass; it is a failure otherwise.

I am personally against this approach, as for each test execution, we literally bash subject under test against the same wall. There is little value of doing it. What I have in mind is:

  1. Randomize the string by injecting different json each time
  2. Record the expected string by reading json
  3. Read the string from a browser window
  4. If the string read from step 3 is identical to the string recorded from step 2, it is a pass otherwise it is not.

Overtime, the second approach will test a larger spectrum of strings instead of a static string.

But, as rule of thumb, there has to be a trade off somewhere. Can anyone see the downsides of the second approach, comparing against the first one?

  • 2
    Why would you do these low level data centric tests from UI? It is expensive, is n't it? – Vishal Aggarwal Aug 8 '18 at 22:46
  • @VishalAggarwal, a very good point, I will review my tests. Thanks – Yu Zhang Aug 8 '18 at 22:49
  • You are welcome, in my opinion this is best suited for unit/ API tests. – Vishal Aggarwal Aug 15 '18 at 11:06
3

Generally I avoid randomness in tests.

I like to test a particular known state. I like to test by example. I like to have tests that are very easy re-runnable and produce the same result every time (deterministic). Thus when app codes that uses it breaks we can fix the app code and the test returns to green.

So for these reasons I nearly always avoid randomness in the data that I test with, even if (not your case) the app has randomness.

I use the basic cases, e.g. for a 5 digit number I always consider having tests for 5 numbers(valid), < 00001, > 99999, alphabetic, blank. Assuming that all the numbers between 00001 and 99999 have the same rules I only test one of them.

I'm ok with only having 4 ways to head bang against the wall and I want to make sure the result (headache) is the same for each of them, every time I supply the same data.

  • 1
    This randomness is not completely random (my own fault for not stating it clearly), it is kind of under control as I can define rules regarding which characters are allowed. Would it still be non-deterministic? – Yu Zhang Aug 8 '18 at 22:40
  • 1
    If you re-run a test does the same value get generated ? – Michael Durrant Aug 8 '18 at 22:43
  • 1
    That basically boils down to having two types of tests: regression tests and tests for finding new defects. In first, determinism is important, in the latter - randomness (or certain degree of randomness). You can use randomness to discover new defects, new test data and thus new test cases. Then test cases that identified defects can be included in your regression test suite. – dzieciou Aug 17 '18 at 7:02
2

I agree mostly with Michael's answer. However, in certain scenarios, randomness not only becomes useful but necessary as well. As recently to re-calculate a price(floating number), I needed to randomize few input values so that application is forced to re-calculate price on every test run which will not be possible with static price value.

Also on randomizing numbers in decimal points up to the large range, I found an issue where the application is incorrectly rounding off the price(after complex algorithm calculation) after certain decimal points which is not acceptable.

This issue was undiscovered over a long period with static values.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.