We have a function that introduces randomness in item ordering, e.g.:

  • the order in which a number of given items are presented to a user is random

For this scenario, there needs to be automated test, I am thinking:

  • Compare the configured item order against the actual presented item order, if they are different, the test will pass.

But there is always a chance that the randomised order happens to be identical with the configured order, e.g.:

  • We have configured 5 items to be presented to a user, the chance for them to be presented exactly the same order as they are configured is 1 in 5*4*3*2*1; for 10 items, the chance is 1 in 10*9*8*7*6*5*4*3*2*1; there is always a tiny chance this automated test will fail.

Is there any certain way to test against randomness?

  • 1
    +1, interesting question, however IMHO you cannot test randomness with certainty. Sep 21, 2021 at 1:39
  • 1
    Can;t you have a retry mechanism , if it is equal then refresh the session once more ?
    – PDHide
    Sep 21, 2021 at 7:49
  • @PDHide, retry would work.
    – Yu Zhang
    Sep 21, 2021 at 10:38

2 Answers 2


The answer is: Isolate the randomness.

You can check this example from Dave Farley where isolated Time: https://www.youtube.com/watch?v=SuDIYk9GBpE&t=1872s

Separate the randomly sorting behavior from the displaying and from the entity definition, using polymorphism.

enter image description here

Then you can validate each one of these parts in isolation:

  • When you want to validate the sorting, you simply inject test-controlled items;
  • When you want to validate the displaying, you provide a set of test-controlled items and a test-controlled fake of the sorting mechanism, with static behavior that you test know about;
  • If you want to validate what constitutes the items, you won't need either a display nor a sorting mechanism, since these are external to the items.

TLDR; Test randomness at the unit level using a Chi Test and then verify at the system level that the values properly propagate from the unit to the system.

By random you probably mean uniform distribution, that each value from a list have the same probability to be chosen.

To decide if something is uniformly distributed and have a meaningful result you need to collect enough samples and then do a mathematical statistical test, or simply eyeball them on a graph. The math is not too complicated, and deciding on "enough" is also simple but both belongs to math stackexchange, this could give you a starting point or here, but I suppose many languages have ready mathematical functions to help you with that, even Excel have a ready Chi Test function.

As to the practicality of such a test, collecting enough results on an end to end test would probably be too slow and cumbersome, the best would be to run a test that collect the chosen values at the unit level and verify randomness there

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.