How do you design UI automated tests to pick alternative steps randomly in a flow?

Question

Background: In cases(mainly in End to End UI tests), where a number of test paths(selection of alternative test steps) can be huge due to combinations however expected results are same and few. In those scenarios, I strongly feel the need to introduce an element of randomness to pick a test path as we are finding more and more issues manually while following a particular random path whereas the standard automated E2E test mostly passes following the primary flow.

Context: In the end-to-end scenario consisting steps let's say [A-Z] where any of the steps may have alternative sub steps[a- g] can be run either on web/mobile app so if we vary any of the sub-steps in user journey, we are finding issues on some specific sub-steps selection but not in general.Also we cannot afford to run each unique combination of selection due to huge combinations either manually or by automation.

Problem statement: What is the standard/accepted common wisdom to design the randomness in alternative sub steps selection in end to end UI tests?

Edit: Adding more meaningful example

Example: An long user journey, consisting of multiple steps where each step independently can be done on either web or mobile app. And cases were found where doing some step in mobile doesn't reflect back in web app or vice versa.

Answer 1

Personally I have come to the conclusion that testing randomness through the UI is a poor testing practice. I have found that it leads to long slow tests that are not repeatable and add questionable value. As Pieter says, UI tests are slow and flaky enough as it is without adding more randomness in!

Based on these lessons my approach now is:

• Use the 'example' based testing that tests a specific condition that you know in advance
• When you need to do data combinatorial testing use unit and integrated tests to test variants
• Avoid using 'Faker' and other random programs. Determine what you want to test and test that

I've found many folks get quite excited about testing randomness through the UI - hey look, with almost no code I can test 1000 different combinations! However my experience is that, exciting as it seems, this can move one away from adding fast specific tests for specific conditions that you want to know work and provide fast feedback if they don't. A lot of random generators that cause failure represent unlikely errors that would be sufficiently serviced with a general error. They can lead to the writing... and thus maintenance... of a lot of unreal scenarios.

$ amounts for example. A company can decide that the acceptable format is 99[,]999.99 i.e. up to 99,999.99 with the comma being optional. Given that there is no need to try ssdfdf.23 123.aa 76.1234 etc even though they might all be generated from a random string generator. I find it is better to say ok test (valid) 23.40 0.30 12,435.34 and (invalid) 'a' 123,456.78 and .401

I have found the above to be true even if the application actually has randomness . For example if the application says "click this button to select a random color" . Well you could test that. You could see if the result is in a valid range of names. But what value would that add? What if the random routine was broke and always gave the same value? This starts you down a rabbit hole of wasted time. Better I have found to say 'red' is selected and the next page shows a red icon

When you say "I strongly feel the need to introduce an element of randomness to pick a test path ." I would like to know more about why you feel that? Also the example given - payment type is an interesting example, after all no actually user picks that randomly! So I say neither should your test, it should, like a real user, know what payment type it wants to use.

Answer 2

Some considerations:

• In my experience it is really difficult to get UI tests stable; even if you run the same set of tests over and over again, some will incidentally randomly fail by unknown cause. This is hard to debug and fix. If you would now randomize the execution paths of your tests, it will be less easy to spot recurring random incidental failures with unknown cause, as the tests are not run in the same way each time.
• When you run the test suite and you get a failure that is caused by a defect in the software under test, it is not obvious for how long the failure has been there. The defect might already be in production, or maybe it has just been introduced. As you don't know when it was introduced, you can't simply check the last commits to see what might have caused it.
• I would suggest looking if you can perhaps split up the tests into smaller and bigger suites of tests. The small suite can be run often and the bigger suite less often (perhaps nightly). In case of any failing test you can see exactly when it was last run successfully and you can check which changes where done in between the two runs.
• Talk with developers about possibilities to cover more in technical tests and less in functional ui tests (if you haven't done so already).
• I would only consider going for a random test execution order, if it is possible to store the last used execution paths. If you do get a failure, you want to be able to run the complete test set EXACTLY the same as before, for debugging and fix confirmation testing.

Answer 3

There are many ways to add randomness - but if you do, you have to deal with randomness :-)

You did not mentioned what is your domain, so let's assume the old favorite: shopping cart.

You can add randomness by:

• searching for one of known product types (select random from known list)
• select one exact type by random, from the found list (select random from calculated list)
• purchase random (small) number of items (plain old random number)
• for bonus points, do above multiple (random) times, where some tries would be using same product type, some the same product item with same (or different) count.

Then, you need to make your code smart enough to interpret the results (to see if they are correct).

OP wanted to add random order of steps: Create substeps/workflows, and select them at random.

Now you have much bigger mountain to climb. Complexity of such solution increases significantly: Cost of creating, maintaining the code, and especially interpreting the failures would require significant research. I cannot imagine a situation where it would be worth the cost. If project has budget for additional testing, IMHO it would be better spent on adding more tests of more predictable kind, not random workflows.