1

Background: In cases(mainly in End to End UI tests), where a number of test paths(selection of alternative test steps) can be huge due to combinations however expected results are same and few. In those scenarios, I strongly feel the need to introduce an element of randomness to pick a test path as we are finding more and more issues manually while following a particular random path whereas the standard automated E2E test mostly passes following the primary flow.

Context: In the end-to-end scenario consisting steps let's say [A-Z] where any of the steps may have alternative sub steps[a- g] can be run either on web/mobile app so if we vary any of the sub-steps in user journey, we are finding issues on some specific sub-steps selection but not in general.Also we cannot afford to run each unique combination of selection due to huge combinations either manually or by automation.

Problem statement: What is the standard/accepted common wisdom to design the randomness in alternative sub steps selection in end to end UI tests?

Edit: Adding more meaningful example

Example: An long user journey, consisting of multiple steps where each step independently can be done on either web or mobile app. And cases were found where doing some step in mobile doesn't reflect back in web app or vice versa.

  • It is not the data but more of random test steps like 'payment by' ( by cash/credit/debit) in a 'online order' journey of a user as a end to end scenario. – Vishal Aggarwal May 2 '18 at 20:38
  • Great question from @VishalAggarwal despite closing vote as usual – Michael Durrant May 2 '18 at 21:57
  • Test steps can be treated as data in parametrized tests. Please describe why steps would be different. – dzieciou May 8 '18 at 10:14
2

Personally I have come to the conclusion that testing randomness through the UI is a poor testing practice. I have found that it leads to long slow tests that are not repeatable and add questionable value. As Pieter says, UI tests are slow and flaky enough as it is without adding more randomness in!

Based on these lessons my approach now is:

  • Use the 'example' based testing that tests a specific condition that you know in advance
  • When you need to do data combinatorial testing use unit and integrated tests to test variants
  • Avoid using 'Faker' and other random programs. Determine what you want to test and test that

I've found many folks get quite excited about testing randomness through the UI - hey look, with almost no code I can test 1000 different combinations! However my experience is that, exciting as it seems, this can move one away from adding fast specific tests for specific conditions that you want to know work and provide fast feedback if they don't. A lot of random generators that cause failure represent unlikely errors that would be sufficiently serviced with a general error. They can lead to the writing... and thus maintenance... of a lot of unreal scenarios.

$ amounts for example. A company can decide that the acceptable format is 99[,]999.99 i.e. up to 99,999.99 with the comma being optional. Given that there is no need to try ssdfdf.23 123.aa 76.1234 etc even though they might all be generated from a random string generator. I find it is better to say ok test (valid) 23.40 0.30 12,435.34 and (invalid) 'a' 123,456.78 and .401

I have found the above to be true even if the application actually has randomness . For example if the application says "click this button to select a random color" . Well you could test that. You could see if the result is in a valid range of names. But what value would that add? What if the random routine was broke and always gave the same value? This starts you down a rabbit hole of wasted time. Better I have found to say 'red' is selected and the next page shows a red icon

When you say "I strongly feel the need to introduce an element of randomness to pick a test path ." I would like to know more about why you feel that? Also the example given - payment type is an interesting example, after all no actually user picks that randomly! So I say neither should your test, it should, like a real user, know what payment type it wants to use.

  • Michael, how about the complex dynamic pricing algorithm verification as an example, where the price shown to the online customer based on the order qty plus tons of additional factors based on user's shopping profile & buying patterns. – Vishal Aggarwal May 2 '18 at 22:57
  • 1
    ok so 'effectively random' or 'might as well be random to the end user'. Well I still think in those you will come up with a range of results and, based on expecting different output for the different ranges you should be able to write enough suitable deterministic tests to represent reasonable expected results. – Michael Durrant May 2 '18 at 23:03
  • 1
    If the different combinations result in significantly different results or side effects that should be tested then you should have test cases for them. Usually I look for boundaries to find good cases. – Michael Durrant May 2 '18 at 23:06
  • Also you can easily end up in automation traps such as sqa.stackexchange.com/q/17022/8992 – Michael Durrant May 2 '18 at 23:10
1

Some considerations:

  • In my experience it is really difficult to get UI tests stable; even if you run the same set of tests over and over again, some will incidentally randomly fail by unknown cause. This is hard to debug and fix. If you would now randomize the execution paths of your tests, it will be less easy to spot recurring random incidental failures with unknown cause, as the tests are not run in the same way each time.
  • When you run the test suite and you get a failure that is caused by a defect in the software under test, it is not obvious for how long the failure has been there. The defect might already be in production, or maybe it has just been introduced. As you don't know when it was introduced, you can't simply check the last commits to see what might have caused it.
  • I would suggest looking if you can perhaps split up the tests into smaller and bigger suites of tests. The small suite can be run often and the bigger suite less often (perhaps nightly). In case of any failing test you can see exactly when it was last run successfully and you can check which changes where done in between the two runs.
  • Talk with developers about possibilities to cover more in technical tests and less in functional ui tests (if you haven't done so already).
  • I would only consider going for a random test execution order, if it is possible to store the last used execution paths. If you do get a failure, you want to be able to run the complete test set EXACTLY the same as before, for debugging and fix confirmation testing.
  • +1 to cover them more under low-level unit tests. – Vishal Aggarwal May 2 '18 at 23:13
  • 1
    I don't have enough reputation yet to comment on others answers. I see in a different comment that you are testing complex pricing algorithms. I've had to deal with similar problems for an e-commerce platform. We ended up creating integration tests for this. The focus of the tests was not the steps that the user would go through, but simply if a given combination of criteria (i.e. products, discounts, shipping method, etc) would lead to the correct calculation. These test could be executed in seconds and debugging is much easier as stacktraces lead you directly into the calculation code – Pieter A May 3 '18 at 5:51
1

There are many ways to add randomness - but if you do, you have to deal with randomness :-)

You did not mentioned what is your domain, so let's assume the old favorite: shopping cart.

You can add randomness by:

  • searching for one of known product types (select random from known list)
  • select one exact type by random, from the found list (select random from calculated list)
  • purchase random (small) number of items (plain old random number)
  • for bonus points, do above multiple (random) times, where some tries would be using same product type, some the same product item with same (or different) count.

Then, you need to make your code smart enough to interpret the results (to see if they are correct).

OP wanted to add random order of steps: Create substeps/workflows, and select them at random.

Now you have much bigger mountain to climb. Complexity of such solution increases significantly: Cost of creating, maintaining the code, and especially interpreting the failures would require significant research. I cannot imagine a situation where it would be worth the cost. If project has budget for additional testing, IMHO it would be better spent on adding more tests of more predictable kind, not random workflows.

  • +1 , I see that by adding randomness ,the tests might become more flakey(if not handled carefully). However my question is around more on technical side instead of domain side where in a end to end user journey, there are numerous alternative steps which all reach ultimately to same user end goal. – Vishal Aggarwal May 8 '18 at 9:38
  • Your points seems more on random data selection instead of steps itself. – Vishal Aggarwal May 8 '18 at 9:41
  • @VishalAggarwal - added blurb about random workflow. Complexity increases significantly, only you can tell if it is worth it. – Peter M. May 8 '18 at 12:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.