what is the best way to write automated end to end user scenarios?

Question

When automating user scenarios, we typically cover an end-to-end user flow. Unit testing recommends that each test method should be independant and only test one thing. But things are different in system acceptance testing, as we have to cover a user flow through the application. Consider an ecommerce application where the user visits typically follows the below steps:

1. visit homepage
2. search for a product
3. select the product
4. add the product to cart
5. click checkout
6. sign in
7. entery peyment details
8. submit order
9. wait for confirmation

Now each of these steps need to be verified as we flow through the user journey. So, there is dependency in test methods, e.g. step 3 depends on step 2 which breaks unit testing practice. Also if we put all steps in one test method it is not easy to detect what went wrong if the test failed. And if we wanted to isolate each test method so they are independant of each other, we have to repeat the initial steps over and over again.

What I have done in the past is to use TestNG to provide this dependency on previous test methods. But I wanted to know how do you handle user journeys in your testing?

Answer 1

I would like to recommend Acceptance Test Driven Development (=Specification by Example) technics. You will be surprised, because acceptance tests will find about 80% of the show stopper bugs (if you have some).

The good point to start and listen a real stories from the experience of different agile teams is the (video) Podcasts section on:

http://skillsmatter.com/go/agile-testing

I am also digging this topic, and here is my collection on BDD/ATDD/Spec by Example http://blog.zhariy.com/search/label/Specification%20by%20Example

Sorry, most of the text descriptions are in Russian, but most of the video and slides are in English.

The tool are not so important for creating acceptance tests. You still can use TestNG, but you could also take a look on Concordion, easyb, JBehave and so on:

http://en.wikipedia.org/wiki/Specification_by_example

Answer 2

I am not a really big fan of automating customer scenarios unless they are only used as baseline to validate a specific scenario works after code churn in the product. Automated tests can be a valuable tool, but they are a poor replacement for emulating actual customer behavior.

That being said, one way to improve test coverage of the various "paths" a customer could take would be to employ model based testing (MBT) (see http://msdn.microsoft.com/en-us/library/ee620469 and http://www.harryrobinson.net/ as starting points)

Answer 3

You can try to use Behavior_Driven_Development BDD for this.

There are different tools to do this in dotnet specflow or ruby cucumber.

Actually you write your tests like this

Scenario 1: Refunded items should be returned to stock

Given a customer previously bought a black sweater from me
and I currently have three black sweaters left in stock
when he returns the sweater for a refund
then I should have four black sweaters in stock

Each given/when/then is executed and/or verified in the bdd tests.

A dotnet Example is documented at Behavior-Driven Development with SpecFlow and WatiN. Doing it with Selenium instead of WaitN should work similar.

Answer 4

My workplace has been able to build flexible and reliable end-to-end automation of user scenarios with the use of user-defined transaction "objects", each of which contains a number of transaction steps (each step is itself an object). All the data lives in CSV files and the scripts are fairly code-intensive.

This approach is pretty much tool-agnostic: our test step files look something like this - TestNumber, StepNumber, StepKind, StepData, Comments 1, 1, LOGON, 1, Log on with user 1 (FK to Users.csv) 1, 2, SELECT, 4, Select sales line 4 - 3x Red Sweater (FK to Sales.csv)

The script code reads in the files, then performs the steps in order, pulling extra data needed to perform the defined actions (LOGON, SELECT etc). At the end of the transaction, there's a baseline check to ensure all amounts are calculated correctly, and each step logs its comment so its easy to tell where something failed.

We've also standardised error handling within our scripts to report which script code unit and file generated the error, so that we can isolate a problem to a specific set of data and code very quickly.

This is a setup that requires a high level of programming ability in the scripting team, and it can be quite opaque to anyone new to the system. It does provide a very useful regression suite that ensures that once the end-to-end scenario is working, it won't break again, and once the framework is mature adding new tests is a matter of adding data and updating baselines. (As an added bonus, if the application being tested changes in a way that breaks the tests, such as adding in a new step, it's a single change to account for it - which is what makes this style of script reasonably robust).