Take the 2-minute tour ×
Software Quality Assurance & Testing Stack Exchange is a question and answer site for software quality control experts, automation engineers, and software testers. It's 100% free, no registration required.

When automating user scenarios, we typically cover an end-to-end user flow. Unit testing recommends that each test method should be independant and only test one thing. But things are different in system acceptance testing, as we have to cover a user flow through the application. Consider an ecommerce application where the user visits typically follows the below steps:

  1. visit homepage
  2. search for a product
  3. select the product
  4. add the product to cart
  5. click checkout
  6. sign in
  7. entery peyment details
  8. submit order
  9. wait for confirmation

Now each of these steps need to be verified as we flow through the user journey. So, there is dependency in test methods, e.g. step 3 depends on step 2 which breaks unit testing practice. Also if we put all steps in one test method it is not easy to detect what went wrong if the test failed. And if we wanted to isolate each test method so they are independant of each other, we have to repeat the initial steps over and over again.

What I have done in the past is to use TestNG to provide this dependency on previous test methods. But I wanted to know how do you handle user journeys in your testing?

share|improve this question
3  
Why are you concerned about breaking unit testing guidelines when it is end-to-end testing that you are doing ? They are testing different aspects of the system –  Phil Kirkham Jun 4 '12 at 1:29

4 Answers 4

My workplace has been able to build flexible and reliable end-to-end automation of user scenarios with the use of user-defined transaction "objects", each of which contains a number of transaction steps (each step is itself an object). All the data lives in CSV files and the scripts are fairly code-intensive.

This approach is pretty much tool-agnostic: our test step files look something like this - TestNumber, StepNumber, StepKind, StepData, Comments 1, 1, LOGON, 1, Log on with user 1 (FK to Users.csv) 1, 2, SELECT, 4, Select sales line 4 - 3x Red Sweater (FK to Sales.csv)

The script code reads in the files, then performs the steps in order, pulling extra data needed to perform the defined actions (LOGON, SELECT etc). At the end of the transaction, there's a baseline check to ensure all amounts are calculated correctly, and each step logs its comment so its easy to tell where something failed.

We've also standardised error handling within our scripts to report which script code unit and file generated the error, so that we can isolate a problem to a specific set of data and code very quickly.

This is a setup that requires a high level of programming ability in the scripting team, and it can be quite opaque to anyone new to the system. It does provide a very useful regression suite that ensures that once the end-to-end scenario is working, it won't break again, and once the framework is mature adding new tests is a matter of adding data and updating baselines. (As an added bonus, if the application being tested changes in a way that breaks the tests, such as adding in a new step, it's a single change to account for it - which is what makes this style of script reasonably robust).

share|improve this answer

You can try to use Behavior_Driven_Development BDD for this.

There are different tools to do this in dotnet specflow or ruby cucumber.

Actually you write your tests like this

Scenario 1: Refunded items should be returned to stock

Given a customer previously bought a black sweater from me
and I currently have three black sweaters left in stock
when he returns the sweater for a refund
then I should have four black sweaters in stock

Each given/when/then is executed and/or verified in the bdd tests.

A dotnet Example is documented at Behavior-Driven Development with SpecFlow and WatiN. Doing it with Selenium instead of WaitN should work similar.

share|improve this answer

I am not a really big fan of automating customer scenarios unless they are only used as baseline to validate a specific scenario works after code churn in the product. Automated tests can be a valuable tool, but they are a poor replacement for emulating actual customer behavior.

That being said, one way to improve test coverage of the various "paths" a customer could take would be to employ model based testing (MBT) (see http://msdn.microsoft.com/en-us/library/ee620469 and http://www.harryrobinson.net/ as starting points)

share|improve this answer
    
I agree with your first paragraph - the idea is attractive, but in my experience, such automated tests require so much maintenance that they are not worth the investment. –  user246 Jun 4 '12 at 15:05
    
I generally agree that poorly desinged UI automation requires a lot of maintenance. However, I have seen well-designed UI automation that doesn't require 'a lot' of maintenance. Of course, some percentage of autoamted tests will require some level of maintenance as the product it is testing evolves. –  Bj Rollison Jun 4 '12 at 16:48
1  
I can't believe you're not a fan of automating customer scenarios! :-). Honestly, I think this is one of two things that web UI automation excels at. One would be automating the testing of any client side code (javascript) and the other would be excercising common user scenarios. I don't like including tons of validations in a single massive test case however. Having tinkered with MBT (I'm not an expert) it seems to lend itself well to API's and flow based apps but not so well to more traditional content web sites and can be overkill for things like automating a single sign up page. –  Sam Woods Jun 6 '12 at 17:41
    
Hey Sam, great to hear from you and hope all is well. I know that an awful lot of Web automation is through the UI. And I also think we often automated common customer scenarios (perhaps more appropriately some defined "happy path" the majority of our customers are likely to follow). And I think these are generally fine as baselines, but automated tests are poor at emulating human behavior, and most automated scripts do a bunch of linear tasks then validate a few specific things. IMHO, our eyes and rational thinking are mcuh better oracles for behavioral issues than many UI automated tests. –  Bj Rollison Jun 8 '12 at 14:45

I would like to recommend Acceptance Test Driven Development (=Specification by Example) technics. You will be surprised, because acceptance tests will find about 80% of the show stopper bugs (if you have some).

The good point to start and listen a real stories from the experience of different agile teams is the (video) Podcasts section on:

http://skillsmatter.com/go/agile-testing

I am also digging this topic, and here is my collection on BDD/ATDD/Spec by Example http://blog.zhariy.com/search/label/Specification%20by%20Example

Sorry, most of the text descriptions are in Russian, but most of the video and slides are in English.

The tool are not so important for creating acceptance tests. You still can use TestNG, but you could also take a look on Concordion, easyb, JBehave and so on:

http://en.wikipedia.org/wiki/Specification_by_example

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.