Should Selenium 2 WebDriver tests of CRUD operations be self-contained and independent?

Question

I am currently writing Selenium 2 WebDriver tests for an ASP.NET MVC 2 website (the tests are written in C# and run with NUnit as the testrunner). I'm familiar with unit testing (and have written unit tests for this project as well). Therefore, I know that unit tests should each be self-contained (should leave the system in the same state that it started in, whether it passed or failed) and independent (order of the tests should not matter, and the tests can be run one at a time or as a full suite without getting different results).

I've been attempting to do this with my UI regression tests as well, but I've been running into cases where doing so seems like I'm doing a lot of repeating myself. For example, in testing the CRUD of something called, for example, a Product. The self-contained tests would go something like this (just listing the steps here, not code):

Test Log On:

• Setup: None.
• Test: Navigate to the Log On page, enter log on information, click submit, verify that log on and redirect was successful.
• Tear Down: Log off.

Test Create:

• Setup: Perform the logic from the Log On test.
• Test: Navigate to the Create Product page, fill in form values, submit, verify that product is created and redirect was successful.
• Tear down: Delete product, log off.

Test Update:

• Setup: Perform the logic from the Log On test and the logic from the Create test.
• Test: Navigate to the Edit Product page, change some values, submit, verify that the update was successful.
• Tear down: Delete product, log off.

Likewise for testing delete, log off, etc.

As you can see, there is a lot of duplication between tests - the actions that are completed as the "Test" part of one test are duplicated in the setup or tear down of other tests. It would reduce duplication greatly if, instead of making each test self-contained, the CRUD tests were always run together - first the log on test, then the create, the update, the delete, and finally the log off test. However, this obviously goes against the very core of unit testing.

Given how many websites are out there with basic CRUD operations, I imagine this situation must come up for a lot of developers. So my question is: what is best practice in this situation? Is it better to have a bunch of independent tests, which overlap one another to varying degrees in their set-up/tear-down code? Or, in the case of automated UI testing/regression testing, is it ok (or even preferable) to have tests depend on one another? Or as a third alternative, should all of the CRUD tests be rolled up in one giant CRUD test?

Or is there some approach that I don't know about that is better overall? I've read about things like PageObjects, but unless I'm missing something, PageObjects really just refactor common actions into stand-alone methods/classes (which I completely agree with for ease of code maintenance and DRY principles), but they don't change the fact that you are still calling the same bits of logic over and over again in setups and tear-downs, rather than logging in once, creating one product, updating one product, deleting one product, and logging off once.

Answer 1

Blobinator, welcome to SQA.

For the purposes of your question, it may help to think of a UI test as two overlapping activities: interacting (e.g. clicking/typing) and verifying (confirming that the site behaves correctly). Interacting and verifying overlap because it may not be possible to interact in the way you intended unless the site behaves correctly. For example, if clicking on the Edit button on page 1 does not take you to page 2, you cannot interact with page 2. Despite the overlap, interacting and verifying are also distinct; for example, the navigation buttons may do the right thing regardless of whether the data on the page is correct.

Your tests should be independent in the sense that a test's success or failure should not depend on whether some other test preceded it. This does not mean that the tests will not share code or share common workflows. If you are concerned about unnecessary work, you might consider coding your tests so that a test can choose whether to verify a particular step. It sounds like you may be doing some of this already.

There are arguments in favor of broad, end-to-end tests as well as more narrow, action-specific tests. End-to-end tests are a better reflection of how you think the software will actually be used. Narrow tests are better for problem isolation. If you are a developer who is pressed for time, and you have a QA team, and you want to choose between concentrating on narrow tests and broad tests, I would recommend concentrating on the narrow tests and leaving the broad tests to your QA team, because they will need to do broad tests manually anyway.

Answer 2

+1 to user246's answer. This is in addition to what he already outlined.

Some other things to keep in mind: You can do setup that is not UI automation, so if you can call a rest API or some other API directly to create your product and save time and save going through the UI steps, go for it (unless of course you are testing that part of the UI). I often create an interface for things like this and implement the UI and non UI version so I can easily pick which one I want to execute in my test.

Reporting is the main reason for breaking things out completely into separate tests. If you get enough info from the failure to know exactly why the test failed and to be able to report on it reliably (get all of the failures for the last month in log in) then stringing together tests may be OK for you, although I still generally try to avoid it.

Answer 3

I write all of our CRUD tests into a single class, broken down into dependent methods. I create and assert the record is present, edit the record and assert the changes persisted, edit and then undo and assert the changes did not persist, then delete the record and assert it is no longer present.

This leaves the application in the state it was when the test began. Like you stated, it is not "unit testing", but it makes the most sense so you are not running the same test 4 times.