What are recommended scenarios to use Selenium Tests for?

Question

My company is introducing BDD into our projects. Our customers have differents processes-cases wich also may fork at specific points. It is a mostly long and tedious work to test each case and each fork manually.

It was my job to try out Selenium. So, I wrote Selenium tests and freed our poor tester from this task. I put quite some thought into my tests like a helper for returning jobs (i.E. click through pagination until you find link foo, always set deadline in the future, generate new e-mail adress for user bar...) or writing classes for stepstones like interview, then including methods ("fail_interview", etc). In the end, I was able to reuse a lot of the code and could expand the tests, once the fundament was writted, relatively fast along the whole processes.

The tests were very appreciated by our management and the poor tester ;-)

However, not all our developers support the way we use Selenium now. Saying they are almost useless because the tests are not directly connected with our source code and adopting the Selenium tests to changes (like a changed button_id or something) would take too long. I might add, that we never used any kind of testing before but manually.

I agreed to the point, that we also should include TDD into our projects. But now, I am not sure there "my" Selenium test should be positioned.

So, what are recommended scenarios to use Selenium Tests for? Was is a good idea to "click through everything"? What should and should not be tests with Selenium?

Answer 1

I agree that your tests will require ongoing maintenance (as does most test automation). There are tactics for organizing your Selenium tests so that maintenance is easier, but they depend upon how the user interface is written, whether the developers help maintain the tests, and the manner in which the user interface changes. In fact, the quicker the user interface changes, the less it makes sense to write automated tests.

A good rule of thumb with any kind of test automation is to start slowly, choose some success criteria beforehand, and afterward, evaluate those criteria as honestly as you can. For example, when I first heard about Selenium, I was concerned about the maintenance cost and the test's sensitivity to timing issues. I picked a small piece of our application that was fairly easy to automate, that historically had a lot of bugs, and which was time-consuming and error-prone to test manually. After the test was written, I used it for several releases and kept track of how much time I spent updating the tests to keep up with application changes. I also adjusted the test several times to address timing issues. Having gone through that experience, I like Selenium for selected tasks but I would never use it for testing our entire application; it requires too much maintenance effort.

Finally, as of this moment, if you search SQA for the word "Selenium", you will turn up 174 questions. It may be worth your time to read some of them.

Answer 2

Another thought to add to the well-written answers: In the question, the author mentions a developer's objection:

adopting the Selenium tests to changes (like a changed button_id or something) would take too long.

Since UI tests are brittle with respect to the underlying structure of the UI, UI automation success depends on successful collaboration between the UI test team and the developers. Let's say you respond to this objection by saying:

Adapting to a button id won't be a problem, because the SW team needs to establish a standard naming convention for all widgets and then adhere to that convention.

Ok, sounds like a reasonable idea, doesn't it? I've encountered 2 responses to that idea:

(response #1): Oh yeah, that's a good idea. That will make everyone's life easier.

(response #2): What are you, crazy!? I've got enough to do. You'll just have to play catch-up with my development.

If your shop has a lot of response #2 folks in it, UI automation is going to be very difficult. If your shop has a lot of response #1 folks in it, you've got a fighting chance.

So make sure to take devops culture into account when deciding on reasonable goals for UI automation.

Answer 3

I agree with Phil completely when he says that it doesn't sound as though you are using BDD/TDD in your project. This would typically include tests written by the developer, either before or immediately after the code was written.

I can also understand part of the uncertainty on the part of the developers. I've had the argument with many of my developers in the past about my use of code to test code. The argument for that however, is that I'm not using it to test the code, but to check that the things that were working, and not changed, are still working.

Your selenium tests should be used firstly, to test/check the things that are extremely monotonous and need to be done often. In addition to this, try automating the checks that take the most time.

I can fully understand the frustration with an element who's identifier has been changed. My recommendation for that, create a library class for each of these (ie: btnFriendlyName) and choose what value you are going to assign to it. Using a page object pattern, if an ID or Name does change, it becomes much more trivial to correct your automation suite.

I also agree with your having your automation separate from the main code. I've heard a lot of other people talk about having their tests bundled with the application code, but, from my experience, it has worked much better to have separated tests in it's own Version Control System.

It sounds as though you are on a very good start!

Answer 4

Let me take BDD out of the equation for a moment and answer your question about what the recommended scenarios are for Selenium automation. I have found that there are 2 primary areas that UI automation should focus on to have the best ROI:

The first is UI specific functionality, for instance anything that is driven by javascript on the page (including ajax/xhtml requests) or dictated by the html: maxlength on a texbox, for example. There are plenty of tests here that are very basic, and others that can be pretty complex, like if you are testing drag and drop functionality, etc.

The second are scenario tests that cover end to end scenarios that a user would typically go through. These longer scenarios are usually much more difficult to automate at lower levels - either unit tests or tests against the API or WebService themselves. These tests can sometimes be difficult however, depending on the site you are testing they may require additional setup or configuration and you would then also need to automate those processes.

Some scenarios that really are not good for UI automation (but are often attempted) is in depth field validation (it's way too slow), security testing (again, way too slow and doesn't give you the flexibility of hitting the service directly), performance tests, layout tests (trying to validate layout across different browsers, this is way easier to just do manually), Localization testing (there are some things you can do, but in general its not a good idea - see this post: Do you verify presence of text in your automated tests?).

Most other testing can be automated at levels other than the UI. Integration tests for integration with other web services, apis, common libraries, etc., can be automated at the API level, or sometimes even as integration unit tests. Much of security testing can be done using httpwebrequests directly or using existing tools that will execute those requests for you. Stress tests definitely should not ever be done with UI automation, although you can add measuring performance as part of the UI tests.

If you add BDD back into the equation, the arguments against UI automation in that process are all just semantics. It is perfectly acceptable and fairly common to have automated UI tests as part of a BDD or TDD process. The big difference is that they require the code to be compiled and the components to be deployed before tests can be run, but depending on how far you go with your integration tests, that may already be the case anyways.

The other arguments against UI automation in general like maintainability all relate to how you approach your automation. If you want to protect yourself from changes in the pages or behavior or elements, etc then ensure you write your automated tests with the proper layers of abstraction. In selenium, the "Page Object Pattern" is used to overcome or reduce a lot of the problems you may run into as far as maintainability.