I need to build a lot of regression tests for our product at work and I plan on using Selenium. I am concerned about building tests that will become quickly outdated and broken based on the fast pace of our product development.
What is the best way to build / structure the tests in a way that will minimize the work needed to keep the tests up to date as the product changes and improves?
You can try building your Selenium tests using the Page Object pattern, more info on: Selenium Wiki - PageObjects.
There are a number of key things that I have learnt are key to have easy to maintain, reliable automated tests.
Map your controls in a single place
Your controls should be mapped into a single location, so that if a control changes, you only need to change it in a single location. Having it in a single location allows you to make a one line code change if a developer re-maps the controls. I also map the control type so that the developer can change from a link to a button, and all I need to do is change this control file.
Abstract the test automation tool where possible
I essentially have three actions that I perform 99% of the time, Invoke, Setvalue and GetValue. Invoke will click a link or button, check or uncheck a checkbox etc. SetValue will put values into controls, like putting text into a text box. GetValue gets data from the controls, which is primarily used for verification.
Each automation action, for example performing a search, will only call these invoke, Set and get methods. This allows almost full abstraction of the underlying automation tool, it also allows you to have a single place in your code where the lower level control specific methods are called.
Use regular expressions to map your control names
A previous commercial tool that I used would cut of control strings at 256 characters. Many applications, like asp.net can dynamically create control names based on their location in page controllers. Using regular expressions lets you define the control in a way that the developer can move it anywhere on the page and your tests will still run.
Create test data objects to hold your test data
I create test data objects that normally map pretty close to 1:1 with the controls on a page, then I create an expected and actual object, then pass these objects around the various layers in the stack. Creating additional tests in some cases can then be as simple as using an existing test with different data.
Separate the test from the implementation
Separation of concerns is a key technical design principle that you should apply from your tests. The intent of the test case should be separated from the physical implementation that does the interaction with the page as much as possible. For example instead of ...
[Test]
public void SearchOnGoogle()
{
using (var browser = new IE("http://www.google.com"))
{
browser.TextField(Find.ByName("q")).TypeText("Stack Exchange");
browser.Button(Find.ByName("btnG")).Click();
Assert.IsTrue(browser.ContainsText("Stack Exchange - Free, Community-Powered Q&A"));
}
}
Your tests should, at the highest level have a domain specific API so.
[Test]
public void SearchOnGoogle()
{
SearchEngine.WebSearch("Stack Exchange");
SearchEngine.Verifcation.WebSearch("Stack Exchange - Free, Community-Powered Q&A");
}
}
...
public void WebSearch(string value)
{
ControlHandler.SetValue(Controls.txtSearchValue, value);
ControlHandler.Invoke(Controls.btnSearch);
}
For indepth information about this topic read Capture The Essence Of Your Test Cases by Michael Hunter
The crucial issue IMO is to acknowledge that your pages will need to be adjusted in order for the tests. To pick up on the Selenium issue, instead of using XPath to structure your selectors, add ids to the pages and use those in the tests.
I've also found the Page Object pattern to give good results:
Within your web app's UI there are areas that your tests interact with. A Page Object simply models these as objects within the test code. This reduces the amount of duplicated code and means that if the UI changes, the fix need only be applied in one place.
I repeatedly apply three principles:
Hide incidental details. Any detail that isn't directly related to the purpose of your test belongs somewhere other than in your test. Hide it somewhere, such as in a variable or in a method, and give it a good name.
Name every important idea. Why does the test code log in in as "F.D.Gumby" rather than as some other user? Because that's the manager who hired the employee you're adding to payroll? Then put "F.D.Gumby" in a variable and call it hiring manager (or whatever), to express its intention.
Eliminate duplication. Each bit of duplicated data is a variable waiting to be born and given an expressive name. Each sequence of duplicated test steps is a method waiting to be born and given an expressive name.
Applying these principles makes automated regression tests far easier to fix when either the requirements or the implementation changes. For further details and examples, see my article "Writing Maintainable Automated Acceptance Tests" (PDF). See also Bob Martin's excellent video of these same ideas.
Discussing your automated test needs with your developers up front can often help.
A developer should not have tip toe around the application with fear of breaking our tests, but if they can proactively make design choices that make it easier for us to maintain them then that can be a huge time saver.
First of all, keep in mind: there will be always some maintenance. Whatever approach you will use, your tests will change over time. Many people will suggest to skip GUI automation as it is very brittle. It is true to some degree; many times it is just something they have heard from someone. Sometimes people say that because they have tried it once, expected for some solution to work out of the box. Sometimes they expected they write some tests once and they will work till the end of time, regardless of app changes. Still GUI tests can be affected by many more things than lower level tests.
As for your tests I would suggest the following readings on the framework design.
First blog from Patrick Wilson
Second would be following materials:
Success in creating a robust Automation framework lies in embedding a Test Framework within your development framework.
One of the common things that is recommended is to create a separate attribute for objects that is unique for each of them (e.g. for a web project anything on the screen is identified by a unique id like TestAutomationdId="HelpMeButton").
This ensures that any changes to that object (other than removing that object) won't affect your script.
In my opinion, automated UI testing is broken by default, the second you create your tests. I'm not gonna get into a lengthy answer as to why, as its limitations are well known already - maintainability being the most crucial one.
I recently attended a talk about the so called "sapient" testing. It means that repetitive, menial stuff that's just following a few steps of a process should be automated, whereas stuff like UI testing, product integration testing should be done by a "sapient" agent - a person.
Its' main philosophy is that automating the work done by a person is actually a degradation of the testing process. Here you can find a few words about it.
So to answer your question - develop fast & agile, build fast & often, execute stable (read: API-based) automatic tests after every build and create a team to handle the UI testing. Each new button added/moved not only means a lot of maintenance of your tests, it means countless more possible ways in which the user might misuse it - are you sure you will be able to keep up with the maintenance and creation of automated UI test cases for each and every possible combination/path?
The most important tip I would give, since it is impossible to create non breaking test cases, is to structure your project so as to be: Flexible & Extensible eg.: - Use ids and use them by retrieving them by a unique property (when it will change you will update only one key). - Create generic methods for actions, never use straight inside your test commands like selenium.click etc. but create another method to do so. In the future you may decide to use a different selenium command, the test case will not have to change, only the method's code. - By creating a test framework with custom methods you will be able to customize the results you will get after the execution and so you may gain time on analyzing your results.
Conventions has made life easier for us! Try to create a standard for how you mark up your pages (HTML) and make sure everyone on your team is aware of the conventions that is decided on. Your conventions should probably include:
I would also recommend using the PageObject pattern as suggested by a few people already.
Selenium is great for testing the UI when the focus is on the UI. If you are interested in testing the functionality I would recommend you use something like NUnit and Moq to automate as much testing of the functionality as you can with code: Mocking HttpContext HttpRequest and HttpResponse for UnitTests (using Moq)
This is a good question, and the gist of it applies to automated tests for other kinds of interfaces (e.g. protocols, APIs, file formats) too. Automation makes the most sense when the implementation is susceptible to change but the interface is stable; conversely, it makes the least sense when the interface is unstable.
I try to be objective about the value of the automated tests vs manual tests; the cost of repeatedly fixing broken tests; the value of the other, less intellectually challenging, manual testing work I could be doing if I weren't repairing automated tests; the amount of test-breaking changes I expect to see; and even how sympathetic the developers are toward building testability into the product. It would be nice to say I could quantify all those variables and plug them into a formula, but really, I can't. I try to be honest with myself, take my best guess, and move forward.
It's hard to make blanket statements about techniques that insulate you from changes, because all changes aren't equal. For example, using page objects might insulate you from structural, semantic-neutral changes within a page, but it will be less helpful, and perhaps even an impediment, when the UI is reorganized in more dramatic ways.
Personally, I don't automate parts of the UI that I think are still undergoing rapid change. I test those parts manually, and then once things look like they're settling down, I re-evaluate. But then I'm the only QA person in our little start-up. If I were a dedicated automation person in a large QA team, I might make different decisions -- although I'd still be skeptical about automating tests against something unstable.
