What are features of MS Visual Studio Coded UI Test framework that are not present in other tools?

Question

I've recently started working at an organization that has no automated testing in place. They are migrating to Team Foundation Server for project management and source control, and use Visual Studio for development of their web and windows projects.

Because of this, I'm seriously considering Visual Studio Coded UI Tests for automated regression.

Since this is a relatively new product, I haven't been able to find much information on the limitations of Coded UI Tests.

What are the major limitations of the VS Coded UI Test approach? Particularly, in what ways are Visual Studio Coded UI Tests more (or less) effective than the more traditional scripted GUI automation tools like TestComplete, QTP, Selenium and so forth?

Answer 1

To start let me be completely transparent and state that I do work at Microsoft. However, in my role I nor my team uses Coded UI, but I do teach the basics of coded UI in some of my classes.

Rather than comparing the features of each tool set I would recommend including other factors in your decision such as:

• As a general rule of thumb we try to minimize abstraction layers between test code and product code. For example, my team primarily tests APIs written in C++ and our test code is in C++. We do use C# in cases where we have UI or WinRT APIs.
• Use a language your developers are familiar with. Our developers code review test code and run our tests to help debug issues.
• Integration with existing tools. It sounds like your team is already using TFS and Visual Studio, and using the test tools within Visual Studio enables a more seamless Continuous Integration cycle.
• Compatibility. While Selenium would work fine for your web based projects, it is not an option for your windows projects. Coded UI (and C#) can be used to test both. It sounds like QTP can also be used to test both Web and windows projects, but is built on VBScript which has its own limitations compared to C#.
• Consider support. Selenium seems to have really good support, as does the C# community. The Coded UI support is available, and I would be more than happy to put you in contact wtih some folks if this is the direction you decide to go and have questions.
• Extensibility. The ability to extend your tests and your test's capabilities using home built wrappers, P/Invoking Win32 APIs, or the new WinRT api's.

There are certainly other long term and short term costs to consider (immediate training costs, etc). But, in a nut shell, I would recommend not just comparing the benefits and limitations of specific features in existing tool-sets. I would recommend expanding to also consider other environmental factors that will increase efficiency in your whole team and the overall business.

Answer 2

I have used Coded UI Tests, but used Selenium Webdriver much more extensively. In my answer I will completely ignore the record and playback capabilities of both because I would not advocate using either except to familiarize yourself with the tool. In addition, I won't comment on features that one has vs the other because they are very nearly identical. Anything you can do with one tool you can do in some way or another with the other tool with very nearly the same amount of effort.

I use Selenium because it is free, and because until recently Coded UI Tests only supported IE by default. Recently they added support out of the box for Chrome and Firefox, which makes me want to re-evaluate it.

What I will focus my answer on is the ease of use of the APIs. I actually prefer the Coded UI API over Selenium WebDriver's API.

Here are some reasons why I prefer the Coded UI API:

1. Elements are strongly typed i.e. a Link is an HtmlHyperLink object. In selenium, everything is an IWebElement which means that every element has the same properties and methods regardless of whether they are actually useful, or even available. Strongly typed elements make the intellisense much more useful because you KNOW if a method or property is exposed that you can actually use it without a runtime "not supported" error.
2. I prefer HtmlHyperLink myLink = new HtmlHyperLink(...) syntax over IWebElement myLink = webdriver.FindElement(By.CssSelector(...)) This is my preference, it seems more intuitive to me and follows standard object oriented coding practices.
3. I really like having the GetChildren, GetParent, GetDesendants methods.

There are probably a few more, but that's generally it. If you're looking for an answer to "What is Coded UI Tests missing that will make my life painful?", I would say it's not really missing anything, all of the standard stuff is there.

One thing to point out is that I built an abstraction layer on top of Selenium and made the API very similar to Coded UI's API, so it is possible to have all of this and still use selenium, just more up front work.

Answer 3

Well Coded UI Tests are hardly "new", though they were new to me as a framework when I was starting to use them early on. I put them in the category of "record and playback" but with extensibility they do add more on once you are able to get some abstraction in your tests and be able to modify the scripts to become more like a proper coded and programmed framework with repeatability and programmable functions. If you are already using Team Foundation Server (TFS) then you get the added benefit of being able to attach test cases to specific tests, as well as some integration with fixes and if you use it the Agile Test Templates so you can link Use Cases up to tests. Although I never used it as that, since we did not use TFS and I found the coded UI rather limited for my own uses, it does give you an environment that is integrated with what your Developers use and a tool they are familiar with if you need code or test case review.

My limits with the Coded UI were that the scripts were hard to maintain with multiple changes, without a significant (for me) up front effort to get the framework up to a point where I could get my automation to a place I wanted it. Testing against SharePoint I thought would be simpler using Visual Studio tools but this was not to be for me, I ended up going with a different process but it definitely looked like something that would be useful for someone who was familiar with the product and had far more integration to do for different pieces than I have. What I ended up with was a combination of WebDriver in C#, utilizing many of the cross-browser add-ons as well as SpecFlow as my high level Test Driver since that allowed me the extensibility to generate many test cases in a language understandable to business owners; which was a driving need for me.

In some ways the choice of tool also depends on what your audience is, and since you seem to have a lot of Visual Studio integration already it may be a plus for you to follow this and be able to give your Developers insight into the tests with a tool they are familiar with.

Answer 4

In my instance I have been doing Coded UI for about 2 years and have found that Coded UI does more in the background for you than some other products. i have also used QTP and Rational Robot and some free tools. The default Object and step repository is a bit of a pain and I would recomend abstracting it out, mainly because it tries too hard to recognise objects and you end up taking alot of the search and filter properties out that is laborious when you first start. Using it with IE the wait for threads state and the object search 'wait' on the fly are very useful.

Coded UI, I think does more than just look at the DOM so you can interact with the GUI as a use would by point, click and typing in rather than changing the properties of a control to achieve the same action as a user would.

However, no matter which tool you pick it is so important to have a symperthetic Development department that will put unique Id's on all your controls and not put invisible controls over controls and are willing to write GUI's with Automation in mind. I can not stress the point enough otherwise you can spend more time working out how to get arround problems that actually coding tests.

Answer 5

I think instead of comparing feature of Coded UI vs other, it would be better if check your application first. You should not use Coded UI just because that is compatible to integrate with TFS?