What tools work well for automating .NET 4.0 web sites using web forms with dynamic data?

Question

What would be the best solution to accommodate automated testing of web forms with code-behind (.NET 4.0)? Specifically, we need a solution that can automate web form testing and handle dynamic data such as database generated IDs across multiple posts.

A little about us

Team: 15 .NET developers, 10 database staff and 1 QC manager (zero QC staff other than manager). Tools Used: TFS, MSDN with Visual Studio Ultimate Tech Used: 99% of our huge codebase is web forms (with code behind/beside) with websites in .NET 4.0 (NOT WebAppProject, NOT MVC). 99% of our apps are dynamic in nature, very minimal static content.

The QC manager is new to our company but comes from previous employment where he had staff with detailed knowledge of http://www.fitnesse.org/. We currently have some automated testing for common code that is manually built to assemblies (not using a build server), but as mentioned above the bulk of what we have is webforms. The web forms are tested manually using MS Word docs and screen shots of manually using the testing server website.

Answer 1

A lot depends on what you want the front-end automation to do.

Given that you've got a lot of web forms with code behind, I'd honestly consider starting with the unit test framework built into Visual Studio, and using that to test the data handling (I'd recommend taking a look at Channel9's TechEd videos for an idea of what you can do with Visual Studio for testing - they're excellent starting points).

For functional testing of web site front ends, you've got multiple options. The database-generated IDs complicates things a little, but can be worked with - if you know the naming pattern or have some other combination of attributes you can use to find the fields you want, you can work with it with any tool. The big caveat here is that you say you have no test specialists other than the manager - the manager is unlikely to have the time to build an automation framework.

Here's how I'd handle things in your situation as I understand it:

1. Work with the developers to get unit tests exercising all the code-behind routines.
2. Use something that's either free or in-house available to build basic UI tests - my choice here would probably be either Selenium 2 or CodedUI - both are code-focused and both are close enough to unit tests that the developers could write the tests with the input of the test manager. These would be as cross-browser as possible (CodedUI runs on multiple browsers but unless you work with a code first solution you need to create the tests with IE).
3. Leave the look and feel testing to be done manually. Humans do this kind of testing better than computers.

Here's a summary of some tools that could be used (mostly the tools I've used or know about):

• Selenium - This is free, open source, and has a large user community. Multiple languages are supported.
• Microsoft CodedUI - This isn't free, but your developer licensing already gives you full access to use this solution. Given the dynamic nature of your sites, I'd recommend you extend with something like the Code First solution. If you're using TFS for issue management and test plan management, you've got integration automagically built in. CodedUI tests can be written with VB or C#.
• HP Quick Test Professional - This is very much not free, but the tool has an extremely large user base. I haven't used it myself, so I don't know how well it adapts to dynamic web content. The only language supported is VB Script.
• SmartBear TestComplete - Another non-free product (but substantially less expensive than QTP). I've used this one extensively, and it can be used with dynamically generated web forms if you're prepared to do a bit of extra work with it. Unlike QTP, TestComplete supports script flavors of multiple languages so if your code behind is C# you could use a script version to build tests.
• AutomationAnywhere TestAnywhere - This is an image-based product that will take screencaps of fields and uses a proprietary script-like language to build and run tests. I doubt it would deal well with your scenario as I found it extremely difficult to work with elements other than the ID to identify fields.

Answer 2

I assume your web application is for external customers (so your users can use web app in many browsers and operating systems outside of your control). Which is very interesting challenge - exactly what we are doing :-)

You are excellently positioned to use new future W3C standard for browser automation, Selenium Webdriver (Se 2). (In a way, you future-proof your QA). It allows you to test your web based app in different browsers, even in parallel. You can avoid building your test farm (virtual machines running different operating systems and browsers) by renting one from cloud providers like SouceLabs. Selenium has bindings for many popular languages, including .NET (We use Python).

And the language for your QA test don't have to be the same as your web app. I found Python much more productive (as dynamically typed "scripting" language, compared to C# or Java) and speed cannot be a concern in QA. Your concern is "speed to market" to develop test, and design patterns (PageObject) to separate page logic and locators from test logic. In fact I know about Wall Street company using Ruby for QA testing their C++ app.

Fitnesse has similar approach as Selenium (and was part of Selenium1), but was using HTML tables to define tests. So your QC manager will feel at home with Selenium Webdriver (similar terminology, method names etc).

I am not sure what would be best test runner for .NET, we use Jenkins (which is in Java), but we are Linux/Python shop as I said.

Read up about PageObject and other best practices to use Selenium, google around or check this.

Answer 3

I can share my experience.

• I initially do functional testing to understand the data flows, pre-requisites, dependencies. Based on this I try to come up with reusable data scripts.
• Example. If data need to be populated for last 2 days, the script would be like (getdate()-2, values). With this every time when you run you would get required data
• Custom Store procs - By having custom stored procs, limited set of web service calls to initialize the system, pre-requisites
• Example - In a order processing system, order creation, updation, cancellation are different flows. For each one, the pre-requisite is (Order creation - No pre-prerequisite, updation - open order exists, cancellation - open order exists). This is an example to identify options to initialize / load the initial data set
• I would suggest analyze workflows, DB states, Identify, isolate, develop and test scripts to reduce manual efforts, slowly this collection of scripts would help you arrive at a semi automated approach and provide better clarity for automation
• Take a Database snapshot of the test data setup, Restore it every time and run your tests. This way it is one time setup and reuse it every time
• For UI testing Telerik Test studio, Coded UI you must be aware of. My suggestion is automation evolves from scripts which eliminate repetitive manual tasks and reduce time for test execution

Answer 4

1. If you're using Visual Studio Ultimate the best option would be using Web Test project type, it can easily handle dynamic stuff and can be converted to Load Test if necessary.

2. If you need browser-based testing, the best choice would be Selenium Webdriver which has .NET/C# client binding.

3. If options 1 and/or 2 for some reasons are unsuitable you can look into Apache JMeter testing solution. It has capability to do correlation (extract and reuse dynamic parameters), provides necessary Assertions and can be used either for single-threaded functional testing acting pretty much like a real browser or can be utilized to create a load test to identify application bottlenecks. ASP.NET Login Testing with JMeter guide seems to be the best place to get started with.