Good strategy for automated regression where initial state can't be guaranteed or predicted

Question

The situation

I need to start building automated regression for a very large web application with thousands of users. It is impossible to start from a known state, or even to be sure that data entered into the application will be saved to a known state. The application is a payroll entry system, where all data is read only if payroll has not been started, and has an insane number of configuration choices.

The application can't be run on a local database with a local setup: it requires multiple interlocking web services (which use - don't ask me why - the system registry to determine where to submit their results) and is not practical to reset (a database refresh can take all day). The test sandbox environment can be accessed by multiple users and configurations changed in ways that completely alter the expected behavior of the system. the development sandbox environment is often out of action for days at a time when large projects are being worked on.

The strategies I see as viable

The potential strategies I'm considering are:

• Set my initial conditions first - This is - alas - a bit more complex than it looks. I would need to make either several database queries or do a number of screenscrapes to determine whether the company has had its configuration changed, and whether I need to start a payroll run, change the configuration back to what it should be, then submit the false run before starting a clean run with the correct configuration. This would also complicate end-of-test checks because I would then need to find and discard the initial submit file in order to check the validity of the one with the data my tests actually entered. On the plus side, the values saved to the pending transaction tables prior to submit and the contents of the submit file would match expectations.
• Dynamic checking of results - by this I mean the much more complex coding needed to check results based on which configuration flags are active when I test, regardless of whether they're what I expect to see. The advantages I see here are that I wouldn't need to try to force the system to the settings I was starting with, and I'd be building in an oblique check that certain fields display only when certain configuration flags are enabled. On the flip side, this is a great deal more complicated to code properly, and as a result, much more error-prone.
• Hybrid approach - where I set certain key values but let the rest do whatever someone else set them to. This would give me some of the disadvantages of each method, but would allow me to limit the extra complexity to areas that tend to be moving targets anyway.

Some other considerations

I have (after six months experience with the system) a reasonable grasp of where the system interacts with web services (these are all built and maintained in-house, but tend not to get much development. They also typically get called from the web application or poll the web application database, so testing them separately isn't on the agenda at this stage). The web services are integral to the system to an extent that it's not possible to simply test only the web application.

The application is written is classic ASP. There is no way to separate front end and business logic. It is not unit tested because it is not unit testable. While there is a plan to replace it with a better engineered system, I will be testing this system for at least another 2 years, potentially longer.

Given these "interesting" complexities, what approach should I take to building automated regression for this application?

Terminology Update In response to @user246 requesting clarification of some of the terminology I used:

• submit the false run - by this I mean to submit a payroll that contains nothing but a change of configuration settings where I don't actually care what gets sent.
• submit file - This one is a bit more complex. All the calculation logic and check generation logic happens in a back end server I have no access to. My only interface to the back end server is a fixed length format text file: the submit file. This file is generated for every payroll submission, with a defined naming convention. When there is more than one payroll submission in a short timeframe for a given customer, it can be difficult to find the one that contains the information I'm interested in.
• pending transaction tables - between the start of a payroll run and submitting the run, everything that happens is recorded to one of a number of database tables. They're known as the pending transaction tables because changes to configuration, payroll, and personnel data are not final until everything has been synched between the web application and the back end server. Until a payroll run is submitted, changes aren't permanent, and depending on the logic involved can exist only in the pending transaction tables.

Answer 1

While the entire system isn't under your control (which is clearly the superior option), it might still be possible for some slice of the system to be under your control.

You say it's a payroll system.

Can you create a "Test Company" that everyone else would agree to leave alone? Can you create "Test Employees" that everyone else would agree to leave alone?

In the cases where I've had to share systems, that's the approach I have taken.

Each tester (manual or automated), had their own subset of the system for their exclusive use. When naming the objects, we preceded them by the initials of the tester.

My "company" would be "JSS Corporation" My "employees" would have names like "JSS Smith", "JSS Jones", etc, etc.

My automation would typically begin by setting up these companies and employees, with the attributes as I needed them (Some single, some married, some full-time, some part-time, etc), then proceed to the other functional steps I needed to test.

Answer 2

From your questions these are Application Related Info / Challenges

• Very large web application with thousands of users
• Data is read only if payroll has not been started, and has an insane number of configuration choices
• Needs to connect to Remote DB / DBs
• Multiple interlocking web services
• Written is classic ASP
• No way to separate front end and business logic

This is a very application Specific Need but I can share some ideas on what worked out for me in a similar situation

• Initial DB set-up was required for (Environment Clean up, Populate / Set-up minimum required Configuration). Having a initial clean-up, pre-requisites setup on DB worked for me

Option #1 for Business Logic Validation

• Validate Cases which can be handled separately by calling stored proc / web services if possible
• Pre populate the required data in DB to minimize UI interactions if possible

Option #2 for Business Logic Validation

• Run your initial DB Setup
• Record and play back required actions using Selenium. The UI actions performed by the app will be handled via selenium. I have not tried to automate ASP app. I hope the id's are available and application works.

Option #3 for Business Logic Validation

• Run your initial DB Setup
• Create stubs custom methods which can call / perform UI functions. You need to duplicate the code. This would be difficult to maintain in my perspective

Other options are, You can target UI automation for

• Run Db Prerequisites / Setup
• Validate Elements at UI level
• Running an End to End Flow at UI Level using Selenium
• At UI level we can automate the priority cases (not all cases can be handled)

I would suggest trying out a prototype to evaluate what works / identify challenges / road blocks at every stage. This would be better to take call which option is viable

Answer 3

My preference is definitely to have the automation be the one in control of the configuration and test that the application is behaving as expected depending on the configuration. If you go with the "Dynamic Checking of Results" approach, you're missing a large portion of test coverage around the app's behavior based on the configuration. With that in mind, here are some options I would pursue to attempt to build something where that is the case:

1. Rather than check to see if the configuration has changed, why not start every set of tests off with a config change to force it into the state you want to test? I could see a number of classes of tests with different setup methods that change the configuration and then test against that specific configuration. If I'm doing this in setup I would try to create a setup method that sends http traffic directly rather than going through a UI, as it would be more reliable and faster.
2. Request an Automated Test Sandbox environment be set up with limited permissions that you can guarantee to be kept in a clean, known state.
3. When you say the Test Sandbox environment can be accessed by multiple users, who are they? Other people at your company? Also, when the configuration is changed, is the configuration system wide or are there many "Customers" set up on a single system each with their own configuration? If there are multiple Customers, can you create one specifically for automation in your test sandbox environment and create a process around it that basically equates to - don't touch this customer - it's for automation only?

Is this really a non-technical problem you are trying to solve with a technical solution? I would probably switch into lobbying/politics mode to get what I actually needed rather than trying to build around the limitations. Only when I was sure I had hit the end of the road would I start considering other technical options and I would be vocal about how it impacts myself and my team.

Hope that helps.

Answer 4

I agree that this seems to be a mix of an organizational and technical problem. Without going into the technical part, I would recommend the following:

1. I would start by thinking of all possible solutions to your problem and write them down. Even if you would rule a solution out at first thought because you think it is not feasible for any reason, write it down without evaluating it. A brainstorming session would be good for this step.
2. If you have a list of solutions, try to think of advantages and disadvantages of each solution (which you already did for some technical alternatives) and write them down.
3. Think about and estimate the needed resources for each solution: Infrastructure (hardware), people, time to implement the solution, cost.
4. Use weighted criteria to evaluate each of the solutions. As a result you have a score for each of your solution alternatives.

It may take a while to get to a list with which you are satisfied. When you are, pick the top two or three candidates and talk to your boss or management about these: Explain/illustrate each of the alternatives, their advantages and disadvantages, needed resources, and recommend to them which alternative is the best in your opinion.

Perhaps you can suggest to do a small prototype implementation for two alternatives, gather experience and then decide for one of them.

Technically I think having a test environment which is under your control, that is, nobody else is using it, should be part of the solution. It doesn't have to be all new hardware, but you could negotiate time frames for your exclusive use of an existing test environment. Of course you need to configure and refresh the environment at the beginning of each of your time frames for your needs. For example, try to get some working days in each week for your exclusive use for implementing and testing your automation. Also get some nights or weekends for some testing which lasts a bit longer.

Maybe you have thought of all of this before, or it was even self-evident for you, but hopefully you find something helpful in my answer.