How do I make sure I test all commonly used applications alongside my application?

Question

I have a Windows application that works on XP, Vista, Win7, Win8, Win 8.1, etc ...

It's a very complicated program that can have computer slowdowns and compatibility issues with other commonly used pieces of software.

How do I make sure I test all commonly used applications alongside my application?

*The combinations of OS's and applications that could run alongside my application is astronomical and impractical to test EVERYTHING. What are some good ways to go about testing for this type of scenario to ensure you are compatibility with all other software and operating systems out there.

**This could be manual or automated testing

Answer 1

By definition testing EVERYTHING will take INFINITE TIME. Which I bet is less than optimal (unless your boss is OK with that - then you just found safe job for life :-)

So you need to agree with your boss about some metrics to limit and prioritize "everything".

Make a survey among your customers which OTHER applications they use, on which platforms. Or make educated guesstimate according to literature, your specific industry etc.

Then, sort by platform and usage. Spend available resources on most widely used platforms/apps. As more testing resources are available (or you are able to automate part of tests), you can test more obscure platforms/apps. Rinse, repeat.

Plan where your application is expected to be 2 years from now (marketing department has idea), and what usage patterns will be expected then.

Some suggestions: XP is dead, I cannot imagine wasting ANY resources on that. Tests/scenarios developed now should be valuable in the future too. If you have limited capacity now, it does not make sense to spent it on platform with limited future. And so on.

If you cannot agree on priorities with your boss, as "where to start". Only one can be first. And go from there.

If one of your requirement is test automation, and you cannot run automated test on one platform: Good news! You just eliminated one dimension of your problem space! (If the platform is obscure enough to be ignored).

Develop some simple sanity tests for more obscure platforms/app combination, and more detailed/specific/automated for more important ones. There is no rule saying that you need to pay equal attention to important platforms and obscure ones, just the opposite. As any engineering task, it is about finding good compromises (less attention to less important stuff).

Answer 2

Since more or less by definition you don't have unlimited time, you can't test everything.

As Peter says, you need to work out which subset of "everything" gets highest priority. I've got a few suggestions on that front:

• If you have data on what operating systems your customers use, great. If not, get it. You want to know which OSs are most popular with your customer base, and which ones your biggest/loudest/most influential customers use.
• If you have data on the other applications your customers use, also great. If not, get it. Here, you want to know the applications they use most, and - by preference - whether they've experienced any issues when using your software alongside those applications. Mine your issue tracking system and support department for this information.
• Now comes the fun part - combinatorial math and grids. I've found it's easiest to map this in a simple spreadsheet:
  • Column A = supported versions of your software (hopefully there's only one, but I've been in a situation where my team was supporting multiple versions of the software)
  • Column B = Supported operating systems, ordered from most used to least used. I'd count the 32 and 64 bit flavors separately, because they are known to behave differently. If this is commercial (business) software rather than consumer software, you should be making a strong argument to your boss that XP support has to go away. Businesses using XP in the USA are not in compliance with multiple regulations relating to information security and privacy. I don't know the legal landscape elsewhere, but I wouldn't be surprised to find the same kind of issues.
  • Column C = Other applications by major version (e.g. IE 8 gets a separate listing from IE 9, etc.), from most used to least used.
  • Columns D through as many as you feel necessary (optional) - repeat the applications list to cover the use of multiple other applications in conjunction with your software. Practically speaking, you probably only need to consider one or at most two extra columns (since most likely the other applications in use are going to be an email client, an office application, and possibly a browser)

Once you've got this information, the total number of combinations to test is the number of items in A multiplied by the number of items in B multiplied by the number of items in C etc... The time you would need is the amount of time it takes to thoroughly test the application (which itself is non trivial) multiplied by the number of combinations - it's likely to be in the months-to-years range for something like this and guaranteed to be more than your release cycle.

Next, do some mining of your issue tracking system and find the combinations that give your software the most problems. I'd highlight these on the spreadsheet (using color-coding).

The next step is to go over this with your boss and give all the numbers as they stand. The goal here is to prioritize.

If you have automation, you want to have configured virtual/actual machines matching the highest priority scenarios, and run automation against those on a regular basis. Depending on resourcing that means one or more virtual/actual machines, since something like this will need to run through GUI automation to simulate usage patterns and you really don't want to be installing/uninstalling software, much less operating systems. How often you run will depend on how often your application is built and what it takes to upgrade only your application.

If you don't have automation, you'll be targeting a much smaller number of combinations and a much less comprehensive set of tests because you're still testing functional changes and other aspects of your software.