Thoughts on pairwise testing

Question

With pairwise testing we're able to reduce exhaustive testing to testing all combinations of pairs. I'm new to a group of software testers testing an huge amount of functions with many input variables. Now my task is to reduce test cases if possible. My problem is with pairwise testing tools I would create more tests than they are already doing, because they're not testing exhaustively (or anything near exhaustive), but just a "few" cases they think are mostly used and know have showed off bugs in the past.

Now I'm thinking about writing down the tests being done and checking whether they already covered all pairwise combinations of this test with some other tests. This would really be a lot of unsystematic work I believe, and I can't imagine there is no better way to help them optimize their work. Furthermore I'm not quite sure if this would work or not. I would appreciate any contribution to my problem.

edit:

consider this scenario: you got to test a software and you already got some testcases with decent coverage for your Problem. Pairwise tools generate testcases covering all pairwise combinations, but thats not what you are looking for, because you dont have the time to test all combinations pairwise tools suggest. Instead I'm thinking about weather its possible or not to drop some of the tests my colleagues created by checking for tests with all pair combinations being already tested in other tests. p.s. does anyone know a tool that could help us with this problem?

edit2:

O.K. now after analysis of the testing-process, I figured out that they are just testing control samples. For example they got 1000 test to do, 4 systems (e.g. first system is Windows XP French, second is Windows 7 englisch etc.) now anyone grabs any feature he wants to and tests it on his mashine (he or she tests that function in just one scenario and if the test passes nobody else has to do it anymore).

I can't imagine how pairwise testing could help them drop any of their tests when they are already testing just control samples while pairwise would suggest to test way more combinations (and testing more or slightly more for better product quality is no option they say). So does maybe anyone see something i might've overlooked or didn't think about?

Answer 1

You might appreciate this podcast from Rex Black who talk about Pairwise Testing:

http://testingpodcast.com/the-myths-of-pairwise-testing-92713/

Answer 2

Pairwise (or combinatorial) testing is a technique to help us systematically produce a subset of tests that test the combinations of variables for 2 (or more) parameters out of all possible combinations of variables. Today there are many tools to help us derive that subset of tests (see http://www.pairwise.org).

Applying this technique assumes the set of tests that are currently applied are providing inadequate coverage, and that the complete set of combinations is impossible to test (mostly due to time/resource constraints).

Pairwise/combinatorial testing is most valuable in complex systems, and these tests are generally very good candidates for data-driven automated tests.

If the total number of combinations is huge (generally the Cartesian Product of all variables), then mapping current tests against the exhaustive list may be more trouble than its worth. Mapping it against a set of tests derived from a tool is likely more feasible.

If your testers already have a set of combinations based on intuition and historical failure indicators that is a good start. An analysis of other 'pairwise' combinations may provide some additional insight.

You might also refer to these posts on combinatorial testing

Answer 3

I'd say that this situation calls for some analysis of the input combinations that are in use with your user base, the total set of input combinations (yes, that part is ugly), the combinations known to be prone to bugs - since bugs tend to cluster - with the ultimate goal being to isolate and prioritize the most common and/or most buggy combinations.

You'll likely find that your team has a good handle on the problematic combinations due to experience with the application. Some data mining of your bug tracking tool will help quantify this information (if you have it - if not, use your team's knowledge as your oracle).

I'd recommend talking to your customer support people to find the most common customer input combinations, particularly since it sounds like your employer doesn't have an actual database of these (I sympathize - I've been there...).

To compare these against the full set of possible combinations I'd recommend using any data analysis tools you can lay hands on. You're going to want to map out the full combination set in some form (this would be an interesting automation project) and then base your priorities on first known buggy, then most common.

Answer 4

I has the similar problem as you; I need test a feature of a network device; the feature will filter out data by upstream/downstream rule, then take 10 possible actions;

the upstream and downstream rule have 127 types; a filter rule can be composed by up to any 8 single rules;

how could I to cover them all ? that's impossible !

so , I call a friend, and ask him how to deal with it ? He gives me a advice as following:

when he faces a big input domain problem during testing, which leads to combination explosion, he likes to break the feature to sub-feature, and build IPM for sub-feature, then use scenario testing to test up-feature which is built on sub-features; when he does this, he will reference the source code to cover every possible path / branch in scenario testing;

I think it's a good ideal, it sounds like to a Google Testing Pyramid ideal;

Because a big IPM model , it's a scenario testing by the nature, so we can break it down to a small unit testing, use less scenario testing, or intergration testing to test whether they can work well together;