Take the 2-minute tour ×
Software Quality Assurance & Testing Stack Exchange is a question and answer site for software quality control experts, automation engineers, and software testers. It's 100% free, no registration required.

With pairwise testing we're able to reduce exhaustive testing to testing all combinations of pairs. I'm new to a group of software testers testing an huge amount of functions with many input variables. Now my task is to reduce test cases if possible. My problem is with pairwise testing tools I would create more tests than they are already doing, because they're not testing exhaustively (or anything near exhaustive), but just a "few" cases they think are mostly used and know have showed off bugs in the past.

Now I'm thinking about writing down the tests being done and checking whether they already covered all pairwise combinations of this test with some other tests. This would really be a lot of unsystematic work I believe, and I can't imagine there is no better way to help them optimize their work. Furthermore I'm not quite sure if this would work or not. I would appreciate any contribution to my problem.

edit:

consider this scenario: you got to test a software and you already got some testcases with decent coverage for your Problem. Pairwise tools generate testcases covering all pairwise combinations, but thats not what you are looking for, because you dont have the time to test all combinations pairwise tools suggest. Instead I'm thinking about weather its possible or not to drop some of the tests my colleagues created by checking for tests with all pair combinations being already tested in other tests. p.s. does anyone know a tool that could help us with this problem?

edit2:

O.K. now after analysis of the testing-process, I figured out that they are just testing control samples. For example they got 1000 test to do, 4 systems (e.g. first system is Windows XP French, second is Windows 7 englisch etc.) now anyone grabs any feature he wants to and tests it on his mashine (he or she tests that function in just one scenario and if the test passes nobody else has to do it anymore).

I can't imagine how pairwise testing could help them drop any of their tests when they are already testing just control samples while pairwise would suggest to test way more combinations (and testing more or slightly more for better product quality is no option they say). So does maybe anyone see something i might've overlooked or didn't think about?

share|improve this question
    
I'm not entirely clear on this... What exactly is your question? "What is the purpose of pairwise testing?" –  user867 Oct 17 '13 at 6:22
    
consider this scenario: you got to test a software and you already got some testcases with decent coverage for your Problem. Pairwise tools generate testcases covering all pairwise combinations, but thats not what you are looking for, because you dont have the time to test all combinations pairwise tools suggest. Instead I'm thinking about weather its possible or not to drop some of the tests my colleagues created by checking for tests with all pair combinations being already tested in other tests. p.s. does anyone know a tool that could help us with this problem? –  Robert.Sie Oct 17 '13 at 7:43
1  
@Robert.Sie - could you please update your question with this information? It helps clarify your question. –  Kate Paulk Oct 17 '13 at 11:32
1  
@Robert.Sie Ah, so your question is "Is it possible to exclude tests from the full set dictated by pairwise testing methods if those tests would be redundant with already existing tests?" If so, then the answer is yes, it is possible; Though it may take some careful analysis to determine what your existing tests already cover, depending on how they're constructed. –  user867 Oct 18 '13 at 5:39
    
No. My Question is: Is it possible to exclude test from the set of tests my colleagues created if some of those tests would be redundant considered pairwise methods. –  Robert.Sie Oct 18 '13 at 6:57
show 2 more comments

3 Answers

up vote 0 down vote accepted

You might appreciate this podcast from Rex Black who talk about Pairwise Testing:

http://testingpodcast.com/the-myths-of-pairwise-testing-92713/

share|improve this answer
1  
Can you summarize the podcast here? –  dzieciou Oct 17 '13 at 20:36
add comment

Pairwise (or combinatorial) testing is a technique to help us systematically produce a subset of tests that test the combinations of variables for 2 (or more) parameters out of all possible combinations of variables. Today there are many tools to help us derive that subset of tests (see http://www.pairwise.org).

Applying this technique assumes the set of tests that are currently applied are providing inadequate coverage, and that the complete set of combinations is impossible to test (mostly due to time/resource constraints).

Pairwise/combinatorial testing is most valuable in complex systems, and these tests are generally very good candidates for data-driven automated tests.

If the total number of combinations is huge (generally the Cartesian Product of all variables), then mapping current tests against the exhaustive list may be more trouble than its worth. Mapping it against a set of tests derived from a tool is likely more feasible.

If your testers already have a set of combinations based on intuition and historical failure indicators that is a good start. An analysis of other 'pairwise' combinations may provide some additional insight.

You might also refer to these posts on combinatorial testing

share|improve this answer
    
+1 for the second paragraph, which cuts to the heart of the matter. –  user867 Oct 17 '13 at 6:21
add comment

I'd say that this situation calls for some analysis of the input combinations that are in use with your user base, the total set of input combinations (yes, that part is ugly), the combinations known to be prone to bugs - since bugs tend to cluster - with the ultimate goal being to isolate and prioritize the most common and/or most buggy combinations.

You'll likely find that your team has a good handle on the problematic combinations due to experience with the application. Some data mining of your bug tracking tool will help quantify this information (if you have it - if not, use your team's knowledge as your oracle).

I'd recommend talking to your customer support people to find the most common customer input combinations, particularly since it sounds like your employer doesn't have an actual database of these (I sympathize - I've been there...).

To compare these against the full set of possible combinations I'd recommend using any data analysis tools you can lay hands on. You're going to want to map out the full combination set in some form (this would be an interesting automation project) and then base your priorities on first known buggy, then most common.

share|improve this answer
    
Hey Kate, thank you for your reply. As im not a softwaretester and really new to all of this i probably didnt understand you correctly. I think the procedure you are discribing is just what my colleagues have been doing so far. They figured out what cases are most common/been buggy in the past and planned their test with that in mind. My task is to apply pairwise testing in order to minimize testcases. Again im sorry if i didnt understand you correctly and thank you for your quick reply. –  Robert.Sie Oct 16 '13 at 12:24
    
Hi, @Robert.Sie, in this case you probably want to apply the pairwise algorithm to the test set that your colleagues are already using. If they've already done the analysis to determine the most common and most risk-prone settings, the pair-wise algorithm applied to those settings should allow a reduction of the number of tests needed - but you should do one pass where half of you run the usual set and the other half run the pair-wise set: this ensures you won't reduce your effectiveness. –  Kate Paulk Oct 16 '13 at 16:48
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.