Hypothetically, lets assume that I am testing the "Search and compare flights" functionality of this website*, and I want to apply a combinational testing technique.

Where should a tester start this testing process? What should their first steps be?

*I am not really testing this site, this was just chosen as an example for this hypothetical question.

  • I hope Justin Hunter will chime in with an answer to this. – user246 May 29 '12 at 14:34
  • @Justin would you care to answer this one? – user246 May 31 '12 at 21:55

Combinatorial testing is an approach to testing an operation that has many inputs. For that website, I would start by identifying the operations. Sometimes there will be a specification for the system, but often there is not, and in any case the specification is unlikely to match the system in every detail. I recommend reading the specification and then carefully working your way through every feature.

Identifying Operations

This is a busy site. The first thing I noticed is that you can search for things. If you fill in some fields and click a button, the site will respond with some search results.

There are smaller operations taking place within the search part of the page. For example, on the flight search page, if you enter a location in the From: to To: boxes, the site attempts to make the location more specific. For instance, "Austin" becomes, "Austin, United States".

The next thing I noticed about that site is that there are several ads positioned below and to the right of the search box. The ads vary in size. Some are static and others scroll. Some have additional text beneath them.

Identifying The Inputs

Having identified the operations, you next want to identify the inputs. For example, field values are inputs to the search operation, but there are other inputs too: there is a database of items to search (e.g. flights, hotel rooms, or rental cars). The search operation may also use inputs that are not readily apparent to the user: a bias in favor of certain vendors based on the site owner's business relationships, or a bias based on your previous history with the site.

Similarly, the To: and From: boxes appear to have a database of place names. Some place names may be unique (I suspect there is only one Kalamazoo in the United States, but there are multiple places named Austin).

I refreshed the page several times and never say the ads change position, but that does not mean the ad layout is static: for example, the content and position of each ad might vary by time of day or by the viewer's geolocation.

Analyzing The Inputs

Having identified the inputs, you next want to analyze them according to their scope and their interdependence. An input's scope is the set of operations that the input affects. Some inputs are specific to one operation (a field value in a search box), whereas others may affect multiple operations (e.g. the user's previous history at the site or the user's geolocation). Sometimes it is convenient to refer to inputs with a large scope as the environment. For a website, the environment might include such factors as the browser, language settings, timezone, and screen resolution.

An input's interdependence is its relationships with other inputs, i.e. whether the input interacts with other inputs. Sometimes an input's permissible values depends on another input. For example, on the flight page the departure date needs to precede the return date, and on the holiday page page, the country and city inputs depend on the region input.

An even more important sense of interdependence is your own belief in which a system's behavior depends on the interaction of two inputs. If you are certain that two inputs are unrelated to each other, write that down. And conversely, if you are certain that the system behaves differently for different combinations of certain variables, write that down too. That information will help you choose which input tuples you ultimately test with.

| improve this answer | |
  • 1
    +1 Nice! What about other dimensions besides operations? E.g. client environment with possible inputs: browser, language settings, timezone, screen resolution, etc. Those inputs can be a subject of combinatorial testing, too. – dzieciou May 29 '12 at 21:29
  • @dzieciou Thank you for the suggestion. I incorporated it into the answer. – user246 May 30 '12 at 1:30

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.