Suitability of functional testing to problems with many interrelated inputs

Question

I have read that functional testing is not that suitable when there is a lot of interrelated inputs. Is my understanding correct that e.g. following program would be really difficult to test using functional testing (test cases):
Car configurator with e.g. 20 fields, each of 20 choices, giving 400 possible combinations.

I would say no one would write 400 test cases for that. But while this is still testing its functionality, we still call it functional testing, correct? But isnt it actually risk-based approach when only some combinations are tested?

Additional question: Is exploratory testing functional testing? I would say it is, yet it does not test against any document..

Answer 1

I live this scenario.

My employer's software has somewhere north of 1000 configuration flags, some with minor effects and others with the ability to dramatically change the system behavior. They support defining some 15 different kinds of item for sale, each with different rules of operation, 8 tax definitions (each of which can be defined in one of four configurations), and have multiple sale interfaces.

Since it's literally impossible to test all the possible combinations, we use a lot of automation and our coverage at the moment is approximately 20% of the system - but that 20% is specifically targeted to the configurations and interfaces that most of our customers use most often.

Because of this, we get very few bugs which can be added to our automation suites. The bugs that come in are typically things that can't be automated without a ridiculous amount of effort (a specific piece of hardware is not playing nicely with the software is a typical example) or that it's simply not worth automating (there's several hours of configuration to perform a 1 minute test that involves a configuration used by one customer who has no intention of upgrading, ever).

As far as exploratory testing goes, yes it is functional, but it's not worth getting hung up on terminology. In the example given, a smart tester will explore a semi-random selection of the potential combinations before targeting likely break points: can they configure the car to be a 2 seater with 4 doors? 6 cylinder electronic? And so forth. Often they'll document while testing and use their documentation to expand the functional documentation for the feature.

It's worth remembering that the label attached to any kind of testing is heavily contextual and one tester's functional testing is another's exploratory or risk-based testing. As long as you end up with a record of what was tested, why, and what the results were, I don't think it's all that important to fuss about whether it was functional, exploratory, risk-based, or some other kind of testing.

Answer 2

This seems to be more of a philosophical question than a practical one. The use of the term "functional testing" is not particularly important; what is more important is what you do, and why you do it.

Yes, what you described is a risk-based approach, but the entire testing practice is an approach to mitigating risk. The fact that we "do not test everything" does not mean we are not doing a particular kind of testing. There is always a bigger context that we selectively sample from.

Answer 3

It really depends on the final output and structure of the fields. If the back-end logic for the fields are all the same, testing each field and combination would be a waste of time. There are 20²⁰ (104,857,600,000,000,000,000,000,000) permutations for 20 items in 20 objects, not the 400 you suggested.

If the combination of fields produce unique results and those results need to be validated, creating an automated test would be the way to go.

Your desired results are what should drive this decision. There is always risk in "passing" something in QA when you have not looked at it, but if the backend logic is the same and it has been tested with other combinations, that risk is much lower than a section of the application that has had zero testing performed.