Why is separating functional and non-functional test important (or not)?

Question

There have been several questions lately along the lines of "Is X a functional or non-functional type of testing?". Although I understand the distinction, it's never been an important distinction for me in my career. Either I was on a team that only did one type of testing, or I have been the only tester on the team and responsible for all types of testing (and I didn't make a big distinction between functional and other types of tests).

What are the pros and cons of separating functional tests from non-functional tests? Are the concerns usually more about dividing work in a team, organizing test suites, applying different policies for different types of testing, or what?

Answer 1

Why seperating functional/non-functional aspects of a system is important for me as an tester

I make such an distinction at the very begining of a project (either during drafting the strategy or implementing it). It helps in making non-functional or 'implicit' (another word for overlooked,forgotten or ignored completely) requirements sufficiently 'explict' to everyone involved. This enables me to ascertain and communicate about where (and how much) we are focussing immediately and how we plan to cater to the critical non functional aspects of the system over a period of time (why I say over a period is in the elaboration of non-functional requirements).

This is certainly good; since as "headlights" of an project you want to illuminate everything that lies ahead.

Few examples/benefits of making such distinction and then including it in your plan could be

• As an tester I do not write unit tests (written by devs as a part of TDD) but since unit tests cater to non-functional requirements like testability,maintainbility or extensibilty; I can always make it a part of my strategy and point to the correct reference or to its lack thereof.
• Similarly I am not an security expert but I might highlight security or vulnerability as legitimate concerns and let experts handle that.

Some eloboration on non-functional requirements

Non-functioanl requirements are largely cross cutting and implicit; and once we pay heed to them; it has to be a continuous effort through the project lifetime. Mike Cohn call these "constraints" and says these are similar to taxes which we have to pay in every agile sprint in this blog.

I largely derive the functional/non-functional distinction from Brian Marick's testing quadrants which has been eloborated in the book Agile Testing by Lisa Crispin and Janet Gregory as well. For me business facing tests are mostly functional and technology facing tests make up for the non-functional requirements.

Answer 2

We categorize things because we believe we need to treat the categories differently. It is hard to express an opinion about why separating functional and non-functional tests is important without knowing how one would use the answer. Important for what purpose? Sometimes people ask such questions in these forums because they did not have an answer in a job interview or a classroom. In such circumstances, the question may seem more important than it really is. Perhaps that is the case here.

It seems to me that "non-functional requirement" is just a clumsy way to say "implied requirement" or "unspoken requirement" or "goes-without-saying requirement".

It may seem important to distinguish functional and non-functional testing for budgeting purposes. Some non-functional requirements are treated differently because they have become specializations, e.g. internationalization testing and security testing. Sometimes specializations are outsourced to other groups or other companies because it is too expensive or time-consuming to maintain expertise in-house.

It seems to me that the test plan for a non-functional requirement must be made explicit just as it is for functional requirements. Once the test plan is explicit, the dichotomy does not matter.

Answer 3

I'd see it as more of a tracking issue, I haven't done much non-functional testing (at least I wouldn't call it that) although I could look back and probably say we covered that in some other kind of testing. Mostly when working on a project I look at the Functional Requirements and start basing tasks/tests/checks off that, anything not in the requirements might be considered non-functional but I've never called it that.

If you go by Wikipedia, we've probably all done it: http://en.wikipedia.org/wiki/Non-functional_testing

Although I never quite grasped their definition of behavior for Requirements either. http://en.wikipedia.org/wiki/Non-functional_requirements

Answer 4

I haven't seen it as an important distinction - perhaps because my test teams are small, and are responsible for all the testing on their projects. I don't have specialization to the level of "this one does functional testing" and "that one does non-functional testing".

It's good to understand the depth and breadth of possible testing though, as you decide what you will test for a given project, and what you won't test.

Answer 5

Any time you group tests into a suite of tests you are likely grouping along some category. For example, we have several suites of functional tests that get ran against every daily build. We also have other tests for security, battery, reliability (non-functional categories). Some of these suites get ran less frequently.

Although we don't necessarily differentiate between structural, functional, non-functional, or behavioral tests explicitly, these are categories that help us model our suites of tests so we can design our tests on specific focus areas.