Why is separating functional and non-functional test important (or not)?

Question

There have been several questions lately along the lines of "Is X a functional or non-functional type of testing?". Although I understand the distinction, it's never been an important distinction for me in my career. Either I was on a team that only did one type of testing, or I have been the only tester on the team and responsible for all types of testing (and I didn't make a big distinction between functional and other types of tests).

What are the pros and cons of separating functional tests from non-functional tests? Are the concerns usually more about dividing work in a team, organizing test suites, applying different policies for different types of testing, or what?

Answer 1

Why separating functional/non-functional aspects of a system is important for me as a tester

I make such a distinction at the very beginning of a project (either during drafting the strategy or implementing it). It helps in making non-functional or 'implicit' (another word for overlooked, forgotten, or ignored completely) requirements sufficiently 'explicit' to everyone involved. This enables me to ascertain and communicate about where (and how much) we are focusing immediately and how we plan to cater to the critical non functional aspects of the system over a period of time (why I say over a period is in the elaboration of non-functional requirements).

This is certainly good; since as "headlights" of a project you want to illuminate everything that lies ahead.

Few examples/benefits of making such distinction and then including it in your plan could be

• As a tester I do not write unit tests (written by devs as a part of TDD) but since unit tests cater to non-functional requirements like testability, maintainability or extensibility; I can always make it a part of my strategy and point to the correct reference or to its lack thereof.
• Similarly, I am not a security expert but I might highlight security or vulnerability as legitimate concerns and let experts handle that.

Some elaboration on non-functional requirements

Non-functional requirements are largely cross cutting and implicit, and once we pay heed to them; it has to be a continuous effort through the project lifetime. Mike Cohn calls these "constraints" and says these are similar to taxes which we have to pay in every agile sprint in this blog.

I largely derive the functional/non-functional distinction from Brian Marick's testing quadrants which has been elaborated in the book Agile Testing by Lisa Crispin and Janet Gregory as well.

For me business facing tests are mostly functional and technology facing tests make up for the non-functional requirements.

Answer 2

We categorize things because we believe we need to treat the categories differently. It is hard to express an opinion about why separating functional and non-functional tests is important without knowing how one would use the answer. Important for what purpose? Sometimes people ask such questions in these forums because they did not have an answer in a job interview or a classroom. In such circumstances, the question may seem more important than it really is. Perhaps that is the case here.

It seems to me that "non-functional requirement" is just a clumsy way to say "implied requirement" or "unspoken requirement" or "goes-without-saying requirement".

It may seem important to distinguish functional and non-functional testing for budgeting purposes. Some non-functional requirements are treated differently because they have become specializations, e.g. internationalization testing and security testing. Sometimes specializations are outsourced to other groups or other companies because it is too expensive or time-consuming to maintain expertise in-house.

It seems to me that the test plan for a non-functional requirement must be made explicit just as it is for functional requirements. Once the test plan is explicit, the dichotomy does not matter.

Answer 3

Any time you group tests into a suite of tests you are likely grouping along some category. For example, we have several suites of functional tests that get ran against every daily build. We also have other tests for security, battery, reliability (non-functional categories). Some of these suites get ran less frequently.

Although we don't necessarily differentiate between structural, functional, non-functional, or behavioral tests explicitly, these are categories that help us model our suites of tests so we can design our tests on specific focus areas.

Answer 4

Let's consider the difference between the two:

• Functional Requirements (FRs) tend to be explicitly requested by the customer/end-user, and are usually expressed in terms of "it must do this" or "it mustn't do that"
• Non-Functional Requirements (NFRs) tend to be vague or generic (eg meet the ISO 12345 standard) or things that experience reminds you to do (eg use a good coding standard, consider security etc).

As the analyst capturing the requirements, we treat them separately because at the requirements analysis level they are different things... but once collated into the specification (in whatever form) they become The Requirements.

Therefore, as a tester, the distinction is unnecessary. What matters is that your tests cover all requirements (functional as well as non-functional), and sufficient test cases exist to demonstrate that all requirements have been adequately fulfilled.

In fact, even as a developer, the distinction is blurry too - what matters is that all requirements are met by the code (or the development process).

--

Of course, in an Agile team, one often wears multiple hats, but when only wearing a tester's hat:

Although I understand the distinction, it's never been an important distinction for me in my career.

I would agree with this.

Answer 5

I'd see it as more of a tracking issue, I haven't done much non-functional testing (at least I wouldn't call it that) although I could look back and probably say we covered that in some other kind of testing. Mostly when working on a project I look at the Functional Requirements and start basing tasks/tests/checks off that, anything not in the requirements might be considered non-functional but I've never called it that.

If you go by Wikipedia, we've probably all done it: Wikipedia article - Non-functional testing.

Although I never quite grasped their definition of behavior for Requirements either. Wikipedia article - Non-functional requirement.

Answer 6

I haven't seen it as an important distinction - perhaps because my test teams are small, and are responsible for all the testing on their projects. I don't have specialization to the level of "this one does functional testing" and "that one does non-functional testing".

It's good to understand the depth and breadth of possible testing though, as you decide what you will test for a given project, and what you won't test.

Answer 7

It's important to make sure both are covered.

Separating them helps ensure that.

It's not so much about the distinction and definition of each being different - others address that - it's more the simple fact that the separation approach makes sure each are approached correctly for their purpose.

Answer 8

Every software testing company performs functional and non-functional testing in projects. The main reason for the separation of functional and non-functional testing is that coverage should be done as per the requirement of the project.

Functional testing ensures that the requirements or specifications are properly satisfied by the application. Whereas Non-functional testing is a type of software testing that is performed to verify the non-functional requirements of the application. It verifies whether the behavior of the system is as per the requirement or not. It tests all the aspects which are not tested in functional testing.