I work in an organisation that is one of many stakeholders in a large software project.

Both functional and non-functional(quality) requirements have been specified years ago, and the solution is now being delivered.

The problem is that the solution does not deliver the non-functional requirements that the stakeholders initially specified. These relate to resilience, availability, life-cycle cost, flexibility in evolution/development etc.

I understand that architecture and design goals are basically nothing more than a bet that our non-functional requirements will be met.

However, I would like to formally document how and to which degree the implemented solution:

  1. Adheres to the specified design and architecture.
  2. Fails to deliver the non-functional requirements.

As I understand the ATAM method is primarily focused validating design and architecture in the design phase and not post-delivery verification of a solution. ISVV from looking at the Wikipedia is primarily focused on functional requirements ... Or am I mistaking?

How can I attack this problem and what widely regarded methods are availble to analyze an existing solution according to spec'd design goals and non-functional requirements?

  • 2
    I believe what you're talking about is just a testing. You have the specifications so you can check if those specs meet the actual state of things. There should not be any special methodology.
    – Alexey R.
    Sep 6, 2017 at 12:12
  • I am trying to test/document the "state of things" that are not part of a solution's observable behaviour. AFAIK normal testing deals with more concrete observable behaviour and properties ... Sep 8, 2017 at 12:43

1 Answer 1


If the non-functional requirements have been documented and have a specified target, then they should be testable. The problem comes in designing the tests to measure non-functional requirements. Resilience of a network, for example, may be specified as how many or what percentage of network connections can be broken before the system stops performing its duties. Availability of a website can be measured in a simple way by the presence or absence of a returned Ping command issued on a regular basis.

If the non-functional requirements have not been documented they are still testable. The cutoff point for success/failure can be found through oracles that would include knowledge of similar systems, human expectation, how the previous version of the system behaved, to name a few. Michael Bolton uses a thought experiment called "Bring Me a Beer" to illustrate implicit non-functional requirements. If you are at a bar and ask a waitress to bring you a beer, and it arrives, the functional requirement has been met. What if the beer costs 10X the expected price? What if the waitress brings the beer an hour after you asked for it? What if the beer is skunky, or has a cigarette butt in the bottom of the glass?

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.