Validating adherence to software architecture/design after implementation

Question

I work in an organisation that is one of many stakeholders in a large software project.

Both functional and non-functional(quality) requirements have been specified years ago, and the solution is now being delivered.

The problem is that the solution does not deliver the non-functional requirements that the stakeholders initially specified. These relate to resilience, availability, life-cycle cost, flexibility in evolution/development etc.

I understand that architecture and design goals are basically nothing more than a bet that our non-functional requirements will be met.

However, I would like to formally document how and to which degree the implemented solution:

1. Adheres to the specified design and architecture.
2. Fails to deliver the non-functional requirements.

As I understand the ATAM method is primarily focused validating design and architecture in the design phase and not post-delivery verification of a solution. ISVV from looking at the Wikipedia is primarily focused on functional requirements ... Or am I mistaking?

How can I attack this problem and what widely regarded methods are availble to analyze an existing solution according to spec'd design goals and non-functional requirements?

Answer 1

If the non-functional requirements have been documented and have a specified target, then they should be testable. The problem comes in designing the tests to measure non-functional requirements. Resilience of a network, for example, may be specified as how many or what percentage of network connections can be broken before the system stops performing its duties. Availability of a website can be measured in a simple way by the presence or absence of a returned Ping command issued on a regular basis.

If the non-functional requirements have not been documented they are still testable. The cutoff point for success/failure can be found through oracles that would include knowledge of similar systems, human expectation, how the previous version of the system behaved, to name a few. Michael Bolton uses a thought experiment called "Bring Me a Beer" to illustrate implicit non-functional requirements. If you are at a bar and ask a waitress to bring you a beer, and it arrives, the functional requirement has been met. What if the beer costs 10X the expected price? What if the waitress brings the beer an hour after you asked for it? What if the beer is skunky, or has a cigarette butt in the bottom of the glass?