Certify in writing that your software will work flawlessly

Question

A client pays you for a job and on delivery he wants you to assure him in writing that your software will work flawlessly.

What if that software is for an airplane;do you deliver it with a notice of 'use it at your own risk' or disclaiming warranty? Wouldnt the client become anxious?

What if that software is for ATM's.How do you certify its flawless operation when it's possible that it can suddenly make the ATM misbehave to hand over extra bills?

In other words, how do you respond to a client request like that, and as already said, how do software houses that build programs for airplanes hand over their software to the client? with strings or no strings attached? Are the clients accustomed to the possibility of fault so that such a request is simply irrational?

Answer 1

A client pays you for a job and on delivery he wants you to assure him in writing that your software will work flawlessly.

It is not possible. Quality is a subjective characteristic - it depends on an individual evaluation about what is and what should be. And testing is the evaluation of a product, under someone's (the tester) perspective.

Now, you can demonstrate to someone that a behavior happens using automated checks - it removes (almost completely) the human aspect of the equation. However, as said, it can only demonstrate what happens - not what doesn't happen - therefore, it cannot ensure that the product is flawlessly.

In other words, how do you respond to a client request like that, and as already said, how do software houses that build programs for airplanes hand over their software to the client?

Any software contract establishes some aspect of success or not. For critical or highly-regulated industries, some standards and external evaluations are required - and you test your software (and create automated checks) with those regulations in mind. Why? Because it is one of the subjective important quality point-of-views of the product.

Answer 2

It should be clearly communicated that failure is an option to avoid the client having any misconceptions. There is no flawless or perfect software. There always be bugs, unexpected behaviors, and corner cases.

how do you respond to a client request like that

You could give a test summary report where you may show what was tested, what activities were performed and under what conditions, manually or in an automated fashion; what business requirements were met and user stories checked and, probably, most importantly what was "in scope" and "out of scope".

Answer 3

Our job as testers is to assure quality and highlight any differences between the requirements and the final product... which means you can't (and shouldn't) guarantee that there are no defects. This is especially true in more complex cases like aviation or finance.

This is why we write test cases, track execution, raise bugs, and obtain sign-off from the decision makers. As the business (or client, in this case) are ultimately the ones signing the testing off then they're taking full ownership of the associated risk instead of you.

You've done your job, it's now up to them to make the decision.

Answer 4

The short answer is, you don’t.

Longer answer: first “works flawlessly” is meaningless. To start with, you need a spec. It should define, with as little ambiguity as possible, how the program should behave. When testing, you test to make sure the software meets the spec. You may well find things the spec doesn’t cover, and need to have things clarified/expanded/whatever, but you need some sort of way to determine what the correct behavior is.

Second, your software is part of a system. And even if the software is flawless, the system can fail. Cosmic radiation is a real thing, it does flip bits randomly. Very rarely, but it happens. More realistically, your program is probably running on top of an OS, or at least on top of hardware, and that hardware can and will fail in different ways. And if the any part of the system changes (patching, hardware upgrade, OS upgrade, etc) it may well cause your program to break, in ways you couldn’t have found before. (This is why some medical equipment still runs on Windows 95, and some ATMs still run OS/2.)

So, you don’t promise flawless behavior. You do say that on this hardware, with this spec, we validated this behavior.

Depending on the importance/severity of failure, there should be system level design that helps improve reliability. Things like multiple execution engines, and/or parallel execution, where multiple programs have to agree on what they should do. Or keeping a human in the loop. Or ECC RAM. Or having auditors. Its up to the user to decide what level of assurance they need.