Quality assurance when people's lives are at stake

Question

We are currently working in an insurance claims processing space and even though we have high quality standards, testing and quality assurance overall has always been, honestly speaking, a supporting non-critical mechanism from a high-level position of our management.

But, I was always wondering, what it would be like to do testing and quality assurance for something much more critical in terms of what is at stake - aviation, aerospace engineering, healthcare, software for nuclear power plants..

The cost of an error is enormously high - we can all remember the famous cases caused by software: Therac 25 incident or the recent Schiaparelli Mars lander crash - resulting into the loss of people's lives, loss of years of work of many people, money loss and many more negative consequences.

How does knowing that your product under test may potentially lead to serious consequences affect a tester and his or her work? What is it like to participate in high-stake testing?

Answer 1

How does knowing that your product under test may potentially lead to serious consequences affect a tester and his or her work? What is it like to participate in high-stake testing?

I test software that handles sterile processing of surgical instruments. Although our software isn't directly responsible for the sterilization, it is responsible for tracking the instruments through the sterilization process. Therefore, bugs have the potential to indirectly affect the patient.

Thankfully, most of the features I've worked on aren't involved in the sterile processing (they aid in the processing, like reports or production managers).

However, when I do get near software that can directly interfere in peoples lives, like when I tested the HL7 ordering functionality...

...Nothing is really that different.

Nervousness is a part of testing. I don't mean sitting there biting your fingernails and then losing sleep at night because your testing may or may not have tragic consequences. I mean that I try to be nervous, on a smaller scale, that what I'm testing doesn't work. That small amount of nervousness is enough to keep me on my toes and not get mislead into the "Everything is fine!" fantasy.

If I catch myself feeling in the shadow of impending doom, I take a step back and reassure myself that I can do a good job if I bring my calm (mostly), collected (mostly) testing mindframe to the task.

Having the criticality of the software at the back of your mind is helpful in the meta-testing decisions (Is this bug critically important? Which feature is the priority? Is my testing Done?), but I find the moment-to-moment testing activity is oblivious to it.

In-fact, the feeling of testing something well is sometimes the best remedy for the stress the responsibility brings! I've previously felt my nerves and concerns about the impact of our release slip away as I mercilessly pursue an evasive bug.

I'll end on a comment that these are my views. The views of others I work with are probably different. I see people get very antsy around nervous times, but I try to remind them that being nervous isn't going to get bugs fixed.

Good testing will.

Answer 2

Such domains are often regulated

In some such industries there are external regulations that require certain processes to be in place, not only for testing but also for the rest of the development cycle. I work in the medical device industry, which in the U.S. is regulated by the FDA. The FDA has three "classes" of medical device based on the level of risk: Class I, Class II, and Class III, with Class III devices being things like implantables that have the highest risk.

The types of things the FDA requires (and if you get audited, you must provide documented evidence for), are:

• Reviews of design artifacts (such as requirements and tests) in addition to code
• Traceability of requirements to tests (and for the higher classes, traceability must also extend to design and code)
• Objective evidence of the testing that is performed
• Validation of any third-party software used
• For products involving hardware, ensuring final testing is performed on production-ready hardware as opposed to earlier prototypes

There is more but I'm not an expert by any means, and as developers we operate based on internal procedures that are derived from the FDA guidance by our regulatory and legal teams, so I can't get too detailed without spending a lot of time reading the actual FDA legal-speak.

Some resources if you care for more details, though there's a lot to wade through:

• Overview of Medical Device Classification and Reclassification
• CFR - Code of Federal Regulations Title 21

---

I'm not sure what regulations require it, but we also have stringent requirements when it comes to defects found in fielded products, whether by customers, or by our internal team after a product is shipped. If any employee discovers or hears about a defect from a customer, they are required to enter it into the system within a short amount of time. If it's a defect that has caused, or could cause harm, it may well cause a "stop ship", recall, or field update.

---

My experience working in this environment

Coming out of school, I had read about the Therac incident and the one field I didn't want to work in was medical devices. Well, here I am. While I haven't worked on any Class III devices, to be honest it hasn't been as scary as I imagined. The regulations, while sometimes requiring more paperwork and process than seems necessary, mostly require things that are good ideas and common sense anyway. I have the privilege of working on a very good team that was committed to quality even before they started building clinical devices; I believe this team commitment is the biggest factor in actually achieving quality. We celebrate finding bugs (because every bug we find and fix is a bug that doesn't harm a customer or patient), but practices such as code reviews, design reviews, pair programming, etc. help prevent many defects before we even have to find them in testing.

When I was an intern at this company, I once ran an overnight experiment on hardware, which my manager walked by in the morning to find the remains of a rather spectacular crash. When I got in, I had an instant message from him saying "so I guess I can call you a certified <product name> engineer now :)". It turned out my simple experiment had uncovered a firmware bug that led to the crash, which a couple more experienced devs then helped me investigate. Essentially being congratulated for my first crash, rather than blamed or scolded, definitely made an impression. Having a blame-free environment takes a lot of the pressure off, especially when the risks and stakes are high.

---

Some of the downsides/risks

I will add, though, that with such regulations there is more opportunity for abuse of power. What we call our "quality" group is really more about regulatory compliance: they're looking at whether we have all our documentation in place and ensuring our testing is compliant with regulations, versus doing the actual testing. Since we can't ship a product without having that documentation in place, this group is effectively a gatekeeper. However, since many of the actual regulations prescribe more of the "what" than the "how", much is often left to the interpretation of the quality staff assigned to your particular project. While every quality person I have personally worked with has been very reasonable and good to work with, I have heard horror stories about other groups in our company, like documents being rejected for using the wrong font, claiming documents are missing required information without providing details about what that information should be, etc.

In such situations, engineers must often dive into the actual regulations to understand what is an actual requirement versus the whim of the particular quality staff you're dealing with. Otherwise the "compliance" or "policy" card will trump the "common sense" card every time.

Also, if you fail an audit by a regulator like the FDA, the company may have to put "remediation" efforts in place that require extra process, and the actual developers and testers may have little say in what is actually a useful remediation versus busy work that makes your process look better without actually improving quality.

Answer 3

I have never participated in such the testing however I think the systems which are critical to ones' lifes are designed in the way of high-modularity, self-diagnostic, functional duplication of components and higl level of component interfaces' formalisation.

I suspect that when a tester tests some particular component they might even be not aware of what would be the E2E case that component would participate in and what would be "the whole" that component is going to be integrated into (or at least have just a guess). So I think the testers do not put either more or less effort that they would put into testing of a, say, log-in form. Moreover I believe that the most part of testing is performed without human participation, and in the parts it isn't the components are tested several times by different teams and people to minimize a human-factor and make people feel comfortable in aspect of possible missing defects (as they would know that this feature will be re-tested by others)..

Functional testing in such systems should not differ a lot from the one we all know and love (hate). But the majority of test and dev resources, I think, are put into system design and architecture (like design a car) as well as into the final e2e tests (like destroy several cars in crash-tests, or ride for 100000km on a stand) or component-integration testing level (like test if the steering-wheel is well-joint to the wheels).

Basically everything should be pretty much like in regualr testing but more formal, standardized and expensive.

Answer 4

Ultimately it is what you feel like at the end of the day, and if you can sleep at night. I have worked with many people that tested medical (Class 1 thru III), and weapons systems. It depends on how well you can manage your sense of responsibility. Remember, QA is only one part of the TEAM and missing something legitimately that was not thought of is not YOUR responsibility, at least alone. There are PhDs and doctors all along the way that are supposed to be doing their jobs too. You are expected to run the testing flawlessly, maintain integrity, and ask questions.

Answer 5

It's a management issue, honestly.

I test an enterprise OS; I'll admit it not at the level, of, say, pacemaker control software, where an individual person/people could die if something goes wrong, but it is at the level of companies failing/people loosing their life's savings and so on. And because it backs almost every financial transaction on the planet, and many airline operations, and other similar transactions, so there's also something along the lines of bad performance leading, in the aggregate, to many lifetimes spent waiting when they didn't have to.

As Alexey says, in a lot of ways, it isn't that different from development/test of any other software, but we do put a lot more effort into it. "Good enough" isn't really an option; we either have to do it right, or not offer the product/function at all. So designing a product that you have to reboot on a regular basis isn't something we do. We spend a lot of time designing and testing error detection and recovery processing; there's more code devoted to error recovery paths than basic functionality in many components, and a lot of our testing is testing error conditions, not happy paths.

We also have to spend a lot more time/resources testing at the system level; one of the things we've been noticing recently is that because the hardware and software is so resilient, it takes a perfect storm to cause problems. It's actually somewhat of a problem; the systems are so reliable that when something does go wrong, the end users don't know what to do, as they probably haven't seen a failure in years. So we've also been putting in a lot more proactive system monitoring/response functionality as well, as we can't trust a lot of end users to do the right thing.

But at some level, it's a question of how much the business is willing to spend. As I said, we're noticing now that it takes a large number of things to go wrong to actually cause a system outage, and after a certain level of complexity, you can't test every possible combination of things going wrong. Reaching successively more reliable systems is exponentially more expensive.

And, as others said, there are regulations/laws to deal with as well; we have to make sure we can prove that we've tested certain things if necessary.

On a day to day basis, I don't think I feel any different than any other tester would; I get frustrated when I can't reproduce problems, I get a great deal of pleasure from tracking down bugs that were hard to find, and I worry about escapes. But I also push back harder than I might in other areas when I think something needs more testing, and I always worry about whether I was responsible if I hear about an ATM network outage or the like.

Answer 6

It requires a bigger sense of attention to detail, respecting the process, and not being afraid of saying something. I've done aircraft maintenance and looked over things, and it's simultaneously rewarding and terrifying, knowing that this is going to hold someone inside of it, and your work has made a difference.