Defending corner cases

Question

Sometimes, we find specific corner cases that cause a failure in our application - for example, broken relationships between the main entities in our application that are not properly handled (that can be reproduced artificially in development environment and have a low probability of happening in production), an extreme or unrealistic value for an input that a regular user would not even consider to enter, problems under certain conditions in a specific timezone with very specific and uncommon steps to reproduce etc.

Most of times, these cases when presented to management and development team are not taken seriously or completely ignored. In some cases, the reasoning for that is perfectly valid and the consequences and symptoms are not critical and are not signs of bigger problems. But, there are also cases, when we, the QA team, think that a specific corner case is a sign of a bigger problem but we cannot prove it.

What can we do to defend a corner case and make it be taken seriously?

Answer 1

I mentioned it in my answer to your other question about bug advocacy in general, but taking steps to "Uncorner your corner cases" as Kamer, Bach, and Pettichord call it can go a long way.

To summarize Lesson 71 by that name in their book, if you find a bug using an edge case, keep testing it with more and more realistic values to identify the full scope of the defect.

In Lesson 70 "Extreme-looking bugs are potential security flaws" they also point out that "idiots aren't the only people who abuse software", so if an "extreme" case is a potential security issue, you could use that argument.

Also, if you suspect a bigger problem, you could look through previous bug reports and see if there are other bugs that seem related. And in general, keep good notes. If the bug is initially rejected, but you later reproduce it under a different configuration, using different values, etc., then be sure to record that new information. And as Peter mentioned in his answer, definitely bring it back up if you have a report from an actual customer.

It would also help if you can find an ally within the development team--if you don't have the technical skills to determine whether a corner case bug is a sign of a more serious issue, but can walk over, explain your concern to a dev, and look at the code together to confirm or refute your hypothesis, you've just gotten much better information to include in your bug report. Definitely requires the right culture and trust, but is invaluable if you have (or can build) those.

Answer 2

To expand on the other answers:

• Note that something is a corner case - don't be afraid to note that an issue is a corner case and is being documented so that when a customer does encounter it, it's on record as a known issue and any workarounds are available.
• Resist temptation - When you advocate for a corner case and are rejected, it can be very tempting to say "I told you so" when a customer reports the problem. No matter what else happens, resist this temptation. Instead, mention that it was reported, and was not prioritized.
• Know the customers - If you know a customer is highly likely to encounter a corner case, say so. This technique can be particularly effective if the customer is a large organization. In my previous position, we had a handful of extremely large, influential customers. If a problem we encountered in testing was going to impact any of these customers in any way, it would be prioritized higher, so if we knew one of those customers was likely to encounter a problem, we would say so in the bug report, no matter how much of a corner case it was.
• Know customer data - Extending the previous point, get as familiar with your customer data as you can. I've found corner cases that made one specific customer's configuration unusable - if you can identify that customer, and reproduce the problem with that customer's setup, you can demonstrate that the problem will occur in the field.
• For security issues, demonstrate on copies of real data - An argument I've found very effective is "I am not a hacker. If I can do this, so can a hacker." In one case I needed to use a test configuration on the production server to prove that I could hack the system and the problem had to be fixed. I rolled those changes back as soon as I'd made them - but I had to make them to demonstrate that there was a problem.
• Know the legal context - If there are security implications to a corner case, make sure you know the legal context around them. Companies are going to be much more inclined to prioritize a fix for a security flaw that could cost them millions in fines plus massive negative publicity than they are to prioritize what seems to be a minor functional glitch.
• Report, then update - If you're not getting any traction, I've found it helps to report the problem and note that you will update it as you find more information, then whenever you have some time, keep researching and looking for more data. I've encountered a number of apparent corner cases that when researched uncovered serious underlying flaws with much broader implications - but I needed to do that research first.

Answer 3

It is completely valid business decision to not waste limited resources to fix situations which cannot happen in PROD, and so are only of theoretical danger.

Valid response would be to agree with management, but track feedback from customers (and be really good and obsessive about it) and fix the issues reported by customers. In bug, you report ID of the feedback from customers, to prove your point (that it happens in PROD, and not only in your imagination).

After a while, when you develop the reputation that your are concerned about valid corner cases, which are later also reported by real-life customers, your "hunches" would be taken more seriously and "corner cases" might be assigned resources and be fixed preemptively. But only after you develop such reputation, not before.

Answer 4

One additional point that I would like to add to the other excellent answers that I have had to resort to a few times is to ask for the person who is minimising the problem to provide an official documented and signed decision to ignore, postpone or back track the issue for your personal records in addition to the companies records.

When they ask for a reason, they usually will, explain that it is your personal policy in such cases so that should the now known problem cause repercussions later and it ends up in court it will be them in the dock rather than you.

As a lot of my test work was in the Avionics sector and a bug, however improbable, that causes a crash or that was a symptom of a larger problem that did so and could with hindsight have been avoided can result in manslaughter charges in several parts of the world this was especially effective for me. However, there is a growing body of law around the world that tends to mirror this level of responsibility and of course many a stellar career has been brought to a halt by a bug that just causes the company financial loss or mere embarrassment.

As a tester it is your task to locate such issues so as to preserve the companies reputation and limit their liability and sometimes you have to point this out.