How should I document tests and requirements when third-party software is broken?

Question

I have a smartphone application that uploads files to a server. This server is third-party software (Minio).

This third-party software has a bug which caused uploads to fail. This breaks our smartphone application in a significant way. Sure, we can handle the error gracefully - but critical data is not uploaded.

So, I fixed a bug in the third-party software and now our smartphone application works again.

Question: In order to prevent regression bugs, how do I document requirements and test in a ISO13485 (or ISO9001) compliant QMS?

Ideally, I would like:

• When something is broken, a test fails (if not, write a test to prevent regression)
• When something is fixed, the failing test will now succeed (if not, fix the test so it will succeed)

But surely, I should not write tests about the internal bugs and behavior for third-party software? (Or I'll end up writing specs for every SOUP i use!)

Answer 1

The short answer: you can't prevent regression in third party software.

More detail:

Third party software, as one of the comments mentioned, is outside your control. Since it's essential to your software, you have a few options in how you handle this.

• Regular smoke tests - Have a set of smoke tests that cover the essential functionality of the third party system and run on a regular basis. These tests won't prevent regressions, but they will notify you as soon as any occur. Your company can then notify your users that there is a problem with the third party provider and are working with them to fix it as soon as possible.
• Use a simulator or test API in testing - For your own testing, you can use a simulator or test API to mimic the third party system. This has risks: my experience is that simulators and test APIs tend not to mimic the third party system exactly. I've encountered bugs with the live systems that aren't reflected to the test system and vice versa. It does allow you to continue testing with your application when there are issues with the third party system's availability, so it's up to you whether the payoff is worth the risks.
• Test failure handling - It should be a given to ensure your system handles failures from the third party system without blowing up, but in my experience that's not always the case. One thing I would aim for with failure handing is a clear statement of the failure. You want your users to see a message that states what happened, and what they should do (e.g. something like "Unable to contact storage provider. Please check {provider's website} for service outage notifications.")
• Flag third party tests - It will help to flag the third party tests in your system so that you know these are the tests that can fail without any changes to your software.

Essentially, because you can't prevent regression in the third party software, your tests are to inform you of the state of the third party service as it pertains to your software.

Answer 2

You can't really account for third party software being broken. It's outside of your control. But you should document it for several reasons:

1. It might happen again and your documentation will give future developers/testers some insight.
2. It justifies that the application is broken due to a third party integration. i.e.: it's not your fault.
3. It creates a paper trail of what you spent your time working on. For instance, if upper management asks why isn't a feature ready yet, you say we ran into an issue in the third party integration which took an entire day to diagnose.

As for tests, in general, you should just test the normal end to end flow of your application works. However, if the bug has a lot of impact (your company is going to get sued/lose thousands of dollars) then yea write a test so that you know immediately when it occurs so you can reach out to the vendor.