3

I have a smartphone application that uploads files to a server. This server is third-party software (Minio).

This third-party software has a bug which caused uploads to fail. This breaks our smartphone application in a significant way. Sure, we can handle the error gracefully - but critical data is not uploaded.

So, I fixed a bug in the third-party software and now our smartphone application works again.

Question: In order to prevent regression bugs, how do I document requirements and test in a ISO13485 (or ISO9001) compliant QMS?

Ideally, I would like:

  • When something is broken, a test fails (if not, write a test to prevent regression)
  • When something is fixed, the failing test will now succeed (if not, fix the test so it will succeed)

But surely, I should not write tests about the internal bugs and behavior for third-party software? (Or I'll end up writing specs for every SOUP i use!)

2
  • What about a unit test on the Client object in your application repository, describing how it handles errors from the server (500's, 400's, etc...)? Jul 20, 2020 at 16:30
  • 3
    You can't prevent regressions, because that system is outside your control. If they deploy another bug, your system will fail again. I've worked on teams that have run regular "enemy tests" to check whether a third party system is working other than as expected (changed API, fallen over, ...) so we can be promptly aware of it and take that into consideration.
    – jonrsharpe
    Jul 20, 2020 at 16:57

2 Answers 2

2

The short answer: you can't prevent regression in third party software.

More detail:

Third party software, as one of the comments mentioned, is outside your control. Since it's essential to your software, you have a few options in how you handle this.

  • Regular smoke tests - Have a set of smoke tests that cover the essential functionality of the third party system and run on a regular basis. These tests won't prevent regressions, but they will notify you as soon as any occur. Your company can then notify your users that there is a problem with the third party provider and are working with them to fix it as soon as possible.
  • Use a simulator or test API in testing - For your own testing, you can use a simulator or test API to mimic the third party system. This has risks: my experience is that simulators and test APIs tend not to mimic the third party system exactly. I've encountered bugs with the live systems that aren't reflected to the test system and vice versa. It does allow you to continue testing with your application when there are issues with the third party system's availability, so it's up to you whether the payoff is worth the risks.
  • Test failure handling - It should be a given to ensure your system handles failures from the third party system without blowing up, but in my experience that's not always the case. One thing I would aim for with failure handing is a clear statement of the failure. You want your users to see a message that states what happened, and what they should do (e.g. something like "Unable to contact storage provider. Please check {provider's website} for service outage notifications.")
  • Flag third party tests - It will help to flag the third party tests in your system so that you know these are the tests that can fail without any changes to your software.

Essentially, because you can't prevent regression in the third party software, your tests are to inform you of the state of the third party service as it pertains to your software.

1

You can't really account for third party software being broken. It's outside of your control. But you should document it for several reasons:

  1. It might happen again and your documentation will give future developers/testers some insight.
  2. It justifies that the application is broken due to a third party integration. i.e.: it's not your fault.
  3. It creates a paper trail of what you spent your time working on. For instance, if upper management asks why isn't a feature ready yet, you say we ran into an issue in the third party integration which took an entire day to diagnose.

As for tests, in general, you should just test the normal end to end flow of your application works. However, if the bug has a lot of impact (your company is going to get sued/lose thousands of dollars) then yea write a test so that you know immediately when it occurs so you can reach out to the vendor.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.