Testing updates of external dependencies

Question

We are having a discussion about when we update a NuGet package or another external dependency. If we should update to this version everywhere, so we only have one version on the whole project. Some think we should, others think we shouldn't. The main reason to not update all location is that we need to re-test everything that uses this.

Can we trust external suppliers to test their code, so that we can upgrade minor versions freely? Is having basic unit-test coverage enough to test the changes.

Checklist:

• Does it build
• Check automated test coverage for dependency usage
• Read change-log in a pair to estimate risks

Or there other things we need to consider when updating libraries?

Answer 1

Personally, I'd prefer to have only one level of any given package in use (or one in development and one in production, if you're going to be moving to a new level of code on the next deployment), as it reduces your vulnerability surface in the case of security bugs (I'm thinking of things like the recent bash and imagemagik issues here).

With that said, though, no, I don't think you can trust external suppliers to test their code sufficiently (again, see the above two bugs which were present for years in the code in question).

Reading the change log may not be sufficient warning of changes, either; what is marked as a bug fix may be changing behavior that you believed was correct before, for example.

I'd say that as full a regression test as possible is called for any time you update libraries you depend on. To reduce the risk, you might want to have a test suite specifically for testing the libraries you use, and if they're open source, you might contribute some test cases to their build infrastructure.

Answer 2

While in theory it's appealing to have the same dependencies and their versions for many apps, in practice this often is not feasible. If each app has its own release cycle it's hardly possible to sync them. May work for 2, but good luck synching 10 apps.

So you need to embrace the fact that dependencies are different and they update from time to time. But changes to the external dependencies are as risky as changes to your own source code.

Couple of thoughts to consider when updating:

• When it comes to the dependencies Unit tests are often useless. E.g. you can check how your code reacts on the situations when you emulate events in some framework. But you don't know if that framework would really emit this kind of event. For that you have to test your code integrated with the framework. This is where Component and System tests play an important role.
• Your tests cover your specific logic. While your code won't treat value 15 as something special, the lib may. This is where you can miss an important bug in the dependencies. To mitigate the risks you can employ Randomized Testing. So if for some field you have boundaries from 5 to 30, instead of cherry-picking and hardcoding your lucky number you can pass parameter integer(5, 30). This will use 15 at some point. Maybe not right away, but it's still better than never.