What are some approaches to testing a major software update?

Question

In the past while at work, we've had to do updates to some major libraries/tools we use in building a web app. For example, we recently upgraded Angular versions and are currently updating our Django version (from 1.10 to 1.11). These updates often have subtle changes that can break existing functionality in unexpected ways.

Is there a good approach or set of approaches to test upgrades like this?

Answer 1

I would say simple run your full regression suite of tests, manual or automated. How do you normally verify that your product still works and that developers didn't break existing functionality in unexpected ways?

What my current team does is the following:

• At the beginning of a new Sprint (every two weeks) we update all version of all packages and dependencies.
• We deploy to the testing environment and run all our automated tests.
• Fix any issues we notice or rollback a version and create a task to research this further.
• Create new features for the Sprint and test these new features with high automated test coverage and exploratory testing session.
• If no issues are found we deploy the new feature and the new versions to production

This has a couple advantages and requirements:

• We are always up-to-date and any backwards compatibility issues are found fast.
• We do it as the first thing in the Sprint. This way we the developers are the first to run into any issues while building and testing these new features.
• We demand from ourselves high test coverage, currently unit + integration tests cover 92% of the code.
• We build at least one end-to-end test per feature. Automate the testing of your main happy-paths. Keep the main user flows working at all time.
• We deploy fast, so if we missed something clients will complain fast and we can rollback small changes. Clients haven't found anything yet in this project.

I would call this the continuous delivery with high automated test coverage approach. Something any sane Agile team should do :)

Answer 2

Review your development process to see if you have all the following steps:

         developer upgrades application
                         |
                        \|/
   developer gets tests passing locally in the upgrade branch
                         |
                        \|/
      developer pushes branch to ci and entire suite passes
                         |
                        \|/
      code review by other developers and changes as needed
                         |
                        \|/
         branch merged into master, ensure tests pass
                         |
                        \|/
        deploy to staging and test manually in staging
                         |
                        \|/
                deploy to production.  
     Consider approaches and monitoring for limited traffic

This all depends on having or writing tests.

Answer 3

They are:

1. Make sure you use version control system
2. Make sure you use continuous integration
3. Make sure you have automated tests (for example using Selenium or derivatives) which cover at least main functionality
4. Make sure you have performance tests for regression purposes (for instance using Apache JMeter)

So when you make a change (update library, add new feature, fix defect, etc) you will have confidence that this new change didn't introduce any errors or other side-effects.