Tips to test product in the production environment?

Question

A lot of experience people says that testing in production is the worst but I observe in testing that a lot of time a few bugs a reproducible only on any one specific environment(like dev, QA, staging, prod...etc) but not in other so what are the tips to test product in production?

Answer 1

That answer is too big to be covered here, and it is highly dependent on the product itself. For an "average" consumer grade, network connected, software upgradable product some keywords you should probably investigate are-

• Testing in production doesn't mean throwing stuff at your users with zero testing, you will need a decent level of tests before you release something.

• In order to test in production, you need a feedback channel, user reviews are the worst kind of such a channel since it comes too late. Telemetry is probably what you are looking for.

• Telemetry raises legal and privacy concerns, but those can be solved

• A/B testing is a way to gradually introducing changes, and be able to revert them if your telemetry indicates there is a problem. You will need feature flags in the code, a framework to remotely manage the experiments (that's a common term used in A/B testing), a way to monitor the results and a little bit of statistical analysis knowledge.

• It might be a good idea to move to a CI/CD model (continuous integration, continuous delivery) model since it will allow you a faster release pace.

• Consider your deployment process and a way to quickly revert changes

Finally, I must stress again- the same way Agile is not "no planning" testing in production doesn't mean "no testing or testers", it just allows you to test in a real environment, and possibly react faster

Answer 2

Testing in the production environment isn’t testing. It’s deploying a product and relying on your end-users to find problems. Which may be fine, depending on the product, the end-user assumptions, and so on. But don’t kid yourself that it’s testing. (This is separate from, say beta testing, where you stage releases to end-users who have agreed to take a incomplete product and use it.)

If problems are being found in the production environment, and not in test environments, then the ideal thing to do is analyze the problems that are being found and determine why they weren’t found earlier (escape analysis). Then you can make an informed decision as to what it would take to find problems in dev/test, and if it’s worth doing so.