Is it ever OK not to test something before release to a production? What a scenario can be, when testing could not be necessarily before releasing the code?
Yes, there are situations where you can release to production without testing.
- Production systems with users that can miss the systems for a couple of days, might want to recieve features without testing.
- Small code changes estimated to be very low risk
- Contract deadlines which could kill your company
- Having no manual testing environment, leading to we test in production situations. Personnaly I do not like testing environments, I would rather test safely in production with feature toggles, but you need 100% test automation, which does not cover the broad term testing.
- Some companies have a continuous deployment pipeline which deploys new features first to a small set of users and then montiors behaviour and does automated rollbacks if needed.
- Feature toggles hide un-tested functional features from most users, so key stakeholders can play with the features, eg test it :)
- Skipping functional testing: Deploy new features which have good unit-test coverage, which had a code-review to production, it works, but could have functional issues. So you could skip functional testing.
- Skip load testing: We will tackle load issues as they arrise mindsets.
- Skip automated testing: ...
Is it ever OK not to test something before release to a production?
It's totally OK to not test something. As most of the times it's not possible to test everything or to execute all the tests in a given time.
So, One of the major tasks of a Test Manager is to prioritize which tests need to be run before release.
What a scenario can be, when testing could not be necessarily before releasing the code?
You can use Risk analysis to find such scenarios. This requires judgemental skills, product knowledge and experience. You can list out scenarios on the basis below:
- Which are the most important functionality in your product?
- Which parts of the code are most complex, and thus most subject to errors?
- Which parts of the application were developed in rush or panic mode?
- Which functionality is most visible to the user?
- What were the problems caused in earlier such releases?
- What do the developers think are the highest-risk aspects of the application?
- What kinds of tests could easily cover multiple functionalities?
- Which tests will have the best high-risk-coverage to time-required ratio?
James Bach says
"Testing is the process of evaluating a product by learning about it through exploration and experimentation, which includes to some degree: questioning, study, modeling, observation, inference, etc."
Some sort of exploration and experimentation can only be done (properly) in production.
I see basically two reasons that would demand test in production:
1 - You need real user activities: A/B testing, monitoring and logging, etc.
2 - You need production environment, which is very rarely can be reproduced: Chaos Engineering, performance testing, configuration testing etc.
Of course, you usually can perform partial experiments to have a first impression in a more controlled environment - particular for the cases pointed at (2), but while you haven't investigated the behavior in production, you can only suppose how the final user will experience the product.
You can read more details about Testing in Production here.
Another aspect that can guide towards testing in production is the evaluation that the impact of any possible error is bearable by your business. Very rare, but it can happen and it would not make sense to waste money on pre-production testing.
In my experience whether or not to test something before release to production is determined by your employer. At my current employer, if something goes into production and isn't right then QA is asked first, did they test it? If they answer yes they're asked, how did it get approved? If they answer no they're asked why not? Also, my employers have let only the most insignificant things go into production without passing through QA. Even spelling corrections get QA's approval before being promoted to production.