Feature toggle on production

Question

We're about to release code on production. The code has some features that have not been fully end-to-end tested due to a number of circumstances. Devs have completed that features but then the scope has changed and we have put those features on hold haven't fully tested them (and we don't have time to do it now). We don't want to remove those features from the code base, because we think those features were only postponed and in the future we will be able to reuse them. We are considering two solutions both of which I think are very risky from my QA perspective.

First is having a flag at runtime that that can toggle a feature on or off. I have read that such approach is often used even to test features on production. A limited audience has this feature enabled (e.g. through a flag in request) and can provide feedback before the feature is enabled for everyone. The risk here is that still on production there will be untested code. Imagine a situation you have service endpoints A and B and some internal code C used by B. You have tested endpoint A and not B so you disable B on production. You're now happy but some time in the future A occasionally fails. In the result of investigation you come up with conclusion that A is also using C.

Second approach would be to remove endpoint B at compilation time so that the build artifact doesn't include it. Still, the problem with endpoint A using C occurs.

So my question is toggling unused feature enough to release untested code?

How if devs will be pushing on that feature how can I evaluate the risk and possible help mitigate it?

Answer 1

The short answer is that you can never be 100% sure.

The longer answer is that this is also true for every other part of the product- you test it but can never test it for every possible scenario.

The even longer answer is risk management and mitigation- your developers and architects need to asses the risk in solutions A and B (feature flags vs. #ifdef-ing it), for example if the feature if an isolated module with few well defined API calls into it than the risk is low, but if the feature involves changes across all of the product code than the risk will be high. As part of risk assessment you will also want to think of ways to mitigate the risks, for example regression testing around the affected areas or A/B testing like SIVA suggested.

BTW totally removing the code is also a risk by itself since it might include bug fixes or other implicit changes.

Answer 2

I am a BIG fan of feature toggles. It doesn't take too much work to mitigate the risks, and the benefits are well worth it.

So my question is toggling unused feature enough to release untested code?

No. Feature toggles are not enough to mitigate the risks posed by releasing completely untested code. However, when you test around that code, feature toggles become a nearly perfect way to release with very minimal risk.

How if devs will be pushing on that feature how can I evaluate the risk and possible help mitigate it?

As QA, testing is obviously a big part of how you will help mitigate risk. But just as important, it’s your job to push for action at each step of the SDLC. For the best risk management, everyone on the team must be involved.

Risk Mitigation in 3 Stages

1. Prioritized and Focused Testing
2. Team Kickoff/Planning and Continuous Collaboration
3. Solid Rollout, Monitoring, and Preparation to Act Plan

1. Prioritized and Focused Testing

Obviously in this situation, time restraints prevent fully testing the new feature. Not a problem. With a pinpointed testing approach, you will quickly hone in on and help eliminate potential risks. While a combination of manual and automated testing is ideal, it can be achieved using just manual.

For the sake of this question, I will describe only this scenario – one where the feature toggles will NOT be turned on during this release.

Checking

• Ensure there’s no possible way they could be toggled on production. Limit permissions.
• Ensure there is no trace of the feature. Test manually navigating to urls.

Testing

• Check for irregularities in error quantities and site response time, not just in your code base, but across the product.
  • Perform Automated Testing: Run it often, early and late in the dev process. Run against old code currently on production versus new. Check for differences.
  • Perform Manual Regression: Test these key areas:
    1. Test areas you know are closely connected in code to the new feature. Then those that are likely connected.
    2. Test functionality that impacts a large percentage of the code baseto find any dependencies you may have missed
       • Your developer should be simultaneously getting code reviews to ensure the feature doesn’t reach further across the full product code in any unexpected ways
    3. Test critical functionality as well as commonly used features. What does your app NEED to be able to do? What do most people use it for? What are the potential blockers that are the only reason some users use your site or app?
    4. Test the potential areas of highest risk. This should be a team discussion hunting to prevent data loss, load/performance issues.

To Be Continued As Soon As Possible...

I apologize, but something has come up and I will have to finish the rest of the steps tomorrow. I’ll cover:

1. The best way to implement feature toggles based on the feature scope
2. An absolutely solid plan for rolling out the release and feature toggle.

These are crucial for success, so I hope you get a chance to come back and take a peak.

As a bonus, I promise to share some of my tips and tricks I’ve learned along the way, as well as a few examples of how feature toggles opened up new worlds of possibilities in some pretty unexpected ways.

Answer 3

Very interesting question.

As per my experience as QA , I would like to suggest following :

1 - If you are releasing BETA version then you can stay with new feature and release it with new feature. So you can get more review about from users.

2 - If you are releasing main version then you should not release features which are untested means not tested by QA team. Because to release things without testing is itself a big risk. And it can damage your business reputation , products other function at any level.

If code is done from developer site then tell them to comment that untested feature code so in release that part will not be visible.

Answer 4

This is usually referred as A/B testing. I have personally tested this.

• With Feature Enabled - Testing performed for new feature and regression to ensure nothing is broken
• With Feature Disabled - Regression performed to ensure functionality remains intact. Automation / Required Manual tests will be run in case if this code is going to remain in production in feature disabled mode.

The challenge was when multiple features Feature1, Feature2, Feature3 are there. They could be enabled / disabled independently using a flag / keyword that would enable / disable feature at code level. In this case the combinations to test becomes more and more coverage is needed.

Another option available is (You can always manipulate to show the feature only to a certain percentage of users X%), This way you will be able to understand user responses for it. This limits large scale change for all users but only minimal set of users able to see the change.

When untested code is pushed (it is usually feature disabled - general regression done on code). Never we had enabled feature which was untested. Once it is tested and certified then feature would be enabled in production.

Advantage is you can quickly disable in case if there is any issue. All leading ecommerce sites have this feature running past few years. Now there are products / tools to aid A/B testing. This will help in quicker turn around, better UX decisions and customer experience. QA has a lot more work to ensure the combinations, feature disabled mode, general regression are working fine. Automation plays important role to handle regression for multiple combinations