When Feature Flags Do And Don’t Make Sense – Software the Hard way

Skip to content

RP

Uncategorized

2019-12-192022-12-26

6 Minutes

Over the past years, I’ve worked in multiple teams adopting very different strategies when it comes to feature flags. I’ve seen the pros and cons of both, and over time, I found myself disagreeing with any fundamentalist position on their use. There is a lot of nuance to this topic, and I think it is worth considering more carefully the various scenarios where feature flags do and do not make sense.

The Reasons For

There are a few major scenarios where feature flags make a lot of sense. The first is when it’s used for A/B testing, where you absolutely do want different behaviors for different users, based on their randomly assigned treatment. I’ve seen this strategy employed extremely well at Amazon where new features are gated by a “feature flag” that is actually controlled by an internal A/B testing framework. The framework randomly exposes some Amazon customers to the new feature, and then monitors their subsequent behavior in order to estimate the business impact of launching the feature.

I was initially skeptical, but was soon won over by how easy the framework was to use, and the valuable insights it provided on the benefits (or drawbacks) of certain features. “Flavor of the month” decisions were replaced with real data. And none of this is possible without the use of “feature flags” to dynamically toggle new features.

Another great use case for feature flags, is when you’re working on a very complex epic that require many different sub-tasks to be completed in different parts of the system. Sub-tasks that are too numerous and invasive to be done in a single pull-request.

In such cases, trying to keep all these disparate changes in side-branches and coordinating a simultaneous merge and deployment, is a recipe for disaster. It’s far more manageable to gate any disruptive changes behind a master flag, merge and deploy all the sub-commits incrementally, and do a flag-flip once all the pieces are in place.

One last use case for feature flags, is when you do not have control over your deployments. For example, consider the Facebook Android app, which contains code contributed by hundreds of different teams, all combined and deployed as a single binary. In such scenarios, performing rollbacks can be infeasible. For practical, political, bureaucratic or even marketing reasons. In such cases, feature flags allow your team to toggle new functionality or mitigate risky changes, without having to rollback or deploy any new binaries.

Someone on Reddit pointed out a similar use-case for feature flags: targeting a very specific launch date for marketing reasons, while still deploying your code much earlier, in order to ensure stability. You can then have a "dynamic" feature flag that automatically enables itself at a specific time. This is also a great use-case for similar reasons – changing functionality in situations where deploying a new binary is impractical.

Risk Aversion

The above are all fantastic use cases for feature flags, but I’ve also seen teams get bogged down by policies that overreach in their use. For example, mandating that every single code change should be behind a feature flag, “just in case we made a mistake”.

Risk management should indeed be a priority for all teams. But there are better ways of doing this than relying on feature flags, especially if your team has control over its own deployments. The vast majority of your bugs should be caught by your automated test suite and/or QA process. And the last few stragglers should be handled using incremental deployments, production alarms and rollbacks.

Besides, as soon as any problem is detected, the recommendation at places like Google is to rollback first and investigate the problem later:

We have seen this at Google any number of times, where a hastily deployed roll-forward fix either fails to fix the original problem, or indeed makes things worse. Even if it fixes the problem it may then uncover other latent bugs in the system; you’re taking yourself further from a known-good state, into the wilds of a release that hasn’t been subject to the regular strenuous QA testing. At Google, our philosophy is that “rollbacks are normal.” When an error is found or reasonably suspected in a new release, the releasing team rolls back first and investigates the problem second

When things are on fire, the last thing you want to do is root-cause the bug and figure out which flag-flip will safely fix the problem. And that may not even fix things – there’s no guarantee that even if your teammate tried to put his changes behind a feature flag, he didn’t inadvertently introduce a bug that cannot be solved by a flag-flip.

Feature flags are a poor man’s alternative to binary rollbacks, and they definitely aren’t a substitute for having a great automated test suite and a robust QA process. If you’re relying on...