Feature Flags. Reducing risks during shipping changes

Aleksandr MakhometUpwork

http://twitter.com/amahomethttps://www.facebook.com/alexander.mahomet

Feature Flags concept.

Feature Flags Concept

● The basic idea is to have a configuration that defines a bunch of toggles for various features you have pending. The running application then uses these flags in order to decide whether or not to show the new feature. © Martin Fowler

● Other names: feature toggles, feature bits, flags, flippers, conditional features

{% if feature_is_enabled('AwesomeRedButtons') %}

<p>Awesome Feature is enabled</p>

{% else %}

<p>Old boring blue buttons</p>

{% endif %}

Feature Flags ideas

● Gradual rollout● Safe rollback● Less feature branches

Gradual rollout

● Dark launching● Easy to enable for a specific number of

users, even for yourself only

Gradual rollout

● Easy to revert if metrics are not good● Automatic disabling in case of spikes

Less Feature Branches

● Less merge/integration hell● Trunk based development● Easy to mix branches● Essential to Continuous Integration

approach

Who does use Feature Flags

● Google● Flickr● Asana● Travis● Rally● More and more

Feature Flags in Upwork

What is Upwork.com

• Formerly odesk.com

• Upwork has nine million registered freelancers and four million registered clients. Three million jobs are posted annually, worth a total of $1 billion USD, making it the world's largest freelancer marketplace.

• Highload. Microservice architecture

Feature Flags. Our choise.

● Gradual rollout - Yes● Graceful degradation - Yes ● Replace feature branches - Not yet

Implementation. Prerequisites

● As simple as possible● Platform agnostic

Configuration file

● INI format● Each feature has code name ● Naming convention● GIT is used● Each change should be reviewed● Validation script

ShowMickeyMouseOnHomePage.enabled = false

Meta data

● Title● Description● Owner● Responsible Team● Reference Url● Permanent

Constraints

● Internal users● Percent of users (1%, 5%, 30%, 50%, 100%)

○ Random within 0 - 100% but constant per user

● Percent of companies

Create any constraints useful for for your project

How do we update flags

● False for start● Gradual update using Etsy Deployinator

○ Staging -> Production(including validation check)

● App servers update● Placing configuration file to s3 storage

How do we use it in PHP

● Take from S3 Cache to APC and create a backup local version.

● Service and Twig function if ($this->container->get('feature')->isEnabled('Feature')) {

// do something ...

}

{% if feature_is_enabled('Feature') %}

<p>Feature is enabled</p>

{% endif %}

● For more information watch and read Benjamin Eberlei - presentation “Feature Flags with Symfony”

Feature flags Audit

● Important to have visibility on changes● Git history● Write logs and send notification to

chat/email/etc when you deploy

Cleanup

● Cleanup as soon as feature is stable● Keep number of conditions invasions

small● Create follow-up ticket to not forget to

remove ● You can try to make cleanup fun :)

Testing

● Feature Flags could complicate testing● Unit tests

○ Just mock dependencies● Integration/Functional tests

We test combinations which should reasonably be expected to happen in production

Feature Flags admin UI

● Important to have visibility on feature flags statuses

● Such UI should provide readable list of feature flags with possibility to search/ filter/sort etc

● We use it for enabling feature for current session

FF lifecycle

● Add feature flag to config with disabled status● Write code. Test your code in branch with

enabled feature flag.● Release code● Enable feature for your own user. Check it● Enable for internal users. Check it● Gradual enabling for group of users. Monitoring● Enable feature for 100% of users. Monitoring● If everything is fine - cleanup feature flag if it is

not permanent.

Feature Flags and A/B testing

● A/B testing is a kind of randomized experiment with two variants, A and B.

● Feature flags approach is actually an A/B testing

● A/B testing system could be much more powerful

● We separated these systems

Side effects and recommendations

Feature Flags problems

● Increase complexity and decrease the readability of code

● Accidental exposure● Increases complexity of testing

Feature Flags. Recommendations. Part 1

● Keep FF implementation simple and modularized

● Use feature flags at minimum amount of points, avoid very invasive cases

● Avoid mixing feature flags● Do monitoring● Be careful with partial enabling many

features at one time

Feature Flags. Recommendations. Part 2

● Reduce number of permanent feature flags● Do cleanup as soon as possible● Don’t leave code in the mainline just in case

you might need it again some day. ● Changes involving data migrations must be

handled with care● Create clear guidelines, describing your

approach, including section like “when to use toggles or not”

Conclusions

● Feature flags are very handy to roll out new features gradually, but they also give you higher control over your system running in production as well.

● On top of that, they give you a means to disable certain features during a production incident.

● At the same time this approach brings new challenges. So decision is up to you, as always :)

More cool reading

General cool tech reading

● http://code.flickr.net/● https://codeascraft.com (Etsy, Rasmus Lerdorf)● https://www.rallydev.com/blog/engineering● https://blog.asana.com/eng/

Feature flags related reading● http://code.flickr.net/2009/12/02/flipping-out/● http://martinfowler.com/bliki/FeatureToggle.html● https://www.youtube.com/watch?v=KJKSCbYrNvY

(Benjamin Eberlei - Feature Flags with Symfony)● http://blog.travis-ci.com/2014-03-04-use-feature-flags-to-ship-changes-with-confidence/● https://blog.asana.com/2011/04/using-flags-to-ease-new-feature-development/● https://www.rallydev.com/blog/authors/ryan-scott?page=1 (4 articles)● http://swreflections.blogspot.com/2014/08/feature-toggles-are-one-of-worst-kinds.html

Le Fin

Questions?Aleksandr Makhomet

http://twitter.com/amahomethttps://www.facebook.com/alexander.mahomet