Building Better FLOSSCommunity Relationships @ FB

DevConf.cz 2017Phil Dibowitz & Davide Cavalca

Production Engineering: Operating SystemsFacebook

Disclaimer:

Lots of teams at Facebook work withopensource and are awesome at it!

Who are you two and why do we care?

Agenda

● Open Source at Facebook● Why do this?● How do you do this?● Our journey, via examples!● Lessons Learned

Open Source @ Facebook

Open Source Team...● Shepherds projects● Builds tooling● Provides guidance/input

Open Source @ Facebook

● Facebook started as a LAMP stack!● HHVM● MySQL● Kernel

Open Source @ Facebook

● Hbase– Forked early…

– Years later, opensource version:● halved avg latency● long tail latency reduced by > 90%● single server failover times cut dramatically● Reduction in long GC pauses

– Now back to upstream – much work later

● Zookeeper– Similar situation in progress

● Chef– Have kept up with development and deployment

– Stayed entirely on open source release

– Great results for 4 years...

Open Source @ Facebook

We also develop our own open-source software in the community...

● Presto● OSQuery● React● RocksDB● McRouter● Buck● Surround360 Processor

Why Build Relationships?

Why? It’s efficient!

● Maintain less internal code– Less man-hours!

● Be able to contribute code– Help community

– Share ownership

– Feel good

● More eyeballs (better code)– More reviews

– More testing

Why? Less waste!

Keeping up with the community means…● Less time wasted…

– Debugging solved problems

– Filing useless bugs

– Porting fixes around

● Less animosity

Why? Direction!

● Know what’s coming soon (... to a theater near you)– Be prepared

– Be part of that discussion

● Influence community direction (eventually)– Start those discussions!

Why? People!

● Happier internal engineers● More engaged community● Better recruiting

● … good relationships are contagious!

How?

Build Social Capital

How? In Person (where possible)

● Conferences● Meet-ups● Their office● Your office● Wherever!

How? Participation

● Be on mailing lists / in SLACK / on IRC… and join in the conversation!

● (again) Go to relevant conferences/meetups● Understand high level project goals● Understand work flows

How? All the context!

● Listen first to get their context● Provide your context before large PRs● Remember: find appropriate medium for the

length and type of conversation

How? Determine speed

● Following “latest”● What’s right for project?

– Consider their release cadence, dependencies, development velocity, etc?

● What’s right for the time?– Consider your resources, deployment cadence, and

rate of contribution you expect to make

How? Finally

Be Honest

Our journey, by example

Common Considerations

● Goal● Speed● Size of change● Previous engagements with the community

A good example: Chef

● We’d already built a relationship with Chef● … but we had a business relationship too.● Goal: (initially) scaling, (later) various features● Speed: custom bleeding edge builds early on… now

quarterly● Got involved in community events and discussion● Benefits: IPv6, multipackage, scaling, systemd support● Also: released various tooling and cookbooks

A place to start: Anaconda

● Goal: Single installer across versions, use newer code● Flew out to meet

– Started with: how to be good community members

– Ended in hackathon

● Kept in touch in IRC● Speed: built from master ~monthly● Benefits: single installer, vastly newer hardware support,

solid IPv6 support, not working in abandoned codebase

Continuing the Journey: systemd

● CentOS 7● Goal: newer features/code, cgroup2● Speed: latest release● Backported (rawhide to CentOS 7) – shared● Conferences● Benefits: cgroup2, mkosi, etc.

Other Examples

● initscripts/network-scripts– IPv6 fixes & features

– Per-device routes

– Re-whitespace

● rpm/yum/dnf– rpmdb scaling issues

– want to help test at scale

Lessons Learned

● Listen first & understand their context● Provide your context● Non-code contributions

Resources

● Rawhide → CentOS 7 backports– Github: https://github.com/facebookincubator/rpm-backports

– (RH) Copr: https://copr.fedorainfracloud.org/coprs/jsynacek/systemd-backports-for-centos-7/

● Kernel blog post– https://code.facebook.com/posts/938078729581886/improving-the-linux-kernel-with-upstream-contributions/

● Systemd talk– https://www.youtube.com/watch?v=LhYd0S3qiMY

● Chef Tooling:– https://github.com/facebook/taste-tester

– https://github.com/facebook/grocery-delivery

– https://github.com/facebook/chef-cookbooks

● MySQL: https://github.com/facebook/mysql-5.6/● HHVM: http://hhvm.com/● Presto: https://prestodb.io/● RocksDB: http://rocksdb.org/

You’ve got questions?

We’ve got answers!(maybe)