Large Scale Open Source Development Models: A Comparative Analysis

LARGE SCALE OPEN SOURCEDEVELOPMENT MODELS

A COMPARATIVE ANALYSISBy Joe Gordon

WHYSaw OpenStack grow from around 60 developers to over 2,100developersUnusual development modelBut how do other projects solve the same problems?

WHY IS PICKING THE RIGHTDEVELOPMENT MODEL

IMPORTANT?

ACCELERATING GROWTHTime till 100 contributors

Linux: 1991 - 2 yearsLinux 2.0 had 190 contributors in 1996 in credits

OpenStack: 2010 - 1 yearDocker: 2013 - several months

over 300 contributors in its first year200 contributors per month

Linux: 1991 - June 2004 (13 years)Debian: 1993 - March 2007 (14 years)OpenStack: 2010 - October 2012 (2 years)

ACCELERATING GROWTHLinux, Debian, Docker, OpenStack (clockwise from top left)

source:

openhub

OPEN SOURCE IS BIG BUSINESSOpen source is the new standard bodiesBalancing corporate interests

Linux foundation Gold and Platinum Members

PICKING THE RIGHT DEVELOPMENT MODELCONWAY'S LAW

organizations which design systems ... are constrainedto produce designs which are copies of the

communication structures of these organizations

DEVELOPMENT MODELSPROJECTS COVERED

Linux KernelApache Software FoundationDebianOpenStackDocker

LINUX KERNEL

LINUX KERNELTime based release100-150 subsystem maintainers

Chain of trustDecentralized review process

Rolling development model

CULTUREChain of trustAbout the individualValue frankness over politenessCorporate friendlyNo single company controlsNot much automated pre commit testing

Failing testing is very bad for author

APACHE SOFTWAREFOUNDATION

APACHE SOFTWAREFOUNDATION

ASF is more of a governance umbrella and cultureSeparate projects

4,400+ committers150+ top level projects

flat (ish) trust model'Review then commit' vs. 'commit then review'

I n o r d e r t o r e d u c e f r i c t i o n a n d a l l o w f o r d i v e r s i t y t o e m e r g e , r a t h e r t h a n f o r c i n ga m o n o c u l t u r e f r o m t h e t o p . . . e a c h p r o j e c t i s d e l e g a t e d a u t h o r i t y o v e r d e v e l o p m e n to f i t s s o f t w a r e , a n d i s g i v e n a g r e a t d e a l o f l a t i t u d e i n d e s i g n i n g i t s o w n t e c h n i c a lc h a r t e r a n d i t s o w n g o v e r n i n g r u l e s .

CULTURELazy consensusFocus is on the team

All decisions are team basedFocus is on contributors not companiesNo monoculture

Within the ASF we worry about any community whichcenters around a few individuals who are working

virtually uncontested.

DEBIAN

DEBIANWhen its ready, not time based. Notoriously slowPackage Maintainers: No review, trust maintainers more

CULTURERotating leadership (elections)Do-ocracy: An individual Developer may make any technical ornontechnical decision with regard to their own workOpen developmentIndependent not 'profit-driven': no imposed decisions by who has money,infrastructure, peopleno benevolent dictator, no oligarchyIt is all about the individual (although individual's can form groups)Territorial

OPENSTACK

OPENSTACKcontinuous delivery + stable releases

No rolling developmentBreak down into services and build teams around each service

5,000 commits per month from 500 contributorsStrong centralized review process (two core reviews per patch)Automated testing to reduce reviewer burdenHaving trouble with scaling the team responsible for a single repository

Can't get past 15 or so members on a core team

CULTUREGroup over individualEgalitarianElectionsWelcoming to new contributorsCorporate friendlyNot controlled by single companyLazy consensusDecentralized designUniform tooling/process across projects

CULTUREOPENSTACK'S 4 OPENS

Open Source, not open coreOpen DesignOpen DevelopmentOpen Community

Lazy consensustechnical governance is a meritocracyput everything in the public

FACTORS LIMITING GROWTHCross project issuesTeam sizeSingle vision

DOCKER

DOCKER'Github' development model

DOCKERRelease every 2 months from stable branch (master isn't frozen)Scaling

split into multiple reposMaintainers / subsystem maintainers35+ maintainers in Docker

1 ) T h e y s h a r e r e s p o n s i b i l i t y i n t h e p r o j e c t ' s s u c c e s s .2 ) T h e y h a v e m a d e a l o n g - t e r m , r e c u r r i n g t i m e i n v e s t m e n t t o i m p r o v e t h e p r o j e c t .3 ) T h e y s p e n d t h a t t i m e d o i n g w h a t e v e r n e e d s t o b e d o n e , n o t n e c e s s a r i l y w h a ti s t h e m o s t i n t e r e s t i n g o r f u n . "

T h i s " c e l l u l a r d i v i s i o n " i s t h e p r i m a r y m e c h a n i s m f o r s c a l i n g m a i n t e n a n c e o f t h ep r o j e c t a s i t g r o w s .

BDFLI d e a l l y , t h e B D F L r o l e i s l i k e t h e Q u e e n o f E n g l a n d : a w e s o m e c r o w n , b u t n o ta n a c t u a l o p e r a t i o n a l r o l e d a y - t o - d a y . T h e r e a l j o b o f a B D F L i s t o N E V E R G O A W A Y .. . . t h e B D F L w i l l a l w a y s b e t h e r e , p r e s e r v i n g t h e p h i l o s o p h y a n d p r i n c i p l e s o f t h ep r o j e c t , a n d k e e p i n g u l t i m a t e a u t h o r i t y o v e r i t s f a t e . T h i s g i v e s u s g r e a tf l e x i b i l i t y i n e x p e r i m e n t i n g w i t h v a r i o u s g o v e r n a n c e m o d e l s , k n o w i n g t h a t w e c a na l w a y s p r e s s t h e " r e s e t " b u t t o n w i t h o u t f e a r o f f r a g m e n t a t i o n o r d e a d l o c k . S e e t h eU S c o n g r e s s f o r a c o u n t e r - e x a m p l e .

B D F L d a i l y r o u t i n e :

* I s t h e p r o j e c t g o v e r n a n c e s t u c k i n a d e a d l o c k o r i r r e v e r s i b l y f r a g m e n t e d ? * I f y e s : r e f a c t o r t h e p r o j e c t g o v e r n a n c e* A r e t h e r e i s s u e s o r c o n f l i c t s e s c a l a t e d b y c o r e ? * I f y e s : r e s o l v e t h e m* G o b a c k t o p o l i s h i n g t h a t c r o w n .

CULTUREEmbraces the BDFLOpen sourceOpen designDocker the company dominates

Most of the maintainers are docker employeesAutomated testingBe Nice and Encourage diversity and participation

THINGS TO CONSIDER FORYOUR OWN PROJECT

There is no one size fits all solution

WHAT DOES OPEN EVEN MEAN?Open SourceOpen CoreOpen DesignOpen DevelopmentOpen Community

TOOLING CONSIDERATIONSBug trackingReview processTestingBarrier to entry for new contributors

DCO vs. CLADeveloper Certificate of OriginContributor Licensing Agreement

Overall workflow

DEVELOPMENT MODEL COMPONENTSCommunicationTeam scaling model

chain of trust modelflat trust modelmaintainers

Release cadenceStabilization periodsRolling developmentCI/CD

Number of artifactsDecision making process -- Consensus modelProject culture

PROBLEMS SPECIFIC TO OPEN SOURCEBDFLVisionManaging competing interestsCorporateConsensus modelTrust and Ownership

Team vs individualFirst come first serve?How do you fire someone?

THANK YOUQUESTIONS?

Slides can be found at jogo.github.io/development-models

Powered by reveal.js