Scaling Puppet Usage to a Global Organization

PuppetCamp Feb 7, 2013

Case Study

Scaling Puppet Usage to a Global Organization


The Presenter

Tor-Åke– Hands-on System Architect– Linux Pro since 2000– DevOps / Operation Automation since 2005– Puppet since 0.25

RedBridge– Open Source Infrastructure Builders– Consulting, Operations and Open Source

Products Since 2003– Partners with PuppetLabs, Redhat, Amazon,

Zimbra, Others


TOC

Part 1: Case description– The Customer and the Challenge

Part 2 : Way of Working– How to Develop Code

Part 3 : Technical Platform– Puppet Masters etc


Part I : Case Description

A Swedish telecommunications companyAbout 10 sites around the worldThousands of usersThousands of systems


System Types (in scope)

Virtual and metal serversSuSE, RedHat, CentOS, Ubuntu and SolarisMostly OSS and Third Party SoftwareDivisions choose from predef:d system types

– Some unique applications

Supporting infrastructure– Network flesystems for applications and user

data– OS-native deployment systems (Satellite, Ops

Center etc)


The Challenge

Bring Home R&D IT From Outsourcing

... and in the process:

Shorten Lead Time– Automate Deployment and (Change)

Management

Increase Cost Efficiency– Solve each problem once– Share the solution globally– A scalable technical platform


Additional Requirements

Leverage existing expertiseMaintain site-local freedom to solve unique problemsKeep site freedom to plan and execute code updates


Puppet?

Puppet is naturally only a part of the solution...

...but an important focal point as it touches all services!


Part II : Way of Working

Why and when is code developed?How and by whom?How is code shared?


The Facilitator

A Global Team, coordinating Puppet DevelopmentKnowledge identifcation and sharingKeeper of the code standardDevelop and support a Puppet architecture


Example Why and When

Site users need a service e.g. ”Hosted Jenkins”Site team calls for Puppet codeGlobal team fnds a Jenkins expertGlobal team helps Jenkins expert write modulesModules are delivered to requesting site team and

users


Another example

Site users need a service e.g. Hosted TomcatSite has a Tomcat expert who can write Puppet

moduleGlobal Team is notifed that Tomcat module existsIf another site requests same service, existing code

is ”globalized” with assistance from Global Team


Development Sync FlowchartNeedcodeNeedcode

Globalavailable?

Globalavailable?

Askaround

Askaround

Anyavailable?

Anyavailable?

Modifyand testModify

and test

Fetchand testFetch

and test

Post globalsuggestionPost globalsuggestion

CodeOK?

CodeOK?

Done.Deploy!Done.

Deploy!

Developand testDevelopand test

No

Yes

Yes

Yes

No

No


Code Sharing

Global Git repository (actually per module)Each site pulls code to site-local repositoryTest locally, and deploy (ITIL Change)Global team is notifed of any local changesIf changes are to be globalized, GT pulls code from

site-local repo


Code Standard

Code structure optimized for sharing some parts, while keeping others site-private

Readability and documentation built inUnit test


Code Structure

Parameters module local to the siteparams::jenkins in

moduleroot/params/manifests/jenkins.pp

All parameters can be overridden per nodePriority:

1)Node defnition (class params)2)Params module3)Module default (in init.pp!)


README

What is the scope of this moduleWhat site and what OS:es has it been tested onExample params fle for params moduleNo description of params!

– Those go in init.pp


Predictable Results

Else-clause with a fail()– e.g.

If $::operatingsystem == Solaris { …} else { fail ( ”we have not tested this OS yet” )

}


Code Review

Members of the Global Team send code for review”Please look at this code and test it on your site”

– Code deemed unreadable = FAIL– Code breaks other modules unit test = FAIL


Adherance to Standard

Lots of code contributors– Varying experience with Puppet

Not always developers– Unfamiliar with peer review, Scrum, XP, Unit

tests etc.

Global Team must fll the gaps– But we are not subject experts!– Educate eachother


Boilerplate

A module with all elementsWell commentedCopy and fll out the blanks


Manifest Patch Strategy

Many small increments?ORTake a big hit when needed?

+ 10 sites with slight differences, ever-evolving+ 7 different OS:es+ System experts distributed on the sites= Regression testing must also be distributed!


Part III : Puppet Platform

Serve thousands of clientsDeployable by Puppet (apply)Support Way of Working


Part III : Puppet Platform

Serve thousands of clientsDeployable by Puppet (apply)Support Way of Working


Deploying Puppet

A global network flesystem (rsync+nfs)Git repos with puppet code and packagesClone itChange parametersBootstrap a frst Puppet Master


Adding More Masters

DNS alternate names in the RR certifcateAdd serverMount shared storageBootstrap server from another serverpuppet agent –server=... --ca_server=...


Puppet Masters

3-10 Masters1 CA ServerShared (NFS) storage

– Manifests– Certifcates

Apache plus PassengerRound Robin DNS Records


Foreman

Just reports (for now...)Masters store Yaml report on diskSpool to foreman db periodically

– Foreman server can be ofine indefnitly w/o losing reports


Agents

Deployed with Kick/JumpstartVersion 2.7Installed under /optRun by cron, hourly with splay


puppet:/// fles

We try to avoid them. Why?– NAS is faster than Passenger

Packages are installed from OS native channels– Available in Global NFS

What to do with Solaris?– Packages directly from Global NFS


Example: installing Solaris pkg

With puppet fle transferif $::custom_fact == install { file

{ “local.pkg”: source => puppet:///remote.pkg

} file { “local.adm”: source => puppet:///remote.adm

} File[“local.pkg”] -> Package[‘VNDRpkg’]}package { ‘VNDRPkg’: source => “local.pkg”, adminfile => “local.adm”,

}

# + the custom fact ruby code!

With NFS Mountpackage { ‘VNDRpkg’ : source =>

‘/net/nfsserver/remote.pkg’, adminfile => ‘/net/nfsserver/remote.adm’,

}

# We’re done!


Orchestration

We don't have it (yet!)Generous ITIL Change WindowsSprawling networks

– Firewall red tape


Lessons Learned

Modules should not depend on modules– 10 sites with prod, dev and test environments– Slightly different module version mix

Puppet is not for Everything!– Template shellscripts are powerful– So is Rpm/Deb/Pkg

The biggest issues are with people– Aligning expectations– Consensus about everything from way-of-

working to variable naming

Technology

Scaling Puppet Usage to a Global Organization