Upload
puppet-labs
View
10.110
Download
1
Embed Size (px)
DESCRIPTION
This talk will show how we deploy Puppet without a Puppetmaster on an autoscaling Amazon Web Services infrastructure. Key points of interest: - Masterless Puppet - Use of Jenkins for Puppet manifest testing and environment promotion (test->staging->production) - Puppet integration with Amazon CloudFormation Sam Bashton Director, Bashton Ltd After working for a number of Internet Service Providers, Sam founded Bashton Ltd in 2004. Focussing exclusively on Linux and Open Source software, Sam and his team provide consultancy, support and 24/7 infrastructure management for a number of high-traffic websites. A serial early adopter, Sam has travelled the world providing training and consultancy and generally spreading the Open Source message. Sam lives in Manchester, UK.
Citation preview
CONTINUOUSLYINTEGRATED PUPPET IN ADYNAMIC ENVIRONMENT
SAM BASHTON, BASHTON LTD
CONTINUOUSLYINTEGRATED PUPPET IN ADYNAMIC ENVIRONMENT
MASTERLESS PUPPET: WHYAND HOW
SAM BASHTON, BASHTON LTD
ABOUT MELinux guy since Slackware, floppy disks and root + bootUsing Puppet since 2007Run a company in Manchester, North West EnglandWe provide outsourced ops for other companies
OUR FULLY MANAGEDENVIRONMENTS
Primarily transactional websites (e-commerce)Majority (70%+) on Amazon Web Services (AWS)Majority using CentOS
HOW WE WORKSimple is better than complexComplexity is worth adding only if it provides obviousfunctional benefits
Re-usabilityResilience
WHY DID WE PICK AWS?Featureset and toolset massively in advance of anyother cloud provider, public or private#1 customer reason for switching to AWS? The ability toscale on demand
TOOLS WE USE FORBUILDING AND MANAGINGDo one thing and do it well
CloudFormation - Amazon tool to manageinfrastructurePuppet - Manage system configurationPulp - centralised repository, manages packagerevisionsJenkins
HOW WE USE PUPPETNo PuppetmasterPuppet manifests, hieradata and modules distributedto all machines via RPMAll machines boot with a common, blank image and getconfigured at first boot
WHAT'S WRONG WITHMASTER BASED PUPPET?
Pets vs Cattle
Puppet designed for a world of servers as petsWe do not live in that world
PUPPET DESIGNED FOR PETSMany assumptions in Puppet presume that yourservers are petsSome of these work against us when managing a herd
MANUAL CERTIFICATESIGNING
Clearly unsuitable when machines are automaticallyprovisioned
POTENTIAL WORKAROUNDS:AutosignUse/write another automated certificate generationmechanism
Possibly tied in with autoscaling
NO MECHANISM FORCLEANING OLD HOSTS
Likely to have host-names reused, causing machines tofail to configurePuppetmaster will fill with certificates for machines thatran for a few hours and went away again
POTENTIAL WORKAROUNDS:Use UUID certificatesAgree not to look in the certificate directoryWrite mechanism for cleaning up old certificates
HOSTS CONFIGURED BASEDON HOSTNAME
Our machines have names like ip-172-26-5-123How does Puppet know what type of machine this is?
POTENTIAL WORKAROUNDSUse an external node classifierUse some mechanism for giving a better hostname, egweb-172-26-5-123 and use regex for node names
PUPPETMASTER IS A SINGLEPOINT OF FAILURE
If the Puppetmaster fails, we can no longer autoscaleupIn particular, this could be a problem if there isavailability zone failure
POTENTIAL WORKAROUNDSClustered Puppetmasters
WORKAROUND RECAPUse/write alternative certificate management softwareWrite an external node classifier / mechanism forsetting hostname appropriatelyCluster multiple Puppetmasters
WHAT WE DID INSTEADDecided using a Puppetmaster was trying to fit a squarepeg into a round holeInstead, decided to run Puppet without a master
APPLYING LOCAL PUPPETMANIFESTS
puppet apply --modulepath=/etc/puppet/modules example.pp
DISTRIBUTING MANIFESTSUse RPMDistribute full set of manifests/modules to eachmachineApply only the manifest relevant to that machine
PACKING PUPPETMANIFESTS IN RPM
Build an RPM containing everything under /etc/puppetMake files readable only by root
APPLY PUPPET MANIFESTSHave an RPM %postinst command apply the Puppetconfig
This isn't as straightforward as running the puppetapply from %postinstPuppet needs to install packages via yum, but yum isrunning installing the Puppet packageInstead, we work around with a dirty hack: have the%postinst create an at script which checks if yumhas finished and then runs the puppet apply
RPM INSTALLATION ANDMANAGEMENT
How do we get these RPMs on our machines?
PULPWe were already using PulpProvides yum repository managementUsed for managing security updates and deployingapplication code
http://pulpproject.org/
WHAT IS PULPRepository managerAllows us to easily audit what packages and versionsare installed whereAllows us to push package installations
Uses qpid message queue
Has concept of 'content distrubtion servers' for easyreplication and clustering
HOW WE USE PULPPuppet contains details of what packages should beinstalledPulp manages which version of the package should beinstalledPulp allows us to clone repos and copy packagesbetween them for easy qa->stage->live environmentmanagement
DEPLOYINGCONFIGURATION AS CODEAllows us to reuse our existing code deploymentinfrastructureManage configuration deployment from Jenkins
HOW WE DEPLOY CODEEverything managed via the Jenkins continuousintegration serverJenkins uses Pulp to install code on remote machines
DETAILS ON HOW WEDEPLOY CODE
Jenkins fetches code from source control (git)An RPM is builtTests are runIf tests pass, the RPM is added to the relevant Pulprepository RPM installed on the target machine(s)
DEPLOYMENT LIFE-CYCLEJenkins also manages deployment life-cycleRPMs are installed on stagingPromoted Builds plugin then used to install the sameRPMs on live once testing is complete
PUPPET DEPLOYMENTPROCESS
Puppet manifests are checked into gitLint tests via Jenkins pulls in modules with librarian-puppet, thenbuilds an RPMDeployment to test environments, functional tests forwider code-base run
Jenkins Warnings plugin
PUTTING IT INTOPRODUCTION
Once suitable tests (automated and manual) have beencarried out, we promote Puppet config into productionWe use the Jenkins 'Promoted Builds' plugin for this
JENKINS: PROMOTION
EXCEPT..How does a machine get from a bare image to the statewhere we can push packages to it from Pulp?How does a machine know what type of machine it is?How do we find other resources, eg databasehostname?
CLOUDFORMATIONAmazon tool for specifying infrastructureEverything* we provision inside AWS is provisioned viaCloudFormationJSON templates
* Everything except for the things Amazon doesn't exposevia CloudFormation..
CLOUD-INITWorks with multiple cloud typesSorts out things like SSH keys, allows us to configurehost namesAlso allows us to provide a bash script to run on startup
PROVISIONING A BAREINSTANCE
cloud-init automatically manually adds the pulp repowhich contains Pulp, Puppet and our Puppetmanifests/modulesInstalls appropriate RPMsPuppet runs, subscribing the machine to the relevantPulp repos, and installing packages in the usual Puppetway
HOW DOES IT KNOW WHATTYPE OF MACHINE IT IS?
We tell it!Use an environmental variable $HOSTTYPESimply run
puppet apply \--modulepath=/etc/puppet/modules ${HOSTTYPE}.pp
EXTRA FACTSCustom facter factsAlso specified in an environmental variable
Data comes from within the CloudFormation templateOn our list of things to look at:
FACTER_HOSTENVIRONMENT=liveFACTER_STACKNAME=customer-web-live
https://github.com/fanduel/hiera-cloudformation
OTHER RESOURCESWe either:
Provide details as a facter factF̀ACTER_DBHOST=xyz
Also use this approach to limit distribution ofsecure details, eg DB passwords
Discover via the EC2 APIEg Varnish servers discover web backends bycalling API and finding hosts tagged appropriately
FREE WINS!
FREE WINS!Greater control over the timing of Puppet runsImproved visibility - for ops and devsConfiguration changes now have to be deployed totesting/staging first
MORE FREE WINS!Puppet configs now have a versionEasy to find config version on the machine itselfConfig changelogs accessible on every machine
(Git changelog added to RPM)
THE DOWNSIDESPuppet manifests and modules on all machines
Potentially a security issue?Mitigated by CloudFormation holding most sensitivedata
ALTERNATIVEIMPLEMENTATIONS
Don't want to use Pulp?Could do basically the same thing with yum s3 plugin
Use mcollective to push package updateshttps://github.com/jbraeuer/yum-s3-plugin
FUTURE IMPROVEMENTSBuild AMIs using Packer instead of configuring at boottime
Decrease time to autoscaleWould probably still need to run Puppet at first bootto configure machine specific settings
QUESTIONS? COMMENTS?Sam Bashton
Twitter: @bashtoni
(Psst.. )http://www.bashton.com/jobs/