Case Study: Green Field Implementation of Puppet 3.0 at ESPN

PuppetConf 2013

CHALLENGES

• Rapid growth– Number of servers being provisioned– New services being provided

• Manual configurations

• Drift

• Silos

• Traditional clouds not reasonable

Objectives

• Push button configuration

• Self service for customers

• Visibility into state of configurations– Reports

• Simplify knowledge transfer

• Data driven

• Elastic

• Build/rebuild a node at anytime

• Support workstations/dev/test/prod

• Allow adhoc manual changes*

• Do not slow down development teams

• Include legacy servers

Application

Middleware

OS

Server

Storage

Network

IAAS

PAAS

SAAS

Puppet

Puppet Versions

• Puppet 3.0.0

• Puppet Dashboard 1.2.12

• PuppetDB 1.0.1

• MCollective 2.2.1

• ActiveMQ 5.5

The Data

• Inherent Facts – Facter– Exists simply because the node exists.

• Derived Facts – Business rules applied to inherent facts.– Puppet custom facts

• Assigned Facts – Exist because we deem it to be true.– Top or node scoped variables

Assigned Facts• espn_role

– Identifes the role class to be applied• espn_cluster

– Identifies nodes in the cluster• espn_env

– workstation, dev, test, qa, prod• espn_owner

– change notifications– security delegation – licensing

What to manage? Everything!

Role

• Aligns with business and or IT needs

• Defines everything on a node

• Used for classification of nodes

• Exactly one role per node

• Includes profiles and releases

• espn_role

Group everything by roles

AA BB CC

Release Environment

• Not puppet environments

• Determined by application development and release cycle

• espn_env

Group by release environment

AA BB CC

ProdProd

QAQA

DevDev

WorkWork

Clusters

• Always identical

• Configured to interact with one another

• espn_cluster

Group by clusters

AA BB CC

ProdProd

QAQA

DevDev

WorkWork

Hiera hierarchy in theoryhostname/%{hostname}

cluster/%{espn_cluster_name}

role_env/%{espn_role}_%{espn_env}

role/%{espn_role}

env/%{espn_env}

network/%{espn_network}

os/%{operationsystem}

default

Hiera hierarchy in practice

role_env/%{espn_role}_%{espn_env}

role/%{espn_role}

env/%{espn_env}

network/%{espn_network}

defaults

Classification

Application

Middleware

OS

Server

Storage

Network

IAAS

PAAS

SAAS

Role

Profiles

Releases

Role

Profiles

Releases Resources

Resources

Classes Resources

Resource Hierarchy

Resources

• The building blocks

• Everything managed is a resource

• Defined in modules, used by profiles

• Two resources never manage the same configuration

Modules

• Isolate resources within the module

• Never reference another module

• No organizational specific logic

• init.pp is a minimal installer

• Reusability is key

R RR RR RR R

R RR RR RR R

R RR RR RR R

R RR RR RR R

R RR RR RR R

Resources by module

R RR RR RR R

R RR RR RR R

R RR RR RR R

R RR RR RR R

R RR RR RR R

Never cross modules

Role

Profiles

Releases Resources

Resources

Classes Resources

Resource Hierarchy

Profile

• Defines the platform

• Cross module references

• Enforces dependencies between modules

• Class parameters prevent hiera overrides

Example Profileclass profile::jboss_eap_6(

$java_version=‘jdk_1_7_u10’,

$mod_cluster_version=‘1.2.3’,

$httpd_version=‘2.2.22’,

){

include “java::${java_version}”

include directories::middleware

class{‘mod_cluster’:

version => $mod_cluster_version,}

class{‘httpd’:

version => $httpd_version,}

class{‘jboss_eap_6’:

java_home => getvar(“java::${java_version}::home”),}

}

Role

Profiles

Releases Resources

Resources

Classes Resources

Resource Hierarchy

Release

• Special type of profile

• Knows how to install on top of a profile

• Deploys resources from an “artifact hash”

• Cleans up removed artifacts

• Driven by versioned release id– espn_release_id

Artifact Hash

• Defines abstract resources in a release

• Contract between developers and operational groups

• Profile agnostic

Example Artifact Hashapplications

‘jee-app.ear’:

version: 1.2.3

espn.war:

version: 1.0.0

datasources:

datasource1:

driver_name: ”oracle”

libraries:

utility.jar:

version: 1.0.0

Example ReleaseClass release::studio_record{

artifact_hash = hiera(‘artifacts’,undef,”release/${espn_release_id}”)

#modify the artifact hash so it can be used with create_resources

#set organization specific parameters such URL’s to the artifact repo

resources {‘jboss7_datasource: purge => true,}

create_resources(jboss7_datasource, $artifact_hash[‘datasources’])

resources {‘jboss7_deployment: purge => true,}

create_resources(jboss7_deployment, $artifact_hash[‘applications’])

}

Role

Profiles

Releases Resources

Resources

Classes Resources

Resource Hierarchy

Example Role

class role::studio_record{

include profile::base

include profile::jboss_eap_6

include profile::mysql_5

include release::studio_record

}

site.pp

# assigned facts retrieved from external datasources

$espn_cluster_nodes = espn_cluster_nodes()

$espn_release_id = hiera(‘espn_release_id’)

# single assigned fact drives 100% of classification

node default {

include “role::${espn_role}”

}

Defining a Node

MCollective

Colonol John Boyd• Military Strategist

• OODA Loop

ObservationsDecision

(Hypothesis)Action(Test)

Observe Orient Decide Act

Feedback

Feedback

UnfoldingCircumstances

OutsideInformation

UnfoldingInteraction

WithEnvironment

CulturalTraditions

Analysis &Synthesis

PreviousExperience

NewInformation

GeneticHeritage

Feedback

ImplicitGuidance& Control


OODA Loop

"Time is the dominant parameter. The pilot who goes through the OODA cycle in the shortest time prevails because his opponent is caught responding to situations that have already changed.“

Harry Hillaker (chief designer of the F-16)




Feedback: Puppet runs on dev/test puppet environments

Feedback


OutsideInformation


WithEnvironment

CulturalTraditions

Analysis &Synthesis

PreviousExperience

NewInformation

GeneticHeritage

Feedback







Feedback: (Test) Noop puppet run on production


OutsideInformation


WithEnvironment

CulturalTraditions

Analysis &Synthesis

PreviousExperience

NewInformation

GeneticHeritage

Feedback



Noop Puppet Runs

• Hourly cron job– “mco puppet runall 20 –noop”

• Dashboard displays unresponsive nodes– no_longer_reporting_cutoff







OutsideInformation


WithEnvironment

PuppetDashboard

CulturalTraditions

Analysis &Synthesis

PreviousExperience

NewInformation

GeneticHeritage

Feedback: (Action) No-noop puppet run on production



Configuration Changes• Always noop first

• Always target nodes with filters

• Always use tags

• Validate changes then apply –no-noop

mco puppet runonce --no-noop -C jboss-eap-6 --tag initscript --tag jboss-eap-6







OutsideInformation

CulturalTraditions

Analysis &Synthesis

PreviousExperience

NewInformation

GeneticHeritage

Feedback: (Action) No-noop puppet run on production

Implicit Guidance & Control:MCollective Agent Plugins



WithEnvironment

PuppetDashboard

20/20 Hindsight

Do Differently?

• Implement MCollective first– Security– Sub-collectives– Availability

• Plan for developer dashboards

• Implement Custom ENC

Nice to have?

• Puppet runs that span multiple nodes– Allocate disk on node A, create shared

filesystem on node B

• Resources automatically tagged with catalog unique identifiers

• Role based access control for dashboard

Questions?

Technology

Case Study: Green Field Implementation of Puppet 3.0 at ESPN