Designing Puppet: Roles/Profiles Pattern

Designing Puppet:

Roles / Profiles Design Pattern

Puppet Camp Stockholm, Feb 2013

Thursday, 7 February 13

• Craig Dunn

• Puppet user since 2008 as an IT contractor

• Started with 0.24

• Joined Puppet Labs in June 2012

• @crayfishX

• Freenode IRC: crayfishx

Agenda

• How people typically design Puppet

• Real-world case study

• Thinking about components

• Designing Puppet for your users

• Node classification

• Data separation

Background

• Originally a blog post written in May 2012

• Advocated by many Puppet Labs Engineers

• Based on a real world solution

• Several community members have adopted with success

Designing Puppet

• You write awesome modules

• You classify them to your node

Designing Puppet

Modules

Node Classification

Down the road...

• Your infrastructure grows

• Business requirements will change

• Your Puppet code feels bulky and high maintenence

• There will always be edge cases eventually

• You decide it needs refactoring

Danger Signs

• Resources being declared in two modules

• You don’t know where your implementation “fits”

• Lot’s of logic at a node level

• Repetition and duplication

• The if statement is your go-to-guy

Write good modules

• Should manage only it’s own resources

• Should be granular

• Should be portable

Thinking beyond the module....

• Puppet is a code base

• How do I design an effective framework

• Gluing everything together

Node-level logic

• Risks duplication and repetition

• No guarantee of consistency

• TMI!

Node-level logicnode basil { class { ‘apache’: version => ‘latest’, } class { ‘motd’: } class { ‘ssh’: } class { ‘users’: default_shell => ‘/bin/false’, } Class[‘ssh’] -> Class[‘users’]}

Node-level logic

• What happens when I have 1000 nodes

• Or 10,000 nodes!!

• That’s a lot of code!

• So where should implement this?

Designing Puppet

• Provide business logic to classification

• Provide an abstraction layer for implementation of components

• Make code adaptable to complex requirements

• Reduce node-level logic

• Reduce functionality overlap

What is the worse thing that is

going to happen to your Puppet code?

Business requirements

Business logic does not align with technology

Case study

• Real world problem

• Solved through design

Case study

“We have 3 applications we

need to deploy using Puppet”

The business view

Application Y Application Z

Application X

Go forth and Puppetize!

And we jumped right in...

Things got painful

Problems

• These applications aren’t that different

• They seem to share a whole bunch of similarities

• Implementation differed on different environments and locations

• Writing 3 separate modules creates conflicts and duplication

Our code was hacky

We are trying to code business logic.

Stop thinking about what it looks like

• Break everything down into components

• Granularity is the key

• Think about what it actually is

What we realised

• Each application stack is a collection of a subset of the same Java apps implemented in different ways

The business view

Application Y Application Z

Application X

The technical reality

Application YApplication Z

Application X

We only have one application

Implemented many different ways

So we had an idea!

• Reduce each Java sub application into granular Puppet modules

• Create a code layer responsible for implementation

• Let’s call them profiles

class profiles::x { include tomcat include mysql include componenta include componentb componentb::resource { ‘name’: ensure => present, }}

class profiles::y { include tomcat include mysql include componenta include componentc include componentd}

class profiles::z { include tomcat include mysql include componenta include componentb include componentd include dependancy Class[‘dependancy’] -> Class[‘componentd’]}

class profiles::application { include tomcat include mysql include componenta}

class profiles::application::x inherits profiles::application { include componentb componentb::resource { ‘name’: ensure => present, }}

class profiles::application::y inherits profiles::application { include componentc include componentd}

class profiles::application::z inherits profiles::application { include componentb include componentd include dependancy Class[‘dependancy’] -> Class[‘componentd’]}

Use inheritance for abstraction within profiles

Profiles and Components

Resources

Components: Resource modelling

Resources

Profiles : Implementation

In reality it was worse

• 2 different deployment types made up of over 15 server types each

• 10+ locations

• 4 environment types

• 10+ locations

• Every installation was an edge case!

• 10+ locations

• Every installation was an edge case!

• My slides weren’t big enough.

Lessons learned

• Granularity is good

• Don’t assume business logic will directly translate to technology

• Abstraction is awesome.... but that’s nothing new....

Abstraction is a core principle of coding

• Functions are abstracted by methods

• Methods abstracted by classes and modules

• They are abstracted with libraries

• Puppet is code!

Puppet is all about abstraction

• Data is abstracted by Hiera

• Providers are abstracted by types

• Resources are abstracted by classes

• Classes are abstracted by modules

Puppet is all about abstraction

• Data is abstracted by Hiera

• Providers are abstracted by types

• Resources are abstracted by classes

• Classes are abstracted by modules

• Modules are abstracted by profiles

Focussing on Abstraction

• We’ve turned business logic into a technology stack

• Can we translate that back into business logic?

• Why would we even want to do that?

UAT Cluster node

include securityinclude usersinclude ntpinclude ssh::serverinclude customappinclude tomcat::server

class { ‘jenkins’: require => Class[‘tomcat::server’],}

include mysqldatabase { ‘apptest’: ensure => present,}

Our example configuration model:

Think about the usersMeet John, Susan and Bill.

John is a Sysadmin

• Wants to ensure all servers have kernel hardening, NTP and SSH Server installed

• Wants to manage what packages, services, files and other resources

• Is responsible for maintaining all the components of a UAT cluster server

Susan is an application specialist

• Cares that a UAT Cluster node requires MySQL Server, Tomcat Server and Jenkins server installed.

Bill is an IT manager

• Bill cares that the server is a UAT Cluster node

What do they care about?

• John cares about modelling all resources

• Susan cares about the technology stack

• Bill cares about the business logic

In Puppet

• Resource modelling is done in component modules

• The technology stack is defined in profiles

• Where do we represent the business logic for Bill?

Introducing Roles

• Represent business logic, not technology

• Define a set of technology stacks (profiles) that make up the logical role

• Allow the business to manage how the infrastructure looks without defining what it is

A node can only have one role

• A role can include as many profiles as required to define itself

• If a node requires two roles, it has by definition become a new role

A node can only have one role

• A role can include as many profiles as required to define itself

• If a node requires two roles, it has by definition become a new role

• Something couldn’t be a lion and a kangaroo at the same time!

It would be a Lingaroo

• One-to-one to nodes

• One-to-many to profiles

• Only implement profiles

Example role

class role::uat_server { include profiles::base include profiles::customapp include profiles::test_tools}

Classification

• Node classification simply assigns roles to nodes

• Roles expose profiles

Classification

node ‘craig.puppetlabs.vm’ { include roles::uat_server}

Classification

The Stack

Resources

The Stack

Resources

The Stack

Resources

The Stack

Resources

Roles : Business Logic

Terminology

• Profiles and Roles are Puppet modules

• Components are Puppet modules responsible for modelling resources

• Everything is a module

Naming conventions

• Components should be named after what they manage (apache, ssh, mysql)

• Profiles should be named after the logical stack they implement (database, bastion, email)

• Roles should be named in business logic convention (uat_server, web_cluster, application, archive)

Hiera OverviewLet’s talk about data!

Managing infrastructure

Production

DC1 DC2 DC3

Managing data in Puppet is hard.

Without Hiera?if ( $::environment == ‘dev’ ) { $ntpserver = ‘192.168.2.1’} else { if ( $::fqdn == ‘host4.mycorp.com’) { $ntpserver = ‘127.0.0.1’ } else { $ntpserver = ‘213.21.6.4’ }}

With Hiera?$ntpserver = hiera(‘ntpserver’)

Hierarchical lookups

• Hiera uses facter facts to determine a hierarchy

• Top down hierarchy for overriding configuration values based on roles, environments, locations.... or anything else

• And do this without any coding!

Separation of data from code

• Puppet modules without hard-coded data are easily shared and more re-usable

• Infrastructure configuration can be managed without needing to edit Puppet code

Pluggable Backends

• Source data from multiple locations

• Data source is abstracted from code

Pluggable Backends

• Source data from multiple locations

• Data source is abstracted from code

• hiera-gpg

• hiera-http

• hiera-mysql

• hiera-redis

• hiera-json

• hiera-zookeeper

Data Separation

• Use Hiera to abstract your data from your code

• Components and profiles can source data from Hiera

Profiles and Hiera

• Use Hiera to model your data

• Use profiles to model your implementation

The Stack

Resources

The Stack

Resources

Hiera:Data

Classification

• Assigning classes to a node

• You can classify within Puppet code (site.pp)

• You can use an External Node Classifier (ENC)

Leveraging an ENC

• You can classify your nodes however you want

• Puppet Dashboard

• Enterprise Console

• Foreman

• Site.pp

• Custom script

Leveraging an ENC

• An ENC should classify a node to it’s role

• Nothing else

The Stack

Resources

Hiera:Data

The Stack

Resources

Hiera:Data

Classifier

Key benefits

• Reduced node-level logic to a role.

• Gain the ability to be flexible with implementation

• Business logic improves managability by non-Puppet users

• Edge cases are now easy to solve

Enough Preaching!

This is not the way to design Puppet... It’s a

Can I implement this design without roles?

• Yes.

• You lose the layer of abstraction that exposes business logic

Can my roles be defined in my ENC?

• Yes.

• Keeping it in code makes it versionable

Can’t I just use Hiera to define profiles?

• Technically yes.

• You lose the flexibility to implement code logic in profiles and it may become restrictive

• You could possibly use: https://github.com/ripienaar/hiera-puppet-nodes

The fundamental concepts....

• Abstraction, abstraction, abstraction

• Decoupling business logic, implementation and resource modelling.

• Separating data and code

• Reducing node-level complexity

Other Resources

• Adrien Thebos’ excellent blog post http://sysadvent.blogspot.co.uk/2012/12/day-13-configuration-management-as-legos.html

• My original blog post http://www.craigdunn.org/2012/05/239/

• Module Structure Redux by R.I.Pienaar http://www.devco.net/archives/2012/12/13/simple-puppet-module-structure-redux.php

Thank you. Questions?

• Follow me at @crayfishX

• Bug me on Freenode: crayfishx

Enjoy the rest of Puppet Camp!

In memory of Giles Constant, who spent many nights debating Puppet design patterns with me over copious amounts of beerand helped me on my journey of discovery learning how to implement Puppet properly. R.I.P

Designing Puppet: Roles/Profiles Pattern

Technology

Certification centered on people and roles...ICT in a job or occupation. 01. 02. 03. CHALLENGE: Professional Digital Profiles GENERAL PROFILES •Administrative staff •Sales representative

State of Puppet - Puppet Camp Silicon Valley 2014

Puppet Camp Düsseldorf 2014: Puppet CA Certificates Explained

Building a deployment pipeline - CloudBees...ruby Developer QA - Jenkins Puppet Servers Puppet Puppet Integration tests Production - Jenkins Puppet Servers Puppet Puppet Trigger MCollective

Puppet Camp Berlin 2014: Advanced Puppet Design

Ottawa Puppet User Group - Puppet Module Testing

PuppetConf. 2016: Puppet Best Practices: Roles & Profiles – Gary Larizza, Puppet

State of Puppet - Puppet Camp Barcelona 2013

WP Puppet Theatre | Puppet Power: Connecting Generations ... › ... › uploads › 2018 › 01 › PP16-Program-LOW-… · WP Puppet Theatre | Puppet Power: Connecting Generations

Profiles, Password Policies, Privileges, and Roles

PuppetConf 2016: Direct Puppet and Application Management for the Puppet Platform – Ryan Coleman, Puppet

Puppet Documentation€¦ · An Introduction to Puppet Supported Platforms Installing Puppet — from packages, source, or gems Conﬁguring Puppet — includes server setup & testing

Puppet Camp Berlin 2015: Puppet Demo (Beginner)

Application security (behind Oracle roles and profiles)

Puppet Camp Berlin 2015: Puppet Keynote

U! Happy Events Core Team PROFILES, ROLES & RESPONSIBILITIES

Printable-Puppet-Dinosaurs - Paging Supermom · Triceratops Puppet Stegosaurus Puppet

Puppet Camp Berlin 2014: Manageable puppet infrastructure

State of Puppet - Puppet Camp Austin

PS Security By Deviprasad. Agenda Components of PS Security Security Model User Profiles Roles Permission List. Dynamic Roles Static Roles Building Roles/Rules