Anatomy of a Reuseable Module

Anatomy of aReusable Module

Alessandro Franceschigithub.com/example42

PuppetConf 2013Saturday, August 24, 13

Be�� patientplease

Saturday, August 24, 13

How�� dowe�� usePuppettoday?

Include classes

Set Params & Variables

DefineBusiness &

Integration Logic

ProvideConfiguration Files

ManageResources


ENC

How do we use Puppet today

Include classesmanifests/

site.pp

Set Parameters / Variables

Integration logic

Resources

ENC HIERA

SITE MODULES

SHARED MODULES

BAD

EDG

EConfiguration files

manifests/site.pp ENC HIERA

manifests/site.pp

BAD

?

BAD

?

BAD

SITE MODULES

SITE MODULES

SHARED MODULES

SITE MODULES

SHARED MODULES

manifests/site.pp

BAD

?

ENC

BAD

?


A�� reusable�� module

is�� all�� aboutCHOICE

Operating Systems

Infrastructures

Scales

Node classifiers

Installation methods

Alternative setups


The�� consof�� a

reusablemodule

Harder & LongerDevelopment

Complexity

Verbosity

Not Optimized for performance


THE PARAMETERSDILEMMA

Managed resources attributesApplication specific config options

Application logic and behaviourIntegration with other modules


Parameters: Resources attributesEnough: $package = $redis::params::package, $service = $redis::params::service, $service_ensure = 'running', $service_enable = true, $file = $redis::params::file, $file_notify = "Service['redis']", $file_source = undef, $file_content = undef,

Too much? $package_provider = undef, $file_owner = $redis::params::file_owner, $file_group = $redis::params::file_group, $file_mode = $redis::params::file_mode, $file_replace = $redis::params::file_replace,

Benefits from: A standard naming convention


Parameters: Application optionsEnough: $puppet_server = “puppet.${::domain}”, $syslog_server = “syslog.${::domain}”, $munin_server = “munin.${::domain}”, $dns_servers = [ '8.8.8.8' , '8.8.4.4' ],

Too much! $anonymous_enable = true, $anon_mkdir_write_enable = true, $anon_upload_enable = false, $chroot_list_enable = true, $chroot_list_file = '/etc/vsftpd/chroot_list',

$resourcefile = $nagios::params::resourcefile, $statusfile = $nagios::params::statusfile, $commandfile = $nagios::params::commandfile, $resultpath = $nagios::params::resultpath, $retentionfile = $nagios::params::retentionfile, $p1file = $nagios::params::p1file,

Benefits from: Template + Options Hash patternSaturday, August 24, 13

Parameters: Application logicExamples: $install_client = true, $install_stomp_server = true, $install_plugins = true, $use_ssl = false, $munin_autoconfigure = true, $service_autorestart = true, $manage_package_repo = true, $run_initdb = undef,

Benefits from: A standard naming convention


Parameters: Modules IntegrationsExamples: $mongo_db_host = $graylog2::params::mongo_db_host, $mongo_db_port = $graylog2::params::mongo_db_port, $mongo_db_name = $graylog2::params::mongo_db_name, $mongo_user = $graylog2::params::mongo_user, $mongo_password = $graylog2::params::mongo_password, $elasticsearch_template = $graylog2::params::elasticsearch_template, $elasticsearch_path = $graylog2::params::elasticsearch_path,

$database = $puppetdb::params::database, $manage_redhat_firewall = $puppetdb::params::manage_redhat_firewall, $db_type = 'mysql',

Benefits from: Shared Stacks


PATTERNSREUSABILITY


Managing�� files

Let user decide how to manage

configuration files.

Alternatives:sourcecontentconcataugeas

custom types


Managing files: source & contentredis/manifests/init.ppclass redis ( $file = $redis::params::file, $file_source = undef, $file_template = undef, $file_content = undef, ) {

[...]

$managed_file_content = $file_content ? { undef => $file_template ? { undef => undef, default => template($file_template), }, default => $file_content, }

[...]

if $redis::file { file { 'redis.conf': path => $redis::file, source => $redis::file_source, content => $redis::managed_file_content, } }}

Provide the Puppet path of an erb templateclass { ‘redis’: file_template => ‘site/redis/redis.conf.erb’,}

Provide directly the content attributeclass { ‘redis’: file_content => “template(‘site/redis/redis.conf.erb’)”,}

Provide a fileserver source pathclass { ‘redis’: file_source => ‘puppet:///modules/site/redis/redis.conf’,}

Manage the configuration file with other methods(augeas, concat...)class { ‘redis’: }


Multiple�� config�� files��

Add parametersto main class

Use a genericconf define

Manage the whole configuration dir


Multiple files: Add parameterselasticsearch/manifests/init.ppclass elasticsearch (

$file = $elasticsearch::params::file, $file_source = undef, $file_template = undef, $file_content = undef,[...]

$init_script_file = '/etc/init.d/elasticsearch', $init_script_file_template = 'elasticsearch/init.erb',

$init_options_file = $elasticsearch::params::init_options_file, $init_options_file_template = 'elasticsearch/init_options.erb',

Provide custom templates for the main file and the init scriptclass { ‘elasticsearch’: file_template => ‘site/elasticsearch/elasticsearch.yml.erb’, init_script_file_template => ‘site/elasticsearch/elasticsearch.init.erb’,}


Multiple files: Generic conf definenova/manifests/conf.ppdefine nova::conf ( $source = undef, $template = undef, $content = undef, $path = undef,[...] $options_hash = undef, $ensure = present ) {

include nova $managed_path = $path ? { undef => "${nova::config_dir}/${name}", default => $path, }[...] file { "nova_conf_${name}": ensure => $ensure, source => $source, content => $managed_content, path => $managed_path, mode => $managed_mode, owner => $managed_owner, group => $managed_group, require => $managed_require, notify => $managed_notify, replace => $managed_replace, }}

Provide a custom template for an alternative config file in config_dirnova::conf { ‘rootwrap.conf’: template => ‘site/nova/rootwrap.conf.erb’,}


Multiple files: Whole config dirredis/manifests/init.ppclass redis ( $dir = $redis::params::dir, $dir_source = undef, $dir_purge = false, $dir_recurse = true, ) {[...]

$dir_ensure = $ensure ? { 'absent' => 'absent', 'present' => 'directory', }

if $redis::dir_source { file { 'redis.dir': ensure => $redis::dir_ensure, path => $redis::dir, source => $redis::dir_source, recurse => $redis::dir_recurse, purge => $redis::dir_purge, force => $redis::dir_purge, notify => $redis::file_notify, require => $redis::file_require, } }

}

Provide a custom source for the whole config_dirclass { ‘redis’: dir_source => ‘puppet:///modules/site/redis/conf/’,}

Provide a custom source for the whole config_dir and purge any not managed config fileclass { ‘redis’: dir_source => ‘puppet:///modules/site/redis/conf/’, dir_purge => true,}


Managing�� Users

Everyone has his own users...

Leave options to decide

if, how and where to manage the ones

the module requires.


Managing Userselasticsearch/manifests/init.ppclass elasticsearch { $ensure = 'present',[...] $user = 'elasticsearch', $user_uid = undef, $user_gid = undef, $user_groups = undef, $user_class = 'elasticsearch::user',[...] if $elasticsearch::user_class { require $elasticsearch::user_class }

elasticsearch/manifests/user.ppclass elasticsearch::user { @user { $elasticsearch::user : ensure => $elasticsearch::ensure, comment => "${elasticsearch::user} user", password => '!', managehome => false, uid => $elasticsearch::user_uid, gid => $elasticsearch::user_gid, groups => $elasticsearch::user_groups, shell => '/bin/bash', } User <| title == $elasticsearch::user |>}

Do not create the requested userclass { ‘elasticsearch’: user_class => undef,}

Provide the user in a different custom classclass { ‘elasticsearch’: user_class => 'site::users',}

Run elasticsearch with a different userclass { ‘elasticsearch’: user => 'apache',}


Managingextra��

resources

Options to specify custom classes

Options to passan hash to

create_resources


Extra Resources: Custom classeselasticsearch/manifests/init.ppclass elasticsearch (

$dependency_class = 'elasticsearch::dependency', $monitor_class = 'elasticsearch::monitor', $firewall_class = 'elasticsearch::firewall', $my_class = undef,

) {

[...]

if $elasticsearch::dependency_class { include $elasticsearch::dependency_class }

if $elasticsearch::monitor and $elasticsearch::monitor_class { include $elasticsearch::monitor_class }

if $elasticsearch::firewall and $elasticsearch::firewall_class { include $elasticsearch::firewall_class }

if $elasticsearch::my_class { include $elasticsearch::my_class }[...]

Provide the modules dependencies with a custom classclass { ‘elasticsearch’: dependency_class => 'site::dep_elasticsearch',}


Extra Resources: Resources Hashelasticsearch/manifests/init.ppclass elasticsearch ( $create_resource = undef, $resources_hash = undef,

) {

[...]

if $create_resource { create_resources( $create_resource , $resources_hash ) }

Alternative: A single hash that includes resources and resources_hash

Provide the modules dependencies with a custom classclass { ‘elasticsearch’: create_resource => 'file', resources_hash => { path => '/etc/elasticsearch/my_file', content => template('site/elasticsearch/my_file.erb), mode => '0600', },}


ManagingPackages

andServices

Names change

Custom packagesare common

Leave choice,optionally


Managing packagesopenssh/manifests/init.ppclass openssh (

$ensure = 'present', $version = undef, $package = $openssh::params::package,[...] ) {

if $version and $ensure == 'present' { $managed_package_ensure = $version } else { $managed_package_ensure = $ensure }

if $openssh::package { package { $openssh::package: ensure => $openssh::managed_package_ensure, } }

openssh/manifests/params.ppclass openssh::params {

$package = $::osfamily ? { Suse => 'openssh', OpenBSD => '', default => 'openssh-server', }

Install a custom company-openssh packageclass { ‘openssh’: package => 'company-openssh',}


Managing servicesopenssh/manifests/init.ppclass openssh ( $service = $openssh::params::service, $service_ensure = 'running', $service_enable = true,[...] ) { if $ensure == 'absent' { $managed_service_enable = undef $managed_service_ensure = stopped } else { $managed_service_enable = $service_enable $managed_service_ensure = $service_ensure }

if $openssh::service { service { $openssh::service: ensure => $openssh::managed_service_ensure, enable => $openssh::managed_service_enable, } }

openssh/manifests/params.ppclass openssh::params {

$service = $::osfamily ? { Debian => 'ssh', default => 'sshd', }[...]

Manage a custom company-openssh serviceclass { ‘openssh’: service => 'company-openssh',}


ManagingInstallation�� options

Let users decide:

OS Packages

Upstream tarballs

Provider


Installation optionselasticsearch/manifests/init.ppclass elasticsearch ( $package_provider = undef,

$install = 'package', $install_base_url = $elasticsearch::params::install_base_url, $install_source = undef, $install_destination = '/opt', ) {[...] $managed_file = $elasticsearch::install ? { package => $elasticsearch::file, default => "${elasticsearch::home_dir}/config/elasticsearch.yml", }[...]case $elasticsearch::install { package: { package { $elasticsearch::package: ensure => $elasticsearch::managed_package_ensure, provider => $elasticsearch::package_provider, } } upstream: { puppi::netinstall { 'netinstall_elasticsearch': url => $elasticsearch::managed_install_source, destination_dir => $elasticsearch::install_destination, owner => $elasticsearch::user, group => $elasticsearch::user, }[...]

Install elasticsearch from upstream sourceclass { ‘elasticsearch’: install => 'upstream', install_source => 'https://download.elasticsearch.org/elasticsearch/elasticsearch/elasticsearch-0.90.3.zip',}


https://download.elasticsearch.org/elasticsearch/elasticsearch/elasticsearch-0.90.3.zip'




Templatesand�� hashes

Managing specific application configs

parameters may get out of control

A single config hashto show them all

A custom templateto use them

Application specific configs



Options Hash: Setupopenssh/manifests/init.ppclass openssh ([...] $file_template = undef, $options_hash = undef,

site/templates/openssh/sshd_config.erb# File Managed by Puppet[...] Port <%= scope.function_options_lookup(['Port','22']) %> PermitRootLogin <%= scope.function_options_lookup(['PermitRootLogin','yes']) %> UsePAM <%= scope.function_options_lookup(['UsePAM','yes']) %>[...]

* Function options_lookup currently in Example42's Puppi module

Alternative site/templates/openssh/sshd_config.erb Port <%= scope.lookupvar('openssh::options_hash')['Port'] ||='22' %> PermitRootLogin <%= scope.lookupvar('openssh::options_hash')['PermitRootLogin'] ||='yes' %> UsePAM <%= scope.lookupvar('openssh::options_hash')['UsePAM'] ||='yes' %>[...]


Options Hash: UsageUsage (with Hiera):include openssh

/etc/puppet/hieradata/global.yml:---openssh::file_template: 'site/openssh/sshd_config.erb'openssh::file_options_hash: Port: '22222' PermitRootLogin: 'no'

Usage (with parametrized class):class { 'openssh': file_template => 'site/openssh/sshd_config.erb' file_options_hash => { Port => '22222', PermitRootLogin => 'no', }


STANDARDSNAMING

Managed resources attributes



TheHandy�� Grailof�� Modules�� Standards

A blog post*

Some discussions on Puppet-Users

github.com/stdmod

Naming standardsfor modules parameters

Community driven

(draft 0.0.2)

* http://www.example42.com/?q=The_handy_Grail_of_Modules_StandardsSaturday, August 24, 13

http://www.example42.com/?q=The_handy_Grail_of_Modules_Standards

http://www.example42.com/?q=The_handy_Grail_of_Modules_Standards

Benefits�� of�� suggested

(and�� shared)naming��

conventions

Saner User Experience

Better modules Interoperability

Reusability Patterns

Predictability in usage and

development


Stdmod Params: Main resources### General parametersensure (enable?) version (package_version?)

### Package - Service - Main configuration filepackage (package_name?)package_ensurepackage_providerpackage_* [any relevant package type attribute]

service (service_name?)service_ensureservice_enableservice_subscribeservice_*

file (file_path? config_file? config?)file_source (source? config_file_source? config_source?)file_template (template? config_file_template? config_template?)file_content (content? config_file_content? config_content?)file_* (config_file_*? config_*?)

file_options_hash (options? options_hash? file_options?)Saturday, August 24, 13

Stdmod Params: Extra resourcesother_packageother_package_*

client_packageclient_package_*

server_packageserver_package_*

other_serviceother_service_*

log_filelog_file_*

pid_filepid_file_*

init_script_fileinit_script_file_*init_config_fileinit_config_file_*


Stdmod Params: Installation ### Parameter related parameters

installinstall_urlinstall_base_urlinstall_sourceinstall_destinationinstall_pre_execinstall_pre_exec_*install_post_execinstall_post_exec_*

install_script_fileinstall_script_file_*install_response_fileinstall_response_file_*


Stdmod Params: Monitormonitormonitor_toolmonitor_hostmonitor_portmonitor_protocolmonitor_urlmonitor_processmonitor_servicemonitor_config_hash


Stdmod Params: Firewallfirewallfirewall_srcfirewall_dstfirewall_portfirewall_protocol


STACKSMODULES

Integrations of modules



Why�� reinventing�� the�� stack?

We always use stacks.

We need them to make something

useful with modules.

What about:Sharing?

Best practices?Standardization?


Stacks - A Simple Sample class stack::logs (

$ensure = 'present',

$syslog_server = false, $syslog_server_port = '5544',

$elasticsearch_server = false, $elasticsearch_server_port = '9200', $elasticsearch_cluster = 'logs', $elasticsearch_java_opts = '-Xmx2g -Xms1g',

$install_logstash = false, $install_elasticsearch = false, $install_kibana = false,

$install_graylog2 = false, $install_graylog2_webinterface = false,

$syslog_config_template = 'stack/logs/syslog.conf.erb', $logstash_config_template = 'stack/logs/logstash.conf.erb', $elasticsearch_config_template = 'stack/logs/elasticsearch.yml.erb', $kibana_config_template = 'stack/logs/config.js.erb', $graylog2_config_template = 'stack/logs/graylog2.conf.erb',

) {

[... TO BE CONTINUED ...]


Stacks - A Simple Sample[...] if $syslog_server { rsyslog::config { 'logstash_stack': content => template($syslog_config_template), } }

if $install_logstash { class { 'logstash': template => $logstash_config_template, } }

if $install_elasticsearch { class { 'elasticsearch': java_opts => $elasticsearch_java_opts, template => $elasticsearch_config_template, } }

[...]


Stacks - UsageOn any host:stack::logs { 'central': syslog_server => 'syslog.example42.com',}

On the Logstash (syslog) server:stack::logs { 'central': syslog_server => 'syslog.example42.com', install_logstash => true, elasticsearch_server => 'el.example42.com',}

On the Elasticsearch server(s), with a custom configuration file:stack::logs { 'central': syslog_server => 'syslog.example42.com', install_elasticsearch => true, elasticsearch_server => 'el.example42.com', elasticsearch_config_template => 'site/logs/elasticsearch.yml.erb',}

On the Kibana server:stack::logs { 'central': syslog_server => 'syslog.example42.com', install_kibana => true, elasticsearch_server => 'el.example42.com',}


TheStacksLogic

Stacks are localModules are shared

Higher level interface

Integrate different set of modules

Preserve moduleslocal change


ENC

How do we use Puppet today

Include classesmanifests/

site.pp

Set Parameters / Variables

Integration logic

Resources

ENC HIERA

SITE MODULES

SHARED MODULES

BAD

EDG

EConfiguration files

manifests/site.pp ENC HIERA

manifests/site.pp

BAD

?

BAD

?

BAD

SITE MODULES

SITE MODULES

SHARED MODULES

SITE MODULES

SHARED MODULES

manifests/site.pp

BAD

?

ENC

BAD

?

STACKS

STACKS

STACKS


Standardsfor

Stacks?

Usual benefits:

User Experience

Interoperability

Higher level API exposure

Possible GUI Integrations


Steps

Define stdmod naming conventions

Explore Stacks design and approach

Create templatesfor stacks and

modules

ExploreGUI integrations


SO�� Long and thanksfor all the fish!

Graphics: www.tatlin.net

@alvagante


SO�� Long and thanksfor all the fish!

Graphics: www.tatlin.net

@alvagante


Technology

Anatomy of a Reuseable Module