Workflow story: Theory versus Practice in large enterprises by Marcin Piebiak

1

Marcin PiebiakSolutions ArchitectLinux Polska Sp. z o.o.

Workflow story – theory versus practice in Large Enterprises

2

What is it?

package { 'ssh': ensure => 'installed', name => 'openssh-server',}

service { 'ssh': ensure => 'running', name => 'sshd', require => Package['ssh'],}

● ResourcesA lot of ready to use resources:

• user • group • host • cron • exec • file • package • service • ...

• Resources are building blocks.• They can be combined to make larger components.• Together they can model the expected state of your system.

3

package { 'ssh': ensure => 'installed', name => 'openssh-server',}

service { 'ssh': ensure => 'running', name => 'sshd',}

● Resources● Declarative language

Rather than listing a series of steps to carry out we can describe the desired final state only.

What is it?

4

package { 'ssh': ensure => 'installed', name => 'openssh-server',}

root@debian ~]# apt-get install openssh-server

root@redhat ~]# yum install openssh-server

● Resources● Declarative language● Abstraction

Resources in Puppet are abstracted from underlying providers.

What is it?

5

root@redhat ~]# facterarchitecture => x86_64domain => linuxpolska.plfacterversion => 1.5.2fqdn => redhat.linuxpolska.plhardwaremodel => x86_64hostname => redhatinterfaces => eth0ipaddress => 172.16.10.1kernel => Linux...

● Resources● Declarative language● Abstraction● Facts

Puppet uses facter to gather information about the host system.

Custom Facts

Facter.add('role') do setcode do 'cat /etc/role' endend

What is it?

6

$pkg_name = hiera('pkg_name')

package { 'apache': ensure => 'installed', name => $pkg_name,}

● Resources● Declarative language● Abstraction● Facts● Data separation

Puppet uses Hiera as its single source of truth data abstraction layer.

Hiera uses Facter facts to determine a hierarchy.

---:backends: - yaml:yaml: :datadir:/etc/hiera:hierarchy: - %{fqdn} - %{osfamily} - %{environment} - common

What is it?

7

class ssh ( $pkg_name = 'openssh-server', $srv_name = 'sshd',) { package { 'ssh': ensure => 'installed', name => $pkg_name, }

service { 'ssh': ensure => 'running', enable => 'true', name => $srv_name, require => Package['ssh'], }}

● Resources● Declarative language● Abstraction● Facts● Data separation● Reusable code

Classes define a collection of resources that are managed together as a single unit.

What is it?

8

ssh├── manifests│ ├── init.pp│ └── server.pp├── files│ └── ssh_config└── templates └── sshd_config.erb

● Resources● Declarative language● Abstraction● Facts● Data separation● Reusable code

Modules are directories that contain your configuration. They are designed to encapsulate all

of the components related to a given configuration in a single folder hierarchy.

Classes are abstracted by modules:

What is it?

9

● Resources● Declarative language● Abstraction● Facts● Data separation● Reusable code

If a more complex deployment is needed, reusing existing classes saves effort and reduces error.

What is it?

10

● Resources● Declarative language● Abstraction● Facts● Data separation● Reusable code

You own abstraction layers...Profiles:class profiles::application { include tomcat include mysql include myapp}class profiles::base { include ssh include ntp include users}

Roles:class role::webapp { include profiles::base include profiles::customapp include profiles::test_tools}

What is it?

11

● Resources● Declarative language● Abstraction● Facts● Data separation● Reusable code● Supports many OS

Supported OS

What is it?

12

● Resources● Declarative language● Abstraction● Facts● Data separation● Reusable code● Supports many OS

Big environment – no problem

Changes?

base/ntp.yaml --- ntp::local_servers: - 192.168.0.45+ - 192.168.0.1

What is it?

13

● Resources● Declarative language● Abstraction● Facts● Data separation● Reusable code● Supports many OS● Provisioning● Orchestration● Puppet Forge● Live Management● Environments● Reporting● Dry-run mode

And more...

VM/Cloud Provisioning

Orchestration

Live Management

MCollectiveReporting

Custom:- types & providers- facts- functions

Environments

Dry-run mode

What is it?

14

● Resources● Declarative language● Abstraction● Facts● Data separation● Reusable code● Supports many OS● Provisioning● Orchestration● Puppet Forge● Live Management● Environments● Reporting● Dry-run mode

What is it?

Thanks to these all superior features, Puppet is:- fast in deployment- easy to use- universal for a lot of operating systems- with unlimited possibilities- easy to expand- flexible

15

VS.

In large enterprises like banks, telco, insurance, etc. those features are not the most relevant. Implementing Puppet in enterprises we must consider another

priority map, another mindset.

We must answer not trivial questions dealing with the core IT Departments way of work.

16

VS.

So... everyone hasaccess to everything?!

17

VS.

So... can I destroythe whole infrastructure

from one place?

18

SolutionPuppet master installation

on hardened system with limited direct access.

tcp:8140

tcp:443

tcp:22

ssh keys

RBAC

For maintenance.

19

Solution

Git as a communication layer between developers and puppet master.

Release manager

developer

developer

developer

Pull request

Fetch

20

VS.

So... everyone now mustuse puppet?!

21

Solution

On beginning each department can have own environments.

Developers

Systems administrators Databases

administrators

Securitydepartment

22

SolutionEach department can have

many environments and its own idea how to organizework with puppet. Security

department

prodtestdev

Integrator

Releasemanager

Testsintegrator

23

SolutionEach department can have

many environments and its own idea how to organizework with puppet. Security

department

prodtestdev

Integrator

Releasemanager

Testsintegrator

v0.2v0.1

commits

Release

24

Solution

After time some departments will start working together in one environment.

Developers

Systems administrators Databases

administrators

Securitydepartment

integrator

25

SolutionDevelopers

Systems administrators Databases

administrators

Securitydepartment

integrator

integrator

26

Solution

At the end all departments will use one environment.

Developers

Systems administrators Databases

administrators

Securitydepartment

integrator

integrator

27

VS.

How can I find outwho made what change

and who approvedthis modification?

28

Solution

prodtestdev

v0.2v0.1

commits

Release

commit 220938c5a2e51ecd4166eb7d75d14974cbcff897Author: Marcin Piebiak <mpp@linuxpolska.pl>Date: Fri Jul 5 11:27:43 2013 +0200

Description....

Person who approvedmodifications.

Using git we have:● date● author● description

We can use git as a place for history of the infrastructure.● git status● git log● git diff● git blame

Git history is cryptographically secured.

29

VS.

If I have a lot ofenvironments how can

I use them?

30

Solution

tcp:8140

prodtestdev

We can specify a set of environments for each host to use.

Database testing system, uses three testing environments from different departments.

31

VS.

Great!But... using commandline I can connect to

different environments!

32

Solution

tcp:8140

puppet agent -t --environment

We use imp module, to control puppet agents behavior and their access to environments.

33

Solutionpuppet.conf

[main]modulepath = /etc/puppetlabs/puppet/modules:/opt/puppet/share/puppet/modulesmanifest = /etc/puppetlabs/manifests/site.pp

[env_sec_prod]modulepath = /var/lib/git/env_sec_prod/modules:/etc/puppetlabs/puppet/modules:/opt/puppet/share/puppet/modules

[env_sec_test]modulepath = /var/lib/git/env_sec_test/modules:/etc/puppetlabs/puppet/modules:/opt/puppet/share/puppet/modules

site.pp

include imp

hiera_include('include')

/etc/puppetlabs/puppet/modules – place for modules from forge and well tested modules and module imp

/etc/puppetlabs/manifests/site.pp - common manifest for all environments

/var/lib/git/$environment/modules – git repository for environment

imp.yaml

---imp::environments: env_sec_prod: order: 'deny,allow' deny: 'all' allow: - 'host1.linuxpolska.lab' - 'www.*.linuxpolska.lab' commiters: - 'john.smith' priority: 1

34

VS.

How can I auditchanges made in

the infrastructure?

35

Solution

Log collector and analyzer.

puppet reports

Puppet reports

When nodes fetch their configurations from the puppet master, they send back inventory data and a report of their run.

36

VS.

How can puppet helpme with audit? How can

I recreate life cycle of each host?

37

Solution

Log collector and analyzer.

● puppet reports● puppet agents catalog● hosts facts● git diffs after commit● hiera configuration for each host● filebuckets● ...

38

Solution

app db system security

file {'/etc/important': ensure => 'file', group => 'apache', mode => '0660',}

file {'/etc/important': ensure => 'file', user => 'root', group => 'root', mode => '0600',}

If we have many environment there is always risk of overwriting someone's changes.

Log collector and analyzer.

39

VS.

How will the modification in puppet

manifestsaffect the wholeinfrastructure?

40

Solution

prodtestdev

v0.2v0.1

commitsLog collector

and analyzer.

Report from puppet normal run.

Report from puppet dry-run.

Using log collector we can analyze the infrastructure modifications before they get to production environment.

41

VS.

All changes are madeautomatically? First I would

like to see what is going to be changed.

42

VS.

How can we rollbackchanges?

43

VS.

After we installpuppet will we know

everything about the infrastructure?

44

THE END

Marcin PiebiakSolutions ArchitectLinux Polska Sp. z o.o.