Technical Architecture - University of Edinburgh€¦ · Web viewA Git hook (a method to trigger tasks when updates are made) is used to trigger R10K to run when a commit has been

Puppet Configuration Management

Technical Architecture

Version 1.0

26/02/2014

Puppet Configuration ManagementTechnical Architecture Version 1.0 26/02/2014

Contents1 Project Context............................................................................................................51.1 APP004-20 Automation Tools - Options Appraisal.......................................................51.1.1 Key Stakeholders..........................................................................................................51.1.2 Key Deliverables...........................................................................................................51.1.3 Summary of technical changes.....................................................................................5

2 Service Description.....................................................................................................62.1 Key Technologies..........................................................................................................7

3 Users, Roles and Groups...........................................................................................8

4 Application / Database Connectors...........................................................................94.1 Live Environment..........................................................................................................9

5 Application Server Requirements.............................................................................9

6 Database Server Requirements.................................................................................9

7 Portal Requirements.................................................................................................10

8 Financial Requirements............................................................................................108.1 Estimated Capital Costs..............................................................................................108.2 Annual Recurrent Costs..............................................................................................10

9 Application URLs......................................................................................................109.1 Live..............................................................................................................................10

10 File System Requirements.......................................................................................1110.1 File System Shares.....................................................................................................11

11 Scheduled Tasks.......................................................................................................1111.1 Live..............................................................................................................................11

12 Technical Diagrams..................................................................................................12

13 Backup Policy............................................................................................................13

14 Disaster Recovery Requirements............................................................................13

15 Resilience Measures.................................................................................................13

16 External Access........................................................................................................13

17 Security Considerations...........................................................................................13

18 Authentication and Authorisation...........................................................................14

19 Load Testing..............................................................................................................14

20 Interfaces and Dependencies..................................................................................14

Page 2 of 16


21 Firewall Configuration..............................................................................................14

22 Software Licence Requirements.............................................................................14

23 Patching and Machine Restarts...............................................................................15

24 Exceptions and Other Issues...................................................................................15

25 Document Sign-off....................................................................................................15

Page 3 of 16


Version ControlDate Version Author Sections Amendments26/02/2014 1.0 Riky Harris All Initial draft

Page 4 of 16


1 Project Context

1.1 APP004-20 Automation Tools - Options Appraisal

1.1.1 Key Stakeholders

Role Unit NameTechnical Architect (Owner) IS Apps Development

TechnologyRiky HarrisEwan Scott

Peer Reviewer IS Apps Development Technology

Iain Fiddes

Project Manager IS Apps Project Services Adam Wadee

Production Management Representative

IS Apps Technology Management

Heather Larnach

IS ITI Representative IS ITI Unix Martin Campbell

1.1.2 Key Deliverables

Deliverable Business BenefitInstallation of a Puppet Master A central server to manage Puppet manifests and Puppet

certificates for client servers to connect to, allowing for the consistency, repeatability, and accuracy of the configuration of servers setup and managed by DevTech and TechMan

Installation of Foreman A web application to allow an easy way for reports and Facts sent from Puppet agents to be viewed, and is able to deploy configurations to new ‘bare metal’ servers

Installation of R10K Puppet DSL, Hiera configuration and third party modules can be imported from Git and Puppet Forge in a more streamlined and manageable workflow allowing for growth of the infrastructure

Installation of MCollective Gives an easy method for commands to be run on multiple servers and can be used to trigger Puppet agent runs

Use of Veeam VM replication The Puppet Master is replicated to a secondary site allowing near-zero data loss and very little downtime in a DR event

Written documentation on Puppet usage and workflow

Allows users to follow a set of standards when writing code and configuration and when deploying these to servers

1.1.3 Summary of technical changes

As part of APP004-20, a Technical Design for a Puppet Master has been produced. This contains all of the proposed services required for the intended future workflow: Puppet, Hiera, R10K, MCollective and Foreman. All of these are installed on a single RHEL VM with Veeam replication of the server to a secondary site.

Page 5 of 16


Page 6 of 16


2 Service DescriptionPuppet is a configuration management system and can be used to apply and enforce the setup of servers from a central location. It works in a master-slave configuration, with a Puppet Agent installed on each client server (known as a ‘node’), requesting its correct configuration (a ‘manifest’) from the Puppet Master. Facter is part of Puppet, and is used to gather information (‘Facts’) about a node, which can be used to control how manifests are applied and is useful to see information about servers.

By using Puppet and the extra tools, a new workflow is being used by IS Applications where server configuration can be represented as ‘code’. This allows the same configurations to be reliably deployed to servers and keeps environments consistent which has become vital as the infrastructure configured and managed by the department has grown so vast, the inefficiencies of handcrafting artisan servers has become too great.

This configuration is represented as a combination of Puppet Domain Specific Language for Puppet modules, YAML for Hiera settings and Ruby for Puppet templates and is all stored centrally within repositories on the IS Applications Git service. It is vital that a Version Control System is used so that changes to configuration can be tracked and easily reverted if issues are found.

Hiera is an accompanying tool to substitute different variables in Puppet DSL dependant on various qualities of the server, in a prioritised, hierarchical manner. This allows servers of different environments or sites to have different settings while being able to use the same Puppet modules.

R10K is used to download requested versions of local and third party modules from Git and Puppet Forge (a central repository of Puppet modules for public use) onto the Puppet Master. A Git hook (a method to trigger tasks when updates are made) is used to trigger R10K to run when a commit has been made to the various Puppet repositories on the IS Applications Git service.

As changes made to the Puppet and Hiera configuration can make massive changes to many servers, peer review takes place before commits to Git are merged into the “Trunk” of the central repositories. There are also Git hooks in place to perform ‘lint testing’ on Puppet manifests to ensure that syntax errors are caught before updates can be committed to Git. This testing can be further developed in the future by using a Continuous Integration style of workflow for the development, automated testing and potentially even deployment of Puppet manifests.

Page 7 of 16


Foreman is used to give a visual representation of the Puppetised infrastructure and is also used for the initial configuration of new servers. It can display reports on the Puppet agents and show discovered Facter facts. It also can deploy configurations onto new ‘bare metal’ servers, including cloud infrastructures and virtual hosting services like IS ITI’s VMware ESX.

MCollective (“The Marionette Collective”) is another tool produced by Puppet Labs which is not required by Puppet nor requires Puppet itself. It is very useful though and is commonly found as part of Puppet deployments, as it gives the ability run commands with elevated privileges on multiple servers. It is being used to trigger Puppet agent refreshes, and it is possible to use this to perform tasks on multiple servers for project Implementation tasks.

The terminology is oddly reversed though, MCollective requires a single Client (which runs on the Puppet Master) to send requests to the many Servers (which each of the Puppet nodes runs one of) via a single Middleware service which handles the communications (also runs on the Puppet Master and uses ActiveMQ to communicate with the STOMP protocol with added SSL).

2.1 Key Technologies

Technology Version New or existingPuppet Labs Puppet 3.5.1 NewPuppet Labs Hiera 1.3.2 NewPuppet Labs MCollective 2.5.0 NewR10K 1.3.0 NewForeman 1.5.1 NewOracle MySQL 5.1.73 Community Edition ExistingApache HTTPD 2.2.15 ExistingRed Hat Enterprise Linux 6.5 x64 ExistingRuby 1.9.3 Existing (as 2.1.0 for GitLab)Apache ActiveMQ 5.8.0 New

Page 8 of 16


3 Users, Roles and Groups

Application Username

Home Directory Description

puppet /var/lib/puppet User for Puppet master and Foreman applicationforeman /usr/share/foreman User for Foreman applicationforeman-proxy /usr/share/foreman-

proxyUser for Foreman proxy daemon

Group Name Members Descriptionpuppet puppet, foreman,

foreman-proxyGroup for the puppet users

foreman foreman Group for the foreman userforeman-proxy foreman-proxy Group for the foreman-proxy user

Database Username Roles Descriptionpuppetlab@[email protected]

“All” User for Foreman application connection

OPS$ Username Roles DescriptionN/A

Database Role Description“All” SELECT, LOCK TABLES, INSERT, UPDATE, DELETE, CREATE, DROP,

INDEX, ALTER - full access to all MySQL privileges

Database Schema Tablespace Descriptionpuppetlabhq_production N/A Database for Foreman application

Page 9 of 16


4 Application / Database ConnectorsNB: There is no Development or Test environment. Ruby applications frequently refer to the Live environment as “Production”.

4.1 Live Environment

Application Name

Puppet Hiera MCollective R10K Foreman

Technology N/A N/A N/A N/A Ruby mysql2 gem

Username puppetlabInstance puppetlabhq_p

roductionAdditional settings

5 Application Server Requirements

Development Test LiveServers puppet.is.ed.ac.uk

Usernames puppet

Application home directories

/var/lib/puppet

Software and versions

Puppet 3.5.1

Additional dependencies

Hiera 1.3.2MCollective 2.5.0

6 Database Server Requirements

Development Test LiveServers puppet.is.ed.ac.ukInstance names puppetlabhq_productionDatabase versions MySQL 5.1.73Project specific requirements

Page 10 of 16


7 Portal Requirements

Development Test LivePortal channel N/A N/A N/A

8 Financial RequirementsThere is a 5 year cost for running puppet.is which is being met by the IS Apps central budget.

8.1 Estimated Capital Costs

Item Development Test Livee.g. hardware £ £ £e.g. disk on SAN £ £ £e.g. backup £ £ £e.g. licences £ £ £

Subtotals £ £ £Estimated Total £

8.2 Annual Recurrent Costs

Item Development Test Livemaintenance £ £ £support £ £ £backups £ £ £

Subtotals £ £ £Total £

9 Application URLs

9.1 Live

Application URLForeman https://puppet.is.ed.ac.uk

Page 11 of 16


10 File System Requirements

Server Volume Size Purposepuppet.is / 20Gb OSpuppet.is /u01 180Gb Data

10.1 File System Shares

Server Path Shared as Users / Groups PermissionsN/A

11 Scheduled Tasks

11.1 Live

Task Schedule Username Descriptioncron: MySQL DB Backup

Nightly oracle mysqldump of puppetlabhq_production database

Something else?

Page 12 of 16


12 Technical Diagrams

Server Estate

Puppet 25/06/2014

APP004-20

CentralFirewall

User connecting to Foremanvia web interface

HTTPS

HTTPS

SSH

SSH

Appleton TowerKings Buildings

CentralFirewall

User running commands on agentsvia MCollective

Veeam VM Replication

HTTPS

Puppet Master VM Offline Replicated Puppet Master VM

Managed Server Estate

STOMP+SSL

Page 13 of 16


13 Backup Policy

Component Detail any variance from the currently agreed backup policyOperating System backup

Veeam replica is the backup

Database backup Veeam replica is the backupFilesystem backup Veeam replica is the backupOther N/A

14 Disaster Recovery Requirements

Application Category Recovery URLPuppet 2/1? N/AMCollective 3 (2/1 if required for deployment) N/AR10K 2/1? N/AForeman 3 (2/1 if required for deployment) N/A

15 Resilience MeasuresThe Puppet Master server is constantly having the changes made to it replicated to a standby VM at the Appleton Tower site. In Disaster Event, the replica VM can be brought online very quickly and with near-zero data loss.

16 External Access

External Details (telephone number, email etc.)

Access Type Authentication and authorisation methods

Description of need

N/A

17 Security ConsiderationsThe Puppet tool is very powerful, and therefore a rigorous process needs to be followed. In the same way, MCollective’s ability to run commands on multiple servers with elevated privileges also makes it very useful, but also needs to be used with care. Thorough peer-review and testing of all Puppet and Hiera code must be used to ensure that mistakes are not made.

Page 14 of 16


Day to day updates of configurations are made via changes to the repositories being pushed through by Git hooks calling R10K. Manual use of MCollective should be used for special tasks and are audited by the Kerberised logins to the Puppet Master server.

18 Authentication and AuthorisationThe Puppet deployment process is mainly done by changes made to the Git repositories. Therefore the authentication and authorisation section of the Git service TAD should be read for further details (K:\ISAPPS\dsg\DevelopmentTechnology\ServiceDocs\Infrastructure\ Git\Technical Architecture\Git Technical Architecture.docx).

User accounts on Foreman will be initially created via the Foreman interface by administrative users (IS Applications team leaders). Users will receive their initial passwords and will be able to log into the Foreman interface.

19 Load TestingNo load testing has been performed on the service, however, as the infrastructure grows and more clients connect to the service, horizontal scaling of the Puppet Master VM may be required. It is possible to run with multiple Puppet Masters, but this greatly increases the complexity of the setup and this vertical scaling should be avoided until it becomes the only sensible method to increase the capacity of the service.

20 Interfaces and DependenciesThe Puppet service depends on the Git hosting service for providing updates to the configurations. If the Git service is unavailable, it won’t be possible to push updates to the configurations to the Git “Origin” and so it won’t be possible to make lasting changes to server setup via R10K.

21 Firewall Configuration

Source Destination Port ProtocolEduninets puppet.is.ed.ac.uk 443 HTTPSpuppet.is.ed.ac.uk Eduninets 8139 HTTPSEduninets puppet.is.ed.ac.uk 8140 HTTPSpuppet.is.ed.ac.uk Eduninets 61614 STOMP+SSL

Page 15 of 16


22 Software Licence Requirements

Software and Supplier Details

Licence Requirements

Already Licensed? (Y/N)

New or Additional Licence Costs

Red Hat Enterprise Linux

1x x64 server Y New (included by IS ITI Unix as part of VM)

23 Patching and Machine Restarts

Has restart been tested to confirm that policy has been achieved? Y / N

24 Exceptions and Other IssuesNone.

25 Document Sign-off

Architect Name Name (date)IS ITI Representative Name (date)Project Manager Acceptance Document URL of email or PPDRBusiness Partner Acceptance Document URL of email or PPDR

Page 16 of 16

Documents

Technical Architecture - University of Edinburgh€¦ · Web viewA Git hook (a method to trigger tasks when updates are made) is used to trigger R10K to run when a commit has been