Upload
puppet
View
97
Download
1
Embed Size (px)
Citation preview
High Availability for Puppet
Russ Mull - @mullr Senior Software Engineer
Zack Smith - @acidprime Principal Professional Services Engineer
Enterprise Readiness: High Availability
Auto scaling Active/ Active
Robust Backup and Restore
tooling
Disaster recovery in Multi
datacenter/ Geo diverse
environments
Eliminate Single Points of Failure
Building capabilities that matter
Puppet Runs continue
High Availability for Puppet - Puppetconf 2016
Building a new catalogClassification, Exported Records, Hiera Data and puppet code being synced from version control
4
run
Puppet Code
Classifier
High Availability for Puppet - Puppetconf 2016
Two of Everything!don’t forget about your external services like git, ldap etc
5
Cl
Pup
Classifier
Pup
Classifier
Balancer“Load”
check
check backup
High Availability for Puppet - Puppetconf 2016
Puppet CA Replication Components
CA private key and cert
Signed Directory
Serial file
Certificate Revocation List (CRL)
8
serial
3E8
crl.pem
signed
101
ca_crt
High Availability for Puppet - Puppetconf 2016
CA Private key
ca
If you don’t care about revocation (security/revocation) this is the only file needed to replicate
9
ca_key.pem
ssl
ca_crt.pem
High Availability for Puppet - Puppetconf 2016
Signed Directory
signed
Used when checking for duplicate CN ( certs with the same name)
10
host1.company.com.pem
ca
High Availability for Puppet - Puppetconf 2016
Serial fileTracking the next numeric serial to be issued to new agent
11
serial
ca
3E8
decimal: 1000
decimal: 10013E9
High Availability for Puppet - Puppetconf 2016
Certificate Revocation ListTracking revoked certificates
12
crl.pem
cadecimal: 1000
decimal: 1001
1000
1001
serial3E9
High Availability for Puppet - Puppetconf 2016
Simply copy your ssldir ahead of the second installation
13
$ssldirscp -r
$ssldir
installer
CA
When using old versions of PE delete the pe-internal* certs post transfer , pre install
DR site
1.
2.
High Availability for Puppet - Puppetconf 2016
Streaming ReplicationThis happens as the postgres database layer
15
PostgreSQL PostgreSQL
PuppetDBPDB PuppetDBPDB
22
Read (Standby)Write
5432
write ahead logs
R
W1
2
16MB
High Availability for Puppet - Puppetconf 2016
Split Reads and writesCan survive temporary failures of the write master
16
PostgreSQL PostgreSQL
PuppetDBPDB
Read (Standby)Write
5432
W R
WWW
queue
High Availability for Puppet - Puppetconf 2016
Promote Standby to WritableThis happens as the postgres database layer
17
PostgreSQL
WriteRead (standby)
PostgreSQL
PostgreSQL
Write
5432
PostgreSQL
Read (standby)
5432
High Availability for Puppet - Puppetconf 2016
Master side FailoverThis is know as “terminus” failover as its handled in the puppetdb terminus package code
19
PuppetDBPDB
[main] server_urls = https://primary:8081, https://replica:8081
Primary
PuppetDBPDB
Replica1 2
High Availability for Puppet - Puppetconf 2016
Command Broadcastcommand_broadcast = true in puppetdb.conf
20
PuppetDBPDBPuppetDBPDB
High Availability for Puppet - Puppetconf 2016
PuppetDB Replication ReconciliationReconciliation happens on an interval
21
PostgreSQL
8081
Write
PostgreSQL
PuppetDBPDB
Write
PuppetDBPDB
Sync Interval
High Availability for Puppet - Puppetconf 2016
Simple HAMonolithic master + Replica
23
P Primary R Replica
High Availability for Puppet - Puppetconf 2016
Large Environment InstallationMonolithic master + Compile masters + Replica
24
R Replica
BalancerLoad
P Primary
BalancerLoad
High Availability for Puppet - Puppetconf 2016
New: Agent Side Failover!Shipping in Puppet 4.6 and higher, PE 2016.4+
25Primary Replica
1 2
High Availability for Puppet - Puppetconf 2016
Use Cases
● Puppet runs keep working
● Promote replica to master
26
When master is unreachable
High Availability for Puppet - Puppetconf 2016
When the master is unreachable
● Run puppet
● Promote replica to master
27
You can:
You can’t● Change classification
● Deploy new puppet code
● Issue new certs
● Use the Puppet Enterprise Console
● Use Application Orchestrator
High Availability for Puppet - Puppetconf 2016
Command Line Interface
29
puppet infra provision replica <hostname>replica.mycorp.net
puppet infra enable replica
replica.mycorp.net
puppet infra status. . .
High Availability for Puppet - Puppetconf 2016
1. Provision Replica
30
puppet infra provision replica <hostname>replica.mycorp.net
High Availability for Puppet - Puppetconf 2016
2. Monitor status of replication
31
puppet infra status> Per-service ‘alerts’> Visible in the UI as well
High Availability for Puppet - Puppetconf 2016
3. Enable replica
32
puppet infra enable replicareplica.mycorp.net
High Availability for Puppet - Puppetconf 2016
What’s a replica?
● Compile Master
● PuppetDB (r/w)
● RBAC, classifier, activity (r/o)
● Orchestrator data (not running)
● CA data (r/o using a proxy)
34
R ReplicaPuppetDBPDB
Console
High Availability for Puppet - Puppetconf 2016
File Sync - Compile Masters
M
Master of Masters
MOM
Compile Master
COMC
Compile Master
COMC
Compile Master
COMC
High Availability for Puppet - Puppetconf 2016
File Sync - CA Replica data
Primary Master Replica Master
R ReplicaP Primary
8140
ssl ssl
High Availability for Puppet - Puppetconf 2016
Puppet Enterprise CA Proxy
Primary Master Replica Master
R ReplicaP Primary
ssl CSR
High Availability for Puppet - Puppetconf 2016
PGLogical Replication
PostgreSQL
RBAC
NCClassifier 5432
5432
PostgreSQL
RBAC
NCClassifier
Write Read (Standby)
PuppetDBPDB
PuppetDBPDBNot synced
High Availability for Puppet - Puppetconf 2016
PE HA - ReplicationPuppetDBPuppetDBPDB PuppetDBPDB
PostgreSQL PostgreSQL
PGlogical
PGlogical
FileSync
Primary Replica
RBAC
NCClassifier
W
W
RBAC
NCClassifier
R
R
High Availability for Puppet - Puppetconf 2016
Puppet Enterprise HA - Beta Signup
Interested in what you heard? Please signup for our HA beta program through the Puppet Enterprise Support portal
42
High Availability for Puppet - Puppetconf 2016
43
https://goo.gl/Z85HLS
PE HA Beta Signup
Support Knowledge base
Z 8 5 H L S