Upload
mrichardson
View
1.590
Download
2
Embed Size (px)
DESCRIPTION
Techniques for allowing your systems to share information with each other. Presented at the October Sydney DevOps Meetup
Citation preview
Node collaboration
Michael Richardson @m_richo
Sharing information between your systems
Agenda1.Chef Search2.Puppet Exported
Resources3.PuppetDB query4.Confd + etcd
Agenda1.Chef Search2.Puppet Exported
Resources3.PuppetDB query4.Confd + etcd
Chef Server
Stores many useful things including• Policies applied to nodes• Metadata that describes each
node (ohai)• And more…
Chef ServerThis data is searchable!
Chef ServerThis data is searchable!
Chef ServerThis data is searchable!
But how?
Chef Server
1. ‘search’ method within recipe.2. Search subcommand of ‘knife’3. /search or /search/INDEX endpoints of
Chef Server API
Chef Server
1. ‘search’ method within recipe.2. Search subcommand of ‘knife’3. /search or /search/INDEX endpoints of
Chef Server API
Chef ServerSearch within recipe – simple.
Chef ServerSearch within recipe – simple.
Chef ServerSearch within recipe – simple.
Chef ServerSearch within recipe – simple.
It’s that simple
Chef ServerSearch within recipe – more complex.
Chef Server
Chef Search is fantastic for
• Searching for nodes with particular attributes
• Searching for nodes with particular profiles
• Using search results to configure services on other nodes.
Chef ServerCommon use cases
• My application talks to a database. What’s its FQDN/IP?• How many memcache nodes are there for my application?
What are there FQDN/IPs?• Where should I send my log files? What’s the FQDN of the
syslog server?• I want to provide a white-list of IP’s that can request
information from a node.• What are the IP addresses of the webservers behind my load
balancer?• My Jenkins slaves need to connect to a Jenkins master.
What’s it’s IP?• What are the SSH Host keys for particular hosts.
Agenda1.Chef Search2.Puppet Exported
Resources3.PuppetDB query4.Confd + etcd
Exported Resources
Definition:An exported resource declaration specifies a
desired state for a resource, does not manage the resource on the target system, and publishes the
resource for use by other nodes. Any node (including the node that exported it) can then
collect the exported resource and manage its own copy of it. *
* https://docs.puppetlabs.com/puppet/latest/reference/lang_exported.html
Exported Resources
Think of it as
• Node(s) “publish” resources to the Puppet Master
• Other Node(s) consume those resources
* https://docs.puppetlabs.com/puppet/latest/reference/lang_exported.html
Exported Resources
Common Examples• Backend servers configured in a load
balancer• Monitoring Servers updated with monitoring
clients (nagios_host / nagios_service)• Distribute public keys (ssh and openssl)
Agenda1.Chef Search2.Puppet Exported
Resources3.PuppetDB query4.Confd + etcd
PuppetDB
Definition:collects data generated by Puppet. It enables
advanced Puppet features like the inventory service and exported resources, and can be the foundation for other applications that use Puppet’s
data. *
PuppetDB
PuppetDB stores:• The most recent facts from every node• The most recent catalog for every
node• Optionally, 14 days of event reports for
every node• Exported Resources
PuppetDB
PuppetDB stores:• The most recent facts from every node• The most recent catalog for every
node• Optionally, 14 days of event reports for
every node• Exported Resources
Useful information
PuppetDB
PuppetDB REST APIV3 API includes the following endpoints:• Facts• Resources• Nodes• Fact-names,• Metrics
• Reports• Events• And more
PuppetDB
PuppetDB
PuppetDB
But this talk is about Node Collaboration.How can I make use of this data?
Awesome
PuppetDB
checkoutPuppetdbquery module
by Erik Dalén
https://forge.puppetlabs.com/dalen/puppetdbqueryhttps://github.com/dalen/puppet-puppetdbquery
PuppetDB
Puppetdbquery module
• Command line tools• Puppet functions to query
PuppetDB• Hiera backend to return query
results from PuppetDB
PuppetDB
Puppetdbquery module
• Command line tools• Puppet functions to query
PuppetDB• Hiera backend to return query
results from PuppetDB
PuppetDB
Puppetdbquery functions
• query_nodes• query_facts
PuppetDB
query fact
PuppetDB
note: Output is abbreviated
array
hash
PuppetDB
PuppetDBquery• Works great and very similar to
Chef Search (searching for specific hosts, and use their facts for configuring other hosts).
• If you use Roles+Profiles pattern, this is a fantastic fit for profiles.
Chef Search and PuppetDB are great!
Chef Search and PuppetDB are great!but what if…
• You want your infrastructure to react more quickly than your Puppet agent / Chef-client run intervals (are you even running the agents continuously).
Chef Search and PuppetDB are great!but what if…
• You want your infrastructure to react more quickly than your Puppet agent / Chef-client run intervals (are you even running the agents continuously).
• You appreciate “desired state” AND “running state” aren’t always the same thing.
Chef Search and PuppetDB are great!but what if…
• You want your infrastructure to react more quickly than your Puppet agent / Chef-client run intervals (are you even running the agents continuously).
• You appreciate “desired state” AND “running state” aren’t always the same thing.
• You don’t want to run a Puppet Master or Chef Server (masterless Puppet / Chef-solo).
Chef Search and PuppetDB are great!but what if…
• You want your infrastructure to react more quickly than your Puppet agent / Chef-client run intervals (are you even running the agents continuously).
• You appreciate “desired state” AND “running state” aren’t always the same thing.
• You don’t want to run a Puppet Master or Chef Server (masterless Puppet / Chef-solo).
• You don’t use Puppet or Chef.
Chef Search and PuppetDB are great!but what if…
Agenda1.Chef Search2.Puppet Exported
Resources3.PuppetDB query4.Confd + etcd
Confd
• “Lightweight configuration management tool”• https://github.com/kelseyhightower/confd• Written in go• Manage local application configuration files using
templates and data from etcd or consul.
etcd
• Highly-available key/value store for shared configuration and service discovery
• https://github.com/coreos/etcd• Written in go• Typical cluster size or 3-9 peers.• CAP theorem
Confd + etcd
etcd1
Distributed etcd cluster
etcd2 etcd3 etcd4 etcd5
Hosts running applications and services configured with confd
Confd + etcd
• Hosts/Services that wish to share information post data to etcd.
• Confd is configured on other hosts to use this data to manage local configuration files and services.
Confd + etcd
Example1. App server runs Confd which is polling the following keys.
1. /myapp/db_ip2. /myapp/db_port3. /myapp/db_adapter4. /myapp/db_name
2. Once up and running DB Server, writes details to etcd cluster.3. Confd reads key/values. Uses template file to write new
application configuration file and reloads application.
etcd1
Distributed etcd cluster
etcd2 etcd3 etcd4 etcd5
App server DB server
21
Confd + etcd
Demo
A “working” demo is worth a thousand powerpoint slides…
Summary
• Share information between your servers.
• Let them discover each other.• Let your nodes collaborate with each
other.
Thank you