Upload
nickchappell
View
115
Download
0
Embed Size (px)
DESCRIPTION
Citation preview
Logstash
MeSysadmin at NetXposure Inc.
“DevOps” because I sit next to the Java engineers (there wasn’t any
desk space next to the other sysadmin)
First grownup job
BS in Microbiology
https://github.com/nickchappell
Splunk
500MB/day limit with demo version
Whaddya do?
tail -f /var/log/**/* | grep somestuff ?
LogstashLog aggregator/router/processor
http://logstash.net/
Elasticsearch +
Logstash +
Kibana
The ELK stack
Logstash internalsWritten in Ruby, but uses JRuby for a runtime
(so it will actually use more than 1 core)
•Logs enter through inputs!•Each incoming log line gets represented internally as an event
with fields!•Filters process events (add, remove or transform fields and their
contents) •Outputs send events elsewhere
Codecs are like filters but can be used inline in inputs/outputs (decode an incoming JSON log line and turn the objects/hashes/arrays into individual event fields)
(show example logstash.conf in repo)
Getting logs into Logstash
rsyslog: http://www.rsyslog.com/ Logstash forwarder: https://github.com/elasticsearch/logstash-forwarder
(show rsyslog config and Puppet code)
...and configure your Logstash server with inputs
(show inpiut section of example logstash.conf)
• TCP/UDP ports for plain syslog • log4J/JMX inputs to take logs directly out of your JVM app • collectd/ganglia • SNMP traps
Set up your machines/devices to send logs....
Processing logs
(show example rsyslog config in VM)
Done with filters
Grok can do lots of heavy lifting!
(show example logstash.conf in repo)
Basic idea: use filters (grok, mutate, etc.) to pull bits of data out of the raw message and put
different pieces into the event's separate fields
You can add, rename and delete an event's fields
http://grokdebug.herokuapp.com/
Exporting logs from Logstash
Tons of different outputs:
One output is Elasticsearch...
•Email •Plain text files on the local machine Logstash is installed on •Nagios/Icinga •Graphite •fluentd/flume/riemann event stream processors •SaaS monitoring/metrics services like Librato, Boundary, PagerDuty
Elasticsearch
http://www.elasticsearch.org/overview/elasticsearch/
ElasticsearchWritten in Java, based on Apache Lucene
Main competitor for Lucene-based search engines is Solr
Can be used as a JSON-based document database
Has HTTP APIs for sending/receiving data
Has several client libraries for sending/receiving data of different languages (Ruby, Python, Java, PHP, etc.)
ElasticSearch
(show ElasticSearch kopf plugin page)
Really easy to cluster (multicast networking FTW)!
Individual documents are grouped into indexes
Indexes are broken into shards
Shards are shared with other nodes in the cluster
Masters coordinate reads/writes and shard replication
(show ElasticSearch node status JSON page)(show example logstash.conf in repo)
Kibana
http://www.elasticsearch.org/overview/kibana/
KibanaHTML5+JS+CSS dashboard/search tool for ElasticSearch
Really easy to deploy and configure to talk to Elasticsearch!
(show Kibana dashboard in browser)
Browser talks directly to Elasticsearch
(show example Kibana config.js)
Rashid Khan's Monitorama PDX 2014 workshop: http://vimeo.com/95297576
https://github.com/elasticsearch/kibana/blob/master/sample/nginx.conf
https://github.com/elasticsearch/kibana/blob/master/sample/apache_ldap.conf
That's cool and all, but so what?
Centralized system logging for admins is one use case...
Centralized application logs for your devs is a lot more compelling!
Demo repository
What the repo has: •1 Puppet master VM •1 Kibana VM •1 Logstash VM •2 Elasticsearch VMs •2 VMs running rsyslog to generate log dataWhat you'll need: •Vagrant •VirtualBox •An Ubuntu 13.10 base box
https://github.com/nickchappell/logstash_pdxdevops
Further instructions in README
Community#logstash on freenode
#elasticsearch on freenode
#kibana on freenode
https://github.com/elasticsearch/
https://github.com/elasticsearch/elasticsearch
https://github.com/elasticsearch/logstash
https://github.com/elasticsearch/kibana