Upload
mantas-klasavicius
View
8.026
Download
6
Embed Size (px)
DESCRIPTION
Logs management using Logstash, ElasticSearch and Kibana, Some extra content about statsd and graphite
Citation preview
Logs Management
infrastructure
logs management
logflow
logstash
Logstash tool for managing events and logs
Logstash written in JRuby
logstash inputs -> filters -> outputs
logstash “all in one” jar file
logstash plugins
Inputseventlog, file, s3, syslog, tcp, udp, websocket, wmi
Filterscsv, dns, geoip, grok, mutate
Outputselasticsearch, email, exec, mongodb, rabbitmq, redis
example
http://logstash.net/docs/1.2.2/
logstash example
input {stdin{
type => “example“ }}
with puppet
=
logstash example
output {redis {
data_type => "list" host => [’10.0.22.26'] key => "logstash”
}}
with puppet
=
logflow
redis
Redis open source, advanced key-value store
Redis buffer new logs from any type of slower parsing
logflow
logflow
logstash example
input {syslog {
port => 5544 type => "hapr" }}
with puppet
=
log4net example
logflow
logstash example
=output {
elasticsearch {cluster => "logs”embedded => false
index => "%{type}-%{+YYYY.MM.dd}”
node_name => "logstash_output" }}
with puppet
logflow
elasticsearch
ES distributed restful search and analytics engine
ES build on top of apache lucene
ES distributed, highly available
ES document oriented, schema free
ES restfull api
elasticsearchData Structure
http://stackoverflow.com/questions/15694724/shards-and-replicas-in-elasticsearch
Index
shards
replica
elasticsearchData Structure
http://stackoverflow.com/questions/15694724/shards-and-replicas-in-elasticsearch
Index
shards
replica
elasticsearchData Structure
http://stackoverflow.com/questions/15694724/shards-and-replicas-in-elasticsearch
Index
shards
replica
elasticsearchDemo
http://logs.adform.com:9200/_plugin/bigdeskhttp://logs.adform.com:9200/_plugin/paramedic/
logflow
kibana 3
kibana: HTML + JavaScript kibana: analytics and search interface to timestamped data sets stored in ElasticSearch
kibana: browser connects directly to ElasticSearch
kibana 3http://logs.adform.com/kibana
logflow
logflow
logflow
statsDstatsD: NodeJS daemon
statsD: extarcts metrics data and flushes to backend
statsD: counters, timers, gouges
450120553994334844675496
=>
mean_90 496upper_90 844
sum_90 3472upper 994lower 120count 8sum 4466mean 558.2
timers example:
http://blog.pkhamre.com/2012/07/24/understanding-statsd-and-graphite/
statsD
.NET 4.0 : https://github.com/robbihun/NStatsD.Client
C#: https://github.com/goncalopereira/statsd-csharp-client
.NET: https://github.com/peschuster/graphite-client
.NET 3.5 – 4.5: http://www.nuget.org/packages/StatsdCsharpClient/
https://github.com/etsy/statsd/wiki
clients
statsD
NStatsD.Client.Current.Increment("testing.increment");NStatsD.Client.Current.Decrement("testing.decrement");NStatsD.Client.Current.Timing("testing.timing", 2345);NStatsD.Client.Current.Gauge("testing.gauge", 45);
https://github.com/robbihun/NStatsD.Client
example
logflow
graphite
graphite: highly scalable real-time graphing system graphite: good when you don’t know the names metrics
graphite
DEMO
web
graphite
.NET: https://github.com/peschuster/graphite-client
• WCF• MSBuild• ELMAH• SQL Server• PerfCounterMonitor.exe (Graphite.System)
• Performance counters• Event log• IIS Application Pools (with)
clients
what's next
what's next
LEARN!!!
what's nextdashboards
http://techblog.netflix.com/2012/12/hystrix-dashboard-and-turbine.htmlhttp://shopify.github.io/dashing/#widgetshttp://fdietz.github.io/team_dashboard/https://github.com/ripienaar/gdash