View
3
Download
0
Category
Preview:
Citation preview
ELKTake care of your logs with
@MattKetmo – IPC Berlin 2015
Matthieu Moquet@MattKetmo
web engineer at
Logs
Why logs?Debug
Understand code Execution workflow
Metrics Monitoring
Which logs?
Syslog
Access Logs
Application Logs
How to view logs?
ssh example.org tail -‐f /var/log/some.log
multitail colortail rainbow
grc …
You should stop doing this!
! Not easily readable ! No aggregations ! Bad performances on concurrent processes
Dataviz > Log messages
Send your logs
AWAY
Open Source
ELK
Elasticsearch Logstash Kibana
3 independent softwaresFlexible
Performant Scalable
« Elasticsearch is a search server based on Lucene. It provides a distributed, multitenant-capable full-text search engine with a RESTful web interface and schema-free JSON documents »
— Wikipedia
elasticsearch
elasticsearch
Aggregations
Real-Time Data
Distributed
RESTful API
Schema-Free
High Availability
elasticsearch
Download at https://www.elastic.co/downloads/elasticsearch
bin/elasticsearch
elasticsearch
See it running…
curl -‐X GET http://localhost:9200/
elasticsearch
Index a document
curl -‐XPUT http://localhost:9200/users/user/mattketmo -‐d '{ "name": "Matthieu Moquet", "age": 26}'
elasticsearch
Get a document
curl -‐XGET http://localhost:9200/users/user/mattketmo
elasticsearchUse aggregations
curl -‐XPOST "http://localhost:9200/users/user/_search" -‐d' { "size": 0, "aggregations": { "age_avg": { "avg": { "field": "age" } } } }'
Logstash
Log Processing / ETL
Input
Output
Filters
echo "[ALERT] Some error message" | sed -‐e "s/^\[\(.*\)\] \(.*\)/\1,\2/" > output.csv !
cat output.csv ALERT,Some error message
Logstash
Logstash
gelf syslog
nginx logs varnish logs
udp …
date dns
geoip grok
urldecode …
elasticsearch redis
graphite nagios zabbix
…
Inputs Filters Outputs
LogstashDownload at https://www.elastic.co/downloads/logstash
bin/logstash -‐e ' input { stdin {} } output { stdout {} }'
LogstashDownload at https://www.elastic.co/downloads/logstash
bin/logstash -‐f /path/to/logstash.conf
Logstashinput { ... } filter { ... } output { elasticsearch_http { host => "elasticsearch.tld" port => 9200 index => "logstash-‐%{+YYYY.MM.dd}" } }
Curator
https://github.com/elastic/curator
Allow you to remove old indexes (via a CRON)
curator -‐-‐host 10.0.0.2 delete indices \ -‐-‐older-‐than 30 -‐-‐time-‐unit days \ -‐-‐timestring '%Y.%m.%d'
Heka
« Heka is an open source stream processing software system developed by Mozilla. Heka is a “Swiss Army Knife” type tool for data processing. »
http://hekad.readthedocs.org
«!Fluentd is an open source data collector for unified logging layer!»
Kibana
! Data visualization web app !Many graphes (histogram, pie chart, geo map, …) ! Built in HTML / CSS / Javascript
KibanaDownload at https://www.elastic.co/downloads/kibana
bin/kibana
upstream es_backend { server 127.0.0.1:9200; keepalive 64; } !server { listen 80; server_name kibana.tld; root /var/www/kibana; try_files $uri $uri/ index.html @elasticsearch; ! location @elasticsearch { proxy_pass http://es_backend; proxy_read_timeout 90; proxy_redirect off; proxy_http_version 1.1; proxy_set_header Connection ""; proxy_set_header X-‐Real-‐IP $remote_addr; proxy_set_header X-‐Forwarded-‐For $proxy_add_x_forwarded_for; proxy_set_header Host $http_host; proxy_pass_header Access-‐Control-‐Allow-‐Origin; proxy_pass_header Access-‐Control-‐Allow-‐Methods; proxy_hide_header Access-‐Control-‐Allow-‐Headers; add_header Access-‐Control-‐Allow-‐Headers 'X-‐Requested-‐With, Content-‐Type'; add_header Access-‐Control-‐Allow-‐Credentials true; } }
Kibana 3 & nginx
Basic pipeline
Indexer
Our current pipeline
Shipper + Indexer
Broker
Kafka pipeline
1M+ messages per second
Logstash >= 1.5.0
How to forward logs?
Forwarding Syslog
RSYSLOGThe rocket-fast system for log processing
*.* @127.0.0.1:514;RYSLOG_ForwardFormat
Logstash input
input { udp { port => 514 type => syslog } }
Logstash filter
filter { if [type] == "syslog" { grok { match => ["message", "<%{POSINT:syslog_pri}>%{TIMESTAMP_ISO8601:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %{GREENDYDATA:syslog_message}"] add_field => ["received_at", "%{@timestamp}"] add_field => ["received_from", "%{host}"] add_tag => ["rsyslog"] } } }
Forwarding Access Log
logstash-forwarderA tool to collect logs locally in preparation for processing elsewhere
https://github.com/elastic/logstash-forwarder
logstash-‐forwarder -‐config conf.json
logstash-forwarder... { "paths": [ "/var/log/nginx/access.log" ], "fields": { "type": "nginx-‐access" } } ...
logtailSmall Perl script to read HTTP access logs
and send it directly to Redis !
https://github.com/shtouff/logtail
Forwarding Application Logs
Monologhttps://github.com/Seldaek/monolog
PSR-3php-fig.org/psr/psr-3
github.com/php-fig/log
<?php !namespace Psr\Log; !interface LoggerInterface { public function log($level, $message, array $context = array()); ! // Shortcuts public function emergency($message, array $context = array()); public function alert($message, array $context = array()); public function critical($message, array $context = array()); public function error($message, array $context = array()); public function warning($message, array $context = array()); public function notice($message, array $context = array()); public function info($message, array $context = array()); public function debug($message, array $context = array()); }
<?php !namespace Psr\Log; !class LogLevel { const EMERGENCY = 'emergency'; const ALERT = 'alert'; const CRITICAL = 'critical'; const ERROR = 'error'; const WARNING = 'warning'; const NOTICE = 'notice'; const INFO = 'info'; const DEBUG = 'debug'; }
Usage: PHPps4,4.!-(%.B6#6&6EUP6EE%+d.-(%.B6#6&6EUm'#>&%+UI)+%'<m'#>&%+d.!$$.j+%')%.'.&6E.*,'##%&.h,'#>&%+.\.#%g.I)+%'<m'#>&%+V?4'),$)6$b6-+W&6E?D.P6EE%+55nOQqkq1Yd.h&6E.\.#%g.P6EE%+V?#'<%?Yd.h&6E/[4-(,m'#>&%+Vh,'#>&%+Yd.!$$.O>>.+%*6+>(.)6.),%.&6E.h&6E/[g'+#"#EV?l66?Yd.h&6E/[%++6+V?u'+?Yd
Usage: Symfonymonolog: handlers: main: type: fingers_crossed action_level: warning handler: file file: type: stream level: debug path: /var/log/symfony.log syslog: type: syslog level: error
Usage: Symfony Controllerps4,4.!#'<%(4'*%.O*<%Uu-#>&%UO44u-#>&%Uj6#)+6&&%+d.!-(%.Ib<M6#bUu-#>&%Ul+'<%g6+=u-#>&%Uj6#)+6&&%+Uj6#)+6&&%+d.!*&'((.l66j6#)+6&&%+.%])%#>(.j6#)+6&&%+.@.....4-!&"*.M-#*)"6#.!'+O*)"6#VY.....@.........h),"(/[E%)V?&6EE%+?Y/["#M6V?2]%*-)"#E.M66.!'+?Yd.!........+%)-+#.h),"(/[+%#>%+V?iO44u-#>&%$l66$!'+W,)<&W)g"E?Yd.....G.G
Usage: Symfony Service
-(%.:(+UP6EUP6EE%+k#)%+M'*%d.-(%.:(+UP6EUq-&&P6EE%+d.!*&'((.O*<%I%+L"*%.@.....4+6)%*)%>.hM66d.....4+6)%*)%>.h&6EE%+d.!....4-!&"*.M-#*)"6#.JJ*6#()+-*)Vl66.hM66D.P6EE%+k#)%+M'*%.h&6EE%+.\.#-&&Y.....@.........h),"(/[M66.\.hM66d.........h),"(/[&6EE%+.\.h&6EE%+.s5.#%g.q-&&P6EE%+VYd.....G.G
Avoid "M.V#-&&.v\\.h&6EE%+Y
Usage: Symfony Service-(%.:(+UP6EUP6EE%+k#)%+M'*%d.-(%.:(+UP6EUq-&&P6EE%+d.!*&'((.O*<%I%+L"*%.@.....4+6)%*)%>.hM66d.....4+6)%*)%>.h&6EE%+d.!....4-!&"*.M-#*)"6#.JJ*6#()+-*)Vl66.hM66Y.....@.........h),"(/[M66.\.hM66d.........h),"(/[&6EE%+.\.#%g.q-&&P6EE%+VYd.....G.!....4-!&"*.M-#*)"6#.(%)P6EE%+VP6EE%+k#)%+M'*%.h&6EE%+Y.....@.........h),"(/[&6EE%+.\.h&6EE%+d.....G.G
40+ HandlersAmqpHandler
ErrorLogHandler FirePHPHandler
GelfHandler MailHandler
NewRelicHandler NullHandler
RedisHandler StreamHandler
SyslogUdpHandler ...
SyslogHandler
<6#6&6E5...,'#>&%+(5......(b(&6E5.........)b4%5.(b(&6E.........&%L%&5.g'+#"#E
Ops problem now!
GelfHandlermonolog: handlers: main: type: fingers_crossed action_level: warning handler: file gelf: type: gelf level: notice publisher: hostname: %logstash_host%
Logstash input
input { gelf { port => 12201 type => gelf } }
Logstash filter
filter {}
Logstash output
output { elasticsearch { host => ["127.0.0.1"] port => 9200 index => "logstash-‐%{+YYYY.MM.dd}" } }
Log
{ "message": "Lorem ipsum", "level": "200", "level_name": "info", "@timestamp": "1432825193000" }
Wait! There is more…
ChannelsUseful to group logs by category
Make it easy to filter Use different rules / handlers
Channels
- assetic - doctrine - event - php - (php_error) - profiler - request - …
Symfony is shipped with many default channels
Usage: Symfony
monolog: channels: -‐ api -‐ business_domain -‐ super_feature -‐ rabbitmq
Create as many as you want
Usage: Symfony
monolog: handlers: main: type: stream path: /var/log/symfony.log channels: !doctrine doctrine: type: stream path: /var/log/doctrine.log channels: doctrine
Usage: Symfony
monolog: handlers: default_notice: type: gelf level: NOTICE channels: [request, security] default_info: type: gelf level: INFO channels: [!request, !security]
Usage: Symfony
services: my_service: class: Acme\Class\Name arguments: ["@logger"] tags: -‐ name: monolog.logger channel: acme
Channels
Visualize proportions Easy filtering
ContextMetadata for your logs
Context
interface LoggerInterface { public function log( $level, $message, array $context = array() ); }
Use it! Abuse it!
Context
h&6EE%+/["#M6V?;(%+.we8.,'(.&6E6-)?Yd
h&6EE%+/["#M6V?;(%+..&6E6-)?D.N?-(%+J">?.\[.e8RY
Instead of building dynamic log messages
Use static strings and add data into the context
@.....A)%<4&')%A5.A&6E()'(,/XAD.....A(%))"#E(A.5.AWWWAD.....A<'44"#E(A5.@.........A(b(&6EA5.AWWWAD.........A,))4A5.AWWWAD.........AE%&MA5.@.............A>b#'<"*J)%<4&')%(A5.N@.................A*)])A5.@.....................A<')*,A5.A*)])JXAD.....................A<')*,J<'44"#EJ)b4%A5.A()+"#EAD.....................A<'44"#EA5.@.........................A)b4%A5.A()+"#EAD.........................A"#>%]A5.A#6)J'#'&bK%>A.....................G.................G.............GRD.............A4+64%+)"%(A.5.@.................Ai)"<%()'<4A.5.@.A)b4%A.5.A>')%AD.A"#>%]A.5.A#6)J'#'&bK%>A.GD.................A<%(('E%A.5.@.A)b4%A.5.A()+"#EAD.A"#>%]A.5.A'#'&bK%>A.GD.................AM'*"&")bA.5.@.A)b4%A.5.A()+"#EAD.A"#>%]A.5.A#6)J'#'&bK%>A.GD.................A)b4%A.5.@.A)b4%A.5.A()+"#EAD.A"#>%]A.5.A#6)J'#'&bK%>A.GD.................A*)])J-(%+J">A.5.@.A)b4%A.5.A()+"#EAD.A"#>%]A.5.A#6)J'#'&bK%>A.G.............G.........G.....G.G
ProcessorAdd metadata for ALL your logs
ProcessorsGitProcessor
IntrospectionProcessor MemoryPeakUsageProcessor
MemoryProcessor MemoryUsageProcessor
ProcessIdProcessor PsrLogMessageProcessor
TagProcessor UidProcessor
WebProcessor
ProcessorCurrent user ID
User-Agent Locale
Country Code IP
Current route Application name / type
Environment (dev / staging / prod) Request UUID
API client …
4-!&"*.M-#*)"6#.JJ"#L6=%V'++'b.h+%*6+>Y.@.....h%#L.....\.h),"(/[*6#)'"#%+/[E%)V?=%+#%&?Y/[E%)2#L"+6#<%#)VYd.....h*6#)%]).\.h),"(/[*6#)'"#%+/[E%)V?*6#)%])?Yd.
....h+%*6+>N?%])+'?RN?%#L?R.....\.h%#Ld.
....h+%*6+>N?%])+'?RN?&6*'&%?R..\.h*6#)%])/[E%)P6*'&%VYd.
....h+%*6+>N?%])+'?RN?<%>"'?R...\.h*6#)%])/[E%)B%>"'VYd.
....h+%*6+>N?%])+'?RN?*&"?R.....\.h*6#)%])/["(j&"VY.s.Z.5.9d.
"M.Vh),"(/[*6#)'"#%+/["(I*64%O*)"L%V?+%C-%()?YY.@.........h+%C-%().\.h),"(/[*6#)'"#%+/[E%)V?+%C-%()J()'*=?Y/[E%)B'()%+Q%C-%()VYd.
"M.Vh+%C-%()/[,%'>%+(/[,'(V?0/Q%C-%()/k>?YY.@.............h+%C-%()k>.\.h+%C-%()/[,%'>%+(/[E%)V?0/Q%C-%()/k>?Yd.............h+%*6+>N?%])+'?RN?+%C-%()J">?R.\.h+%C-%()k>d.........G.
........h+%*6+>N?%])+'?RN?+6-)%?R.\.h+%C-%()/[E%)V?J+6-)%?Yd.
........h+%*6+>N?%])+'?RN?*&"%#)J"4?R.\.h+%C-%()/[E%)j&"%#)k4VYd.
....G.
"M.V#-&&.v\\.h-(%+.\.h-(%+B'#'E%+/[E%)j-++%#);(%+VYY.@.........h+%*6+>N?%])+'?RN?-(%+J">?R.\.h-(%+/[E%)k>VYd.....G.
$$.WWW.
+%)-+#.h+%*6+>d.G
Context + Processor
h4+6t%*).\.#%g.:+6t%*)Ve8D.?M66!'+?Yd.!h&6EE%+/["#M6V?:+6t%*).*+%')%>?D.N.....?4+6t%*)?.\[.h4+6t%*).RYd.
Use processor to pretty format the context
Context + Processor*&'((.:+6t%*):+6*%((6+..@.....4-!&"*.M-#*)"6#.JJ"#L6=%V'++'b.h+%*6+>(Y.....@........."M.Vv"((%)Vh+%*6+>(N?*6#)%])?RN?4+6t%*)?RYY.@.............+%)-+#d.........G.!........h4+6t%*).\.h+%*6+>(N?*6#)%])?RN?4+6t%*)?Rd........."M.Vvh4+6t%*)."#()'#*%6M.:+6t%*)Y.@.............+%)-+#d.........G.!........h+%*6+>(N?*6#)%])?RN?4+6t%*)?R.\.N.............?">?...\[.h4+6t%*)/[E%)k>VYD.............?#'<%?.\[.h4+6t%*)/[E%)q'<%VYD.........R.!........+%)-+#.h+%*6+>(d.....G.G
ELK is very powerful & flexible
Other usages
Business Metrics
Business Metrics
http://david.pilato.fr/blog/2015/04/28/exploring-capitaine-train-dataset/
PacketBeat: Performance Management
Suricate: Security
https://home.regit.org/tag/suricata/
Graphana: Monitoring
Thank You
Slides available at
moquet.net/talks/ipc-2015-elk
Leave feedbacks at @MattKetmo
Recommended