Tame your logs with (an) ELK - suse.com · Tame your logs with (an) ELK State-of-the-art monitoring...

Tame your logs with (an) ELKState-of-the-art monitoring and log analysis

Klaus Kämpf

Product Owner SUSE Manager

SUSE Linux

kkaempf@suse.com

2

Preface

This is a research project and technology preview.

It may or may not result in a maintained product offering in the future.

3

4

5

Problem Statement

6

127.0.0.1 - - [05/Oct/2016:15:30:18 +0200] "GET /cgi-bin/translate_key.cgi?scout_shared_key=7a6a63848194 HTTP/1.1" 200 1 "-" "SatIDXL8r/1.0 libwww-perl/5.816"10.160.4.230 - - [05/Oct/2016:15:30:18 +0200] "POST /satconfig/cgi-mod-perl/accept_status_log.cgi HTTP/1.1" 200 -127.0.0.1 - - [05/Oct/2016:15:30:39 +0200] "GET /cgi-bin/translate_key.cgi?scout_shared_key=7a6a63848194 HTTP/1.1" 200 1 "-" "SatIDXL8r/1.0 libwww-perl/5.816"10.160.4.230 - - [05/Oct/2016:15:30:39 +0200] "POST /tsdb HTTP/1.1" 200 82 "-" "libwww-perl/5.816"10.160.4.230 - - [05/Oct/2016:15:30:39 +0200] "POST /cgi-bin/eventHandler.cgi HTTP/1.1" 200 82127.0.0.1 - - [05/Oct/2016:15:30:40 +0200] "GET /cgi-bin/translate_key.cgi?scout_shared_key=7a6a63848194 HTTP/1.1" 200 1 "-" "SatIDXL8r/1.0 libwww-perl/5.816"10.160.4.230 - - [05/Oct/2016:15:30:40 +0200] "GET /satconfig/cgi-mod-perl/fetch_commands.cgi?cluster_id=7a6a63848194&node_id=2&role=lead&version=1.0 HTTP/1.1" 200 2310.160.4.230 - - [05/Oct/2016:15:30:41 +0200] "GET /satconfig/cgi-mod-perl/fetch_commands.cgi?cluster_id=7a6a63848194&node_id=2&role=lead&version=1.0 HTTP/1.1" 200 2310.160.4.230 - - [05/Oct/2016:15:30:50 +0200] "POST /tsdb HTTP/1.1" 200 164 "-" "libwww-perl/5.816"10.160.4.230 - - [05/Oct/2016:15:30:50 +0200] "POST /cgi-bin/eventHandler.cgi HTTP/1.1" 200 16410.162.166.1 - - [05/Oct/2016:15:30:58 +0200] "POST /XMLRPC HTTP/1.1" 200 16310.162.166.1 - - [05/Oct/2016:15:30:58 +0200] "POST /XMLRPC HTTP/1.1" 200 73110.160.4.230 - - [05/Oct/2016:15:31:00 +0200] "POST /cobbler_api HTTP/1.1" 200 144 "-" "Java/1.7.0"10.160.4.230 - - [05/Oct/2016:15:31:00 +0200] "POST /cobbler_api HTTP/1.1" 200 129 "-" "Java/1.7.0"10.160.4.230 - - [05/Oct/2016:15:31:00 +0200] "POST /tsdb HTTP/1.1" 200 111 "-" "libwww-perl/5.816"10.160.4.230 - - [05/Oct/2016:15:31:00 +0200] "POST /cgi-bin/eventHandler.cgi HTTP/1.1" 200 11110.160.4.230 - - [05/Oct/2016:15:31:05 +0200] "POST /tsdb HTTP/1.1" 200 87 "-" "libwww-perl/5.816"10.160.4.230 - - [05/Oct/2016:15:31:05 +0200] "POST /cgi-bin/eventHandler.cgi HTTP/1.1" 200 87127.0.0.1 - - [05/Oct/2016:15:31:18 +0200] "GET /cgi-bin/translate_key.cgi?scout_shared_key=7a6a63848194 HTTP/1.1" 200 1 "-" "SatIDXL8r/1.0 libwww-perl/5.816"10.160.4.230 - - [05/Oct/2016:15:31:18 +0200] "POST /satconfig/cgi-mod-perl/accept_status_log.cgi HTTP/1.1" 200 -10.160.4.230 - - [05/Oct/2016:15:31:41 +0200] "GET /satconfig/cgi-mod-perl/fetch_commands.cgi?cluster_id=7a6a63848194&node_id=2&role=lead&version=1.0 HTTP/1.1" 200 2310.160.4.230 - - [05/Oct/2016:15:32:00 +0200] "POST /cobbler_api HTTP/1.1" 200 144 "-" "Java/1.7.0"10.160.4.230 - - [05/Oct/2016:15:32:00 +0200] "POST /cobbler_api HTTP/1.1" 200 129 "-" "Java/1.7.0"127.0.0.1 - - [05/Oct/2016:15:32:18 +0200] "GET /cgi-bin/translate_key.cgi?scout_shared_key=7a6a63848194 HTTP/1.1" 200 1 "-" "SatIDXL8r/1.0 libwww-perl/5.816"10.160.4.230 - - [05/Oct/2016:15:32:18 +0200] "POST /satconfig/cgi-mod-perl/accept_status_log.cgi HTTP/1.1" 200 -10.162.166.1 - - [05/Oct/2016:15:32:26 +0200] "POST /XMLRPC HTTP/1.1" 200 16310.162.166.1 - - [05/Oct/2016:15:32:27 +0200] "POST /XMLRPC HTTP/1.1" 200 731

7

8

Elasticsearch

Kibana

Logstash

9Apache Logos: The Apache Software Foundation - http://svn.apache.org/viewvc/jakarta/site/xdocs/images/logos/tomcat.eps, Apache License 2.0, https://commons.wikimedia.org/w/index.php?curid=11302180

...

Elasticsearch

Logstash Kibana

10

Elasticsearch

11

Elasticsearch

Full text database

Scalable

Terminology● Index: Database● Mapping: Schema● Document: Record● Field: key-value pair

12

Elasticsearch – raw data

13

Elasticsearch – Kibana fields

14

Elasticsearch – internal fields

15

Logstash

16

Logstash - Overview

Log server

Scalable

Focus on time-based events

JRuby

Main components● Input - typically text or json● Filter - parse and manipulate● Output - usually Elasticsearch

17

Logstash - input.conf

input {

stdin {}

}

18

Logstash - input.conf

input {

tcp {

port => 9000

type => "access_log"

}

tcp {

port => 9001

type => "error_log"

tags => ["tag1", "tag2"]

...

19

Logstash - filter.conf

filter {

if ([type] == "osa-dispatcher") {

grok {

match => {

"message" => "\d\d\d\d\/\d\d\/\d\d\s\d\d:\d\d:\d\d\s[+-]\d\d:\d\d:timestamp ...”

}

}

}

}

20

Logstash - filter.pattern

# osa-dispatcher

# 2015/06/12 11:39:04 +02:00 14117 0.0.0.0: osad/jabber_lib.main('ERROR',...)

TIMESTAMP \d\d\d\d\/\d\d\/\d\d\s\d\d:\d\d:\d\d\s[+-]\d\d:\d\d

PID [\d]+

FUNCTION [\w_\.]+

ARGS \([^\)]+\)

21

Logstash - filter.conf

filter {

if ([type] == "osa-dispatcher") {

grok {

match => {

"message" => "%{TIMESTAMP:timestamp} %{PID:pid:int} %{IPV4:clientip}: ...”

}

}

}

}

22

Logstash - output.conf

output {

stdout { codec => rubydebug }

}

23

Logstash - output.conf

output {

elasticsearch {

hosts => [“localhost:9200”]

}

}

24

Logstash - Start

# ls

filter.conf input.conf osa-dispatcher.pattern output.conf rhn_web_api.pattern

# logstash -f ./\*.conf --auto-reload

25

Kibana

26

Kibana

Visualization frontend to Elasticsearch

Web frontend

Focus on time-based events

Comfortable query interface

Dashboard management

● Settings● Discover● Visualize● Dashboard

27

Kibana - Settings

● Select index pattern– wildcards possbile

● Time based ?– Time-field name

Loads mapping● field names● field types● analyzed ?

28

Kibana - Discover

● No results found ?– Expand your time range

● Explore fields● Include/Exclude

● Create query● Save search● Visualize !

29

Kibana - Visualize

● Create new● Select visualization type

● New/Saved search● Graph-specific parameters

30

Kibana - Dashboard

Collection of visualization tiles

Can be saved/shared

31

SUSE Manager

32

SUSE Manager - Components

Apache Web Server

Tomcat Application Server

PostgreSQL database

Java Application Stack

Python API

Salt

33

SUSE Manager - Apache

# grep ErrorLog /etc/apache2/httpd.conf

#ErrorLog /var/log/apache2/error_log

# cat /etc/apache2/sysconfig.d/logstash.conf

CustomLog "|/usr/bin/nc logstash.mgr.suse.de 9000" combined

ErrorLog "|$/usr/bin/tee -a /var/log/apache2/error_log

| /usr/bin/nc logstash.mgr.suse.de 9001"

34

SUSE Manager - Tomcat

# /etc/tomcat/log4j.properties

log4j.rootLogger=debug, R, LogstashAppender

...

log4j.appender.LogstashAppender=org.apache.log4j.net.SocketAppender

log4j.appender.LogstashAppender.port=9006

log4j.appender.LogstashAppender.remoteHost=logstash.mgr.suse.de

35

SUSE Manager - Java

# /usr/share/rhn/classes/log4j.properties

log4j.rootLogger=WARN,RootAppender,LogstashAppender

...

log4j.appender.LogstashAppender=org.apache.log4j.net.SocketAppender

log4j.appender.LogstashAppender.port=9007

log4j.appender.LogstashAppender.remoteHost=logstash.mgr.suse.de

36

Elastic Beats

37

Beats

Formerly ‘logstash-forwarder’

Unobtrusive (log) file forwarder

Written in Go, fast

Simple configuration

38

SUSE Manager - Filebeat

# /usr/filebeat/filebeat.yml

filebeat:

prospectors:

-

paths:

- /var/log/apache2/access_log

encoding: utf-8

input_type: log

document_type: access_log

...

output:

logstash:

hosts: ["logstash.mgr.suse.de:5045"]

39

Packages

40

Packages

Watch this space for packages

https://build.opensuse.org/package/show/security:logging/elasticsearch

41

Outlook

42

Outlook

● Complete packaging● Package queries ?● Package visualizations ?

● Document best practices

● Use to observe testing

● Automated analysis of supportconfigs

● Productize ?