The ELK Stack - Conygreneueda.conygre.com/citi/content/elk/elk.pdf · The ELK Stack Elastic...

The ELK Stack

Elastic Logging

Agenda

1. Logging and analysis2. The ELK stack3. Logs & Elasticsearch Lab 14. Beats Lab 25. Kibana Lab 36. Logstash & Filebeat Lab 47. Kibana Custom Visualisations Lab 5

‹#›

Elastic Stack: Logging Lab 1

Section 1

Installing, configuring and starting Elasticsearch & Kibana

Installing Elasticsearch: Steps

4

Install Elasticsearch

Start Elasticsearch

Test

Install Java

5

•

•

Elasticsearch is written in the Java programming language and requires a JVM (Java Virtual Machine) to run. If you do not have Java installed you must first download and install the Java JDK located here:

• http://www.oracle.com/technetwork/java/javase/downloads/index.html If

you're running the Windows OS or Linux you must also set theJAVA_HOME path so Elasticsearch can find the Java runtime. Please seethe next slide for a screenshot of how to set this variable in Windows. This is not necessary if you are using Mac OS.

Set the Windows JAVA_HOME variable

Make sure you set the path that your JDK is at

6

Create A WorkspaceYou will want to create a place to work out for this course.

7

# Mac OS or Linux:~/course

# Windows: C:\course

Download Elasticsearch

8

Download Elasticsearch from the following site. www.elastic.co/downloads/elasticsearch

The course will work with Elasticsearch 5 or 2.x – we recommend 5 !

Unzip Elasticsearch and place into your "course" directory. In your terminal, change directory into the course/elasticsearch directory and start Elasticsearch:

# run in the foreground on Mac OS or Linux./bin/elasticsearch

# run in the foreground on Windows bin\elasticsearch.bat

# to quit Elasticsearch type ctrl-c, but don't quit now!

Test Elasticsearch

Using your Web Browser, test to see if Elasticsearch has started:

• localhost:9200

9

Installing Kibana: Steps

10

Install Kibana

Start Kibana Optionally install "Sense" if using Kibana 4.x

Restart Kibana

Test

Download Kibana

11

Download Kibana. You may choose to do this course using Kibana 5 or 4.x. If you are using Elasticsearch 2.x then you must use Kibana 4.x.We recommend Kibana 5

www.elastic.co/downloads/kibana Make sure you download the version for your operating system.

Unzip Kibana and place into your "course" directory. Open a new terminal window, change directory into the course/kibana directory and start Kibana:

# run in the foreground on Mac OS or Linux./bin/kibana

# run in the foreground on Windows bin\kibana.bat

Install "Sense"

This step is only mandatory if you are using Kibana 4.x. Kibana 5.x has a "Console" already installed which allows you to send requests to Elasticsearch. For Kibana 4.x you will need to install a plugin called "Sense" to send Elasticsearch requests:

# first you must kill Kibana. In the terminal you# started it in type ctrl-c to kill it# install "sense" on Mac OS or Linux Kibana distributionsbin/kibana plugin --install elastic/sense

12

# install "sense" on Windows Kibana distributions bin\kibana.bat plugin --install elastic/sense

# restart Kibana

Test Kibana 4.x (See next slide for Kibana 5)

Using your Web Browser, go to Kibana and issue a test command:

• localhost:5601/app/sense

13

Test Kibana 5 Using your Web Browser, go to Kibana and issue a test command:

localhost:5601

Click on the "console" icon on the left side navigation: </>

14

Congratulations, first lab completed.

Now we can get ready to load some data.

13

‹#›

Installing, configuring and starting Topbeat

Elastic Stack: Logging Lab 2

Section 2

Installing Topbeat: Steps

2

Download Topbeat

Configure Topbeat

Start Topbeat

Test

Install Topbeat

3

Download Topbeat. Use the latest version.

www.elastic.co/downloads/beats/topbeat

Unzip topbeat and place it in your course directory.

Before we can start Topbeat we will want to configure it.

Also, make a copy of the config in case something goes wrong:# Mac OS, Linuxcp topbeat.yml bkp-topbeat.yml# Windowscopy topbeat.yml bkp-topbeat.yml#We will use topbeat.yml in this lab

# open the topbeat.yml file in a text editor vim topbeat.yml

# modify the config file as explained in the# following slides

Configure Topbeat Input SettingsSetup to collect metrics every 10s from system, filesystem, and all processes

4

# read the input setup (no changes needed)

input:# In seconds, defines how often to read server statistics period: 10

# Regular expression to match the processes that are monitored procs: [".*"]

# Statistics to collect (we are collecting all the explained statistics)

stats:system: true proc: true filesystem: true

Configure Topbeat Shipper SettingsSetup to tag every document with the server properties

5

tags)# read and modify the# feel free to define

shipper settings (modify and add your own tags

to group servers by different logicalshipper:

# Tags make it easy properties.

tags: ["front-end", "web-tier", "mac", "notebook"]

Configure Topbeat Output SettingsSetup to send data to the Elasticsearch running on localhost:9200

6

# read and modify the output settings (hosts, protocol, username & password)

output: elasticsearch:# add your ES host addresshosts: ["localhost:9200"]

Configure Topbeat To Load The TemplateUse the Topbeat template Topbeat ships with

7

#This template will ensure Elasticsearch has the correct mapping#This setting is located in#Uncomment the template and

the elasticsearch output section path settings:

filetemplate:

# Path to template name : "topbeat"path: "topbeat.template.json"

#save the topbeat.yml file now

Running TopbeatLet's get data flowing......................

8

# Mac OS, Linux./topbeat -e -c topbeat.yml -d "elasticsearch"

# or, execute with admin privileges, to collect more data sudo ./topbeat -e -c topbeat.yml -d "elasticsearch"

# Windows.\topbeat.exe -e -c topbeat.yml -d "elasticsearch"

# make sure to let Topbeat run for awhile so you collect enough data

Query The topbeat* IndexUse "Sense" or "Console" to view Topbeat data

9

Congratulations, second lab completed.

Now we can get ready to load some Kibana Dashboards

10

‹#›

Loading Beats dashboards into Kibana

Elastic Stack: Logging Lab 3

Section 3

Visualize Computer's Health: Steps

2

Load Beats Dashboards

Setup Index Pattern

Explore Data In Kibana Discover

Explore Topbeat Dashboard

Download & Install The Dashboards

3

Go to this URL to download the dashboards. There are instructions for Mac OS, Linux, Windows, etc:

https://www.elastic.co/guide/en/beats/libbeat/1.3/load-kibana-dashboards.html

# Mac OS, Linux# grab the dashboards, unzip them and run the load scriptcurl -L -O http://download.elastic.co/beats/dashboards/beats-dashboards-1.3.1.zip unzip beats-dashboards-1.3.1.zipcd beats-dashboards-1.3.1/./load.sh

# Windows - See Website instructions:# https://www.elastic.co/guide/en/beats/libbeat/current/load-kibana-dashboards.html

Configure The topbeat-* Index PatternOpen "Settings" to create the topbeat-* index pattern

4

Set Time Picker & Open Discover InterfaceView the Topbeat data

Index Pattern is pointing to topbeat-*2

You are looking into the last 15 minutes of data. Click here to change.

1

5

Click here to see your dashboards.

Discover InterfacePlay around a bit and then go to Dashboard

1

6

Click here to load existing dashboards.

Find the Topbeat DashboardLet's explore it

1

You have many pre- loaded dashboards. Start typing topbeat to filter it. Then, click on it.

2

7

The Topbeat Dashboard

8

Explore Topbeat Dashboard

9

1. To move visualizations click in the header, drag and drop.2. To resize visualizations click in the bottom-right corner, drag and drop.

3. To remove visualizations click on the X (top-right corner).

4. Click, drag and drop inside a chart to drill down the time range.

5. Use the time-picker to set the time range to 1 hour and the auto-refresh to 10 seconds.

6. Click on the Topbeat Dashboard link to reload default dashboard (you will lose unsaved changes).

7. Save the modified dashboard with the same name (overwrite) or with a new name (new dashboard).

Explore Topbeat Dashboard

10

Congratulations, third lab completed.

Now we are ready to ingest log data using Logstash & Filebeat

11

‹#›

Installing, configuring and starting Filebeat and LogstashLoading Apache log data

Elastic Stack: Logging Lab 4

Section 4

Logstash & Filebeat Logging: Steps

2

Install Logstash

Test Logstash

Prepare Log Data

Install Filebeat

Test Filebeat

Configure Logstash & Filebeat

Ship Apache Log Data, Transform It, Load It

Download & Install Logstash

3

Download Logstash.

www.elastic.co/downloads/logstash Ask your instructor for the Logdata.zip file. Put the logstash_configs and log_data folders in your course directory. Unzip Logstash into your course directory and open the "logstash_simple.conf" file

in a text editor

# open the logstash_simple.conf file in a text editor vim logstash_configs/logstash_simple.conf

# look at the file

logstash_simple.confStart Logstash against this config

4

# use the standard input input {stdin {}

}

# no filters filter {}

# just output

output to the terminal{

stdout {codec => rubydebug

}}

Run LogstashLogstash will start listening to port 5044

5

terminal and start logstash# open a logstash# Mac OS, Linux./bin/logstash -f ../logstash_configs/logstash_simple.conf

# windows.\bin\logstash -f ..\logstash_configs\logstash_simple.conf

# you should see something like the following

Settings: Default pipeline workers: 4 Logstash startup completed

Send Message To Logstash stdin Input

With Logstash running against the logstash_simple.conf file, type some text in the terminal Logstash is running in:

# type "Hello world!" in the terminal Hello world!

6

notice: the notice: the

"message" field has the inputother fields are meta information fields

# you should see the following output##{"message" => "Hello world!", "@version"

=> "1","@timestamp" => "2016-06-15T23:09:11.981Z","host" => "your-host-name"

}

Modify logstash_simple.conf Using grok

7

Change the "logstash_simple.conf" to contain a grok filter as follows:

input { stdin {} }

filter { grok {match => {

"message" => '%{HTTPDATE:timestamp} %{IP:ip} <%{DATA:msg}>'}

}}

output { stdout { codec => rubydebug } }

Start Logstash Against logstash_simple.conf Again

8

Now that you have added a filter to the configuration, restart Logstash and give it this input in the terminal:

# copy and paste the following in the terminal 22/Mar/2016:16:38:00 -0700 183.60.215.50 <text at the end>

# you should see the following output# notice: the <> plays an important part in the grok# notice: there are 2 timestamp fields with 2 different values{

"message" => "22/Mar/2014:16:38:00 -0700 183.60.215.50 <This>","@version" => "1","@timestamp" => "2016-06-15T23:13:39.481Z","host" => "your-host-name","timestamp" => "22/Mar/2016:16:38:00 -0700", "ip" => "183.60.215.50","msg" => "text at the end"

}

Logstash Is Running And You Are Transforming Data Into Documents;

Now Lets Prepare The Apache Log Data…

9

Prepare Apache Log Data

10

We want the log data to be relevant to the time you are taking this course. This way we can use the Time Picker in Kibana with settings like "Last 30

Days". Otherwise you would be stuck using "old" log data - and that's no fun!

First, stop Logstash since we will use Logstash to create the log data for us using a prepared Logstash config file

# to stop the logstash execution there are two options ctrl+C and then press enterctrl+D

Go Into The log_data Directory

11

Look inside the "log_data" directory provided to you. It should be in your "course" directory if you downloaded and extracted it there:

# list the log_data directory~/course/log_data$ ls convert_times.conf original.log

Convert The DataNow convert the original.log file using convert_times.con with Logstash. Go into you Logstash directory and run this:

# Mac OS, Linuxcat original.log | ../logstash-5.0.0/bin/logstash

12

-f ./convert_times.conf

# Windowstype original.log | ..\logstash-5.0.0\bin\logstash -f .\convert_times.conf

starts running:# you will see this on your screen after Logstash Settings: Default pipeline workers: 4Pipeline main started.............................................................................................................................................................................................................................................# the dots show that the process is working

Verify Conversion Worked

13

When done you will see this in your terminal:

And if you list the directory you should have a new file called "access.log":

.................................................................

.................................................................

....................Pipeline main has been shutdown

~/course/log_data$ lsaccess.log convert_times.conf original.log

The Log File Is PreparedFirst We Need To Configure &

Start Filebeat…

14

Download & Install Filebeat

15

Download Filebeat:

www.elastic.co/downloads/beats/filebeat

Unzip Filebeat into your course directory and open the "filebeat.yml" file:

# open the filebeat.yml file (e.g. vim, emacs, sublime) vim filebeat.yml

# make the config changes explained in following slides

Configure Filebeat input settingsRead apache web server log file

16

to the# 1) Change the log file in paths from "- /var/log/*.log" file below

filebeat:# List of prospectors to fetch data. prospectors:

# Each - is a prospector-paths:

- <path_to_home_directory>/course/log_data/access.log# if using Windows, use a patch that matches where the access.log is:# - c:\course\log_data\access.log

Configure Filebeat shipper settingsSetup to tag every document (log line) with the server properties

17

# read and modify the shipper settings (tags)

to group servers by different logicalshipper:

# Tags make it easy properties.

tags: ["front-end", "apache", "web-tier", "access-log"]

Configure Filebeat output settingsSetup to send data to Elasticsearch running on your localhost

18

elasticsearch configuration

host address - logstash listens

# comment out every line of the previous

#elasticsearch:#hosts: ["localhost:9200"]

# uncomment the logstash line and change# on port 5044

logstash:hosts: ["localhost:5044"]

Examine logstash_stdout.conf

19

If Logstash is currently running, stop it.

We will start it with a new config.

Open the "logstash_configs/logstash_stdout.conf" file and examine it:input { beats {

host => "localhost" port => 5044congestion_threshold => 30

}}

filter { grok {match => {"message" => '%{IPORHOST:clientip} %{USER:ident} %{USER:auth} \[%{HTTPDATE:timestamp}\] "%{WORD:verb} %{DATA:request} HTTP/%{NUMBER:httpversion}" %{NUMBER:response:int} (?:-|%{NUMBER:bytes:int}) %{QS:referrer} %

{QS:agent}'}

}

date {match => [ "timestamp", "dd/MMM/YYYY:HH:mm:ss Z" ] locale => enremove_field => timestamp

}geoip {source => "clientip"

}

useragent {source => "agent" target => "useragent"

}}

output {stdout{ codec => rubydebug }

}

Start Logstash Against The logstash_stdout.conf

20

Working out of the "logstash" directory:

# Mac OS, Linux:./bin/logstash -f ../logstash_configs/logstash_stdout.conf

# Windows:.\bin\logstash -f ..\logstash_configs\logstash_stdout.conf

# you should see something like the following

Settings: Default pipeline workers: 4 Logstash startup completed

Run FilebeatLogstash is waiting for Filbeat to ship it logs - start Filebeat

21

start# go back to the filebeat directory in a new terminal and filebeat# linux, mac./filebeat -e -c filebeat.yml -d "logstash"

# windows.\filebeat.exe -e -c filebeat.yml -d "logstash"

Test the Filebeat/Logstash IntegrationCheck if documents are being printed in the terminal

# in the logstash-terminal-window you should see documents being printed

{"message" => "46.105.14.53 - - [14/Jan/2016:22:36:26 +0000] \"GET /blog/

tags/puppet?flav=rss20 HTTP/1.1\" 200 14872 \"-\" \"UniversalFeedParser/4.2- pre-314-svn +http://feedparser.org/\"",

"@version" => "1","@timestamp" => "2016-02-10T01:05:18.520Z","beat" => { "hostname" => "your_host_name", "name" => "front001" }, "count" => 1,"fields" => nil,"input_type" => "log","offset" => "source" =>"tags""type""host"

3799864,"/Users/user//course/log_data",

=> [ [0] "front-end", [1] "apache", [2] "web-tier", ... ],=>=>

"log", "your_host_name"

}

22

Almost ThereNow You Only Need To Change

The Logstash Output To Elasticsearch…

23

Stop Filebeat and LogstashKill Logstash and Filebeat and clean reading state

24

process# in the logstash-terminal-window kill the logstash Ctrl-C

process# in the filebeat-terminal-window kill the filebeat Ctrl-C# Attention: filebeat creates a '.filebeat' file with last reading state.# To read the entire log file in the next execution, you need to delete# .filebeat before restart.# We want all the data in Elasticsearch, so let's delete it.

rm .filebeat

Configure Logstash output to ElasticsearchSend data to you Elasticsearch instance

25

# there is a file in the sample folder called logstash_elasticsearch.conf# with a more complex config. Check it out!

vim ../logstash_configs/logstash_elasticsearch.conf

# also, edit the output to add your Elasticsearch host output {# for each event prints a dot (.) in the stdoutstdout { codec => dots }

elasticsearch {hosts => 'localhost:9200'

}}

Run LogstashLogstash will start listening to port 5044, but no data yet...

26

# execute logstash with elasticsearch config from the logstash directory:# linux, mac./bin/logstash -f ../logstash_configs/logstash_elasticsearch.conf

# windows.\bin\logstash -f ..\logstash_configs\logstash_elasticsearch.conf

# you should see something like the following

Settings: Default pipeline workers: 4 Logstash startup completed

Run FilebeatStart sending apache log events to Logstash (one line is one event)

27

# in the filebeat-terminal-window start filebeat

# linux, mac./filebeat -e -c filebeat.yml -d "logstash"

# windows.\filebeat.exe -e -c filebeat.yml -d "logstash"

Test Logstash!

28

Dots!

# you should see something like the following................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................

Look At Data In KibanaOpen "Sense"/"Console" to view some logs on the logstash-* index:

29

Congratulations, 4th lab completed.

Now it is time to build custom visualizations in Kibana

30

‹#›

Creating custom Kibana Visualizations and Dashboards for the Apache log data

Elastic Stack: Logging Lab 5

Section 5

Visualize Apache Access Logs Lab: Overview

2

Configure 'logstash-*' Index Pattern

Create a Metrics Visualization

Create a Pie Chart

Create a Bar Chart

Create a Line Chart

Create a Tile Map

Create a Dashboard

Create Extra Visualizations

Set The Time Picker To "Last 30 Days"

We want to make sure that Kibana is working over a time-range where there is data:

3

Configure 'logstash-*' Index PatternLog in to Kibana and go to Settings

1 Click here to change Settings.

2 There 5 Index Patterns pre-loaded.

3Let's create a new Index Pattern for documents indexed by Logstash.

4

Configure Logstash Index PatternConfigure Kibana to look into the entire Logstash data

1 You want to analyze all Logstash indices.

2

3

This field will be used to filter events with the global time filter.

Create the index pattern

5

Logstash Index Pattern Is SetLet's create visualizations

1 Click here to create new visualizations.

6

Visualization typeLet's start simple and create a Metric visualization

1 Click here to create a metric visualization.

7

Data sourceSelect which data will be used to create visualization

1

Create a visualization based on a new search (no search defined). Choose this one!

X

Create a visualization based on a saved search, for example: geoip.country_name:america

8

Data sourceSelect which data will be used to create visualization

1

Choose the Index Pattern that contains the data that you want to visualize (logstash-*).

9

Metric visualizationCheck the number of events, the sum of bytes served, and much more...

You can change the font size in the options menu.

2

Remember to use the3 green play button to

apply changes.

1

By default you will see the count of documents. You can change the metric to average, sum, min. Or even add multiple metrics in the same visualizations.

10

Metric visualizationSave it!

1 Click here to open the save menu.

2 Visualization name.

3 Click to save.

4After saving, click here to create a new visualization.

11

Pie chartCreate a new visualization From a new search and choose Pie chart

12

Pie chartSplit Slices by the top 10 responses

Create a terms aggregation

Set the size to 10 and click Apply Changes

Select the response field13

Pie chartDo not forget to save and name your visualization (e.g. Response Codes)

2

You can hover over the pie chart to get detailed information about the data. For example, the count and percentage. And click a specific color to create a filter.

1

14

You can hover over the legend to set the color or create a filter.

Bar chartCreate a new visualization of type Vertical bar chart

1 All documents in a single bar (bucket).

3

2 Y-Axis represents the count.

15

Split the X-Axis into many buckets.

Bar chartSplit the X-Axis into multiple bars based on the event time

4 Click apply.

Select Date Histogram aggregation.

On the field @timestamp.

Set the bucket Interval to Auto.

1

2

3

5 Let's sub-divide each bar.

16

Bar chartSub-divide each bar into 5 bars based on the response code

6 Click apply. And do not forget to save the visualization (e.g. Events over Time by Response Code).

Select Terms aggregation.

On the field response.

Order by the number of events.

Sub-divide each bar into a maximum of 5 bars.

1

2

3

4

5 Order in descending order.

17

Line chartCreate a new visualization of type Line chart

1 All documents in a single point (bucket).

3

2 Y-Axis represents the count.

Split the X-Axis into many buckets.

18

Line chartInstead of documents, let's "count" bytes served

1 Open the Y-Axis metric, so we can change it.

4 After the apply button, you will still see one bucket, but with a very different value.

2 Sum values.

3 On the field bytes.

5 Split the X-Axis into many buckets (points).

19

Line chartSplit the X-Axis into multiple bars based on the event time

4 Click apply.

Select Date Histogram aggregation.

On the field @timestamp.

Set the bucket Interval to Auto.

1

2

3

5 Let's sub-divide each point.

20

Line chartBandwidth over Time by Country

1

6 Click apply.

2

3

4

5

Select Terms aggregation.

On the field geoip.country_name.raw. Ask an instructor why the .raw inner field!!

Order by the Sum of bytes.

Sub-divide each point into a maximum of 5 points.

Order in descending order.

7Why do we have more than 5 countries? Try clicking the down arrow (changes the aggregation execution order) and click apply again. And do not forget to save the visualization (e.g. Top 5 Countries Bandwidth over Time).

21

Tile MapCreate a visualization that uses geo-coordinates and a map

1. Create a "Tile Map" Visualization2. Add "Geo Coordinates" based on the "geoip.location" field 3. In the "Options" select "Shaded Circle Markers"4. Apply and save the visualization

22

Finally, The Hard Work Is Completed.

Now You Can Put All Of Them TogetherIn A Dashboard…

23

Create a new DashboardNow that we have some visualizations, let's view them together

1 List your visualizations.

2 Use the search bar to locate the visualization you want and then click on it to add to the dashboard.

24

The dashboard is not looking good!You can drag your visualizations around and re-size them

25

Now we have the Apache Access Logs dashboardSave your dashboard!

1 Save your dashboard.

26

Visualizations – Technical ChallengesHere's some good ideas on visualizations to make - try and solve them!

27

(Bar chart) Histogram on response codes

with interval 100(Line chart)

BytesDate Histogram

(Pie chart) Doughnut chart for User agents by device by country

(Data table) Top 10 IPs

(Data table) Top 10 Requests

(Line chart)Split Lines of country

fields by response codes (exclude USA)

Congratulations - 5th Lab Completed

You have a fully functional Elastic Stack running.

28