Upload
steve-behrendt
View
100
Download
4
Embed Size (px)
Citation preview
ELK - StackA perfect match for your Log Management
Steve Behrendt
@derStevedersteve.com
2
The problem Distributed systems Service-oriented Architectures Microservices Multi-language systems Multi-technology stack Multiple Datastores (SQL, noSQL, File stores)
3
Traditional Architecture
4
Browser
IIS
Store App
SQL Server
Product Schema
Invoice Schema
Billing Schema
Customer Schema
Product Module
Customer Module
Azure DocumentDB
MySQL
Azure SQL Server
Microservices
5
Browser
IIS
Apache / Tomcat
Azure
Product UI
Service
Customer UI
Service
Invoicing
Service
CMS Service
Customer Schema
SQL Server
Product Schema
Content Schema
Billing Schema
Azure DocumentDB
MySQL
Azure SQL Server
Be the logs with you
6
Browser
IIS
Apache / Tomcat
Azure
Product UI
Service
Customer UI
Service
Invoicing
Service
CMS Service
Customer Schema
SQL Server
Product Schema
Content Schema
Billing Schema
The challenges Different log formats Each log has its expert Different log locations (machines/servers) Different date formats Internet of things – decentralised log creation
and storing Searching files by keyword is hard Combination of different messages Setting the log context
7
One solution:
ELK
8
ELK E - Elasticsearch L - Logstash K – Kibana … Lucene Shield Marvel
9
The ELK architecture
10
LogsLogstas
h Elasticsearch Kibana
IIS
Syslog
EntLib
Broker
Indexer SearchStorage
Visualize
Visualize
Visualize
Visualize
Logstash Collecting, Filtering, Normalizing, Sending logs to a central location
Understands the logs
11
Logstash Pipeline
12
• Log FileInput• grok• date• geoip• useragent
Filter
• Elasticsearch• Console
Output
Grok debugger
http://grokdebug.herokuapp.com/
13
Forwarder - lumberjack Separate service to forward messages to a
remote endpoint, e.g. logstash instance or elasticsearch
14
Logstash DEMO
15
Elasticsearch Based on lucene for indexing and searching - but
lucene is just a library and very complex Provides (simple) Restful - API abstraction on top
of lucene Stores documents in json format
16
Elasticsearch - Scaling Supports vertical (bigger hardware) and
horizontal scaling (more hardware)
Horizontal scaling is hard, but Elasticsearch is distributed by nature
17
The empty cluster
18
Node: Is a running instance of elasticsearch
Cluster: A cluster consists of one or more nodes with the same cluster name that are working together to share their data and workload
Index and shards
19
Shard: low-level worker holding a slice of data. A single instance of lucene.
Index: logical namespace that points to one or more physical shards
Replicas / Failover
20
Primary and replica shards: Primary and associated replica shard store the same documents.
Newly indexed document first stored on a primary shard, then copied in parallel to the associated replica shard(s).
Horizontal scaling
21
3 shards spread across 3 from 2 nodes.
Each shard is full fledged search engine. Scaling by increasing number of replica shards.
Cluster Discovery Discovering nodes inside a cluster and electing a
master node
Zen discovery
22
Types, Documents, Fields
23
Relational Database
Elasticsearch
Databases
Indices
Tables
Types
Rows
Documents
Columns
Fields
Storing documents
PUT /customer/employee/1 { "first_name" : "John", "last_name" : "Smith", "age" : 25, "about" : "I love to go rock climbing", "interests": [ "sports", "music" ] }
24
Retrieving document
GET /customer/employee/1
Search lite
GET /customer/employee/search
25
Query DSL
GET /customer/employee/_search{ "query" : { "match" : { "last_name" : "Smith" } }}
26
Elasticsearch
DEMO
27
Kibana Data Visualization + Data Discovery
28
Kibana DEMO
29
What’s missing? Security Alerting
30
Alternatives?
31
32
Go and grok some logs