20
Log Analysis – Logstash, Elastic Search, Kibana Avinash Ramineni Shantanu Mirajkar

Log analysis using Logstash,ElasticSearch and Kibana

Tags:

Embed Size (px)

DESCRIPTION

Log analysis using Logstash, ElasticSearch and Kibana

Citation preview

Page 1: Log analysis using Logstash,ElasticSearch and Kibana

Log Analysis – Logstash, Elastic Search, Kibana Avinash Ramineni Shantanu Mirajkar

Page 2: Log analysis using Logstash,ElasticSearch and Kibana

• Logging• Pains of Log Management• Introducing Logstash• Elasticsearch• Kibana• Demo• Installing Logstash, Elasticsearch Kibana• Questions

Agenda

Page 3: Log analysis using Logstash,ElasticSearch and Kibana

• Why do we need Logging ?– Troubleshoot Issues– Security

• Analyze logs to detect patterns• Detect Malware Activity - Intrusion Detection, Denial of Service• Unauthorized Resource Usage

– Monitoring• Monitor Resource Usage

• Developers and Logging– Logging Aids in Development ?– Forget about Production !!!!!

Logging

Page 4: Log analysis using Logstash,ElasticSearch and Kibana

• “Capture-it-all” Approach• What to Log? Everything • DevOps Movement• Logs are archived for years• Big Data • Application Usage Statistics

Logging

Page 5: Log analysis using Logstash,ElasticSearch and Kibana

• Searching the logs– Command line, cat, tail, sed, grep, awk– Regular Expressions

• Multiple Servers behind the load balancer• Multi-Tier Architecture

– Web Application– Service Layer– Correlation between various components in a System

• Geographically distributed– Timestamps

Log management

Page 6: Log analysis using Logstash,ElasticSearch and Kibana

• Centralize all the Logs– Too much information to go through– Increasingly hard to correlate the contextual Data

• Add Searching and Indexing Technology– grep– Custom logging frameworks , custom integration of logging, searching

technologies• Monitor the Logs

Log management

Page 7: Log analysis using Logstash,ElasticSearch and Kibana

• Logstash to the Rescue– Integration Framework• Log Collection• Centralization• Parsing • Storage and Search

Logstash

Page 8: Log analysis using Logstash,ElasticSearch and Kibana

• JRuby– Run on Java Virtual Machine (JVM)– Simple Message Based Architecture– Single Agent that can be configured for multiple things– OPEN SOURCE

• Four Components– Shipper– Broker and Indexer– Search and Storage– Web Interface

Logstash

Page 9: Log analysis using Logstash,ElasticSearch and Kibana

Architecture

Image courtesy of Logstashbook

Page 10: Log analysis using Logstash,ElasticSearch and Kibana

Architecture - Broker

• Acts as Temp Buffer between Logstash Agents and the Central server– Enhance Performance by providing caching buffer

for log events– Adds Resiliency • Incase the Indexing fails, the events are held in a queue

instead of getting lost

• AMQP,0MQ, Redis

Page 11: Log analysis using Logstash,ElasticSearch and Kibana

• Indexing and Searching Tool– Built on Lucene

• Search and Index data available Restfully as JSON over HTTP• Comes bundled with Logstash – embedded• Text indexing Search Engine

– Searches on the Index rather than on the content• Creates Indexes of the incoming content

– Uses Apache Lucene to create Indexes• ElasticSearch can have a schema – Fields on which Indexes are

created

ElasticSearch

Page 12: Log analysis using Logstash,ElasticSearch and Kibana

• Indexes are stored in Lucene Instances called “Shards”

• ElasticSearch can have multiple nodes• Two Types of Shards– Primary– Replica

• Replicas of Primary Shards– Protect the data– Make Searches Faster

ElasticSearch

Page 13: Log analysis using Logstash,ElasticSearch and Kibana

• Wouldn’t it be good to have a webpage to do search on ElasticSearch instead of searching it through a Service

• Kibana provides a Simple but Powerful web Interface– Customizable Dashboards– Search the log events

• Support Lucene Query Syntax

– Creation of tables, graphs and sophisticated visualizations

Kibana

Page 14: Log analysis using Logstash,ElasticSearch and Kibana

Kibana

Page 15: Log analysis using Logstash,ElasticSearch and Kibana

Kibana

Page 16: Log analysis using Logstash,ElasticSearch and Kibana

Demo

Page 17: Log analysis using Logstash,ElasticSearch and Kibana

• Send Alerts– Emails– Instant Messaging– Other Monitoring System

• Collect and Deliver Metrics to metric engine

Alerts / Monitoring Support

Page 18: Log analysis using Logstash,ElasticSearch and Kibana

• Small VMs with limited memory• Outsourced managed servers• Java not installed• Alternatives

– Syslog• Rsyslog• Syslogd• Syslog-NG

– Logstash Forwarder (Lumber Jack)

Shipping Logs with Logstash Agent

Page 19: Log analysis using Logstash,ElasticSearch and Kibana

• Scale each component as needed• Can be built into using chef and puppet scripts

Scaling / Deployment