Log analysis using Logstash,ElasticSearch and Kibana

Log Analysis – Logstash, Elastic Search, Kibana Avinash Ramineni Shantanu Mirajkar

• Logging• Pains of Log Management• Introducing Logstash• Elasticsearch• Kibana• Demo• Installing Logstash, Elasticsearch Kibana• Questions

Agenda

• Why do we need Logging ?– Troubleshoot Issues– Security

• Analyze logs to detect patterns• Detect Malware Activity - Intrusion Detection, Denial of Service• Unauthorized Resource Usage

– Monitoring• Monitor Resource Usage

• Developers and Logging– Logging Aids in Development ?– Forget about Production !!!!!

Logging

• “Capture-it-all” Approach• What to Log? Everything • DevOps Movement• Logs are archived for years• Big Data • Application Usage Statistics

Logging

• Searching the logs– Command line, cat, tail, sed, grep, awk– Regular Expressions

• Multiple Servers behind the load balancer• Multi-Tier Architecture

– Web Application– Service Layer– Correlation between various components in a System

• Geographically distributed– Timestamps

Log management

• Centralize all the Logs– Too much information to go through– Increasingly hard to correlate the contextual Data

• Add Searching and Indexing Technology– grep– Custom logging frameworks , custom integration of logging, searching

technologies• Monitor the Logs

Log management

• Logstash to the Rescue– Integration Framework• Log Collection• Centralization• Parsing • Storage and Search

Logstash

• JRuby– Run on Java Virtual Machine (JVM)– Simple Message Based Architecture– Single Agent that can be configured for multiple things– OPEN SOURCE

• Four Components– Shipper– Broker and Indexer– Search and Storage– Web Interface

Logstash

Architecture

Image courtesy of Logstashbook

Architecture - Broker

• Acts as Temp Buffer between Logstash Agents and the Central server– Enhance Performance by providing caching buffer

for log events– Adds Resiliency • Incase the Indexing fails, the events are held in a queue

instead of getting lost

• AMQP,0MQ, Redis

• Indexing and Searching Tool– Built on Lucene

• Search and Index data available Restfully as JSON over HTTP• Comes bundled with Logstash – embedded• Text indexing Search Engine

– Searches on the Index rather than on the content• Creates Indexes of the incoming content

– Uses Apache Lucene to create Indexes• ElasticSearch can have a schema – Fields on which Indexes are

created

ElasticSearch

• Indexes are stored in Lucene Instances called “Shards”

• ElasticSearch can have multiple nodes• Two Types of Shards– Primary– Replica

• Replicas of Primary Shards– Protect the data– Make Searches Faster

ElasticSearch

• Wouldn’t it be good to have a webpage to do search on ElasticSearch instead of searching it through a Service

• Kibana provides a Simple but Powerful web Interface– Customizable Dashboards– Search the log events

• Support Lucene Query Syntax

– Creation of tables, graphs and sophisticated visualizations

Kibana

Kibana

Kibana

Demo

• Send Alerts– Emails– Instant Messaging– Other Monitoring System

• Collect and Deliver Metrics to metric engine

Alerts / Monitoring Support

• Small VMs with limited memory• Outsourced managed servers• Java not installed• Alternatives

– Syslog• Rsyslog• Syslogd• Syslog-NG

– Logstash Forwarder (Lumber Jack)

Shipping Logs with Logstash Agent

• Scale each component as needed• Can be built into using chef and puppet scripts

Scaling / Deployment

Industry ExperienceQuestions ?

avinash@clairvoyantsoft.comTwitter:@avinashramineni

shantanu@clairvoyantsoft.com