Log Analysis using Open Source Scalable SystemsGurvinder SinghVerktøysamling Oct'2013

Motivation

● Distributed Systems

● In the moment of heat

● Centeralized interface to logs

● Easier access

10/2/13 SLIDE 2

http://www.themeparkreview.com/tatsumediaday/tatsumediaday57.jpg

Challenges

● Almost every component generates logs

● Different Formats and logging methods

● Different requirements for processing

● Dashboards

● Alerts

10/2/13 SLIDE 3

http://img72.imageshack.us/img72/3885/nephew2logs.jpg

Components

● Logstash

● Redis

● Elasticsearch

● Kibana

● Beaver

● lumberjack

● .....

10/2/13 SLIDE 4

Architecture

Architecture

Architecture

Architecture

Architecture

Demo

Possible Input/Processing/Output Options

10/2/13SLIDE

11

Thanks:-)

Questions?

10/2/13SLIDE

12