Phil Cryer, September 2014

Moving towards unified logging

goal

=> decouple data sources from backend systems by providing a unified logging layer to route logs as needed

currently

=> A host runs a Splunk app and forwards all of its logs to Splunk

Host

a host

Splunk

Host

sends logs to splunk

Splunk

=> this works, but we want a more flexible, open source solution that doesn’t restrict us with specific tools or size quotas

why is this a problem

=> create a unified logging layer to handle logs with FluentD, an open source, flexible and lightweight alternative to route logs

idea

http://www.fluentd.org/

Host

the ELK stack in development…

Elasticsearch

Host

Logstash

logstash writes to elasticsearch

Elasticsearch

Host Host

Logstashrsyslogd

but this can be done just with rsyslogd

Elasticsearch

Host Host

Logstash FluentD

and can also be done with FluentD

Host

rsyslogd

Elasticsearch

Host Host Host Host

FluentDHost

Logstash FluentD Dockerrsyslogd

FluentD

but FluentD can be used for more, like routing

Elasticsearch

Host Host Host

FluentDHost

Logstash FluentD Docker

FluentD

Host

rsyslogd

and handle input from various data sources

Elasticsearch

Host Host Host

FluentDHost

Splunk

Logstash FluentD rsyslogd

FluentD FluentD

Host

rsyslogd

and output to various backends, even Splunk

Elasticsearch

Host Host Host

FluentDHost

Splunk

Host

Logstash FluentD rsyslogd Splunk

FluentD

Host

rsyslogd

it could do this independently of Splunk

Elasticsearch

Host Host Host

FluentDHost

Splunk

Host

Logstash FluentD rsyslogd Splunk

FluentD FluentD

Host

rsyslogd

or in parallel

Elasticsearch

DB Memcache DNS

FluentDHost

Splunk

IDS

Logstash FluentD rsyslogd

FluentD

App

rsyslogd

and these could be from a variety applications

rsyslogd

FluentD

Docker TCP Socket

FluentDHost

MySQL Scala App

Data sources

AWS

using various data sources

http://www.fluentd.org/datasources

AWS, Docker Containers, Flume, Java Apps, MySQL SlowQuery Logs, Scala Apps, TCP Socket, and more

DB Memcache DNS IDSApp

FluentDHost

KafkaFluentD

HDFSFluentD

Data outputs

sent to various data outputs

http://www.fluentd.org/dataoutputs

AWS, Kafka, CouchDB, Elasticsearch, Hbase, HDFS, Mongo DB, MySQL, Redis, Riak, Splunk, and more

Moving towards unified logging

Thanks.

so let’s start…