Embed Size (px)
Citation preview
Sadayuki Furuhashi!uentd.org
Fluentdthe missing log collector
Self-introduction
> Sadayuki Furuhashitwitter/github: @frsyuki
> Treasure Data, Inc.Founder & Software Architect
> Open source projectsMessagePack - “It’s like JSON. but fast and small”
Fluentd - “Log everything in JSON”
Today’s topic:
Make log collection easyusing Fluentd
Reporting & Monitoring
Reporting & Monitoring
Collect Store Process Visualize
Collect Store Process Visualize
easier & shorter time
Hadoop / Hive
MongoDB
Treasure Data
Tableau
Excel
R
Collect Store Process Visualize
easier & shorter timeHow to shorten here?
Hadoop / Hive
MongoDB
Treasure Data
Tableau
Excel
R
Collect Store Process Visualize
easier & shorter timeHow to shorten here?
Hadoop / Hive
MongoDB
Treasure Data
Tableau
Excel
R
Fluentd Users
How Fluentd works?
=Fluentd
syslogd+
many
=Fluentd
syslogd+
many
✓ Plugins
✓ JSON
Nagios
MongoDB
Hadoop
Alerting
Amazon S3
Analysis
Archiving
MySQL
Apache
Frontend
Access logs
syslogd
App logs
System logs
Backend
Databasesfilter / buffer / routing
Nagios
MongoDB
Hadoop
Alerting
Amazon S3
Analysis
Archiving
MySQL
Apache
Frontend
Access logs
syslogd
App logs
System logs
Backend
Databasesfilter / buffer / routing
Nagios
MongoDB
Hadoop
Alerting
Amazon S3
Analysis
Archiving
MySQL
Apache
Frontend
Access logs
syslogd
App logs
System logs
Backend
Databasesfilter / buffer / routing
Input Plugins Output Plugins
Buffer Plugins(Filter Plugins)
Input Plugins Output Plugins
2012-02-04 01:33:51myapp.buylog { “user”: ”me”, “path”: “/buyItem”, “price”: 150, “referer”: “/landing”}
timetag
record
JSON
log
in_tail: reads !le and parses lines
fluentdapache
access.log
✓ read a log file✓ custom regexp✓ custom parser in Ruby
in_tail
failure handling & retrying
fluentdapache
access.log buffer
✓ retry automatically✓ exponential retry wait✓ persistent on a file
in_tail
routing / copying
fluentdapache
access.log buffer
✓ routing based on tags✓ copy to multiple storages
in_tail
Amazon S3
Hadoop
# logs from a file<source> type tail path /var/log/httpd.log format apache2 tag web.access</source>
# logs from client libraries<source> type forward port 24224</source>
# store logs to MongoDB and S3<match **> type copy
<match> type mongo host mongo.example.com capped capped_size 200m </match>
<match> type s3 path archive/ </match></match>
Fluentd
forwarding
fluentd
fluentd
fluentd
fluentd
fluentd
fluentdfluentd
send / ackFluentd
=Fluentd
syslogd+
many
✓ Plugins
✓ JSON
Fluentd - plugin distribution platform
$ fluent-gem search -rd fluent-plugin
$ fluent-gem install fluent-plugin-mongo
Fluentd - plugin distribution platform
$ fluent-gem search -rd fluent-plugin
$ fluent-gem install fluent-plugin-mongo
117 plugins!
Collect Store Process Visualize
Hadoop / Hive
MongoDB
Treasure DataTableau
Excel
R
our company provides
Treasure Data?
We’re Hiring!
http://www.treasure-data.com/careers/
Backup slides
Fluentd and Flume NG - con!guration
<source> type forward port 24224</source>
<match **> type file path /var/log/logs</match>
# source
host1.sources = avro-source1
host1.sources.avro-source1.type = avro
host1.sources.avro-source1.bind = 0.0.0.0
host1.sources.avro-source1.port = 41414
host1.sources.avro-source1.channels = ch1
# channel
host1.channels = ch_avro_log
host1.channels.ch_avro_log.type = memory
# sink
host1.sinks = log-sink1
host1.sinks.log-sink1.type = logger
host1.sinks.log-sink1.channel = ch1
Fluentd and Flume NG - topology
fluentd
fluentd
fluentd
fluentd
fluentd
fluentdfluentd
send / ack
Fluentd
Agent
Agent
Agent
Agent
Collector
CollectorCollector
send / ack
Flume NG
out_hdfs
fluentdapache
access.log buffer
✓ retry automatically✓ exponential retry wait✓ persistent on a file
✓ slice files based on time
in_tail
2013-01-01/01/access.log.gz2013-01-01/02/access.log.gz2013-01-01/03/access.log.gz...
fluentd
fluentd
fluentd
✓ automatic fail-over✓ load balancing
out_s3
fluentdapache
access.log buffer
✓ retry automatically✓ exponential retry wait✓ persistent on a file
Amazon S3
✓ slice files based on time
in_tail
2013-01-01/01/access.log.gz2013-01-01/02/access.log.gz2013-01-01/03/access.log.gz...
out_hdfs
fluentdapache
access.log buffer
✓ retry automatically✓ exponential retry wait✓ persistent on a file
✓ slice files based on time
in_tail
2013-01-01/01/access.log.gz2013-01-01/02/access.log.gz2013-01-01/03/access.log.gz...
HDFS
✓ custom text formater
