1.5 Million Log Lines per Second
Big Data Everywhere Chicago 2014
Mike Keane [email protected]
Building and maintaining Flume flows at Conversant
• Quicker insight into production data
• Reduce complexity of administering/managing new servers,
data centers, etc.
• Scalable
• No data loss or duplication
• Replace TSV files with Avro objects
• Able to be monitored by Network Operations Center (NOC)
• Able to recover from downtime quickly
R SLA for Event Driven Logging with Flume
• A Flume Flow is a series of flume agents data follows from origination
to final destination
• Data on a Flume Flow is packaged in FlumeEvent Avro objects
• A FlumeEvent is composed of
• Headers – A map of string value pairs
• Body – A byte array
• A FlumeEvent is an atomic unit of data
• FlumeEvents are sent in batches
• When a batch of FlumeEvents only partially makes it to the next flume
agent in the flow, the entire batch is resent resulting in duplicates
R Simplistic Flume Overview
R Simplistic Flume Overview
Flume Agent
R Simplistic Flume Overview
EmbeddedAgent Compressor Agent
Landing Agent
Overview of existing network topology
• 3 data centers divided into 12 lanes participating in the OpenRTB market
• 6 lanes in the east coast data center
• 4 lanes in the west coast data center
• 2 lanes in the European data center
• Each lane has approximately 75 servers handling OpenRTB
operations.
• 30 different logs
• Over 60,000,000,000 log lines per day
Overview of existing network topology.
• 2 Server Flume Flow from East Coast (IAD) to Chicago (ORD) with
over 250K TSV lines per second
• No Data Loss
• Failover
• Compression performance
P.O.C. Can Flume handle our log volume reliably?
P.O.C. Overview
P.O.C. passes
• Larger Batch sizes helped, but could not reach 250K per second
• Multiple TSV lines Per FlumeEvent hits over 360K per second
• Failover passed with duplicates
• Compression passed but needed to parallelize 7X sinks
Taking Flume to Production
• Embedding the EmbeddedAgent in existing servers
• Modify EmbeddedAgent
• Properties from existing infrastructure
• Implement Monitoring
• Create “Flume”Implementation of proprietary logging interface
• Replace POJO to TSV with Avro to AvroDataFile
• Preventing duplicates, not removing
• Add LogType header
Taking Flume to Production
• Custom Sink for AvroDataFile body (based on HDFSEventSink)
• Check if UUID header is in HBase
• Yes – increment duplicate count metric
• No
• Write AvroDataFile body to HDFS using Custom Writer
• Put UUID to HBase
Taking Flume to Production
• Custom Selector based on MultiplexingChannelSelector
• Route FlumeEvents to channels by log type or groups of log
types
• Bifurcate to multiple locations each log and each location
with its own percentage of data to bifurcate
Configuring Flume Flows
• Configuring Flume can be tedious, use a templating engine
• In Q2 2014 Conversant expanded from 7 lanes in 2 data centers
to 12 lanes in 3 data centers (~400 more servers to configure).
• Static headers useful for tracking flows
• 15 minutes to configure all Q2 expansion CompressorLane('iad6', [CompressorAgent("dtiad06flm01p"),
CompressorAgent("dtiad06flm02p"),
CompressorAgent("dtiad06flm03p")])
compressor.list = dtiad06flm01p, dtiad06flm02p,dtiad06flm03p
Monitoring the Flume Flows
• Flume metrics are available by JMX or Json over HTTP
• Metrics to monitor
• ChannelFillPercentage
• Rate of change on EventDrainSuccessCount on failover sinks
• FLUME-2307 – File channel deletes fail after timeout (fixed 1.5)
• Publishing metrics to TSDB provides great visual insight
Monitoring the Flume Flows
ChannelFillPercentage
Monitoring the Flume Flows
Rate of taking events off “Critical Logs” file channel
Monitoring the Flume Flows
Rate of Flume Events by data center East Coast, West Coast, Europe
Monitoring the Flume Flows
Monitoring by Groups
Benefits of migrating to Flume
• Business has insight into data in under 10 minutes
• Configuring expansion trivial
• Failover enables automatic recovery from down time
• Bifurcation
• enables scaled constant regression lane(s)
• Subset of data to analytics development cluster
Benefits of migrating to Flume
5 minute aggregations to business within 10 minutes
Gotchas…
• Scaling for Compression
• Auto reloading of properties inconsistent
• “It is recommended (though not required) to use a separate disk
for the File Channel checkpoint.”
RAID-6 raid array, Force Write Back
• Bad configurations not easy to see, not always clear in log file.
• NetcatSource – Not too useful beyond trivial usage
Gotchas…
• POM file edits
• JUnits are not deterministic
• Hadoop jars added to classpath by startup script – IDE
• Avoiding cost of Avro schema evolution
What is next
• Upgrade to Flume 1.5
• Bifurcate to micro batch (Storm? Spark?)
• Disable sink switch
Recommended
