Embed Size (px)
DESCRIPTION
In this talk we’ll explain how we implemented “update-less updates” (not a typo!) for HBase using append-only approach. This approach uses HBase core strengths like fast range scans and the recently added coprocessors to enable real-time analytics. It shines in situations where high data volume and velocity make random updates (aka Get+Put) prohibitively expensive. Apart from making real-time analytics possible, we’ll show how the append-only approach to updates makes it possible to perform rollbacks of data changes and avoid data inconsistency problems caused by tasks in MapReduce jobs that fail after only partially updating data in HBase.
Citation preview
Real-time Analytics with HBase
Alex Baranau, Sematext International
Sunday, May 20, 12
Alex Baranau, Sematext International, 2012
About me
Software Engineer at Sematext International
http://blog.sematext.com/author/abaranau
@abaranau
http://github.com/sematext (abaranau)
Sunday, May 20, 12
Alex Baranau, Sematext International, 2012
Plan
Problem background: what? why?
Going real-time with append-only updates approach: how?
Open-source implementation: how exactly?
Q&A
Sunday, May 20, 12
Alex Baranau, Sematext International, 2012
Background: our servicesSystems Monitoring Service (Solr, HBase, ...)
Search Analytics Service
data collector
data collector
data collector
Data Analytics &
Storage
Reports
0
25
50
75
100
2007 2008 2009 2010
Sunday, May 20, 12
Alex Baranau, Sematext International, 2012
Background: Report ExampleSearch engine (Solr) request latency
Sunday, May 20, 12
Alex Baranau, Sematext International, 2012
Background: Report ExampleHBase flush operations
Sunday, May 20, 12
Alex Baranau, Sematext International, 2012
Background: requirements
High volume of input data
Multiple filters/dimensions
Interactive (fast) reports
Show wide range of data intervals
Real-time data changes visibility
No sampling, accurate data needed
Sunday, May 20, 12
Alex Baranau, Sematext International, 2012
Background: serve raw data?simply storing all data points doesn’t work
to show 1-year worth of data points collected every second 31,536,000 points have to be fetched
pre-aggregation (at least partial) needed
Data Analytics & Storage
input data aggregated data
data processing (pre-aggregating)
Reports
Sunday, May 20, 12
Alex Baranau, Sematext International, 2012
Background: pre-aggregation
input data item
aggregated value
aggregated value
aggregated value
processinglogic
aggregation rules* filters/dimensions* time range granularities* ...
OLAP-like Solution
Sunday, May 20, 12
Alex Baranau, Sematext International, 2012
Background: pre-aggregation
input data itemtime: 1332882078sensor: sensor55value: 80.0
minute: 22214701value: 30.0
by minute
day: 2012-04-26value: 10.5
by day
minute: 22214701sensor: sensor55cpu: 70.3
by minute & sensor
......
...
...
aggregated record groupsSimplified Example
processinglogic
aggregation rules * by sensor * by minute/day
Sunday, May 20, 12
Alex Baranau, Sematext International, 2012
Background: RMW updates are slow
more dimensions/filters -> greater output data vs input data ratio
individual ready-modify-write (Get+Put) operations are slow and not efficient (10-20+ times slower than only Puts)
... sensor2value:15.0
sensor2value:41.0
sensor1<...> ... input
storage (HBase)
sensor1<...>
sensor2avg : 28.7min: 15.0max: 41.0
Get Put Get Put Get Put
... reportssensor2Get/Scan
...
Sunday, May 20, 12
Alex Baranau, Sematext International, 2012
Background: improve updates
Using in-place increment operations? Not fast enough and not flexible...
Buffering input records on the way in and writing in small batches? Doesn’t scale and possible loss of data...
Sunday, May 20, 12
Alex Baranau, Sematext International, 2012
Background: batch updates
More efficient data processing: multiple updates processed at once, not individually
Decreases aggregation output (per input record)
Reliable, no data loss
Using “dictionary records” helps to reduce number of Get operations
Sunday, May 20, 12
Alex Baranau, Sematext International, 2012
Background: batch updates
Using data de-normalization to reduce random Get operations while doing “Get+Put” updates:
Keep compound records which hold data of multiple “normal” records that are usually updated together
N Get+Put operations replaced with M (Get+Put) and N Put operations, where M << N
“Dictionary Records”
Sunday, May 20, 12
Alex Baranau, Sematext International, 2012
Background: batch updates
Not real-time
If done frequently (closer to real-time), still a lot of costly Get+Put update operations
Bad (any?) rollback support
Handling of failures of tasks which partially wrote data to HBase is complex
Sunday, May 20, 12
Going Real-timewith
Append-based Updates
Sunday, May 20, 12
Alex Baranau, Sematext International, 2012
Append-only: main goals
Increase record update throughput
Process updates more efficiently: reduce operations number and resources usage
Ideally, apply high volume of incoming data changes in real-time
Add ability to roll back changes
Handle well high update peaks
Sunday, May 20, 12
Alex Baranau, Sematext International, 2012
Append-only: how?
1. Replace read-modify-write (Get+Put) operations at write time with simple append-only writes (Put)
2. Defer processing of updates to periodic jobs
3. Perform processing of updates on the fly only if user asks for data earlier than updates are processed.
Sunday, May 20, 12
Alex Baranau, Sematext International, 2012
Append-only: writing updates
... sensor2value:15.0
sensor2value:41.0
sensor1... ... input
sensor1<...>
sensor2avg : 22.7max: 31.0
Put Put Put
...
...
...
sensor1<...>
sensor2value: 15.0
sensor2value: 41.0
...
...
Replace update (Get+Put) operations at write timewith simple append-only writes (Put)1
storage (HBase)
Sunday, May 20, 12
Alex Baranau, Sematext International, 2012
Append-only: writing updates
sensor1<...>
sensor2avg : 22.7max: 31.0
...
...
...
sensor1<...>
sensor2value: 15.0
sensor2value: 41.0
...
processing updates with MR job
sensor1<...>
sensor2avg : 23.4max: 41.0
... ......
Defer processing of updates to periodic jobs 2
Sunday, May 20, 12
Alex Baranau, Sematext International, 2012
Append-only: writing updates
sensor1<...>
sensor2avg : 22.7max: 31.0
...
...
...
sensor1<...>
sensor2value: 15.0
sensor2value: 41.0
...
... reportssensor1
sensor2avg : 23.4max: 41.0
...
storage
Perform aggregations on the fly if user asks for data earlier than updates are processed3
Sunday, May 20, 12
Alex Baranau, Sematext International, 2012
Append-only: benefits
High update throughput
Real-time updates visibility
Efficient updates processing
Handling high peaks of update operations
Ability to roll back any range of changes
Automatically handling failures of tasks which only partially updated data (e.g. in MR jobs)
Sunday, May 20, 12
Alex Baranau, Sematext International, 2012
Append-only: high update throughput
Avoid Get+Put operations upon writing
Use only Put operations (i.e. insert new records only) which is very fast in HBase
Process updates when flushing client-side buffer to reduce the number of actual writes
1/6
Sunday, May 20, 12
Alex Baranau, Sematext International, 2012
Append-only: real-time updates
Increased update throughput allows to apply updates in real-time
User always sees the latest data changes
Updates processed on the fly during Get or Scan can be stored back right away
Periodic updates processing helps avoid doing a lot of work during reads, making reading very fast
2/6
Sunday, May 20, 12
Alex Baranau, Sematext International, 2012
Append-only: efficient updatesTo apply N changes:
N Get+Put operations replaced with
N Puts and 1 Scan (shared) + 1 Put operation
Applying N changes at once is much more efficient than performing N individual changes
Especially when updated value is complex (like bitmaps), takes time to load in memory
Skip compacting if too few records to process
Avoid a lot of redundant Get operations when large portion of operations - inserting new data
3/6
Sunday, May 20, 12
Alex Baranau, Sematext International, 2012
Append-only: high peaks handling
Actual updates do not happen at write time
Merge is deferred to periodic jobs, which can be scheduled to run at off-peak time (nights/week-ends)
Merge speed is not critical, doesn’t affect the visibility of changes
4/6
Sunday, May 20, 12
Alex Baranau, Sematext International, 2012
Append-only: rollbackRollbacks are easy when updates were not processed yet (not merged)
To preserve rollback ability after they are processed (and result is written back), updates can be compacted into groups
processing updates... sensor2...
sensor2...
sensor2...sensor2...
sensor2...
... ... sensor2...
sensor2...
sensor2...
...written at:
9:00
10:00
11:00...
...
... ...
5/6
Sunday, May 20, 12
Alex Baranau, Sematext International, 2012
Append-only: rollbackExample:* keep all-time avg value for sensor* data collected every 10 second for 30 days
Solution:* perform periodic compactions every 4 hours* compact groups based on 1-hour interval
Result:At any point of time there are no more than24 * 30 + 4 * 60 * 6 = 2160 non-compacted records that needs to be processed on the fly
5/6
Sunday, May 20, 12
Alex Baranau, Sematext International, 2012
Append-only: idempotencyUsing append-only approach helps recover from failed tasks which write data to HBase
without rolling back partial updates
avoids applying duplicate updates
fixes task failure with simple restart of task
Note: new task should write records with same row keys as failed one
easy, esp. given that input data is likely to be same
Very convenient when writing from MapReduce
Updates processing periodic jobs are also idempotent
6/6
Sunday, May 20, 12
Alex Baranau, Sematext International, 2012
Append-only: cons
Processing on the fly makes reading slower
Looking for data to compact (during periodic compactions) may be inefficient
Increased amount of stored data depending on use-case (in 0.92+)
Sunday, May 20, 12
Alex Baranau, Sematext International, 2012
Append-only + Batch?Works very well together, batch approach benefits from:
increased update throughput
automatic task failures handling
rollback ability
Use when HBase cluster cannot cope with processing updates in real-time or update operations are bottleneck in your batch
We use it ;)
Sunday, May 20, 12
Append-only updates implementation
HBaseHUT
Sunday, May 20, 12
Alex Baranau, Sematext International, 2012
HBaseHUT: Overview
Simple
Easy to integrate into existing projects
Packed as a singe jar to be added to HBase client classpath (also add it to RegionServer classpath to benefit from server-side optimizations)
Supports native HBase API: HBaseHUT classes implement native HBase interfaces
Apache License, v2.0
Sunday, May 20, 12
Alex Baranau, Sematext International, 2012
HBaseHUT: OverviewProcessing of updates on-the-fly (behind ResultScanner interface)
Allows storing back processed Result
Can use CPs to process updates on server-side
Periodic processing of updates with Scan or MapReduce job
Including processing updates in groups based on write ts
Rolling back changes with MapReduce job
Sunday, May 20, 12
Alex Baranau, Sematext International, 2012
HBaseHUT vs(?) OpenTSDB“vs” is wrong, they are simply different things
OpenTSDB is a time-series database
HBaseHUT is a library which implements append-only updates approach to be used in your project
OpenTSDB uses “serve raw data” approach (with storage improvements), limited to handling numeric values
HBaseHUT is meant for (but not limited to) “serve aggregated data” approach, works with any data
Sunday, May 20, 12
Alex Baranau, Sematext International, 2012
HBaseHUT: API overview
Writing data:
Put put = new Put(HutPut.adjustRow(rowKey));// ...hTable.put(put);
Reading data:
Scan scan = new Scan(startKey, stopKey);ResultScanner resultScanner = new HutResultScanner(hTable.getScanner(scan), updateProcessor);
for (Result current : resultScanner) {...}
Sunday, May 20, 12
Alex Baranau, Sematext International, 2012
HBaseHUT: API overviewExample UpdateProcessor:
public class MaxFunction extends UpdateProcessor { // ... constructor & utility methods
@Override public void process(Iterable<Result> records, UpdateProcessingResult result) { Double maxVal = null; for (Result record : records) { double val = getValue(record); if (maxVal == null || maxVal < val) { maxVal = val; } }
result.add(colfam, qual, Bytes.toBytes(maxVal)); }}
Sunday, May 20, 12
Alex Baranau, Sematext International, 2012
HBaseHUT: how we use itData Analytics & Storage
input data
aggregated data
initial data processing
Reports
HBase
HBas
eHUT
HBas
eHUT
periodic updates
processing
HBaseHUT
MapReducejobs
HBaseHUT
Sunday, May 20, 12
Alex Baranau, Sematext International, 2012
HBaseHUT: Next Steps
Wider CPs (HBase 0.92+) utilization
Process updates during memstore flush
Make use of Append operation (HBase 0.94+)
Integrate with asynchbase lib
Reduce storage overhead from adjusting row keys
etc.
Sunday, May 20, 12
Alex Baranau, Sematext International, 2012
Qs?http://github.com/sematext/HBaseHUT
http://blog.sematext.com
@abaranau
http://github.com/sematext (abaranau)
http://sematext.com, we are hiring! ;)
Sunday, May 20, 12
