Upload
others
View
19
Download
0
Embed Size (px)
Citation preview
2
HBase Cache ▪HBase Cache for better read performance
• Keep data local to the process
▪L1 on heap LRU Cache• Larger Cache Sizes => Larger Java Heap => GC issues
▪L2 Bucket Cache• LRU• Backed by off heap memory / File• Can be larger than L1 allocation• Not constrained by Java Heap Size
▪ L2 offheap cache can give large cache size
▪ 4 MB physical memory buffers.
▪ Different sized buckets 5 KB, 9 KB,… 513 KB. Each bucket having at least 4 slots
▪ HFile blocks placed in appropriate sized bucket
▪ One Block may span across 2 ByteBuffers.
▪ Read path assumption of data being in a byte array.
▪ Cells having assumption of data parts being in byte array. (ie. Rowkey, family, value etc)
Bucket Cache overview4 MB
5 KB buckets
9 KB buckets
513 KB buckets
Bucket Cache - Reads
4
▪ Read block by block
▪ On heap block copy
▪ Less throughput vs L1
▪ GC Impacts
▪ Predictable latency problems
Region1
Region2
Read request
Read request
RegionServer
Scanner layers
Scanner layers
Copy
Copy
Get block
On heap Block
Off heap Bucket Cache
Java ByteBuffer
Bucket slot
Get block
On heap Block
5
Building the offheap read path
▪ Cellify the read path▪ Avoid the byte[] assumption and usage of KeyValue throughout the read path
▪ Add ByteBufferedCell extension in the server side
▪ Create NIO ByteBuffer wrapper called ByteBuff• SingleByteBuff, MultiByteBuff
§ Ref count based block cache and block eviction mechanism• Increment ref count when reader hits a block in L2 cache
• Decrement once response is created for RPC using a RPC callback
• Evict if/when ref count = 0
SingleByteBuff
MultiByteBuff
Complete Picture
Region1
Region2
Read request
Read request
HRegionServer
Off heap Bucket Cache
Refcount++
Read response
Read response
Scanner layers
Scanner layersRefcount++
callback
callback
Refcount--
Refcount--MultiByteBuff
SingleByteBuff
End to End off heap - from bucket cache till RPC
Bucket Cache – Off heap Reads
▪ Delivers data directly from off heap cache
▪ Lesser Heap needed
▪ Better throughput
▪ Predictable latency
▪ Available in HBase 2.0
▪ JIRA # HBASE-11425
8
HBase at Alibaba
▪ Core storage system in Search Infrastructure
▪ Query HBase to provide real time search index updates
▪ Data Store for online machine learning system
▪ Biggest HBase cluster is 1500+ nodes, 200,000+ regions, tens ofmillions QPS
▪ Importance of consistent Latency and Throughput
▪ Use HBase Cache for better read performance– L1 LRU Cache
– & L2 Off Heap Cache
Data Platform for Search Indexingl Data Storage for Batch and Streaming processing
Data Source
Hadoop cluster
HBase
Batch & Streaming
Event
Offline & Real Time Processing Exporting
Ali ODPS MySQLSearch Engines
HBase HBase
HDFS HDFS HDFS
Latency matters
Database and Queue for Machine Learning
UDF UDF UDFHQueue
Online log
Parsing Log Training
User Models
Training Item ModelsItem ID
User ID
HQueue
Aggregate
Updates
Machine Learning Models
Online System
Δw Export Model
Updates
Model
Flink+MR Processing over Yarn
Throughput matters
11
Old Performance Data
▪ From A/B test cluster with 400+ nodes and on simulated data
▪ 12 GB off heap L2 cache
▪ Inconsistent throughput. Dips in latency
▪ HBase read from L2 does data copy to on heap– More garbage and GC workload
12
Performance Data After Off heaping read path
▪ Backported patches to hbase-1.2 based codebase
▪ Average throughput improved from 18 to 25 million QPS – 30% improvement
▪ Linear throughput. Predictable latency.
▪ In production since Oct 2016
▪ Used in 2016 Singles’ Day, with peaks at 100K QPS per single RegionServer
13
Related features in HBase 2.0
▪ Off heaping write path (HBASE-11579)
▪ Compacting Memstore (HBASE-14920)§ References
● https://blogs.apache.org/hbase/● https://blogs.apache.org/hbase/entry/offheap-read-path-in-production
Questions?