Upload
others
View
15
Download
0
Embed Size (px)
Citation preview
DruidKylin
2017-08-05
OLAP
Druid
Kylin
OLAP
ROLAP (Relational OLAP)
RDBMS/
Presto / Impala / SparkSQL / Drill
MOLAP (Multi-dimensional OLAP)
Kylin / Druid
Search Engines
Elasticsearch / Solr
ROLAP (Presto / SparkSQL)
SQL
MOLAP (Kylin, Druid)
OLAP
Search Engine (ES)
Join
OLAP
1.
2. /
3.
OLAP
SQL
KylinDruid
Presto / Spark Kylin Druid ES
SQL
OLAP
Druid
Kylin
Druid
OLAP
40100Datasource20 TB
Kafka
150TP99~1
Druid/JVM
JVM
Coordinator/Overlord 2 x 8g VMs 6g Heap
Broker 8 x 8core 16g VMs 11g Heap, 3g Non-Heap
Historical 19 x 40core 128g 12disk 12g Heap, 10g Non-Heap
MiddleManager 19 x 40core 128g 8 x 6g Heap Peons
Tranquility Clients 2 x 40core 128g 2~3g per JVM
Druid
DataSourceQPSLatencyIngestion Rate
/CPUI/OJVM
DashboardOLAP
Kafka ESEmitter
druid
Spark Streaming
DruidDashboard
Druid
Filter
Agg
Grouping
Tier
historical historical
historical historical
broker broker
Load Balance
_default_tier
historical historical
historical historical
broker broker
Load Balance
workload
(1/2)
Historicalsys cpu
NUMAzone reclaim modedirect page scan
echo 0 > /proc/sys/vm/zone_reclaim_mode
(2/2)
CentOS 6.5
echo never > /sys/kernel/mm/transparent_hugepage/enabled echo never > /sys/kernel/mm/transparent_hugepage/defrag
SQL
OLAP
Druid
Kylin
Kylin
KylinHadoopOLAPSQL
KylinMOLAPCube
CubeCube Cube
Kylin
Kylin
OLAP
20+Project350Cube
8000Cube140TB
130TP99~1
Kylin
RS RS
job jobHA
A
query queryB
query query
HBase
RS
RS RSRS
Cube
1. 2. 3. 4. Union
1. 2. HBaseHA 3. 4.
1. 2. 3. 4. JMX
3Kylin Committer
Cube