Batch and Stream processing with SQL

Batch and Stream processingwith SQL

2013/11/07 Cloudera World Tokyo 2013

TAGOMORI Satoshi @tagomorisLINE Corp.

13年11月7日木曜日

SQL、宇宙、すべての答え

SELECT 42 FROM anywhere


TAGOMORI Satoshi (@tagomoris)LINE Corp.

Hadoop, Fluentd, Norikra, ...





Data Collecting,Aggregation, Analytics,

Visualization


SQL好きですか？


How to write M/R(or Storm app, or...)

Java (or Scala, Clojure, JRuby, ...)

Hadoop Streaming

Pig

Hive, Impala (SQL!)


Our log trafficDaily

2.1+ TB (non compressed)

6.8+ Billion lines / day

Peak time

150,000+ lines / sec

380+ Mbps


Our Hadoop cluster

CDH 4.2.0

Master Nodes: 3 (NameNodeHA+QJM)

NameNode, JournalNode, JobTracker

Slave Nodes: 20


What we want to doCOUNT PV,UU and others (daily/realtime)

COUNT Service metrics (daily/hourly/realtime)

FIND Surprising Errors [4xx,5xx] (immediately)

CHECK Response Times (immediately)

SERCH Logs in troubles (hourly/immediately)

VISUALIZE/NOTIFY App Status (realtime)


Batches and StreamsHadoop is for batchesHigh performance batch is important

HDFS has good performance

Stream log writing and calculationsare also VERY VERY IMPORTANT

Hybrid System:Stream processing + Batch


System OverviewWeb Servers Fluentd

Cluster

ArchiveStorage(scribed)

FluentdWatchers

GraphTools

Notifications(IRC)

Hadoop Cluster(HDFS, MR)

webhdfs

HuahinManager

hiveserver

STREAM

Shib ShibUI

BATCH SCHEDULEDBATCH

Norikra


Data analytics players

StoragesHadoop Cluster

Visualization Tools

ADMINISTRATOR

Raw Log FormatsApplication Logs

Data SizesData Semantics

PROGRAMMER

SERVICE DIRECTORSALES

Whatever Metrics They Want

BOARD MEMBER

........


Data analytics players

StoragesHadoop Cluster

Visualization Tools

ADMINISTRATOR

Raw Log FormatsApplication Logs

Data SizesData Semantics

PROGRAMMER

SERVICE DIRECTORSALES

Whatever Metrics They Want

BOARD MEMBER

........

WE NEED THE QUERY LANGUAGEWHAT THEY ALL CAN

RUN AND UNDERSTAND!!!!!!!!!!


SQL: Hive


SQL: Hive


Hive

SQL: w/o compile, w/o deployment

HiveServer: w/o server login

Shib: Select only



Hive:

Simplify versioning problems

Hive 0.10 of CDH 4.2.0

Upgrade CDH for only Hive version


Hive: Pros

Many Scheduled Queries Metrics OnDemand Queries


Hive: Cons

Too Many Scheduled Queries for short time window


Stream processingQueries for fixed Window

every 1hour, 10minutes, 1minutes, ...latest 10evens, ...all events

Once query registered, Runs foreverResults appear automatically

NO MORE STORAGES


Stream processing

And

SQL13年11月7日木曜日

Norikra:Schema-less Stream Processing with SQL


Norikra(1):Schema-less event stream:

Add/Remove data fields whenever you want

SQL:No more restarts to add/remove queriesw/ JOINs, w/ SubQueriesw/UDF

Truly Complex events:Nested Hash/Array, accessible directly from SQL


Norikra(2):Open source software:

Licensed under GPLv2Based on EsperUDF plugins from rubygems.org

Ultra-fast bootstrap & small start:3mins to install/start1 server


Norikra Queries: (1)

SELECT name, ageFROM events




{“name”:”tagomoris”, “age”:34, “address”:”Tokyo”, “corp”:”LINE”, “current”:”Meguro”}

{“name”:”tagomoris”,”age”:34}




{“name”:”tagomoris”, “address”:”Tokyo”, “corp”:”LINE”, “current”:”Meguro”}

nothing




WHERE current=”Meguro”


{“name”:”tagomoris”,”age”:34}


Norikra Queries: (2){“name”:”frsyuki”, “age”:25, “address”:”MountainView”, “corp”:”TD”, “current”:”BayArea”}


WHERE current=”Meguro”

nothing



SELECT age, COUNT(*) as cntFROM events.win:time_batch(5 mins)

GROUP BY age




GROUP BY age


{”age”:34,”cnt”:3}, {“age”:33,”cnt”:1}, ...

every 5 mins



SELECT age, COUNT(*) as cntFROM

events.win:time_batch(5 mins)GROUP BY age


{”age”:34,”cnt”:3}, {“age”:33,”cnt”:1}, ...every 5 mins

SELECT max(age) as maxFROM

events.win:time_batch(5 mins)

{“max”:51}




GROUP BY age

{“name”:”tagomoris”, “user:{“age”:34, “corp”:”LINE”, “address”:”Tokyo”}, “current”:”Meguro”, “speaker”:true, “attend”:[true,true,false, ...]}



SELECT user.age, COUNT(*) as cntFROM events.win:time_batch(5 mins)

GROUP BY user.age




SELECT user.age, COUNT(*) as cntFROM events.win:time_batch(5 mins)

WHERE current=”Meguro” AND attend.$0 AND attend.$1GROUP BY user.age



Before: Hive EVERY HOUR!SELECT yyyymmdd, hh, campaign_id, region, lang, count(*) AS click, count(distinct member_id) AS uuFROM ( SELECT yyyymmdd, hh, get_json_object(log, '$.campaign.id') AS campaign_id, get_json_object(log, '$.member.region') AS region, get_json_object(log, '$.member.lang') AS lang, get_json_object(log, '$.member.id') AS member_id FROM applog WHERE service='myservice' AND yyyymmdd='20131101' AND hh='00' AND get_json_object(log, '$.type') = 'click') xGROUP BY yyyymmdd, hh, campaign_id, region, lang


After: NorikraSELECT campaign.id AS campaign_id, member.region AS region, count(*) AS click, count(distinct member.id) AS uuFROM myservice.win:time_batch(1 hours)WHERE type="click"GROUP BY campaign.id, member.region


Norikra: Current Status

v0.1.0: Released at 2013/11/01

by tagomoris

http://norikra.github.io/

Documents: under development

Just started to use in production


http://norikra.github.io

http://norikra.github.io

SQL Queries for batches for streams


企画・開発幅広く募集中

•データマーケティング•データベースエンジニア• BI企画・開発• etc…


MUSIC�

SHOPPING� COOKING�

MOVIE�

GAME�

TRAVEL�

NEWS�

MOM&KIDS�

SPORTS�

BOOK�

GIRLS�

Variety Volume Velocity�


企画・開発幅広く募集中

コーポレートサイトからどうぞ応募を！

データ分析・解析規模拡大/強化中


See Also:Log analysis system with Hadoop in livedoor 2013

http://www.slideshare.net/tagomoris/log-analysis-with-hadoop-in-livedoor-2013

Norikrahttp://norikra.github.io/https://github.com/norikra

Shibhttps://github.com/tagomoris/shib

Fluentdhttp://fluentd.org/https://github.com/fluent/fluentd






https://github.com/norikra

https://github.com/norikra

https://github.com/tagomoris/shib

https://github.com/tagomoris/shib

http://fluentd.org/

http://fluentd.org/

https://github.com/fluent/fluentd

https://github.com/fluent/fluentd

Technology

Batch and Stream processing with SQL