Norikra in action

Data/Stream Processing Meetup (2013/06/28)TAGOMORI Satoshi (@tagomoris)

13年6月29日土曜日

TAGOMORI Satoshi (@tagomoris)LINE corp.

Ruby, Perl, Node.js, Hadoop, ...

13年6月29日土曜日

13年6月29日土曜日

System OverviewWeb Servers Fluentd

Cluster

ArchiveStorage(scribed)

FluentdWatchers

GraphTools

Notifications(IRC)

Hadoop Cluster(HDFS, YARN)

webhdfs

HuahinManager

hiveserver

STREAM

Shib ShibUI

BATCH SCHEDULEDBATCH

Norikra

13年6月29日土曜日

Stream queryCustom fluentd plugin: not so casual enoughxQL: declarative languagestreams processing

for optional data fieldsno more schema management

connectivity with Fluentd

13年6月29日土曜日

Stream query: vs stored data query

No more query wait time

Immediate result for time batch

No more storages

No more query execution management

Once register query, runs forever

13年6月29日土曜日

Esper

"Esper and Event Processing Language (EPL) provide a highly scalable, memory-efficient, in-memory computing, SQL-standard, minimal latency, real-time streaming Big Data processing engine for medium to high-velocity and high-variety data."

http://esper.codehaus.org/

13年6月29日土曜日

Norikra: is not for only Fluentd.

13年6月29日土曜日

Norikra query: vs Fluentd custom plugin

SQL!!!

No more restart for new queries

register queries whenever we want

No more private plugins

No more fat Fluentd configurations

13年6月29日土曜日

Norikra

Full feature of Esper over JRuby

Simple RPC: msgpack-rpc-over-http

Simple RPC Server: mizuno (jetty + rack)

Simple Client Library: norikra-client

Just same code for cruby/jruby

13年6月29日土曜日

Norikra

Norikra Server (on JVM)

Esper Instance (Query Engine)

Type DefinitionManager

Output Event Pool

Norikra Engine

RPC Servermizuno (Jetty + Rack)

Rack RPC HandlerNorikraClient

NorikraClient

JRUBY

CRUBY

msgpack-rpc-over-http

13年6月29日土曜日

Norikra Query: target "sales"

goods_id:5 price:49.8 num:1 shop:"LINE"goods_id:2 price:12.5 num:3 shop:"Cookpad"goods_id:4 price:36.6 num:10 shop:"Cookpad"

SELECT shop, sum(price*num) AS amountFROM sales.win:time_batch(10 minutes)GROUP BY shop

goods_id:5 price:49.8 num:1 shop:"LINE"

goods_id:2 price:12.5 num:3 shop:"Cookpad" affiliate:"BiS"

SELECT affiliate, count(*) AS cntFROM sales.win:time_batch(1 hour)GROUP BY affiliate

13年6月29日土曜日

Esper and NorikraEsper:

queries for streamsstream: a set of field-type pairs of eventsusers need to know for field set variations(or manage 'map subtypes' on your own)

Norikra:queries for targetstarget: virtual name of union of field set variationusers don't need to know for detail of target

13年6月29日土曜日

automated stream inheritanceof norikra's target

Base typedef

Query typedef

Data typedef

b_xxxxxxxxx

minimal fieldset definition:

name: 'string'id: 'long'

valid: 'boolean'action_type: 'string'

13年6月29日土曜日

automated stream inheritanceof norikra's target

Base typedef

Query typedef

Data typedef

b_xxxxxxxxx

event data fieldset definition:

name: 'string'id: 'long'

valid: 'boolean'action_type: 'string'

product_code: 'string'charge: 'integer'shop_code: 'long'e_xxxxxxxx1

13年6月29日土曜日

automated stream inheritanceof norikra's target

Base typedef

Query typedef

Data typedef

b_xxxxxxxxx

e_xxxxxxxx1 e_xxxxxxxx2

event data fieldset definition:name: 'string'

id: 'long'valid: 'boolean'

action_type: 'string'product_code: 'string'

charge: 'integer'shop_code: 'long'affiliate: 'string'

13年6月29日土曜日

automated stream inheritanceof norikra's target

Base typedef

Query typedef

Data typedef

b_xxxxxxxxx

e_xxxxxxxx1 e_xxxxxxxx2

new query:SELECT count(*)

FROM target.win:time_batch(1min)WHERE affiliate.length() > 0

13年6月29日土曜日

automated stream inheritanceof norikra's target

Base typedef

Query typedef

Data typedef

b_xxxxxxxxx

e_xxxxxxxx1 e_xxxxxxxx2'

event data fieldset definition:

name: 'string'id: 'long'

valid: 'boolean'action_type: 'string'

affiliate: 'string'

q_xxxxxxxx0

new query:SELECT count(*)

FROM target.win:time_batch(1min)WHERE affiliate.length() > 0

13年6月29日土曜日

automated stream inheritanceof norikra's target

Base typedef

Query typedef

Data typedef

b_xxxxxxxxx

e_xxxxxxxx1 e_xxxxxxxx2'

q_xxxxxxxx0

Registered EPL:SELECT count(*)

FROM q_xxxxxxxx0.win:time_batch(1min)WHERE affiliate.length() > 0

13年6月29日土曜日

automated stream inheritanceof norikra's target

Base typedef

Query typedef

Data typedef

b_xxxxxxxxx

e_xxxxxxxx1' e_xxxxxxxx2'

q_xxxxxxxx0

e_xxxxxxxx3'

q_xxxxxxxx1

13年6月29日土曜日

Output data pooling

Output event data: pushed

Event pushing brings many problems

Pooling + fetch

typical usecase: aggregation

-> not so many outputs

13年6月29日土曜日

fluent-plugin-norikra

Fluentd plugin to use Norikra

Norikra server autostart

Automatically defined target

Pre-defined queries for each targets

13年6月29日土曜日

fluent-plugin-norikra

installation

`gem install fluent-plugin-norikra`

configuration

see DEMO

13年6月29日土曜日

Demo: bootstrap

rbenv shell jruby-1.7.4gem install norikrawhich norikrarbenv shell 2.0.0-pxxxgem install fluent-plugin-norikravi demo.conffluentd -c demo.conf

13年6月29日土曜日

Demo: query streams

some messages over fluent-cat

register queries with norikra-client

more messages over fluent-cat & norikra-client

13年6月29日土曜日

roadmap of norikraNorikra is still UNDER DEVELOPMENT

Norikra feature updates (JOINs, etc)Web GUI

query & target list managementsave & restore metadata

Distributed & orchestrated nodes

13年6月29日土曜日

See also:http://fluentd.org/http://fluentd.org/plugin/https://github.com/tagomoris/norikrahttps://github.com/tagomoris/norikra-clienthttps://github.com/tagomoris/fluent-plugin-norikrahttp://esper.codehaus.org/

"Fluentd: The ruby based middleware across the world"http://www.slideshare.net/tagomoris/fluentd-in-tkrk10

"Log analysis system with Hadoop in livedoor 2013 Winter"http://www.slideshare.net/tagomoris/log-analysis-with-hadoop-in-livedoor-2013

13年6月29日土曜日