Twitter Storm

Twitter StormRealtime distributed computations

Sergey Lukjanov <[email protected]>Dmitry Mescheryakov <[email protected]>

Wednesday, October 3, 12

mailto:[email protected]




Real-time data processing

2Wednesday, October 3, 12

Real-time data processingbefore Twitter Storm:

network of queues and workers




2

MESSAGES QUEUE




2

MESSAGES QUEUE

Message routing can be

complex!



3

MESSAGES QUEUE

QUEUE

QUEUE

MESSAGES

MESSAGES



3

MESSAGES QUEUE

QUEUE

QUEUE

MESSAGES

MESSAGESQueues replication is needed for reliability



3

MESSAGES QUEUE

QUEUE

QUEUE

MESSAGES


Hard to maintain queues



3

MESSAGES QUEUE

QUEUE

QUEUE

MESSAGES


Hard to maintain queues

Each new computation branch

requires routing reconfiguration


Twitter Storm


Twitter Stormdistributed;



fault-tolerant;



fault-tolerant;

real-time;



fault-tolerant;

real-time;

computation;



fault-tolerant;

real-time;

computation;

fail-fast components.


(Very) basic info


(Very) basic infocreated by Nathan Marz from Backtype/Twitter;



Eclipse Public License 1.0;




open sourced at September 19th, 2011;





about 16k Java and 7k Clojure LoC;






most watched Java repo at Github (> 4k watchers);






most watched Java repo at Github (> 4k watchers);

active UG.


Current status


Current statuscurrent stable release: 0.8.1;



0.8.2 with small bug fixes is already on the way;




0.9.0 with major core improvements is planned;





not very active contributions, we can try to get into;





not very active contributions, we can try to get into;

used by over 30 companies (such as Twitter, Groupon, Alibaba, GumGum, etc).


Key properties


Key propertiesextremely broad set of use cases:

streams processing;

database updating;

distributed rpc;



streams processing;

database updating;

distributed rpc;

scalable and extremely robust;



streams processing;

database updating;

distributed rpc;


guarantees no data loss;



streams processing;

database updating;

distributed rpc;



fault-tolerant;



streams processing;

database updating;

distributed rpc;



fault-tolerant;

programming language agnostic.7


Key conceptsTuples (ordered list of elements)


Key conceptsTuples (ordered list of elements)

8

( “Saratov”, “slukjanov”, “event1”, “10/3/12 16:20”)


Key conceptsStreams (unbounded sequence of tuples)


Key conceptsStreams (unbounded sequence of tuples)

9

TUPLE TUPLE TUPLE TUPLE TUPLE


Key conceptsSpouts (source of streams)



10




10


Spouts can talk with:queues;

logs;

API calls;

event data.


Key conceptsBolts (process tuples and create new streams)


Key conceptsBolts (process tuples and create new streams)

11


TUPLETUPLE

TUPLETUPLE

TUPLE

TUPLE

TUPLE

TUPLE

TUPLE

TUPLE


Key conceptsYou can do the following things in Bolts:

12

apply functions / transformations;

filter;

aggregation;

streaming joins;

access DBs, APIs, etc...


Key conceptsTopologies (a directed graph of Spouts and Bolts)



13


TUPLETUPLE

TUPLETUPLE

TUPL

TUPLE

TUPLE

TUPLE

TUPLE

TUPLE



14



TUPLE

TUPLE

TUPLE

TUPLE

TUPLE



TUPLE

TUPLE

TUPLE

TUPLE

TUPLE


Key conceptsTasks (instances of spouts and bolts)


Key conceptsTasks (instances of spouts and bolts)

15

Task 1

Task 2

Task 3

Task 4


Key conceptsCluster

16

Nimbus

Zookeeper

Zookeeper

Zookeeper

Supervisor

Supervisor

Supervisor

Supervisor

Supervisor


Key conceptsCluster

16

Nimbus

Zookeeper

Zookeeper

Zookeeper

Supervisor

Supervisor

Supervisor

Supervisor

Supervisor

UI


Key conceptsCluster

16

Nimbus

Zookeeper

Zookeeper

Zookeeper

Supervisor

Supervisor

Supervisor

Supervisor

Supervisor

Hadoop’s Job tracker

UI


Key conceptsCluster

16

Nimbus

Zookeeper

Zookeeper

Zookeeper

Supervisor

Supervisor

Supervisor

Supervisor

Supervisor

Hadoop’s Job tracker

Hadoop’s Task tracker

UI


Based on


Based onApache Zookeeper (maintaining configs);



∅MQ (transport layer);




Apache Thrift (cross-language bridge, rpc);





LMAX Disruptor (bounded prod-cons queue);





LMAX Disruptor (bounded prod-cons queue);

Kryo (serialization framework).


Grouping


Groupingshuffle (randomly and evenly distributed);



local or shuffle (local workers are preferred);




fields (the stream is partitioned by specified fields);





all (the stream is replicated across all the bolt’s tasks);






global (the entire stream goes to a single bolt’s task);







direct (producers could directly emit tuples);







direct (producers could directly emit tuples);

custom (implement interface CustomStreamGrouping).


WordCount sample


WordCount samplerandom sentence generator;



sentence splitter;



sentence splitter;

word counter;



sentence splitter;

word counter;

ping spout (metronome).


WordCount sample


WordCount sample

20

SENTENCE SENTENCE

SENTENCE GENERATOR


WordCount sample

20

SENTENCE SENTENCE

SENTENCE GENERATOR

WORD WORD

SENTENCESPLITTER


WordCount sample

20

SENTENCE SENTENCE

SENTENCE GENERATOR

WORD WORD

SENTENCESPLITTER

WORDCOUNTER


WordCount sample

20

SENTENCE SENTENCE

SENTENCE GENERATOR

WORD WORD

SENTENCESPLITTER

WORDCOUNTER

GROUP BY WORD


WordCount sample

20

SENTENCE SENTENCE

SENTENCE GENERATOR

WORD WORD

SENTENCESPLITTER

WORDCOUNTER

PING

PINGPINGGENERATOR

GROUP BY WORD


WordCount sample

20

SOUT

SENTENCE SENTENCE

SENTENCE GENERATOR

WORD WORD

SENTENCESPLITTER

WORDCOUNTER

PING

PINGPINGGENERATOR

GROUP BY WORD


WordCount sample

20

SENTENCE SENTENCE

SENTENCE GENERATOR

WORD WORD

SENTENCESPLITTER

WORDCOUNTER

PING

PINGPINGGENERATOR

DB

GROUP BY WORD


Sentence generator

21

public class RandSentenceGenerator extends BaseRichSpout { private SpoutOutputCollector collector; private Random random; private String[] sentences; @Override public void open(Map map, TopologyContext ctx, SpoutOutputCollector collector) { this.collector = collector; this.random = new Random(); this.sentences = <sentences array>; } @Override public void nextTuple() { Utils.sleep(10); String sentence = sentences[random.nextInt(sentences.length)]; collector.emit(new Values(sentence)); }

@Override public void declareOutputFields(OutputFieldsDeclarer declarer) { declarer.declare(new Fields("sentence")); } }


Sentence splitter

22

public class SplitSentence extends BaseBasicBolt { @Override public void execute(Tuple tuple, BasicOutputCollector collector) { String sentence = tuple.getString(0); for (String word : sentence.split("\\s")) { collector.emit(new Values(word)); } } @Override public void declareOutputFields(OutputFieldsDeclarer declarer) { declarer.declare(new Fields("word")); } }


Word count

23

public class WordCount extends BaseBasicBolt { private HashMultiset<String> words = HashMultiset.create();

@Override public void prepare(Map conf, TopologyContext ctx) { super.prepare(conf, ctx); this.logger = Logger.getLogger(this.getClass()); this.name = ctx.getThisComponentId(); this.task = ctx.getThisTaskIndex(); }

@Override public void execute(Tuple tuple, BasicOutputCollector collector) { String source = tuple.getSourceComponent(); if ("split".equals(source)) { words.add(tuple.getString(0)); } else if ("ping".equals(source)) { logger.warn("RESULT " + name + ":" + task + " :: " + words); } }

@Override public void declareOutputFields(OutputFieldsDeclarer declarer) { declarer.declare(new Fields("word", "count")); } }


Topology builder

24

public class WordCounter { public static void main(String[] args) throws Exception { TopologyBuilder builder = new TopologyBuilder(); builder.setSpout("source", new RandSentenceGenerator(), 3); builder.setSpout("ping", new PingSpout()); builder.setBolt("split", new SplitSentence(), 8) .shuffleGrouping("source"); builder.setBolt("count", new WordCount(), 12) .fieldsGrouping("split", new Fields("word")) .allGrouping("ping");

<topology submitting> } }


Topology submitter

25

public class WordCounter { public static void main(String[] args) throws Exception { TopologyBuilder builder = new TopologyBuilder(); <building topology> Config conf = new Config(); conf.setDebug(true); conf.setNumWorkers(3); StormSubmitter.submitTopology("tplg-name", conf, builder.createTopology()); } }


Multilang support


Multilang supportDSLs for Scala, JRuby and Clojure;



ShellSpout, ShellBolt;



ShellSpout, ShellBolt;

json-based protocol:

receive/emit tuples;

ack/fail tuples;

write to logs.


Online logs processing



27

Application

Application

Application QUEUE

Twitter Storm Cluster

RABBITMQ

STORAGE CASSANDRA





28

RABBITMQ



28

RABBITMQ

QUEUECONSUMER

LOGMESSAGES



28

MESSAGE MESSAGE

MESSAGEPARSERSHUFFLE GROUPING

RABBITMQ

QUEUECONSUMER

LOGMESSAGES



28

MESSAGE MESSAGE


EVENT EVENT

FIELDS GROUPINGEVENT

AGGREGATOR

RABBITMQ

QUEUECONSUMER

LOGMESSAGES



28

MESSAGE MESSAGE


EVENT EVENT

FIELDS GROUPINGEVENT

AGGREGATOR

CASSANDRA

REALTIMEINFO & STATS

RABBITMQ

QUEUECONSUMER

LOGMESSAGES


Storm fault-tolerance


Storm fault-tolerance

29

Parts of Storm cluster:

Zookeeper nodes;

Nimbus (master) node;

Supervisor nodes.


Nimbus as a point of failure



30

when Nimbus is down:

topologies continue to work;

tasks from failing nodes aren’t respawned;

can’t upload a new topology or rebalance an old one;



30

when Nimbus is down:

topologies continue to work;

tasks from failing nodes aren’t respawned;

can’t upload a new topology or rebalance an old one;

impossible to run Nimbus at another node:

either fix the failed node;

or create new and resubmit all topologies.


Tuple types

31

spout tuple - emitted from Spouts;

child tuple - emitted from Bolts, based on parent tuple(s) (child or spout ones).


Tuple types

31

spout tuple - emitted from Spouts;

child tuple - emitted from Bolts, based on parent tuple(s) (child or spout ones).

[“the cow jumped over the moon”]

[“the”]

[“cow”]

[“jumped”]

[“over”]

[“the”]

[“moon”]

[“the”, 1]

[“cow”, 1]

[“jumped”, 1]

[“over”, 1]

[“the”, 2]

[“moon”, 1]


Reliability API Guaranties

32

public class QueueConsumer extends BaseRichSpout {

...

@Override public void nextTuple() { Message msg = queueClient.popMessage(); collector.emit(msg.getPayload(), msg.getId()); }

@Override public void ack(Object msgId) { queueClient.ack(msgId); } @Override public void fail(Object msgId) { queueClient.fail(msgId); }

...

}


Tuple tree tracking


Tuple tree tracking

33

spout tuple creation:

collector.emit(values, msgId);


Tuple tree tracking

33



child tuple creation:

collector.emit(parentTuples, values);


Tuple tree tracking

33





tuple end of processing:

collector.ack(tuple);


Tuple tree tracking

33





tuple end of processing:

collector.ack(tuple);

tuple failed to process:

collector.fail(tuple);


Disabling reliability API



34

globally:

Config.TOPOLOGY_ACKER_EXECUTORS = 0;



34

globally:


on topology level:




34

globally:


on topology level:


for a single tuple:



Acker system impl

35

every tuple is assigned a random 64-bit ID


Acker system impl

35


Spout Bolt A Bolt B Bolt C[1] [2] [3]


Acker system impl

35



[1] emit


Acker system impl

35



[1] emit[2] emit


Acker system impl

35



[1] emit[2] emit[1] ack


Acker system impl

35



[1] emit[2] emit[1] ack[3] emit


Acker system impl

35



[1] emit[2] emit[1] ack[3] emit[2] ack


Acker system impl

35



[1] emit[2] emit[1] ack[3] emit[2] ack[3] ack


Acker - simplified algo



36

tuple tree:

{ Spout tuple ID, set };



36

tuple tree:

{ Spout tuple ID, set };

message processing:

[tuple_id] emit: set.add(tuple_id);

[tuple_id] ack: set.remove(tuple_id);

if (set.size == 0): send ack to parent spout.


Acker - real algo

37

tuple tree:

{ Spout tuple ID, ackVal: int64 };

message processing:

[tuple_id] emit: ackVal ^= tuple_id;

[tuple_id] ack: ackVal ^= tuple_id;

if (ackVal == 0): send ack to parent spout.


Correctness of the tracking



38

bolt fails before sending ack for a tuple:

no ack arrive before timeout, spout tuple fails;



38



acker fails before acking tuple tree processing:

-- the same as above --;



38



acker fails before acking tuple tree processing:

-- the same as above --;

spout fails before acking message:

the message source should handle client’s death.


Reliability API - Conclusion



39

easy to dismiss:

on message - at most one processing;



39

easy to dismiss:


if using, little overhead and high durability:

one message - at least one processing;



39

easy to dismiss:


if using, little overhead and high durability:

one message - at least one processing;

with some further work (transactions, Trident API):

one message - exactly one processing.


Transactional approach: design #1



40

Spout Bolt ATUPLE COMMITMESSAGE



40

input provides messages in strong order;




40


each message is assigned Transaction ID;




40


each message is assigned Transaction ID;

if (curr_tx_id > prev_tx_id) commit(result, curr_tx_id).




41


each batch of messages is assigned Transaction ID;

if (curr_tx_id > prev_tx_id) commit(result, curr_tx_id).

Spout Bolt A

BATCH OF TUPLES COMMIT

BATCH OFMESSAGES



42

the same as #2, but each transaction is split:

processing phase;

commit phase;



42


processing phase;

commit phase;

process phases might intersect for difference transactions;



42


processing phase;

commit phase;

process phases might intersect for difference transactions;

commit phases go in strong order.


Trident API: Intro


Trident API: Intro

43

high-level abstraction for doing realtime computations;


Trident API: Intro

43


high throughput (millions of messages per second);


Trident API: Intro

43



stateful stream processing;


Trident API: Intro

43




low latency distributed querying;


Trident API: Intro

43





different semantics (including exactly-once one);


Trident API: Intro

43





different semantics (including exactly-once one);

smth. like Pig or Cascading.


Trident API: Operations



44

partition-local operations (w/o network transfer):

function, filter, partitionAggregate, stateQuery, etc;



44



repartitioning operations (grouping);



44




aggregations operations:

aggregate, persistentAggregate;



44






operations on grouped streams;



44






operations on grouped streams;

merges and joins.


Trident API: Demo

45

TridentTopology topology = new TridentTopology(); TridentState wordCounts = topology.newStream("spout1", new FixedBatchSpout()) .each(new Fields("sentence"), new Split(), new Fields("word")) .groupBy(new Fields("word")) .persistentAggregate(new MemoryMapState.Factory(), new Count(), new Fields("count")) .parallelismHint(6); topology.newDRPCStream("words") .each(new Fields("args"), new Split(), new Fields("word")) .groupBy(new Fields("word")) .stateQuery(wordCounts, new Fields("word"), new MapGet(), new Fields("count")) .each(new Fields("count"), new FilterNull()) .aggregate(new Fields("count"), new Sum(), new Fields("sum")); Config config = new Config(); config.setMaxSpoutPending(100); cluster.submitTopology("word-count-tplg", config, topology.build()); DRPCClient client = new DRPCClient("drpc.server.host", 3772); System.out.println(client.execute("words", "cat dog the man")); System.out.println(client.execute("words", "cat")); // prints the JSON-encoded result, e.g.: "[[5078]]"


Q & A


Documents

Twitter Storm