Twitter Storm

Twitter StormRealtime distributed computations

Sergey Lukjanov <slukjanov@mirantis.com>Dmitry Mescheryakov <dmescheryakov@mirantis.com>

Wednesday, October 3, 12

Real-time data processing

2Wednesday, October 3, 12

Real-time data processingbefore Twitter Storm:

network of queues and workers

MESSAGES QUEUE

Message routing can be

complex!

MESSAGES QUEUE

MESSAGES

MESSAGES QUEUE

MESSAGES

MESSAGESQueues replication is needed for reliability

MESSAGES QUEUE

MESSAGES

Hard to maintain queues

MESSAGES QUEUE

MESSAGES

Hard to maintain queues

Each new computation branch

requires routing reconfiguration

Twitter Storm

Twitter Stormdistributed;

fault-tolerant;

real-time;

fault-tolerant;

real-time;

computation;

fault-tolerant;

real-time;

computation;

fail-fast components.

(Very) basic info

(Very) basic infocreated by Nathan Marz from Backtype/Twitter;

Eclipse Public License 1.0;

open sourced at September 19th, 2011;

about 16k Java and 7k Clojure LoC;

most watched Java repo at Github (> 4k watchers);

active UG.

Current status

Current statuscurrent stable release: 0.8.1;

0.8.2 with small bug fixes is already on the way;

0.9.0 with major core improvements is planned;

not very active contributions, we can try to get into;

used by over 30 companies (such as Twitter, Groupon, Alibaba, GumGum, etc).

Key properties

Key propertiesextremely broad set of use cases:

streams processing;

database updating;

distributed rpc;

streams processing;

database updating;

distributed rpc;

scalable and extremely robust;

streams processing;

database updating;

distributed rpc;

guarantees no data loss;

streams processing;

database updating;

distributed rpc;

fault-tolerant;

streams processing;

database updating;

distributed rpc;

fault-tolerant;

programming language agnostic.7

Key conceptsTuples (ordered list of elements)

( “Saratov”, “slukjanov”, “event1”, “10/3/12 16:20”)

Key conceptsStreams (unbounded sequence of tuples)

TUPLE TUPLE TUPLE TUPLE TUPLE

Key conceptsSpouts (source of streams)

Spouts can talk with:queues;

API calls;

event data.

Key conceptsBolts (process tuples and create new streams)

TUPLETUPLE

Key conceptsYou can do the following things in Bolts:

apply functions / transformations;

filter;

aggregation;

streaming joins;

access DBs, APIs, etc...

Key conceptsTopologies (a directed graph of Spouts and Bolts)

TUPLETUPLE

Key conceptsTasks (instances of spouts and bolts)

Task 1

Task 2

Task 3

Task 4

Key conceptsCluster

Nimbus

Zookeeper

Supervisor

Key conceptsCluster

Nimbus

Zookeeper

Supervisor

Key conceptsCluster

Nimbus

Zookeeper

Supervisor

Hadoop’s Job tracker

Key conceptsCluster

Nimbus

Zookeeper

Supervisor

Hadoop’s Job tracker

Hadoop’s Task tracker

Based on

Based onApache Zookeeper (maintaining configs);

∅MQ (transport layer);

Apache Thrift (cross-language bridge, rpc);

LMAX Disruptor (bounded prod-cons queue);

Kryo (serialization framework).

Grouping

Groupingshuffle (randomly and evenly distributed);

local or shuffle (local workers are preferred);

fields (the stream is partitioned by specified fields);

all (the stream is replicated across all the bolt’s tasks);

global (the entire stream goes to a single bolt’s task);

direct (producers could directly emit tuples);

custom (implement interface CustomStreamGrouping).

WordCount sample

WordCount samplerandom sentence generator;

sentence splitter;

word counter;

sentence splitter;

word counter;

ping spout (metronome).

WordCount sample

SENTENCE SENTENCE

SENTENCE GENERATOR

WordCount sample

SENTENCE SENTENCE

SENTENCE GENERATOR

WORD WORD

SENTENCESPLITTER

WordCount sample

SENTENCE SENTENCE

SENTENCE GENERATOR

WORD WORD

SENTENCESPLITTER

WORDCOUNTER

WordCount sample

SENTENCE SENTENCE

SENTENCE GENERATOR

WORD WORD

SENTENCESPLITTER

WORDCOUNTER

GROUP BY WORD

WordCount sample

SENTENCE SENTENCE

SENTENCE GENERATOR

WORD WORD

SENTENCESPLITTER

WORDCOUNTER

PINGPINGGENERATOR

GROUP BY WORD

WordCount sample

SENTENCE SENTENCE

SENTENCE GENERATOR

WORD WORD

SENTENCESPLITTER

WORDCOUNTER

PINGPINGGENERATOR

GROUP BY WORD

WordCount sample

SENTENCE SENTENCE

SENTENCE GENERATOR

WORD WORD

SENTENCESPLITTER

WORDCOUNTER

PINGPINGGENERATOR

GROUP BY WORD

Sentence generator

public class RandSentenceGenerator extends BaseRichSpout { private SpoutOutputCollector collector; private Random random; private String[] sentences; @Override public void open(Map map, TopologyContext ctx, SpoutOutputCollector collector) { this.collector = collector; this.random = new Random(); this.sentences = <sentences array>; } @Override public void nextTuple() { Utils.sleep(10); String sentence = sentences[random.nextInt(sentences.length)]; collector.emit(new Values(sentence)); }

@Override public void declareOutputFields(OutputFieldsDeclarer declarer) { declarer.declare(new Fields("sentence")); } }

Sentence splitter

public class SplitSentence extends BaseBasicBolt { @Override public void execute(Tuple tuple, BasicOutputCollector collector) { String sentence = tuple.getString(0); for (String word : sentence.split("\\s")) { collector.emit(new Values(word)); } } @Override public void declareOutputFields(OutputFieldsDeclarer declarer) { declarer.declare(new Fields("word")); } }

Word count

public class WordCount extends BaseBasicBolt { private HashMultiset<String> words = HashMultiset.create();

@Override public void prepare(Map conf, TopologyContext ctx) { super.prepare(conf, ctx); this.logger = Logger.getLogger(this.getClass()); this.name = ctx.getThisComponentId(); this.task = ctx.getThisTaskIndex(); }

@Override public void execute(Tuple tuple, BasicOutputCollector collector) { String source = tuple.getSourceComponent(); if ("split".equals(source)) { words.add(tuple.getString(0)); } else if ("ping".equals(source)) { logger.warn("RESULT " + name + ":" + task + " :: " + words); } }

@Override public void declareOutputFields(OutputFieldsDeclarer declarer) { declarer.declare(new Fields("word", "count")); } }

Topology builder

public class WordCounter { public static void main(String[] args) throws Exception { TopologyBuilder builder = new TopologyBuilder(); builder.setSpout("source", new RandSentenceGenerator(), 3); builder.setSpout("ping", new PingSpout()); builder.setBolt("split", new SplitSentence(), 8) .shuffleGrouping("source"); builder.setBolt("count", new WordCount(), 12) .fieldsGrouping("split", new Fields("word")) .allGrouping("ping");

<topology submitting> } }

Topology submitter

public class WordCounter { public static void main(String[] args) throws Exception { TopologyBuilder builder = new TopologyBuilder(); <building topology> Config conf = new Config(); conf.setDebug(true); conf.setNumWorkers(3); StormSubmitter.submitTopology("tplg-name", conf, builder.createTopology()); } }

Multilang support

Multilang supportDSLs for Scala, JRuby and Clojure;

ShellSpout, ShellBolt;

json-based protocol:

receive/emit tuples;

ack/fail tuples;

write to logs.

Online logs processing

Application

Application QUEUE

Twitter Storm Cluster

RABBITMQ

STORAGE CASSANDRA

RABBITMQ

QUEUECONSUMER

LOGMESSAGES

MESSAGE MESSAGE

MESSAGEPARSERSHUFFLE GROUPING

RABBITMQ

QUEUECONSUMER

LOGMESSAGES

MESSAGE MESSAGE

EVENT EVENT

FIELDS GROUPINGEVENT

AGGREGATOR

RABBITMQ

QUEUECONSUMER

LOGMESSAGES

MESSAGE MESSAGE

EVENT EVENT

FIELDS GROUPINGEVENT

AGGREGATOR

CASSANDRA

REALTIMEINFO & STATS

RABBITMQ

QUEUECONSUMER

LOGMESSAGES

Storm fault-tolerance

Parts of Storm cluster:

Zookeeper nodes;

Nimbus (master) node;

Supervisor nodes.

Nimbus as a point of failure

when Nimbus is down:

topologies continue to work;

tasks from failing nodes aren’t respawned;

can’t upload a new topology or rebalance an old one;

when Nimbus is down:

topologies continue to work;

tasks from failing nodes aren’t respawned;

can’t upload a new topology or rebalance an old one;

impossible to run Nimbus at another node:

either fix the failed node;

or create new and resubmit all topologies.

Tuple types

spout tuple - emitted from Spouts;

child tuple - emitted from Bolts, based on parent tuple(s) (child or spout ones).

Tuple types

spout tuple - emitted from Spouts;

child tuple - emitted from Bolts, based on parent tuple(s) (child or spout ones).

[“the cow jumped over the moon”]

[“the”]

[“cow”]

[“jumped”]

[“over”]

[“the”]

[“moon”]

[“the”, 1]

[“cow”, 1]

[“jumped”, 1]

[“over”, 1]

[“the”, 2]

[“moon”, 1]

Reliability API Guaranties

public class QueueConsumer extends BaseRichSpout {

@Override public void nextTuple() { Message msg = queueClient.popMessage(); collector.emit(msg.getPayload(), msg.getId()); }

@Override public void ack(Object msgId) { queueClient.ack(msgId); } @Override public void fail(Object msgId) { queueClient.fail(msgId); }

Tuple tree tracking

spout tuple creation:

collector.emit(values, msgId);

Tuple tree tracking

child tuple creation:

collector.emit(parentTuples, values);

Tuple tree tracking

tuple end of processing:

collector.ack(tuple);

Tuple tree tracking

tuple end of processing:

collector.ack(tuple);

tuple failed to process:

collector.fail(tuple);

Disabling reliability API

globally:

Config.TOPOLOGY_ACKER_EXECUTORS = 0;

globally:

on topology level:

globally:

on topology level:

for a single tuple:

Acker system impl

every tuple is assigned a random 64-bit ID

Acker system impl

Spout Bolt A Bolt B Bolt C[1] [2] [3]

Acker system impl

[1] emit

Acker system impl

[1] emit[2] emit

Acker system impl

[1] emit[2] emit[1] ack

Acker system impl

[1] emit[2] emit[1] ack[3] emit

Acker system impl

[1] emit[2] emit[1] ack[3] emit[2] ack

Acker system impl

[1] emit[2] emit[1] ack[3] emit[2] ack[3] ack

Acker - simplified algo

tuple tree:

{ Spout tuple ID, set };

tuple tree:

{ Spout tuple ID, set };

message processing:

[tuple_id] emit: set.add(tuple_id);

[tuple_id] ack: set.remove(tuple_id);

if (set.size == 0): send ack to parent spout.

Acker - real algo

tuple tree:

{ Spout tuple ID, ackVal: int64 };

message processing:

[tuple_id] emit: ackVal ^= tuple_id;

[tuple_id] ack: ackVal ^= tuple_id;

if (ackVal == 0): send ack to parent spout.

Correctness of the tracking

bolt fails before sending ack for a tuple:

no ack arrive before timeout, spout tuple fails;

acker fails before acking tuple tree processing:

-- the same as above --;

acker fails before acking tuple tree processing:

-- the same as above --;

spout fails before acking message:

the message source should handle client’s death.

Reliability API - Conclusion

easy to dismiss:

on message - at most one processing;

easy to dismiss:

if using, little overhead and high durability:

one message - at least one processing;

easy to dismiss:

if using, little overhead and high durability:

one message - at least one processing;

with some further work (transactions, Trident API):

one message - exactly one processing.

Transactional approach: design #1

Spout Bolt ATUPLE COMMITMESSAGE

input provides messages in strong order;

each message is assigned Transaction ID;

if (curr_tx_id > prev_tx_id) commit(result, curr_tx_id).

each batch of messages is assigned Transaction ID;

if (curr_tx_id > prev_tx_id) commit(result, curr_tx_id).

Spout Bolt A

BATCH OF TUPLES COMMIT

BATCH OFMESSAGES

the same as #2, but each transaction is split:

processing phase;

commit phase;

processing phase;

commit phase;

process phases might intersect for difference transactions;

processing phase;

commit phase;

process phases might intersect for difference transactions;

commit phases go in strong order.

Trident API: Intro

high-level abstraction for doing realtime computations;

Trident API: Intro

high throughput (millions of messages per second);

Trident API: Intro

stateful stream processing;

Trident API: Intro

low latency distributed querying;

Trident API: Intro

different semantics (including exactly-once one);

Trident API: Intro

different semantics (including exactly-once one);

smth. like Pig or Cascading.

Trident API: Operations

partition-local operations (w/o network transfer):

function, filter, partitionAggregate, stateQuery, etc;

repartitioning operations (grouping);

aggregations operations:

aggregate, persistentAggregate;

operations on grouped streams;

merges and joins.

Trident API: Demo

TridentTopology topology = new TridentTopology(); TridentState wordCounts = topology.newStream("spout1", new FixedBatchSpout()) .each(new Fields("sentence"), new Split(), new Fields("word")) .groupBy(new Fields("word")) .persistentAggregate(new MemoryMapState.Factory(), new Count(), new Fields("count")) .parallelismHint(6); topology.newDRPCStream("words") .each(new Fields("args"), new Split(), new Fields("word")) .groupBy(new Fields("word")) .stateQuery(wordCounts, new Fields("word"), new MapGet(), new Fields("count")) .each(new Fields("count"), new FilterNull()) .aggregate(new Fields("count"), new Sum(), new Fields("sum")); Config config = new Config(); config.setMaxSpoutPending(100); cluster.submitTopology("word-count-tplg", config, topology.build()); DRPCClient client = new DRPCClient("drpc.server.host", 3772); System.out.println(client.execute("words", "cat dog the man")); System.out.println(client.execute("words", "cat")); // prints the JSON-encoded result, e.g.: "[[5078]]"

Twitter Storm

Documents

Storm - NodeBB · Distributed realtime computation system Originated at BackType/Twitter, open sourced in late 2011 ... sudo tar -zxvf apache-storm-1.0.2.tar.gz -C /opt/ 4. Fill in

Advanced Twitter Twitter

Twitter: @ketab nphoto.goodreads.com/documents/1357836988books/16130753.pdf · 2013. 1. 10. · Twitter: @ketab_n. Twitter: @ketab_n. Twitter: @ketab_n

Emerging threats by SANS Internet Storm CenterEmerging threats by SANS Internet Storm Center 3 About me - Bojan Ždrnja, Twitter: @bojanz - CTO at INFIGO IS- or testing team lead-

Twitter-Top Twitter Techniques

Introductie in Twitter Cards, Twitter Ads en Twitter Analytics

lillypreserveapartments.com...storm water management storm water management storm water management storm water management storm water management Storm water management 3125 LILLY RD

Storm @Twitter - cs.brown.edu · Storm @Twitter Ankit Toshniwal, Siddarth Taneja, Amit Shukla, Karthik Ramasamy, Jignesh M. Patel*, Sanjeev Kulkarni, Jason Jackson, Krishna Gade,

How to Spot First Stories on Twitter using Storm

bnetcmsus-a.akamaihd.net · THANK YOU FOR USING THIS KIT! Please share your creations with us on Twitter (@Blizzheroes) or Heroes of the Storm Facebook

Storm @Twitter - GitHub Pageslintool.github.io/UMD-courses/bigdata-2015-Spring/... · 2016-01-29 · graph), and thus needs a framework that allows extensibility. 4. Efficient: Since

Big Data Processing - cs.princeton.edu€¦ · • Apache/Twitter Storm: Record acknowledgment • Spark Streaming: Micro-batches • Google Cloud dataflow: transactional updates

Twitter: @ketab n egtema3.pdf · Twitter: @ketab_n. Twitter: @ketab_n. Twitter: @ketab_n

Twitter : ketab nphoto.goodreads.com/documents/1345890648books/15847622.pdf · Twitter : ketab_n. Twitter : ketab_n. Twitter : ketab_n

DECISION RIGHTS WORKSHOPWays to Use Twitter for Advocacy • Spread information about your issue or cause. Consider using a “tweetchat” or a Twitter “storm” to broaden the

D2 Storm Drain Restoration and Storm Water Mitigation Job... · D2 Storm Drain Restoration and Storm Water Mitigation ... D2 Storm Drain Restoration and Storm Water Mitigation

Pulsar Realtime Analytics At Scale · PDF fileNetty . Pulsar Integration with Kylin ... – Real-time analytics dashboard ... Twitter Storm/Spark Streaming vs Pulsar

CS535 Big Data 1/28/2019 Week 2-A Sangmi Lee Pallickaracs535/slides/week2-A.pdfApache Storm and Twitter Heron 5. ScalableDistributedFileSystems:GoogleFileSystemsIandII (W6) 1/28/2019

Storm @Twitter - Berkeley Database Researchdb.cs.berkeley.edu/cs286/papers/storm-sigmod2014.pdf · · 2014-08-28Storm @Twitter Ankit Toshniwal, Siddarth Taneja, Amit Shukla, Karthik

Storm Reports via Twitter Project By Tim Brice WFO El Paso April 29, 2010