Upload
viirya
View
5.017
Download
3
Embed Size (px)
DESCRIPTION
The slides "Real-time Big Data Processing with Storm: Using Twitter Streaming as Example" for the presentation in Hadoop in Taiwan 2013.
Citation preview
Real-time Big Data Processing with Storm: Using Twitter
Streaming as Example
Liang-Chi HsiehHadoop in Taiwan 2013
1
In Today’s Talk
• Introduce stream computation in Big Data
• Introduce current stream computation platforms
• Storm
• Architecture & concepts
• Use case: analysis of Twitter streaming data
2
Recap, the Four V’s of Big Data
• To help us talk ‘big data’, it is common to break it down into four dimensions
• Volume: Scale of Data
• Velocity: Analysis of Streaming Data
• Variety: Different Forms of Data
• Veracity: Uncertainty of Data
http://dashburst.com/infographic/big-data-volume-variety-velocity/
3
• Velocity: Data in motion
• Require realtime response to process, analyze continuous data stream
http://www.intergen.co.nz/Global/Images/BlogImages/2013/Defining-big-data.png
4
Streaming Data
• Data coming from:
• Logs
• Sensors
• Stock trade
• Personal devices
• Network connections
• etc...
5
Batch Data Processing Architecture
6
Data Store Hadoop
Data Flow Batch Run
Batch View
Query• Views generated in batch maybe out of date
• Batch workflow is too slow
Data Processing Architecture: Batch and Realtime
7
Data Store Hadoop
Batch Run
Realtime Processing
Batch View
Realtime View
Query
Data Flow
• Generate realtime views of data by using stream computation
Current Stream Computation Platforms
• S4
• Storm
• Spark Streaming
• MillWheel
8
S4
• General-purpose, distributed, scalable, fault-tolerant, pluggable platform for processing data stream
• Initially released by Yahoo!
• Apache Incubator project since September 2011
• Written in Java
9
AdapterPEs &
Streams
Storm
• Distributed and fault-tolerant realtime computation
• Provide a set of general primitives for doing realtime computation
10
http://storm-project.net/
Spark Streaming• (Near) real-time processing of stream data
• New programming model
• Discretized streams (D-Streams)
• Built on Resilient Distributed Datasets (RDDs)
• Based on Spark
• Integrated with Spark batch and interactive computation modes
11
Spark Streaming• D-Streams
• Treat a streaming computation as a series of deterministic batch computations on a small time intervals
• Latencies can be as low as a second, supported by the fast execution engine Spark
val ssc = new StreamingContext(sparkUrl, "Tutorial", Seconds(1), sparkHome, Seq(jarFile))
val tweets = ssc.twitterStream(twitterUsername, twitterPassword)
val statuses = tweets.map(status => status.getText())
statuses.print()
batch@t batch@t+1 batch@t+2Twitter Streaming Data
D-Streams: RDDs
12
MillWheel• Google’s computation framework for low-latency
stream data-processing applications
• Application logic is written as individual nodes in a directed computation graph
• Fault tolerance
• Exactly-once delivery guarantees
• Low watermarks is used to prevent logical inconsistencies caused by data delivery not in order
13
Storm: Distributed and Fault-Tolerant Realtime Computation
• Guaranteed data processing
• Every tuple will be fully processed
• Exactly-once? Using Trident
• Horizontal scalability
• Fault-tolerance
• Easy to deploy and operate
• One click deploy on EC2
14
Storm Architecture
• A Storm cluster is similar to a Hadoop cluster
• Togologies vs. MapReduce jobs
• Running a topology:
• Killing a topology
15
storm jar all‐my‐code.jar backtype.storm.MyTopology arg1 arg2
storm kill {topology name}
Storm Architecture
• Two kinds of nodes
• Master node runs a daemon called Nimbus
• Each worker node runs a daemon called Supervisor
• Each worker process executes a subset of a topology
16https://github.com/nathanmarz/storm/wiki/images/storm-cluster.png
Topologies• A topology is a graph of computation
• Each node contains processing logic
• Links between nodes represent the data flows between those processing units
• Topology definitions are Thrift structs and Nimbus is a Thrift service
• You can create and submit topologies using any programming language
17
Topologies: Concepts• Stream: unbounded
sequence of tuples
• Primitives
• Spouts
• Bolts
• Interfaces can be implemented to run your logic
18
https://github.com/nathanmarz/storm/wiki/images/topology.png
Data Model
• Tuples are used by Storm as data model
• A named list of values
• A field in a tuple can be an object of any type
• Storm supports all the primitive types, strings, and byte arrays
• Implement corresponding serializer for using custom type
19
Tuples
Stream Grouping• Define how streams are distributed to downstream
tasks
• Shuffle grouping: randomly distributed
• Fields grouping: partitioned by specified fields
• All grouping: replicated to all tasks
• Global grouping: the task with lowest id
20
https://github.com/nathanmarz/storm/wiki/images/topology-tasks.png
Simple TopologyTopologyBuilder builder = new TopologyBuilder(); builder.setSpout("words", new TestWordSpout(), 10); builder.setBolt("exclaim1", new ExclamationBolt(), 3) .shuffleGrouping("words");builder.setBolt("exclaim2", new ExclamationBolt(), 2) .shuffleGrouping("exclaim1");
“words:” TestWordSpout
“exclaim1”: ExclamationBolt
“exclaim2”: ExclamationBolt
shuffleGrouping
shuffleGrouping
shuffle grouping: tuples are randomly distributed to the boltʼs tasks21
Submit TopologyConfig conf = new Config();conf.setDebug(true);conf.setNumWorkers(2);
LocalCluster cluster = new LocalCluster();cluster.submitTopology("test", conf, builder.createTopology());Utils.sleep(10000);cluster.killTopology("test");cluster.shutdown();
Local mode:
Distributed mode:Config conf = new Config();conf.setNumWorkers(20);conf.setMaxSpoutPending(5000);StormSubmitter.submitTopology("mytopology", conf, topology);
22
Guaranteeing Message Processing
• Every tuple will be fully processed
• Tuple tree
Fully processed: all messages in the tree must to be processed.
23
Storm Reliability API• A Bolt to split a tuple containing a sentence to the
tuples of wordspublic void execute(Tuple tuple) {
String sentence = tuple.getString(0);
for(String word: sentence.split(" ")) {
_collector.emit(tuple, new Values(word));
}
_collector.ack(tuple);
}
“Anchoring” creates a new link in the
tuple tree.
Calling “ack” (or “fail”) makes the tuple as complete (or failed).
24
Storm on YARN
• Enable Storm clusters to be deployed on Hadoop YARN
25
http://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/yarn_architecture.gif
Use Case: Analysis of Twitter Streaming Data
• Suppose we want to program a simple visualization for Twitter streaming data
• Tweet visualization on map: heatmap
• Since there are too many tweets at same time, we are like to group tweets by their geo-locations
26
Heatmap: Tweet Visualization on Map
• Graphical representation of tweet data
• Clear visualization of the intensity of tweet count by geo-locations
• Static or dynamic
27
Batch Approach: Hadoop
• Generating static tweet heatmap
• Continuous data collecting
• Batch data processing using Hadoop Java programs, Hive or Pig
28
Twitter Storage Batch Processing by Hadoop
Simple Geo-location-based Tweet Grouping
• Goal
• To group geographical near tweets together
• Using Hive
29
Data Store & Data Loading
• Simple data schema
CREATE EXTERNAL TABLE tweets ( id_str STRING, geo STRUCT< type:STRING, coordinates:ARRAY<DOUBLE>>) ROW FORMAT SERDE 'com.cloudera.hive.serde.JSONSerDe'LOCATION '/user/hduser/tweets';
load data local inpath '/mnt/tweets_2013_3.json' overwrite into table tweets;
• Loading data in Hive
30
Hive Query
• Applying Hive query on collected tweets data
insert overwrite local directory '/tmp/tweets_coords.txt' select avg(geo.coordinates[0]), avg(geo.coordinates[1]), count(*) as tweet_count from tweets group by floor(geo.coordinates[0] * 100000), floor(geo.coordinates[1] * 100000) sort by tweet_count desc;
31
Static Tweet Heatmap
• Heatmap visualization of partial tweets collected in Jan, 2013
32
Streaming Approach: Storm
• Generate realtime Twitter usage heatmap view
• Higher level Storm programming by using DSLs
• Scala DSL here
33
class ExclamationBolt extends StormBolt(outputFields = List("word")) {
def execute(t: Tuple) = {
t emit (t.getString(0) + "!!!")
t ack
}
}
Bolt DSL
class MySpout extends StormSpout(outputFields = List("word", "author")) {
def nextTuple = {}
}Spout DSL
Stream Computation Design
Tweets
Defined Time Slot
Calculate some statistics, e.g. average geo-locations,
for each group
Group geographical near tweets
Perform predication tasks such as classification, sentiment analysis
Send/Store results
34
Create Topology
val builder = new TopologyBuilder
builder.setSpout("tweetstream", new TweetStreamSpout, 1) builder.setSpout("clock", new ClockSpout) builder.setBolt("geogrouping", new GeoGrouping, 12) .fieldsGrouping("tweetstream", new Fields("geo_lat", "geo_lng")) .allGrouping("clock")
• Two Spouts
• One for produce tweet stream
• One for generate time interval needed to update tweet statistics
• Only one Bolt; Stream grouping by lat, lng for tweet stream
35
Tweet Spout & Clock Spout
class TweetStreamSpout extends StormSpout(outputFields = List("geo_lat", "geo_lng", "lat", "lng", "txt")) {
def nextTuple = { ...
emit (math.floor(lat * 10000), math.floor(lng * 10000), lat, lng, txt) ...
}}
class ClockSpout extends StormSpout(outputFields = List("timestamp")) { def nextTuple { Thread sleep 1000 * 1 emit (System.currentTimeMillis / 1000) }}
36
GeoGrouping Boltclass GeoGrouping extends StormBolt(List("geo_lat", "geo_lng", "lat", "lng", "txt")) { def execute(t: Tuple) = t matchSeq { case Seq(clockTime: Long) => // Calculate statistics for each group of tweets // Perform classification tasks // Send/Store results case Seq(geo_lat: Double, geo_lng: Double, lat: Double, lng: Double, txt: String) => // Group tweets by geo-locations
}}
37
Demo
38