Upload
huguk
View
590
Download
4
Embed Size (px)
Citation preview
Recent HistoryApril ‘14 December ‘14
v0.5 v0.6 v0.7
April ‘15
Project Incubation
Top Level Project
v0.8 v0.9
Currently moving towards 0.10 and 1.0 release.
What is Flink?
StreamingTopologies
Stream TimeWindow Count
Low Latency
Long Batch PipelinesResource Utilization
1.2
1.4
1.5
1.2
0.8
0.9
1.0
0.8
Rating Matrix User Matrix Item Matrix
1.5
1.7
1.2
0.6
1.0
1.1
0.8
0.4
W X Y ZW X Y Z
A
B
C
D
4.0
4.5
5.0
3.5
2.0
3.5
4.0
2.0
1.0
= XUse
r
Machine LearningIterative Algorithms
Graph Analysis
53
1 2
4
0.5
0.2 0.9
0.3
0.1
0.40.7
Mutable State
Overview
Deployment Local (Single JVM) · Cluster (Standalone, YARN)
DataStream API Unbounded Data
DataSet API Bounded Data
Runtime Distributed Streaming Data Flow
Libraries Machine Learning · Graph Processing · SQL-like API
Stream ProcessingReal world data is unbounded and is pushed to
systems.
BatchStreaming
Stream Platform Architecture
Server Logs
Trxn Logs
Sensor Logs
Downstream Systems
Flink
– Analyze and correlate streams – Create derived streams
Kafka
– Gather and backup streams – Offer streams
Cornerstones of Flink
Low Latency for fast results.
High Throughput to handle many events per second.
Exactly-once guarantees for correct results.
Intuitive APIs for productivity.
DataStream APIStreamExecutionEnvironment env = StreamExecutionEnvironment .getExecutionEnvironment()
DataStream<String> data = env.fromElements( "O Romeo, Romeo! wherefore art thou Romeo?”, ...);
// DataStream Windowed WordCount DataStream<Tuple2<String, Integer>> counts = data .flatMap(new SplitByWhitespace()) // (word, 1) .keyBy(0) // [word, [1, 1, …]] for 10 seconds .timeWindow(Time.of(10, TimeUnit.SECONDS)) .sum(1); // sum per word per 10 second window
counts.print();
env.execute();
DataStream APIStreamExecutionEnvironment env = StreamExecutionEnvironment .getExecutionEnvironment()
DataStream<String> data = env.fromElements( "O Romeo, Romeo! wherefore art thou Romeo?”, ...);
// DataStream Windowed WordCount DataStream<Tuple2<String, Integer>> counts = data .flatMap(new SplitByWhitespace()) // (word, 1) .keyBy(0) // [word, [1, 1, …]] for 10 seconds .timeWindow(Time.of(10, TimeUnit.SECONDS)) .sum(1); // sum per word per 10 second window
counts.print();
env.execute();
DataStream APIStreamExecutionEnvironment env = StreamExecutionEnvironment .getExecutionEnvironment()
DataStream<String> data = env.fromElements( "O Romeo, Romeo! wherefore art thou Romeo?”, ...);
// DataStream Windowed WordCount DataStream<Tuple2<String, Integer>> counts = data .flatMap(new SplitByWhitespace()) // (word, 1) .keyBy(0) // [word, [1, 1, …]] for 10 seconds .timeWindow(Time.of(10, TimeUnit.SECONDS)) .sum(1); // sum per word per 10 second window
counts.print();
env.execute();
DataStream APIStreamExecutionEnvironment env = StreamExecutionEnvironment .getExecutionEnvironment()
DataStream<String> data = env.fromElements( "O Romeo, Romeo! wherefore art thou Romeo?”, ...);
// DataStream Windowed WordCount DataStream<Tuple2<String, Integer>> counts = data .flatMap(new SplitByWhitespace()) // (word, 1) .keyBy(0) // [word, [1, 1, …]] for 10 seconds .timeWindow(Time.of(10, TimeUnit.SECONDS)) .sum(1); // sum per word per 10 second window
counts.print();
env.execute();
DataStream APIStreamExecutionEnvironment env = StreamExecutionEnvironment .getExecutionEnvironment()
DataStream<String> data = env.fromElements( "O Romeo, Romeo! wherefore art thou Romeo?”, ...);
// DataStream Windowed WordCount DataStream<Tuple2<String, Integer>> counts = data .flatMap(new SplitByWhitespace()) // (word, 1) .keyBy(0) // [word, [1, 1, …]] for 10 seconds .timeWindow(Time.of(10, TimeUnit.SECONDS)) .sum(1); // sum per word per 10 second window
counts.print();
env.execute();
DataStream APIStreamExecutionEnvironment env = StreamExecutionEnvironment .getExecutionEnvironment()
DataStream<String> data = env.fromElements( "O Romeo, Romeo! wherefore art thou Romeo?”, ...);
// DataStream Windowed WordCount DataStream<Tuple2<String, Integer>> counts = data .flatMap(new SplitByWhitespace()) // (word, 1) .keyBy(0) // [word, [1, 1, …]] for 10 seconds .timeWindow(Time.of(10, TimeUnit.SECONDS)) .sum(1); // sum per word per 10 second window
counts.print();
env.execute();
DataStream APIStreamExecutionEnvironment env = StreamExecutionEnvironment .getExecutionEnvironment()
DataStream<String> data = env.fromElements( "O Romeo, Romeo! wherefore art thou Romeo?”, ...);
// DataStream Windowed WordCount DataStream<Tuple2<String, Integer>> counts = data .flatMap(new SplitByWhitespace()) // (word, 1) .keyBy(0) // [word, [1, 1, …]] for 10 seconds .timeWindow(Time.of(10, TimeUnit.SECONDS)) .sum(1); // sum per word per 10 second window
counts.print();
env.execute();
DataStream APIStreamExecutionEnvironment env = StreamExecutionEnvironment .getExecutionEnvironment()
DataStream<String> data = env.fromElements( "O Romeo, Romeo! wherefore art thou Romeo?”, ...);
// DataStream Windowed WordCount DataStream<Tuple2<String, Integer>> counts = data .flatMap(new SplitByWhitespace()) // (word, 1) .keyBy(0) // [word, [1, 1, …]] for 10 seconds .timeWindow(Time.of(10, TimeUnit.SECONDS)) .sum(1); // sum per word per 10 second window
counts.print();
env.execute();
Pipelining
s1 t1 w1
s2 t2 w2
Source Tokenizer Window Count
Complete Pipeline Online Concurrently.
Pipelining
s1 t1 w1
s2 t2 w2
Source Tokenizer Window Count
Complete Pipeline Online Concurrently.
Chained tasks
Pipelining
s1
s2 t2 w2
t1 w1
Source Tokenizer Window Count
Complete Pipeline Online Concurrently.
Chained tasks Pipelined Shuffle
Streaming Fault Tolerance
At Least Once • Ensure that all operators see all events.
Exactly Once• Ensure that all operators see all events. • Do not perform duplicates updates to operator state.
Streaming Fault Tolerance
At Least Once • Ensure that all operators see all events.
Exactly Once• Ensure that all operators see all events. • Do not perform duplicates updates to operator state.
Flink guarantees exactly once processing.
Distributed SnaphotsBarriers flow through the topology in line with data.
Flink guarantees exactly once processing.
Part of snapshot
Distributed SnaphotsBarriers flow through the topology in line with data.
Flink guarantees exactly once processing.
Part of snapshot
Distributed SnaphotsBarriers flow through the topology in line with data.
Flink guarantees exactly once processing.
Part of snapshot
Distributed SnaphotsBarriers flow through the topology in line with data.
Flink guarantees exactly once processing.
Part of snapshot
Distributed SnaphotsBarriers flow through the topology in line with data.
Flink guarantees exactly once processing.
Part of snapshot
Distributed SnaphotsBarriers flow through the topology in line with data.
Flink guarantees exactly once processing.
Part of snapshot
Distributed SnaphotsBarriers flow through the topology in line with data.
Flink guarantees exactly once processing.
Part of snapshot
Distributed SnaphotsBarriers flow through the topology in line with data.
Flink guarantees exactly once processing.
Part of snapshot
Distributed SnaphotsBarriers flow through the topology in line with data.
Flink guarantees exactly once processing.
Part of snapshot
Distributed SnaphotsBarriers flow through the topology in line with data.
Flink guarantees exactly once processing.
Part of snapshot
Distributed Snaphots
Flink guarantees exactly once processing.
JobManager
Master
State Backend
Ceckpoint DataSource 1: State 1:
Source 2: State 2:
Source 3: Sink 1:
Source 4: Sink 2:
Offset: 6791
Offset: 7252
Offset: 5589
Offset: 6843
Distributed Snaphots
Flink guarantees exactly once processing.
JobManager
Master
State Backend
Ceckpoint DataSource 1: State 1:
Source 2: State 2:
Source 3: Sink 1:
Source 4: Sink 2:
Offset: 6791
Offset: 7252
Offset: 5589
Offset: 6843
Start CheckpointMessage
Distributed Snaphots
Flink guarantees exactly once processing.
JobManager
Master
State Backend
Ceckpoint DataSource 1: 6791 State 1:
Source 2: 7252 State 2:
Source 3: 5589 Sink 1:
Source 4: 6843 Sink 2:
Emit Barriers
Acknowledge withPosition
Distributed Snaphots
Flink guarantees exactly once processing.
JobManager
Master
State Backend
Ceckpoint DataSource 1: 6791 State 1:
Source 2: 7252 State 2:
Source 3: 5589 Sink 1:
Source 4: 6843 Sink 2:
Received barrier at each input
Distributed Snaphots
Flink guarantees exactly once processing.
JobManager
Master
State Backend
Ceckpoint DataSource 1: 6791 State 1:
Source 2: 7252 State 2:
Source 3: 5589 Sink 1:
Source 4: 6843 Sink 2:
s1 Write Snapshotof its state
Received barrier at each input
Distributed Snaphots
Flink guarantees exactly once processing.
JobManager
Master
State Backend
Ceckpoint DataSource 1: 6791 State 1: PTR1
Source 2: 7252 State 2: PTR2
Source 3: 5589 Sink 1:
Source 4: 6843 Sink 2:
s1
Acknowledge withpointer to state
s2
Distributed Snaphots
Flink guarantees exactly once processing.
JobManager
Master
State Backend
Ceckpoint DataSource 1: 6791 State 1: PTR1
Source 2: 7252 State 2: PTR2
Source 3: 5589 Sink 1: ACK
Source 4: 6843 Sink 2: ACK
s1 s2
Acknowledge CheckpointReceived barrier
at each input
Distributed Snaphots
Flink guarantees exactly once processing.
JobManager
Master
State Backend
Ceckpoint DataSource 1: 6791 State 1: PTR1
Source 2: 7252 State 2: PTR2
Source 3: 5589 Sink 1: ACK
Source 4: 6843 Sink 2: ACK
s1 s2
Operator State
User-defined state • Flink’s transformations are long running operators • Feel free to keep objects around • Hooks to include into system’s checkpoint
Windowed streams• Time, count, and data-driven windows • Managed by the system
Batch on Streaming
DataStream API Unbounded Data
DataSet API Bounded Data
Runtime Distributed Streaming Data Flow
Libraries Machine Learning · Graph Processing · SQL-like API
Batch on StreamingRun a bounded stream (data set) on
a stream processor.
Bounded data set
Unbounded data stream
Batch on Streaming
Stream Windows
PipelinedData Exchange
Global View
Pipelined or BlockingData Exchange
Infinite Streams Finite Streams
Run a bounded stream (data set) ona stream processor.
Batch Pipelines
Batch Pipelines
Data exchange is mostly streamed
Batch Pipelines
Data exchange is mostly streamed
Some operators block (e.g. sort, hash table)
DataSet APIExecutionEnvironment env = ExecutionEnvironment .getExecutionEnvironment()
DataSet<String> data = env.fromElements( "O Romeo, Romeo! wherefore art thou Romeo?”, ...);
// DataSet WordCount DataSet<Tuple2<String, Integer>> counts = data .flatMap(new SplitByWhitespace()) // (word, 1) .groupBy(0) // [word, [1, 1, …]] .sum(1); // sum per word for all occurrences
counts.print();
DataStream APIExecutionEnvironment env = ExecutionEnvironment .getExecutionEnvironment()
DataSet<String> data = env.fromElements( "O Romeo, Romeo! wherefore art thou Romeo?”, ...);
// DataSet WordCount DataSet<Tuple2<String, Integer>> counts = data .flatMap(new SplitByWhitespace()) // (word, 1) .groupBy(0) // [word, [1, 1, …]] .sum(1); // sum per word for all occurrences
counts.print();
DataStream APIExecutionEnvironment env = ExecutionEnvironment .getExecutionEnvironment()
DataSet<String> data = env.fromElements( "O Romeo, Romeo! wherefore art thou Romeo?”, ...);
// DataSet WordCount DataSet<Tuple2<String, Integer>> counts = data .flatMap(new SplitByWhitespace()) // (word, 1) .groupBy(0) // [word, [1, 1, …]] .sum(1); // sum per word for all occurrences
counts.print();
DataStream APIExecutionEnvironment env = ExecutionEnvironment .getExecutionEnvironment()
DataSet<String> data = env.fromElements( "O Romeo, Romeo! wherefore art thou Romeo?”, ...);
// DataSet WordCount DataSet<Tuple2<String, Integer>> counts = data .flatMap(new SplitByWhitespace()) // (word, 1) .groupBy(0) // [word, [1, 1, …]] .sum(1); // sum per word for all occurrences
counts.print();
DataStream APIExecutionEnvironment env = ExecutionEnvironment .getExecutionEnvironment()
DataSet<String> data = env.fromElements( "O Romeo, Romeo! wherefore art thou Romeo?”, ...);
// DataSet WordCount DataSet<Tuple2<String, Integer>> counts = data .flatMap(new SplitByWhitespace()) // (word, 1) .groupBy(0) // [word, [1, 1, …]] .sum(1); // sum per word for all occurrences
counts.print();
DataStream APIExecutionEnvironment env = ExecutionEnvironment .getExecutionEnvironment()
DataSet<String> data = env.fromElements( "O Romeo, Romeo! wherefore art thou Romeo?”, ...);
// DataSet WordCount DataSet<Tuple2<String, Integer>> counts = data .flatMap(new SplitByWhitespace()) // (word, 1) .groupBy(0) // [word, [1, 1, …]] .sum(1); // sum per word for all occurrences
counts.print();
DataStream APIExecutionEnvironment env = ExecutionEnvironment .getExecutionEnvironment()
DataSet<String> data = env.fromElements( "O Romeo, Romeo! wherefore art thou Romeo?”, ...);
// DataSet WordCount DataSet<Tuple2<String, Integer>> counts = data .flatMap(new SplitByWhitespace()) // (word, 1) .groupBy(0) // [word, [1, 1, …]] .sum(1); // sum per word for all occurrences
counts.print();
Batch-specific optimizations
Managed memory • On- and off-heap memory • Internal operators (e.g. join or sort) with out-of-core
support • Serialization stack for user-types
Cost-based optimizer• Program adapts to changing data size
Getting Started
Project Page: http://flink.apache.org
Quickstarts: Java & Scala API
Getting Started
Project Page: http://flink.apache.org
Docs: Programming Guides
Getting Started
Project Page: http://flink.apache.org
Get Involved: Mailing Lists, Stack Overflow, IRC, …
Blogs http://flink.apache.org/blog http://data-artisans.com/blog
Twitter @ApacheFlink
Mailing lists (news|user|dev)@flink.apache.org
Apache Flink