2
Some properties of “Big Data”
•Big data is inherently immutable, meaning it is not supposed to updated once generated.
•Mostly the operations are coarse grained when it comes to write
•Commodity hardware makes more sense for storage/computation of such enormous data,hence the data is distributed across clusterof many such machines
• The distributed nature makes the programming complicated.
3
Brush up for Hadoop concepts
Distributed Storage => HDFS
Cluster Manager => YARN
Fault tolerance => achieved via replication
Job scheduling => Scheduler in YARN
Mapper
Reducer
Combiner
4http://hadoop.apache.org/docs/r1.2.1/images/hdfsarchitecture.gif
5
Map Reduce Programming Model
6https://twitter.com/francesc/status/507942534388011008
7http://www.admin-magazine.com/HPC/Articles/MapReduce-and-Hadoop
8
http://www.slideshare.net/JimArgeropoulos/hadoop-101-32661121
9
MapReduce pain points
• considerable latency
• only Map and Reduce phases
• Non trivial to test
• results into complex workflow
• Not suitable for Iterative processing
10
Immutability and MapReduce model
• HDFS storage is immutable or append-only.
• The MapReduce model lacks to exploit the immutable nature of
the data.
• The intermediate results are persisted resulting in huge of IO,
causing a serious performance hit.
11
Wouldn’t it be very nice if we could have• Low latency
• Programmer friendly programming model
• Unified ecosystem
• Fault tolerance and other typical distributed system properties
• Easily testable code
• Of course open source :)
12
What is Apache Spark
• Cluster computing Engine
• Abstracts the storage and cluster management
• Unified interfaces to data
• API in Scala, Python, Java, R*
13
Where does it fit in existing Bigdata ecosystem
http://www.kdnuggets.com/2014/06/yarn-all-rage-hadoop-summit.html
14
Why should you care about Apache Spark
• Abstracts underlying storage,
• Abstracts cluster management
• Easy programming model
• Very easy to test the code
• Highly performant
15
• Petabyte sort record
https://databricks.com/blog/2014/10/10/spark-petabyte-sort.html
16
• Offers in memory caching of data
• Specialized Applications
• GraphX for graph processing
• Spark Streaming
• MLib for Machine learning
• Spark SQL
• Data exploration via Spark-Shell
17
Programming model
for
Apache Spark
18
Word Count example
val file = spark.textFile("input path")
val counts = file.flatMap(line => line.split(" "))
.map(word => (word, 1))
.reduceByKey((a, b) => a + b)
counts.saveAsTextFile("destination path")
19
Comparing example with MapReduce
20
Spark Shell Demo
• SparkContext
• RDD
• RDD operations
21
RDD
• RDD stands for Resilient Distributed Dataset.
• basic abstraction for Spark
22
• Equivalent of Distributed collections.
• The interface makes distributed nature of underlying data transparent.
• RDD is immutable
• Can be created via,
• parallelising a collection,
• transforming an existing RDD by applying a transformation function,
• reading from a persistent data store like HDFS.
23
RDD is lazily evaluated
RDD has two type of operations
• Transformations
Create a DAG of transformations to be applied on the RDD
Does not evaluating anything
• Actions
Evaluate the DAG of transformations
24
RDD operations
Transformations
map(f : T ⇒ U) : RDD[T] ⇒ RDD[U]
filter(f : T ⇒ Bool) : RDD[T] ⇒ RDD[T]
flatMap(f : T ⇒ Seq[U]) : RDD[T] ⇒ RDD[U]
sample(fraction : Float) : RDD[T] ⇒ RDD[T] (Deterministic sampling)
union() : (RDD[T],RDD[T]) ⇒ RDD[T]
join() : (RDD[(K, V)],RDD[(K, W)]) ⇒ RDD[(K, (V, W))]
groupByKey() : RDD[(K, V)] ⇒ RDD[(K, Seq[V])]
reduceByKey(f : (V,V) ⇒ V) : RDD[(K, V)] ⇒ RDD[(K, V)]
partitionBy(p : Partitioner[K]) : RDD[(K, V)] ⇒ RDD[(K, V)]
25
Actions
count() : RDD[T] ⇒ Long
collect() : RDD[T] ⇒ Seq[T]
reduce(f : (T,T) ⇒ T) : RDD[T] ⇒ T
lookup(k : K) : RDD[(K, V)] ⇒ Seq[V] (On hash/range partitioned RDDs)
save(path : String) : Outputs RDD to a storage system, e.g., HDFS
26
Job Execution
27
Spark Execution in Context of YARN
http://kb.cnblogs.com/page/198414/
28
Fault tolerance via lineage
MappedRDD
FilteredRDD
FlatMappedRDD
MappedRDD
HadoopRDD
29
Testing
30
Why is Spark more performant than MapReduce
31
Reduced IO
• No disk IO between phases since phases themselves are pipelined
• No network IO involved unless a shuffle is required
32
No Mandatory Shuffle
• Programs not bounded by map and reduce phases
• No mandatory Shuffle and sort required
33
In memory caching of data
• Optional In memory caching
• DAG engine can apply certain optimisations since when an action is called, it knows what all transformations as to be applied
34
Questions?
35
Thank You!