Upload
others
View
1
Download
0
Embed Size (px)
Citation preview
1
Main Memory Map Reduce (M3R)
PL Day, September 23, 2013 Avi Shinnar, David Cunningham, Ben Herta, Vijay Saraswat, Wei Zhang In collaboraLon with: Yan Li*, Dave Grove, Mikio Takeuchi**, Salikh Zakirov**, Juemin Zhang * IBM China Research Lab ** IBM Tokyo Research Lab Otherwise IBM TJ Watson Research Lab, New York
© 2013 IBM CorporaLon
resilience performance
low latency
M3R (engine) Hadoop
X10 Java
out-‐of-‐core in-‐memory
scalability
© 2013 IBM CorporaLon
• “Java”-‐like language designed for performance and producLvity at scale
• Asynchronous ParLLoned Global Address Space programming model – async S: run S as a separate ac/vity – at (P) S: run S at place P – finish S: wait for termina/on of children ac/vi/es – MPI style barriers, local “atomic” synchronizaLon…
• Advanced type system – reified generics, closures, dependent types…
• MulLple backends: Java, C++, CUDA hap://x10-‐lang.org/
X10
© 2013 IBM CorporaLon
Main-‐Memory Map Reduce in X10
secondary sorLng (values)
sorLng (keys)
combiners
map-‐only jobs
mulL-‐threading
iteraLve jobs
user controlled serializaLon out of core shuffle
debugging
profiling
© 2013 IBM CorporaLon
M3R/Hadoop Architecture Java Hadoop App Hadoop
Map Reduce Engine
HDFS data
M3R Engine
HDFS
X10 M3R jobs
M3R/Hadoop adaptor
X10 Java
mulLple jobs
mulLple jobs
HDFS
JVM only
JVM/Na/ve
Java M3R jobs
© 2013 IBM CorporaLon
M3R / Hadoop
Performance
CompaLbility
© 2013 IBM CorporaLon
Hadoop Job
Input (InputFormat/
RecordReader/
InputSplit)
File System
(HDFS)
Map (Mapper)
Reduce (Reducer)
Output (OutputFormat/
RecordWriter
OutputCommi:er)
Shuffle File
System
File
System
© 2013 IBM CorporaLon
M3R/Hadoop Job: cache
Input (InputFormat/
RecordReader/
InputSplit)
File System
(HDFS)
Map (Mapper)
Reduce (Reducer)
Output (OutputFormat/
RecordWriter
OutputCommi:er)
Shuffle File
System
File
System
Cache
© 2013 IBM CorporaLon
M3R/Hadoop Job: in-‐memory
Input (InputFormat/
RecordReader/
InputSplit)
File System
(HDFS)
Map (Mapper)
Reduce (Reducer)
Output (OutputFormat/
RecordWriter
OutputCommi:er)
Shuffle File
System
File
System
Cache
© 2013 IBM CorporaLon
M3R/Hadoop Job: co-‐locaLon
Input (InputFormat/
RecordReader/
InputSplit)
File System
(HDFS)
Map (Mapper)
Reduce (Reducer)
Output (OutputFormat/
RecordWriter
OutputCommi:er)
Shuffle File
System
File
System
Cache
© 2013 IBM CorporaLon
Iterated Matrix Vector mulLplicaLon
• Algorithm (“standard HPC”) – Row block parLLon G – Replicate V – In parallel, at each place, mulLply each
row of G with V. – In parallel, each place broadcasts its
segment of V to all others • This reassembles V for next phase.
• Performance key – Read the appropriate part of G once, never communicate it.
– Reassembly is local.
G V V
© 2013 IBM CorporaLon
M3R/Hadoop Job: locality
Input (InputFormat/
RecordReader/
InputSplit)
File System
(HDFS)
Map (Mapper)
Reduce (Reducer)
Output (OutputFormat/
RecordWriter
OutputCommi:er)
Cache
Shuffle ParLLoner
© 2013 IBM CorporaLon
ParLLon Stability in M3R
• The reducer associated with a given parLLon number will always be run at the same place – Assuming the number of reducers and the number of places remains the same,
• The number of reducers is determined by the applicaLon.
• The number of places is fixed for the duraLon of the M3R server.
© 2013 IBM CorporaLon
Sparse Matrix Vector MulLplicaLon
0
200
400
600
800
1000
1200
1400
1600
1800
2000
0 200000 400000 600000 800000 1000000 1200000 1400000 1600000
Tim
e (s
)
Size M (G is an MxM matrix with sparsity 0.001)
Sparse Matrix Vector Multiplication
M3R/Hadoop Hadoop Expon. (M3R/Hadoop) Expon. (Hadoop)
50X 0
5
10
15
20
25
30
35
40
45
0 200000 400000 600000 800000 1000000 1200000 1400000 1600000
Tim
e (s
)
Size M (G is an MxM matrix of sparsity 0.001)
Sparse Matrix Vector Multiplication
M3R/Hadoop Expon. (M3R/Hadoop)
© 2013 IBM CorporaLon
M3R / Hadoop
Performance
CompaLbility
© 2013 IBM CorporaLon
JobTracker
© 2013 IBM CorporaLon
© 2013 IBM CorporaLon
© 2013 IBM CorporaLon
© 2013 IBM CorporaLon
© 2013 IBM CorporaLon
M3R / Hadoop
Performance
CompaLbility
© 2013 IBM CorporaLon
DML results (Nov. 2011)
PageRank Hadoop M3R Speedup 100K 880s 452s 1.9x 200K 885s 574s 1.5x 400K 872s 530s 1.7x
Linear Regression
Hadoop M3R Speedup
1000K 1272s 120s 10.6x 3000K 1438s 185s 7.8x 5000K 1473s 275s 5.4x
GNNMF Hadoop M3R Speedup 100K 1489s 115s 13x 200K 1492s 185s 8.1x 400K 1481s 300s 4.9x
© 2013 IBM CorporaLon
Pig unit tests
© 2013 IBM CorporaLon
Current Status / Future Work
• VLDB ‘12 – Avraham Shinnar, David Cunningham, Vijay Saraswat, and Benjamin
Herta. 2012. M3R: increased performance for in-‐memory Hadoop jobs. Proc. VLDB Endow. 5, 12 (August 2012), 1736-‐1747.
• Things generally work quite well • Working on out-‐of-‐core shuffle
– Performance degradaLon instead of crashing
• Working on dynamic class loading
© 2013 IBM CorporaLon