Upload
cachet
View
65
Download
5
Tags:
Embed Size (px)
DESCRIPTION
Analyzing Yellowstone’s Network with a Raspberry Pi Cluster . Lauren Patterson. Objective of the Project. Using a low cost Raspberry Pi cluster to find the interconnect path between two nodes on Yellowstone in order to analyze the performance of jobs. Assembling the Raspberry Pi cluster. - PowerPoint PPT Presentation
Citation preview
Analyzing Yellowstone’s Network with a Raspberry Pi Cluster
Lauren Patterson
Using a low cost Raspberry Pi cluster to find the interconnect path between two nodes on Yellowstone in order to analyze the performance of jobs.
Objective of the Project
Assembling the Raspberry Pi cluster
Yellowstone Interconnect
Credit: Siddhartha Ghosh
Files Used
• job1_nodes.txt– Gives the job ID and nodes used
• ibnetdiscover.log (Discover File)– Lists connections between switches
• LFTS.txt– Routing table for each switch
What is Hadoop?• HDFS • MapReduce
HDFSName Node
Job TrackerTask TrackerMap/Reduce
Data Node
Task TrackerMap/Reduce
Data NodeTask TrackerMap/Reduce
Data NodeTask TrackerMap/Reduce
InputData
Map Phase
Shuffle phase
Reduce phase
OutputData
MapReduce
Pig• Apache Pig • Pig Latin • Grunt
Pig Latin Script• Created Pig Latin Script to find the path
between two nodes in Yellowstone
http://www.edureka.in/blog/pig-programming-create-your-first-apache-pig-script/
JOIN Operations in PIG Default, Inner Join returns intersection of A and B
Set BSet A A BU
Full, Right and Left Outer Joins return A and B with different parts nulled out (white)
Full
Right
Left
Join
Path Finder Code Flow
Six Hops (seconds) Four Hops (seconds) Two Hops (seconds)
Raspberry Pi Cluster
3216 2514 1584
Virtual Box 450 351 214
250
750
1250
1750
2250
2750
3250
Pig (Single Path)
Seco
nds
Results
±3
±82 ±19 ±15
±3±4
Python• Single Path Python• Parallel Python– Mpi4py 1.3.1
Six Hop (seconds) Four Hop (seconds) Two Hop (seconds)
Raspberry Pi 6.39 4.136 2.226
Virtual Box 0.273 0.174 0.099
0.5
1.5
2.5
3.5
4.5
5.5
6.5
Serial Python (Single Path)Se
cond
s
±0.02
±0.07
±0.006
±0.11
±0.004
±0.11
16*15 Paths (sec-onds)
12*11 Paths (sec-onds)
8*7 Paths (seconds)
One Proces-sor
933 503 197
Two Proces-sors
493 275 108
Four Proces-sors
258 134 57
50250450650850
Parallel Python on Raspberry PiSe
cond
s
±18
±4
±20
±2±7 ±4
±1
±2 ±0.5
What Do All Of These Have In Common?
• Raspberry Pi• Hadoop • Pig • Python
Acknowledgments
Richard LoftKarina Hauser Stephanie BarrBruce Chittenden Amogh SimhaRaghu Raj Prasanna Kumar
Questions?