ICS 556ICS 556Parallel AlgorithmsParallel Algorithms

Ebrahim MalallaEbrahim Malalla

malalla@kfupm.edu.samalalla@kfupm.edu.sa

Office:Office: Bldg 22, Room 124-8 Bldg 22, Room 124-8

Sequential ComputersSequential Computers

"…sequential computers are approaching a "…sequential computers are approaching a fundamental physical limit on their potential fundamental physical limit on their potential power. Such a limit is the speed of light…"power. Such a limit is the speed of light…"

``o'' are uniprocessors, ``+''denotes modestly parallel computers with 4--16 processors,  ``x'' denotes massively parallel computers with 100s or 1000s of processors

ParallelismParallelism

Is and can be done easily in real life Is and can be done easily in real life (nature)(nature)

E.g., ants and pees colony.E.g., ants and pees colony.

So why not use it to solve human’s So why not use it to solve human’s problems!problems!

Parallel ComputersParallel Computers

Supercomputers (high performance Supercomputers (high performance computers) are needed tocomputers) are needed to

Speedup sequential algorithmsSpeedup sequential algorithmsSolve large complex problems that are Solve large complex problems that are

almost not practically solvable.almost not practically solvable.

Applications of ParallelismApplications of Parallelism Weather forecastingWeather forecasting Simulations (DNA, biomedical, flight control)Simulations (DNA, biomedical, flight control) Scientific ComputationsScientific Computations

Fourier transform, waveletsFourier transform, wavelets Applied linear algebra, matrix factorization and Applied linear algebra, matrix factorization and

multiplicationmultiplication Image and signal processingImage and signal processing Numerical analysisNumerical analysis PDEsPDEs Huge DatabasesHuge Databases Real-time systems with large live stream inputsReal-time systems with large live stream inputs

Unfortunately .. Unfortunately ..

Not every algorithm can be parallelized Not every algorithm can be parallelized

““One woman can have a child in 9 monthsOne woman can have a child in 9 months

But 9 women can’t have a child in 1 month!”But 9 women can’t have a child in 1 month!”

Courses in Parallelism Courses in Parallelism

Parallel ArchitectureParallel Architecture Parallel ProgrammingParallel Programming Parallel AlgorithmsParallel Algorithms Parallel Algorithms for scientific computationsParallel Algorithms for scientific computations Concurrent ProgrammingConcurrent Programming Distributed systemsDistributed systems Distributed AlgorithmsDistributed Algorithms Cluster ComputingCluster Computing

Parallel ArchitectureParallel Architecture

Focuses on the design of hardwaresFocuses on the design of hardwaresE.g., processors, memory, E.g., processors, memory,

communications and communications and interconnections, ...etcinterconnections, ...etc

Parallel ProgrammingParallel Programming Code parallel programs using parallel programming Code parallel programs using parallel programming

languages, e.g., PVM (Parallel Virtual Machine), MPI languages, e.g., PVM (Parallel Virtual Machine), MPI (Massage Passing Interface), OpenMP(Massage Passing Interface), OpenMP

Study: Parallel machine and programming models, Study: Parallel machine and programming models, Programming techniques (partitioning, pipelined Programming techniques (partitioning, pipelined computations, synchronous computations, load balancing computations, synchronous computations, load balancing and termination detection) and termination detection)

Textbooks: Textbooks: Parallel Computing, by Grama, Gupta, Karypis, Kumar, Addison-Parallel Computing, by Grama, Gupta, Karypis, Kumar, Addison-

Wesley, 2003. Wesley, 2003. Parallel Programming, by Wilkinson and Allen, Pearson-Prentice Parallel Programming, by Wilkinson and Allen, Pearson-Prentice

HallHall Fundamentals of Parallel Processing by Jordan and Alaghband, Fundamentals of Parallel Processing by Jordan and Alaghband,

Prentice Hall Prentice Hall

Parallel AlgorithmsParallel Algorithms

Focuses on the design and analysis of Focuses on the design and analysis of parallel algorithmsparallel algorithms

Given a problemGiven a problempartition it into sub-problems, partition it into sub-problems, Distribute these sub-problems to the Distribute these sub-problems to the

processorsprocessorsSolve the communication problemSolve the communication problemCombine the solutionCombine the solution

Parallel Computation ModelsParallel Computation Models

Parallel algorithms are designed to run on Parallel algorithms are designed to run on a specific parallel computera specific parallel computer

But there many different practical models But there many different practical models of them (unlike the sequential computer)of them (unlike the sequential computer)

Shared Memory ModelShared Memory Model

Interconnection Network ModelInterconnection Network Model

Network structure could be array, tree, hypercube, or mesh based design

Combinatorial Circuits ModelCombinatorial Circuits Model

Other optionsOther options

Synchronous or asynchronous Synchronous or asynchronous computationcomputation

SIMD MIDMSIMD MIDM ........

Massively Parallel Massively Parallel ComputerComputer

Top 10 Supercomputers (2007)Top 10 Supercomputers (2007)

ManufacturManufacturer er

Computer Computer TFLOPS TFLOPS CountrCountry y

1 1 IBM IBM BlueGene/L eServer Blue BlueGene/L eServer Blue Gene Gene

280.6 280.6 USA USA

2 2 Sandia/Cray Sandia/Cray Red Storm Cray XT3 Red Storm Cray XT3 101.4 101.4 USA USA

3 3 IBM IBM BGW eServer Blue Gene BGW eServer Blue Gene 91.29 91.29 USA USA

4 4 IBM IBM ASC Purple eServer pSeries ASC Purple eServer pSeries p575 p575

75.76 75.76 USA USA

5 5 IBM IBM MareNostrum JS21 Cluster, MareNostrum JS21 Cluster, Myrinet Myrinet

62.63 62.63 Spain Spain

6 6 Dell Dell Thunderbird PowerEdge Thunderbird PowerEdge 1850, IB 1850, IB

53.00 53.00 USA USA

Google Map of TOP 100 HPCs

Intel Promised Teraflops chipIntel Promised Teraflops chip

Intel has introducedIntel has introduced

a prototype for a prototype for

A supercomputerA supercomputer

Chip capable of Chip capable of

teraflops Performance teraflops Performance

While While consuming consuming

Very little power. Very little power. It has 80 cores and consumes only 62 watts.It has 80 cores and consumes only 62 watts.

Approaches for studyingApproaches for studying Parallel Algorithms Parallel Algorithms

1.1. Model Approach:Model Approach: divide the subject divide the subject according to the parallel computation according to the parallel computation models and for each study different models and for each study different parallel algorithms design techniques parallel algorithms design techniques and consider different application and consider different application problemsproblems

Approaches for studyingApproaches for studying Parallel Algorithms Parallel Algorithms

2.2. Problem Approach:Problem Approach: Divide the subject Divide the subject according to the type of problems that can be according to the type of problems that can be solved by parallel algorithms likesolved by parallel algorithms like

Sorting and searchingSorting and searching Scientific computationScientific computation Graph problemsGraph problems Computational geometryComputational geometry Simulations, graphics and visionSimulations, graphics and vision Biomedical modelingBiomedical modeling

For each type we study design techniques for For each type we study design techniques for models of computations that are best suited.models of computations that are best suited.

Approaches for studyingApproaches for studying Parallel Algorithms Parallel Algorithms

3.3. Models and Methods Approach: Models and Methods Approach: mix of mix of the above two approaches.the above two approaches.

Models:Models: Combinatorial circuits, Combinatorial circuits, interconnected networks, shared interconnected networks, shared memorymemory

Design Methods:Design Methods: prefix computation, list prefix computation, list ranking, divide and conquer, split and ranking, divide and conquer, split and plan, .... etcplan, .... etc

TextbookTextbook

Parallel ComputationParallel Computation

By Selim AklBy Selim Akl

Supplementary Supplementary

Lecture notes, Lecture notes,

current researchcurrent research

paperspapers

AssessmentAssessment

4-5 assignments4-5 assignments2 exams2 examsPresentation and report based on Presentation and report based on

research papersresearch papers