View
218
Download
0
Category
Tags:
Preview:
Citation preview
1
HPCA
2001
Message Passing Interface Message Passing Interface (MPI) and Parallel Algorithm (MPI) and Parallel Algorithm DesignDesign
Message Passing Interface Message Passing Interface (MPI) and Parallel Algorithm (MPI) and Parallel Algorithm DesignDesign
2
HPCA
2001
What is MPI?
A message passing library specification
– message-passing model
– not a compiler specification
– not a specific product
For parallel computers, clusters and
heterogeneous networks.
Full-featured
3
HPCA
2001
Why use MPI? (1)
Message passing now mature as
programming paradigm
– well understood
– efficient match to hardware
– many applications
4
HPCA
2001
Who Designed MPI ?
Venders
– IBM, Intel, Sun, SGI, Meiko, Cray, Convex,
Ncube,…..
Research Lab.
– PVM, p4, Zipcode, TCGMSG, Chameleon,
Express, Linda, PM (Japan RWCP), AM
(Berkeley), FM (HPVM at Illinois)
5
HPCA
2001
Vender-Supported MPI HP-MPI HP; Convex SPP MPI-F IBM SP1/SP2 Hitachi/MPI Hitachi SGI/MPI SGI PowerChallenge series MPI/DE NEC. INTEL/MPI Intel. Paragon (iCC lib) T.MPI Telmat Multinode Fujitsu/MPI Fujitsu AP1000 EPCC/MPI Cray & EPCC, T3D/T3E.
6
HPCA
2001
Research MPI
MPICH Argonne National Lab. &
Mississippi State U. LAM Ohio Supercomp. center MPICH/NT Mississippi State U. MPI-FM Illinois (Myrinet) MPI-AM UC Berkeley (Myrinet) MPI-PM RWCP, Japan (Myrinet) MPI-CCL Calif. Tech.
7
HPCA
2001
Research MPI
CRI/EPCC MPI Cray Research and Edinburgh
(Cray T3D/E)Parallel Computing Centre
MPI-AP Australian National U.-
(AP1000) CAP Research Program
W32MPI Illinois, Concurrent Systems
RACE-MPI Hughes Aircraft Co.
MPI-BIP INRIA, France (Myrinet)
8
HPCA
2001
Language Binding
MPI 1: C, Fortran (for MPICH-based implementation)
MPI 2: C, C++, Fortran Java :
– Through Java native method interface (JNI): mpiJava JavaMPI
– Implement the MPI package by pure Java: MPIJ: (DOGMA project)
– JMPI (by MPI Software Technology)
9
HPCA
2001
Main Features of MPIMain Features of MPIMain Features of MPIMain Features of MPI
10
HPCA
2001
“Communicator”
Identify the process group and context with respect to which the operation is to be performed
In a parallel environment, processes need to know each others (“naming”: machine name, IP address, process ID)
11
HPCA
2001
Process
Process
Process
Process
ProcessProcess
Process
Process
Process
Process
Process
Communicator (2)Four communicatorsProcess in different communicators cannot communicate
Process
Process
Process
Process
ProcessProcess
Communicator within Communicator
Process
Process
Same process can be existed in different communicators
Process
12
HPCA
2001
Point-to-point Communication
The basic point-to-point communication operators are send and receive.
Communication Modes : – normal mode (blocking and non-blocking), – synchronous mode, – ready mode (to allow access to fast protocols), – buffered mode– ….
13
HPCA
2001
Collective Communication
Communication that involves a group of processes. E.g, broadcast, barrier, reduce, scatter, gather, all-to-all, ..
14
HPCA
2001 MPI ProgrammingMPI ProgrammingMPI ProgrammingMPI Programming
15
HPCA
2001
Writing MPI programs
MPI comprises 125 functions Many parallel programs can be written
with just 6 basic functions
16
HPCA
2001
Six basic functions (1)
1. MPI_INIT: Initiate an MPI computation
2. MPI_FINALIZE: Terminate a computation
3. MPI_COMM_SIZE: Determine number of processes in a communicator
4. MPI_COMM_RANK: Determine the identifier of a process in a specific communicator
5. MPI_SEND: Send a message from one process to another process
6. MPI_RECV: Receive a message from one process to another process
17
HPCA
2001
Program main
begin
MPI_INIT()
MPI_COMM_SIZE(MPI_COMM_WORLD, count)
MPI_COMM_RANK(MPI_COMM_WORLD, myid)
print(“I am ”, myid, “ of ”, count)
MPI_FINALIZE()
end
A simple program
Initiate computationFind the numberof processesFind the process ID of
current process Each process prints out its outputShut down
18
HPCA
2001
Result
I’m 0 of 4
I’m 2 of 4
I’m 1 of 4
I’m 3 of 4
Process 2 Process 3
Process 1Process 0
19
HPCA
2001
Another program (2 nodes)
…..MPI_COMM_RANK(MPI_COMM_WORLD, myid)if myid=0 MPI_SEND(“Zero”,…,…,1,…,…) MPI_RECV(words,…,…,1,…,…,…)else MPI_RECV(words,…,…,0,…,…,…) MPI_SEND(“One”,…,…,0,…,…)END IFprint(“Received from %s”,words)……
I’m process 0!if myid=0 MPI_SEND(“Zero”,…,…,1,…,…) MPI_RECV(words,…,…,1,…,…,…)……
I’m process 1!else MPI_RECV(words,…,…,0,…,…,…) MPI_SEND(“One”,…,…,0,…,…)
20
HPCA
2001
Result
Received from One
Received from Zero
Process 0
Process 1
21
HPCA
2001
Collective Communication
Three Types of Collective Operations Barrier
• for process synchronization• MPI_BARRIER
Data movement• moving data among processes• no computation• MPI_BCAST, MPI_GATHER, MPI_SCATTER
Reduction operations• Involve computation• MPI_REDUCE, MPI_SCAN
22
HPCA
2001
Barrier
MPI_BARRIER Used to synchronize execution of a group of
processes
wait
compute
Continue execution
All members reach the same point before any can proceed
Process 1 Process 2 Process p
Perform barrier
Blocking time
23
HPCA
2001
Data Movement Broadcast:
– one member sends the same message to all members
Scatter: – one member sends a different message to each member
Gather: – every member sends a message to a single member
All-to-all broadcast: – every member performs a broadcast
All-to-all scatter-gather (Total Exchange): – every member performs a scatter (and gather)
24
HPCA
2001
MPI Collective Communications
Broadcast (MPI_Bcast) Combine-to-one (MPI_Reduce) Scatter (MPI_Scatter) Gather (MPI_Gather) Collect (MPI_Allgather) Combine-to-all (MPI_Allreduce) Reduce: (MPI_Reduce) Scan: (MPI_Scan) All-to-All: (MPI_Alltoall)
25
HPCA
2001
FACEFACE
Process 0 Process 1 Process 2 Process 3
BCAST BCAST BCAST BCAST
FACE FACE FACE
Data movement (1)
MPI_BCAST One single process sends the same data to
all other processes, itself included
26
HPCA
2001
Process 0 Process 1 Process 2 Process 3
GATHER GATHER GATHER GATHER
EA C EF F A CFACE
Data movement (2)
MPI_GATHER All process (include the root process) send
the same data to one process and store them in rank order
27
HPCA
2001
Process 0 Process 1 Process 2 Process 3
SCATTER SCATTER SCATTER SCATTER
FACEF C EA
Data movement (3)
MPI_SCATTER A process sends out a message, which is
split into several equals parts, and the ith portion is sent to the ith process
28
HPCA
2001
Process 0 Process 1 Process 2 Process 3
REDUCE REDUCE REDUCE REDUCE
9 3 789
8 9 3 7max
Data movement (4)
MPI_REDUCE (e.g., find maximum value) combine the values of each process, using a
specified operation, and return the combined value to a process
29
HPCA
2001
MPI_SCAN
Scan Op: +
Input
Result
Scan (parallel prefix): “partial” reduction based upon relative process number
Process 0 Process 3 Process 5
+ + + +
30
HPCA
2001
Example program (1)
Calculating the value of by:
1
02
dxx1
4
31
HPCA
2001
Example program (2)
……
MPI_BCAST(numprocs, …, …, 0, …)
for (i = myid + 1; i <= n; i += numprocs)
compute the area for each interval
accumulate the result in processes’
program data (sum)
MPI_REDUCE(&sum, …, …, …, MPI_SUM, 0, …)
if (myid == 0)
Output result
……
32
HPCA
2001
Calculated by process 0Calculated by process 1Calculated by process 2Calculated by process 3
OK!
OK!
OK!
OK!
=3.141...
Start calculation!
Recommended