View
213
Download
0
Category
Preview:
Citation preview
Lecture 5MPI Introduction
Lecture 5MPI Introduction
Topics Topics …
ReadingsReadings
January 19, 2012
CSCE 713 Advanced Computer Architecture
– 2 –CSCE 713 Spring 2012
A distributed memory systemA distributed memory system
Copyright © 2010, Elsevier Inc. All rights Reserved
– 3 –CSCE 713 Spring 2012
A shared memory systemA shared memory system
Copyright © 2010, Elsevier Inc. All rights Reserved
– 4 –CSCE 713 Spring 2012
Identifying MPI processesIdentifying MPI processes
Common practice to identify processes by nonnegative Common practice to identify processes by nonnegative integer ranks.integer ranks.
pp processes are numbered processes are numbered 0, 1, 2, .. p-10, 1, 2, .. p-1
Copyright © 2010, Elsevier Inc. All rights Reserved
– 5 –CSCE 713 Spring 2012
Our first MPI programOur first MPI program
Copyright © 2010, Elsevier Inc. All rights Reserved
– 6 –CSCE 713 Spring 2012
CompilationCompilation
mpicc -g -Wall -o mpi_hello mpi_hello.c
– 7 –CSCE 713 Spring 2012
ExecutionExecution
Copyright © 2010, Elsevier Inc. All rights Reserved
mpiexec -n <number of processes> <executable>
mpiexec -n 1 ./mpi_hello
mpiexec -n 4 ./mpi_hello
run with 1 process
run with 4 processes
– 8 –CSCE 713 Spring 2012
ExecutionExecution
Copyright © 2010, Elsevier Inc. All rights Reserved
mpiexec -n 1 ./mpi_hello
mpiexec -n 4 ./mpi_hello
Greetings from process 0 of 1 !
Greetings from process 0 of 4 !Greetings from process 1 of 4 !Greetings from process 2 of 4 !Greetings from process 3 of 4 !
– 9 –CSCE 713 Spring 2012
MPI ProgramsMPI ProgramsWritten in C.Written in C.
Has main. Uses stdio.h, string.h, etc.
Need to add Need to add mpi.hmpi.h header file. header file.
Identifiers defined by MPI start with “MPI_”.Identifiers defined by MPI start with “MPI_”.
First letter following underscore is uppercase.First letter following underscore is uppercase. For function names and MPI-defined types. Helps to avoid confusion.
Copyright © 2010, Elsevier Inc. All rights Reserved
– 10 –CSCE 713 Spring 2012
MPI ComponentsMPI Components
MPI_InitMPI_Init Tells MPI to do all the necessary setup.
MPI_FinalizeMPI_Finalize Tells MPI we’re done, so clean up anything allocated for this
program.
Copyright © 2010, Elsevier Inc. All rights Reserved
– 11 –CSCE 713 Spring 2012
Basic OutlineBasic Outline
Copyright © 2010, Elsevier Inc. All rights Reserved
– 12 –CSCE 713 Spring 2012
Communicators Communicators A collection of processes that can send messages to A collection of processes that can send messages to
each other.each other.
MPI_Init defines a communicator that consists of all the MPI_Init defines a communicator that consists of all the processes created when the program is started.processes created when the program is started.
Called Called MPI_COMM_WORLDMPI_COMM_WORLD..
Copyright © 2010, Elsevier Inc. All rights Reserved
– 13 –CSCE 713 Spring 2012
CommunicatorsCommunicators
Copyright © 2010, Elsevier Inc. All rights Reserved
number of processes in the communicator
my rank (the process making this call)
– 14 –CSCE 713 Spring 2012
SPMDSPMD
Single-Program Multiple-DataSingle-Program Multiple-Data
We compile We compile oneone program. program.
Process 0 does something different.Process 0 does something different. Receives messages and prints them while the other
processes do the work.
The The if-elseif-else construct makes our program SPMD. construct makes our program SPMD.
Copyright © 2010, Elsevier Inc. All rights Reserved
– 15 –CSCE 713 Spring 2012
CommunicationCommunication
Copyright © 2010, Elsevier Inc. All rights Reserved
– 16 –CSCE 713 Spring 2012
Data typesData types
Copyright © 2010, Elsevier Inc. All rights Reserved
– 17 –CSCE 713 Spring 2012
CommunicationCommunication
Copyright © 2010, Elsevier Inc. All rights Reserved
– 18 –CSCE 713 Spring 2012
Message matchingMessage matching
Copyright © 2010, Elsevier Inc. All rights Reserved
MPI_Sendsrc = q
MPI_Recvdest = r
r
q
– 19 –CSCE 713 Spring 2012
Receiving messagesReceiving messages
A receiver can get a message without knowing:A receiver can get a message without knowing: the amount of data in the message, the sender of the message, or the tag of the message.
Copyright © 2010, Elsevier Inc. All rights Reserved
– 20 –CSCE 713 Spring 2012
status_p argumentstatus_p argument
Copyright © 2010, Elsevier Inc. All rights Reserved
MPI_SOURCEMPI_TAG
MPI_ERROR
MPI_Status*
MPI_Status* status;
status.MPI_SOURCEstatus.MPI_TAG
– 21 –CSCE 713 Spring 2012
How much data am I receiving?How much data am I receiving?
Copyright © 2010, Elsevier Inc. All rights Reserved
– 22 –CSCE 713 Spring 2012
Issues with send and receiveIssues with send and receive
Exact behavior is determined by the MPI Exact behavior is determined by the MPI implementation.implementation.
MPI_Send may behave differently with regard to buffer MPI_Send may behave differently with regard to buffer size, cutoffs and blocking.size, cutoffs and blocking.
MPI_Recv always blocks until a matching message is MPI_Recv always blocks until a matching message is received.received.
Know your implementation;Know your implementation;don’t make assumptions!don’t make assumptions!
Copyright © 2010, Elsevier Inc. All rights Reserved
– 23 –CSCE 713 Spring 2012
TRAPEZOIDAL RULE IN MPITRAPEZOIDAL RULE IN MPI
Copyright © 2010, Elsevier Inc. All rights Reserved
– 24 –CSCE 713 Spring 2012
The Trapezoidal RuleThe Trapezoidal Rule
Copyright © 2010, Elsevier Inc. All rights Reserved
– 25 –CSCE 713 Spring 2012
The Trapezoidal RuleThe Trapezoidal Rule
Copyright © 2010, Elsevier Inc. All rights Reserved
– 26 –CSCE 713 Spring 2012
One trapezoidOne trapezoid
Copyright © 2010, Elsevier Inc. All rights Reserved
– 27 –CSCE 713 Spring 2012
Pseudo-code for a serial programPseudo-code for a serial program
Copyright © 2010, Elsevier Inc. All rights Reserved
– 28 –CSCE 713 Spring 2012
Parallelizing the Trapezoidal RuleParallelizing the Trapezoidal Rule
1.1. Partition problem solution into tasks.Partition problem solution into tasks.
2.2. Identify communication channels between tasks.Identify communication channels between tasks.
3.3. Aggregate tasks into composite tasks.Aggregate tasks into composite tasks.
4.4. Map composite tasks to cores.Map composite tasks to cores.
Copyright © 2010, Elsevier Inc. All rights Reserved
– 29 –CSCE 713 Spring 2012
Parallel pseudo-codeParallel pseudo-code
Copyright © 2010, Elsevier Inc. All rights Reserved
– 30 –CSCE 713 Spring 2012
Tasks and communications for Trapezoidal RuleTasks and communications for Trapezoidal Rule
Copyright © 2010, Elsevier Inc. All rights Reserved
– 31 –CSCE 713 Spring 2012
First version (1)First version (1)
Copyright © 2010, Elsevier Inc. All rights Reserved
– 32 –CSCE 713 Spring 2012
First version (2)First version (2)
Copyright © 2010, Elsevier Inc. All rights Reserved
– 33 –CSCE 713 Spring 2012
First version (3)First version (3)
Copyright © 2010, Elsevier Inc. All rights Reserved
– 34 –CSCE 713 Spring 2012
Dealing with I/ODealing with I/O
Copyright © 2010, Elsevier Inc. All rights Reserved
Each process justprints a message.
– 35 –CSCE 713 Spring 2012
Running with 6 processesRunning with 6 processes
Copyright © 2010, Elsevier Inc. All rights Reserved
unpredictable output
– 36 –CSCE 713 Spring 2012
Input Input Most MPI implementations only allow process 0 in Most MPI implementations only allow process 0 in
MPI_COMM_WORLD access to MPI_COMM_WORLD access to stdinstdin..
Process 0 must read the data (scanf) and send to the Process 0 must read the data (scanf) and send to the other processes.other processes.
Copyright © 2010, Elsevier Inc. All rights Reserved
– 37 –CSCE 713 Spring 2012
Function for reading user inputFunction for reading user input
Copyright © 2010, Elsevier Inc. All rights Reserved
– 38 –CSCE 713 Spring 2012
COLLECTIVE COMMUNICATIONCOLLECTIVE COMMUNICATION
Copyright © 2010, Elsevier Inc. All rights Reserved
– 39 –CSCE 713 Spring 2012
Tree-structured communicationTree-structured communication1.1. In the first phase: In the first phase:
(a) Process 1 sends to 0, 3 sends to 2, 5 sends to 4, (a) Process 1 sends to 0, 3 sends to 2, 5 sends to 4, and 7 sends to 6. and 7 sends to 6. (b) Processes 0, 2, 4, and 6 add in the received (b) Processes 0, 2, 4, and 6 add in the received values. values. (c) Processes 2 and 6 send their new values to (c) Processes 2 and 6 send their new values to processes 0 and 4, respectively.processes 0 and 4, respectively.(d) Processes 0 and 4 add the received values into (d) Processes 0 and 4 add the received values into their new values.their new values.
2.2. (a) Process 4 sends its newest value to process 0.(a) Process 4 sends its newest value to process 0.(b) Process 0 adds the received value to its newest (b) Process 0 adds the received value to its newest value.value.
Copyright © 2010, Elsevier Inc. All rights Reserved
– 40 –CSCE 713 Spring 2012
A tree-structured global sumA tree-structured global sum
Copyright © 2010, Elsevier Inc. All rights Reserved
– 41 –CSCE 713 Spring 2012
OverviewOverviewLast TimeLast Time
Posix Pthreads: create, join, exit, mutexes /class/csce713-006 Code and Data
Readings for todayReadings for today http://csapp.cs.cmu.edu/public/1e/public/ch9-preview.pdf
NewNew Website alive and kicking; dropbox too! From Last Lecture’s slides: Gauss-Seidel Method, Barriers,
Threads Assignment Next time performance evaluation, barriers and MPI intro
– 42 –CSCE 713 Spring 2012
Threads programming AssignmentThreads programming Assignment
1.1. Matrix addition (embarassingly parallel)Matrix addition (embarassingly parallel)
2.2. VersionsVersionsa. Sequential
b. Sequential with blocking factor
c. Sequential Read without conversions
d. Multi threaded passing number of threads as command line argument (args.c code should be distributed as an example)
3.3. Plot of several runsPlot of several runs
4.4. Next timeNext time
– 43 –CSCE 713 Spring 2012
Next few slides are from Parallel Programming by Next few slides are from Parallel Programming by PachecoPacheco
/class/csce713-006/Code/Pacheco contains the code /class/csce713-006/Code/Pacheco contains the code (again only for use with this course do not distribute)(again only for use with this course do not distribute)
ExamplesExamples
1.1. Simple use of mutex in calculating sumSimple use of mutex in calculating sum
2.2. SemaphoreSemaphore
– 44 –CSCE 713 Spring 2012
Global sum function that uses a mutex (2)Global sum function that uses a mutex (2)
Copyright © 2010, Elsevier Inc. All rights Reserved
– 45 –CSCE 713 Spring 2012
Barriers - synchronizing threadsBarriers - synchronizing threads
Synchronizing threads after a period of computationSynchronizing threads after a period of computation
• No thread proceeds until all others have reached the No thread proceeds until all others have reached the barrierbarrier
E.g., last time iteration of Gauss-Siedel, sync check for E.g., last time iteration of Gauss-Siedel, sync check for convergenceconvergence
Examples of barrier uses and implementationsExamples of barrier uses and implementations
1.1. Using barriers for testing convergence, i.e. satisfying Using barriers for testing convergence, i.e. satisfying a completion criteriaa completion criteria
2.2. Using barriers to time the slowest threadUsing barriers to time the slowest thread
3.3. Using barriers for debuggingUsing barriers for debugging
– 46 –CSCE 713 Spring 2012
Using barriers for testing convergence, i.e. satisfying a completion criteriaUsing barriers for testing convergence, i.e. satisfying a completion criteriaSlide 43 of Lecture 3 (slightly modified)Slide 43 of Lecture 3 (slightly modified)
General computationGeneral computation
Stop when all xStop when all xii from this iteration calculated from this iteration calculated
ii
i
ii
i,
n
j1j
jj,
a
xac
x
n
1i
2i )x- (berror i
– 47 –CSCE 713 Spring 2012
Using barriers to time the slowest threadUsing barriers to time the slowest thread
Copyright © 2010, Elsevier Inc. All rights Reserved
– 48 –CSCE 713 Spring 2012
Using barriers for debuggingUsing barriers for debugging
Copyright © 2010, Elsevier Inc. All rights Reserved
– 49 –CSCE 713 Spring 2012
Barrier ImplementationsBarrier Implementations
1.1. Mutex , threadCounter, busy-waitsMutex , threadCounter, busy-waits
2.2. Mutex plus barrier semaphoreMutex plus barrier semaphore
– 50 –CSCE 713 Spring 2012
Busy-waiting and a MutexBusy-waiting and a Mutex
Implementing a barrier using busy-waiting and a mutex Implementing a barrier using busy-waiting and a mutex is straightforward.is straightforward.
We use a shared counter protected by the mutex.We use a shared counter protected by the mutex.
When the counter indicates that every thread has When the counter indicates that every thread has entered the critical section, threads can leave the entered the critical section, threads can leave the critical section.critical section.
Copyright © 2010, Elsevier Inc. All rights Reserved
– 51 –CSCE 713 Spring 2012
Busy-waiting and a MutexBusy-waiting and a Mutex
Copyright © 2010, Elsevier Inc. All rights Reserved
We need one counter variable for each
instance of the barrier,
otherwise problemsare likely to occur.
– 52 –CSCE 713 Spring 2012
Implementing a barrier with semaphoresImplementing a barrier with semaphores
Copyright © 2010, Elsevier Inc. All rights Reserved
– 53 –CSCE 713 Spring 2012
Condition VariablesCondition Variables
A condition variable is a data object that allows a A condition variable is a data object that allows a thread to suspend execution until a certain event or thread to suspend execution until a certain event or condition occurs.condition occurs.
When the event or condition occurs another thread can When the event or condition occurs another thread can signal the thread to “wake up.”signal the thread to “wake up.”
A condition variable is always associated with a mutex.A condition variable is always associated with a mutex.
Copyright © 2010, Elsevier Inc. All rights Reserved
– 54 –CSCE 713 Spring 2012
Condition VariablesCondition Variables
Copyright © 2010, Elsevier Inc. All rights Reserved
– 55 –CSCE 713 Spring 2012
Implementing a barrier with condition variablesImplementing a barrier with condition variables
Copyright © 2010, Elsevier Inc. All rights Reserved
– 56 –CSCE 713 Spring 2012
POSIX Semaphores POSIX Semaphores
$ man -k semaphore$ man -k semaphoresem_close (3) - close a named semaphoresem_close (3) - close a named semaphoresem_destroy (3) - destroy an unnamed semaphoresem_destroy (3) - destroy an unnamed semaphoresem_getvalue (3) - get the value of a semaphoresem_getvalue (3) - get the value of a semaphoresem_init (3) - initialize an unnamed semaphoresem_init (3) - initialize an unnamed semaphoresem_open (3) - initialize and open a named semaphoresem_open (3) - initialize and open a named semaphoresem_overview (7) - Overview of POSIX semaphoressem_overview (7) - Overview of POSIX semaphoressem_post (3) - unlock a semaphoresem_post (3) - unlock a semaphoresem_timedwait (3) - lock a semaphoresem_timedwait (3) - lock a semaphoresem_trywait (3) - lock a semaphoresem_trywait (3) - lock a semaphoresem_unlink (3) - remove a named semaphoresem_unlink (3) - remove a named semaphoresem_wait (3) - lock a semaphoresem_wait (3) - lock a semaphore
$ man –s 7 semaphores$ man –s 7 semaphoresNo manual entry for semaphore in section 7No manual entry for semaphore in section 7
– 57 –CSCE 713 Spring 2012
Implementing a barrier with condition variablesImplementing a barrier with condition variables
Copyright © 2010, Elsevier Inc. All rights Reserved
– 58 –CSCE 713 Spring 2012
System V Semaphores (Sets)System V Semaphores (Sets)
In System V IPC there are semaphore setsIn System V IPC there are semaphore setsCommandsCommands• ipcs - provide information on ipc facilities ipcs - provide information on ipc facilities • ipcrm - remove a message queue, semaphore set or shared ipcrm - remove a message queue, semaphore set or shared
memory segment etc.memory segment etc.
System Calls:System Calls:semctl (2) - semaphore control operationssemctl (2) - semaphore control operationssemget (2) - get a semaphore set identifiersemget (2) - get a semaphore set identifiersemop (2) - semaphore operationssemop (2) - semaphore operationssemtimedop (2) - semaphore operationssemtimedop (2) - semaphore operations
– 59 –CSCE 713 Spring 2012
Recommended