Lecture 5 Barriers and MPI Introduction Topics Barriers Uses implementations MPI Introduction Readings – Semaphore handout dropboxed January 24, 2012 CSCE

Lecture 5Barriers and MPI Introduction

Lecture 5Barriers and MPI Introduction

Topics Topics Barriers

Uses implementations

MPI Introduction

Readings – Semaphore handout Readings – Semaphore handout dropboxeddropboxed

January 24, 2012

CSCE 713 Advanced Computer Architecture

– 2 –CSCE 713 Spring 2012

OverviewOverviewLast TimeLast Time

Posix Pthreads: create, join, exit, mutexes /class/csce713-006 Code and Data

Readings for todayReadings for today http://csapp.cs.cmu.edu/public/1e/public/ch9-preview.pdf

NewNew Website alive and kicking; dropbox too! From Last Lecture’s slides: Gauss-Seidel Method, Barriers,

Threads Assignment Next time performance evaluation, barriers and MPI intro


Threads programming AssignmentThreads programming Assignment

1.1. Matrix addition (embarassingly parallel)Matrix addition (embarassingly parallel)

2.2. VersionsVersionsa. Sequential

b. Sequential with blocking factor

c. Sequential Read without conversions

d. Multi threaded passing number of threads as command line argument (args.c code should be distributed as an example)

3.3. Plot of several runsPlot of several runs

4.4. Next timeNext time

5.5. Test Data in /class/csce713-006/DataTest Data in /class/csce713-006/Data• A100, A400, B100, B400


Next few slides are mostly from Parallel Programming Next few slides are mostly from Parallel Programming by Pachecoby Pacheco

/class/csce713-006/Code/Pacheco contains the code /class/csce713-006/Code/Pacheco contains the code (again only for use with this course do not distribute)(again only for use with this course do not distribute)


Recall use of mutex to control access to a variableRecall use of mutex to control access to a variable

Copyright © 2010, Elsevier Inc. All rights Reserved


Barriers - synchronizing threadsBarriers - synchronizing threads

Synchronizing threads after a period of computationSynchronizing threads after a period of computation

• No thread proceeds until all others have reached the No thread proceeds until all others have reached the barrierbarrier

E.g., last time iteration of Gauss-Siedel, sync check for E.g., last time iteration of Gauss-Siedel, sync check for convergenceconvergence

Examples of barrier uses and implementationsExamples of barrier uses and implementations

1.1. Using barriers for testing convergence, i.e. satisfying Using barriers for testing convergence, i.e. satisfying a completion criteriaa completion criteria

2.2. Using barriers to time the slowest threadUsing barriers to time the slowest thread

3.3. Using barriers for debuggingUsing barriers for debugging


Using barriers for testing convergence, i.e. satisfying a completion criteriaUsing barriers for testing convergence, i.e. satisfying a completion criteriaSlide 43 of Lecture 3 (slightly modified)Slide 43 of Lecture 3 (slightly modified)

General computationGeneral computation

Stop when all xStop when all xii from this iteration calculated from this iteration calculated

ii

i

ii

i,

n

j1j

jj,

a

xac

x

n

1i

2i )x- (berror i


Using barriers to time the slowest threadUsing barriers to time the slowest thread



Using barriers for debuggingUsing barriers for debugging


– 10 –CSCE 713 Spring 2012

Barrier ImplementationsBarrier Implementations

1.1. Mutex , threadCounter, busy-waitsMutex , threadCounter, busy-waits

2.2. Mutex plus barrier semaphoreMutex plus barrier semaphore

3.3. Mutex and condition variablesMutex and condition variables

– 11 –CSCE 713 Spring 2012

Busy-waiting and a MutexBusy-waiting and a Mutex

Implementing a barrier using busy-waiting and a mutex Implementing a barrier using busy-waiting and a mutex is straightforward.is straightforward.

We use a shared counter protected by the mutex.We use a shared counter protected by the mutex.

When the counter indicates that every thread has When the counter indicates that every thread has entered the critical section, threads can leave the entered the critical section, threads can leave the critical section.critical section.


– 12 –CSCE 713 Spring 2012

Busy-waiting and a MutexBusy-waiting and a Mutex


We need one counter variable for each

instance of the barrier,

otherwise problemsare likely to occur.

– 13 –CSCE 713 Spring 2012

Implementing a barrier with semaphoresImplementing a barrier with semaphores


– 14 –CSCE 713 Spring 2012

Condition VariablesCondition Variables

A condition variable is a data object that allows a A condition variable is a data object that allows a thread to suspend execution until a certain event or thread to suspend execution until a certain event or condition occurs.condition occurs.

When the event or condition occurs another thread can When the event or condition occurs another thread can signal the thread to “wake up.”signal the thread to “wake up.”

A condition variable is always associated with a mutex.A condition variable is always associated with a mutex.


– 15 –CSCE 713 Spring 2012

Condition VariablesCondition Variables


– 16 –CSCE 713 Spring 2012

Implementing a barrier with condition variablesImplementing a barrier with condition variables


– 17 –CSCE 713 Spring 2012

POSIX Semaphores POSIX Semaphores

$ man -k semaphore$ man -k semaphoresem_close (3) - close a named semaphoresem_close (3) - close a named semaphoresem_destroy (3) - destroy an unnamed semaphoresem_destroy (3) - destroy an unnamed semaphoresem_getvalue (3) - get the value of a semaphoresem_getvalue (3) - get the value of a semaphoresem_init (3) - initialize an unnamed semaphoresem_init (3) - initialize an unnamed semaphoresem_open (3) - initialize and open a named semaphoresem_open (3) - initialize and open a named semaphoresem_overview (7) - Overview of POSIX semaphoressem_overview (7) - Overview of POSIX semaphoressem_post (3) - unlock a semaphoresem_post (3) - unlock a semaphoresem_timedwait (3) - lock a semaphoresem_timedwait (3) - lock a semaphoresem_trywait (3) - lock a semaphoresem_trywait (3) - lock a semaphoresem_unlink (3) - remove a named semaphoresem_unlink (3) - remove a named semaphoresem_wait (3) - lock a semaphoresem_wait (3) - lock a semaphore

$ man –s 7 semaphores$ man –s 7 semaphoresNo manual entry for semaphore in section 7No manual entry for semaphore in section 7

– 18 –CSCE 713 Spring 2012

Implementing a barrier with condition variablesImplementing a barrier with condition variables


– 19 –CSCE 713 Spring 2012

Pthread BarriersPthread Barriers

// Barrier variable pthread_barrier_t barr;// Barrier variable pthread_barrier_t barr;

// Barrier initialization // Barrier initialization

if(pthread_barrier_init(&barr, NULL, THREADS)) if(pthread_barrier_init(&barr, NULL, THREADS)) { printf("Could not create a barrier\n"); { printf("Could not create a barrier\n");

return -1; return -1;

} }

// Create Threads// Create Threads

for(int i = 0; i < THREADS; ++i) { for(int i = 0; i < THREADS; ++i) {

if(pthread_create(&thr[i], NULL, &entry_point, (void*)i)) if(pthread_create(&thr[i], NULL, &entry_point, (void*)i)) ……

– 20 –CSCE 713 Spring 2012

Wait on Barrier in thread functionWait on Barrier in thread function

// Synchronization point // Synchronization point

int rc = pthread_barrier_wait(&barr); int rc = pthread_barrier_wait(&barr);

if(rc != 0 && rc !=PTHREAD_BARRIER_SERIAL_THREAD){ if(rc != 0 && rc !=PTHREAD_BARRIER_SERIAL_THREAD){

printf("Could not wait on barrier\n"); printf("Could not wait on barrier\n");

exit(-1); exit(-1);

}}

– 21 –CSCE 713 Spring 2012

System V Semaphores (Sets)System V Semaphores (Sets)

In System V IPC there are semaphore setsIn System V IPC there are semaphore setsCommandsCommands• ipcs - provide information on ipc facilities ipcs - provide information on ipc facilities • ipcrm - remove a message queue, semaphore set or shared ipcrm - remove a message queue, semaphore set or shared

memory segment etc.memory segment etc.

System Calls:System Calls:semctl (2) - semaphore control operationssemctl (2) - semaphore control operationssemget (2) - get a semaphore set identifiersemget (2) - get a semaphore set identifiersemop (2) - semaphore operationssemop (2) - semaphore operationssemtimedop (2) - semaphore operationssemtimedop (2) - semaphore operations

– 22 –CSCE 713 Spring 2012

MPI IntroductionMPI Introduction

– 23 –CSCE 713 Spring 2012

A distributed memory systemA distributed memory system


– 24 –CSCE 713 Spring 2012

A shared memory systemA shared memory system


– 25 –CSCE 713 Spring 2012

A tree-structured global sumA tree-structured global sum


– 26 –CSCE 713 Spring 2012

An alternative tree-structured global sumAn alternative tree-structured global sum


– 27 –CSCE 713 Spring 2012

Identifying MPI processesIdentifying MPI processes

Common practice to identify processes by nonnegative Common practice to identify processes by nonnegative integer ranks.integer ranks.

pp processes are numbered processes are numbered 0, 1, 2, .. p-10, 1, 2, .. p-1


– 28 –CSCE 713 Spring 2012

Our first MPI programOur first MPI program


– 29 –CSCE 713 Spring 2012

CompilationCompilation

mpicc -g -Wall -o mpi_hello mpi_hello.c

– 30 –CSCE 713 Spring 2012

ExecutionExecution


mpiexec -n <number of processes> <executable>

mpiexec -n 1 ./mpi_hello


run with 1 process

run with 4 processes

– 31 –CSCE 713 Spring 2012

ExecutionExecution




Greetings from process 0 of 1 !





– 32 –CSCE 713 Spring 2012

MPI ProgramsMPI ProgramsWritten in C.Written in C.

Has main. Uses stdio.h, string.h, etc.

Need to add Need to add mpi.hmpi.h header file. header file.

Identifiers defined by MPI start with “MPI_”.Identifiers defined by MPI start with “MPI_”.

First letter following underscore is uppercase.First letter following underscore is uppercase. For function names and MPI-defined types. Helps to avoid confusion.


– 33 –CSCE 713 Spring 2012

MPI ComponentsMPI Components

MPI_InitMPI_Init Tells MPI to do all the necessary setup.

MPI_FinalizeMPI_Finalize Tells MPI we’re done, so clean up anything allocated for this

program.


– 34 –CSCE 713 Spring 2012

Basic OutlineBasic Outline


– 35 –CSCE 713 Spring 2012

Communicators Communicators A collection of processes that can send messages to A collection of processes that can send messages to

each other.each other.

MPI_Init defines a communicator that consists of all the MPI_Init defines a communicator that consists of all the processes created when the program is started.processes created when the program is started.

Called Called MPI_COMM_WORLDMPI_COMM_WORLD..


– 36 –CSCE 713 Spring 2012

CommunicatorsCommunicators


number of processes in the communicator

my rank (the process making this call)

Documents

Lecture 5 Barriers and MPI Introduction Topics Barriers Uses implementations MPI Introduction Readings – Semaphore handout dropboxed January 24, 2012 CSCE