Chapter 3

MPI

• MPI = Message Passing Interface• Specification of message passing libraries for developers

and users– Not a library by itself, but specifies what such a library

should be– Specifies application programming interface (API) for

such libraries– Many libraries implement such APIs on different

platforms – MPI libraries• Goal: provide a standard for writing message passing

programs– Portable, efficient, flexible

• Language binding: C, C++, FORTRAN programs

The Program

#include <stdio.h>

#include <string.h>

#include "mpi.h"

main(int argc, char* argv[]) {

int my_rank; /* rank of process */

int p; /* number of processes */

int source; /* rank of sender */

int dest; /* rank of receiver */

int tag = 0; /* tag for messages */

char message[100]; /* storage for message */

MPI_Status status; /* return status for */

/* receive */

/* Start up MPI */

MPI_Init(&argc, &argv);

/* Find out process rank */

MPI_Comm_rank(MPI_COMM_WORLD, &my_rank);

The Program

/* Find out number of processes */

MPI_Comm_size(MPI_COMM_WORLD, &p);

if (my_rank != 0) {

/* Create message */

sprintf(message, "Greetings from process %d!",

my_rank);

dest = 0;

/* Use strlen+1 so that '\0' gets transmitted */

MPI_Send(message, strlen(message)+1, MPI_CHAR,

dest, tag, MPI_COMM_WORLD);

} else { /* my_rank == 0 */

for (source = 1; source < p; source++) {

MPI_Recv(message, 100, MPI_CHAR, source, tag,

MPI_COMM_WORLD, &status);

printf("%s\n", message);

}

}

/* Shut down MPI */

MPI_Finalize();

} /* main */

General MPI programs

#include <mpi.h>

main( int argc, char** argv ) { MPI_Init( &argc, &argv );

/* main part of the program */

/* Use MPI function call depend on your data

partitioning and the parallelization architecture*/

MPI_Finalize();}

MPI Basics

• MPI’s pre-defined constants, function prototypes, etc., are included in a header file. This file must be included in your code wherever MPI function calls appear (in “main” and in user subroutines/functions) :– #include “mpi.h” for C codes– #include “mpi++.h” * for C++ codes– include “mpif.h” for f77 and f9x codes

• MPI_Init must be the first MPI function called• Terminates MPI by calling MPI_Finalize• These two functions must only be called once in user

code.

MPI Basics

• MPI’s pre-defined constants, function prototypes, etc., are included in a header file. This file must be included in your code wherever MPI function calls appear (in “main” and in user subroutines/functions) :– #include “mpi.h” for C codes– #include “mpi++.h” * for C++ codes– include “mpif.h” for f77 and f9x codes

• MPI_Init must be the first MPI function called• Terminates MPI by calling MPI_Finalize• These two functions must only be called once in user

code.

MPI Basics

• C is case-sensitive language. MPI function names always begin with “MPI_”, followed by specific name with leading character capitalized, e.g., MPI_Comm_rank. MPI pre-defined constant variables are expressed in upper case characters, e.g., MPI_COMM_WORLD.

Basic MPI Datatypes

MPI datatype C datatype

MPI_CHAR signed char

MPI_SIGNED_CHAR signed char

MPI_UNSIGNED_CHAR unsigned char

MPI_SHORT signed short

MPI_UNSIGNED_SHORT unsigned short

MPI_INT signed int

MPI_UNSIGNED unsigned int

MPI_LONG signed long

MPI_UNSIGNED_LONG unsigned long

MPI_FLOAT float

MPI_DOUBLE double

MPI_LONG_DOUBLE long double

MPI is Simple

• Many parallel programs can be written using just these six functions, only two of which are non-trivial:

– MPI_INIT

– MPI_FINALIZE

– MPI_COMM_SIZE

– MPI_COMM_RANK

– MPI_SEND

– MPI_RECV

Initialization

• Initialization: MPI_Init() initializes MPI environment– Must be called before any other MPI routine (so put it at the

beginning of code)– Can be called only once; subsequent calls are erroneous.

int MPI_Init(int *argc, char ***argv)

Termination

• MPI_Finalize() cleans up MPI environment– Must be called before exits.– No other MPI routine can be called after this call,

even MPI_INIT()

Termination

• MPI_Finalize() cleans up MPI environment– Must be called before exits.– No other MPI routine can be called after this call,

even MPI_INIT()

Processes

• MPI is process-oriented: program consists of multiple processes, each corresponding to one processor.

• MIMD: Each process runs its own code. In practice, runs its own copy of the same code (SPMD)..

• MPI processes are identified by their ranks:– If total nprocs processes in computation, rank ranges from 0, 1, …, nprocs-1.

– nprocs does not change during computation.

Communicators

• Communicator: is a group of processes that can communicate with one another.

• Most MPI routines require a communicator argument to specify the collection of processes the communication is based on.

• All processes in the computation form the communicator MPI_COMM_WORLD.– MPI_COMM_WORLD is pre-defined by MPI, available anywhere

• Can create subgroups/subcommunicators within MPI_COMM_WORLD.– A process may belong to different communicators, and have

different ranks in different communicators.

Size and Rank

• Number of processors: MPI_COMM_SIZE()• Which processor: MPI_COMM_RANK()• Can compute data decomposition etc.

– Know total number of grid points, total number of processors and current processor id; can calculate which portion of data current processor is to work on.

• Ranks also used to specify source and destination of communications.

int my_rank, ncpus;

MPI_Comm_rank(MPI_COMM_WORLD, &my_rank);

MPI_Comm_size(MPI_COMM_WORLD, &ncpus);

Compile and Run Program

• Compile the MPI programmpicc –o greetings greetings.c

• After compiling, a executable file greetings is generated.

• If running on the head nodempirun –np 4 ./greetingsGreetings from process 1!Greetings from process 2!Greetings from process 3!This is NOT allowed in HPC supercomputers.

PBS scripts

• PBS: Portable Batch System

• A cluster is shared with others– Need to use a job submission system

• PBS will allocate the job to some other computer, log in as the user, and execute it

• Useful Commands– qsub : submits a job – qstat : monitors status – qdel : deletes a job from a queue

A Job with PBS scripts

vi myjob1

#!/bin/bash

#PBS -N job1

#PBS -q production

#PBS -l select=4:ncpus=1

#PBS -l place=free

#PBS -V

 

cd $PBS_O_WORKDIR

 

mpirun -np 4 -machinefile $PBS_NODEFILE ./greetings

Submit Jobs

• Submit the job

qsub myjob1

283724.service0

• Check the job status

Qstat

PBS Pro Server andy.csi.cuny.edu at CUNY CSI HPC Center

Job id Name User Time Use S Queue

---------------- ---------------- ---------------- -------- - -----

276540.service0 methane_g09 michael.green 10265259 R qlong8_gau

276544.service0 methane_g09 michael.green 10265100 R qlong8_gau

277189.service0 BEAST_serial edward.myers 2373:38: R qserial

277828.service0 2xTDR e.sandoval 0 H qlong16_qdr

Submit Jobs

• See the output

cat job1.o283724

Greetings from process 1!

Greetings from process 2!

Greetings from process 3!

• See the error file

Cat job1.e283724

PBS scripts

PBS Description

#PBS -N jobname Assign a name to job

#PBS -M email_address Specify email address

#PBS -m b Send email at job start

#PBS -m e Send email at job end

#PBS -m a Send email at job abort

#PBS -o out_file Redirect stdout to specified file

#PBS -e errfile Redirect stderr to specified file

#PBS -q queue_name Specify queue to be used

#PBS -l select=chunk specification Specify MPI resource requirements

#PBS -l walltime=runtime Set wallclock time limit