Software Tools

Using PBS

Software toolsPortland compilers pgf77 pgf90 pghpf pgcc pgCC Portland debugger

GNU compilers g77 gcc

Intel ifort icc

…software toolsMPICH (MPI-1, MPI-2MPI compilers mpicc mpiCC mpif77 Mpif90

mpirunYour favorite text editorPBS

Using PBS

PBS an open source job scheduler* and queue manager

PBS – Portable Batch System

Others Torque Sun Grid Engine Platform LSF IBM Load Leveler

PBS/Torque widely used

PBS

openPBS – open source versionPBSpro – commercial versionTorque

qsub [options] script

Submits jobs to a queue (execution)Job specifications in the file scriptMore on pbs scripts laterPbs will queue, schedule and launch job specified in the command/scriptqsub returns jobid -jobnumber.servername (9999.master)

qsub – common options

-q queuename -- submit to the queue queuename -q parallel@master

-N name -- give the job the name name -N mybigjob1

-I – run the job in interactive mode rather than batch

qsub – common options

-o path -- location of pbs output, defaults to ~owner -e path -- location of pbs error file, defaults to ~owner -a datetime --places job in a wait state until datetime datetime is [[[[CC]YY]MM]DD]hhmm[.SS] -- run a job on/after 5:00 pm on Feb. 14 -a 0214 1700

qsub – common options

-m [a,b,e]a --mail is sent when the job is

aborted by the batch system*.

b --mail is sent when the job begins execution.

e --mail is sent when the job terminates.

qsub – common options

-M useraddress - defines mail addresses for email (see –m) -M Don.McLaughlin@mail.wvu.edu

-S path define the location of the shell to interpret the script-S /bin/csh

qsub – common options

-l job resource list (important)nodes=nppn=m-l nodes=2:ppn=2

qstat - get status of jobs/queues

qstat –a – get status report in alternative format

qstat –a – statesE - Job is exiting after having run. H - Job is held. Q - job is queued, eligible to run or routed. R - job is running. T - job is being moved to new location. (not

here)W - job is waiting for its execution time

(qsub -a option) to be reached.

qstat – other options

qstat –an -- lists jobs in queue/running. For running jobs shows nodes/processors assigned to the job.

qstat – options

qstat –q -- list all queues and that status

qdel – deleting a job from the queue

qdel jobidjobid is sequence_number[.server_name]for example –

qdel 9999.master or

qdel 9999

other PBS commands

qhold jobid -- places job jobid on holdqrls jobid -- moves job jobid from hold to queued or executing statusqalter jobid – modifies attributes of the job jobid specified in qsub after it is submitted

** not all attributes may be modified

other PBS commands

qmove dest jobid -- moves job jobid from current queue to the queue dest for example –

qmove short 9999

PBS qsub scripts

Standard (almost) script in whatever shell is supported on the system.Must include the line –

mpirun myprogmay include other statements/directives – like…

PBS qsub scripts – cont.

a simple script – demtest.sh

#!/bin/sh#PBS -N dem#PBS -q parallel#PBS -l nodes=2:ppn=2mpirun -np 4 -machinefile $PBS_NODEFILE

hellocat $PBS_NODEFILE

PBS scripts cont…

then –

qsub demtest.sh

PBS scripts cont… and your get dem.o3975 ---

Hello world! I'm node 1 Hello world! I'm node 2 Hello world! I'm node 3 Hello world! I'm node 0node7node7node6node6

Xpbs

For more information on PBS commands-

System man pages “man qstat”http://www-wisconsin.cern.ch/pbs/commands.htmlhttp://rsusu1.rnd.runnet.ru/opbs/commands.htmlhttp://www.sao.nrc.ca/imsb/rcsg/hpc/sgi_pbs2000/pbs/sld001.htmhttp://www.ccr.buffalo.edu/documents/pbs.PDF

Logging in to the cluster

using a secure shell clientlogin in to-

energy.cluster.wvu.edu

Compiling a MPI program

mpicc myprog.c –o myprog

or

gcc myprog.c –o myprog –lmpi –L/usr/local/packages/mpich/lib –I/usr/local/packages/mpich/include