The Campus Cluster

What is the Campus Cluster?

• Batch job system• High throughput• High latency• Available resources:– ~450 nodes– 12 Cores/node– 24-96 GB memory– Shared high performance filesystem– High speed multinode message passing

What isn’t the Campus Cluster?

• Not: Instantly available computation resource– Can wait up to 4 hours for a node

• Not: High I/O Friendly– Network disk access can hurt performance

• Not: ….

Getting Set Up

Getting started

• Request an account: https://campuscluster.illinois.edu/invest/user_form.html

• Connecting:ssh to taub.campuscluster.illinois.eduUse netid and AD password

Where to put data• Home Directory ~/

– Backed up, currently no quota (in future 10’s of GB)• Use /scratch for temporary data - ~10TB

– Scratch data is currently deleted after ~3 months– Available on all nodes– No backup

• /scratch.local - ~100GB– Local to each node, not shared across network– Beware that other users may fill disk

• /projects/VisionLanguage/ - ~15TB– Keep things tidy by creating a directory for your netid– Backed up

• Current Filesystem best practices (Should improve for Cluster v. 2):– Try to do batch writes to one large file– Avoid many little writes to many little files

Backup = Snapshots(Just learned this yesterday)

• Snapshots taken daily

• Not intended for disaster recovery – Stored on same disk as data

• Intended for accidental deletes/overwrites, etc.– Backed up data can be accessed at:/gpfs/ddn_snapshot/.snapshots/<date>/<path>

e.g. recover accidentally deleted file in home directory: /gpfs/ddn_snapshot/.snapshots/2012-12-24/home/iendres2/christmas_list

Moving data to/from cluster

• Only option right now is sftp/scp

• SSHFS lets you mount a directory from remote machines– Haven’t tried this, but might be useful

Modules

[iendres2 ~]$ modules load <modulename>

Manages environment, typically used to add software to path:– To get the latest version of matlab:

[iendres2 ~]$ modules load matlab/7.14– To find modules such as vim, svn:

[iendres2 ~]$ modules avail

Useful Startup Options

Appended to the end of my bashrc:– Make default permissions the same for user and

group, useful when working on a joint project• umask u=rwx,g=rwx

– Safer alternative – don’t allow writing• umask u=rwx,g=rx

– Load common modules• module load vim• module load svn• module load matlab

Submitting Jobs

Queues

– Primary (VisionLanguage)• Nodes we own (Currently 8)• Jobs can last 72 hours• We have priority access

– Secondary (secondary)• Anyone else’s idle nodes (~500)• Jobs can only last 4 hours, automatically killed• Not unusual to wait 12 hours for job to begin runing

Scheduler

• Typically behaves as first come first serve

• Claims of priority scheduling, we don’t know how it works…

Types of job

– Batch job• No graphics, runs and completes without user

interaction

– Interactive Jobs• Brings remote shell to your terminal• X-forwarding available for graphics

• Both wait in queue the same way

Scheduling jobs

– Batch job• [iendres2 ~]$ qsub <job_script>

• job_script defines parameters of job and the actual command to run• Details on job scripts to follow

– Interactive Jobs• [iendres2 ~]$ qsub -q <queuename> -I -l walltime=00:30:00,nodes=1:ppn=12

• Include –X for X-forwarding• Details on –l parameters to follow

Configuring Jobs

Basics

• Parameters of jobs are defined by a bash script which contains “PBS commands” followed by script to execute

#PBS -q VisionLanguage#PBS -l nodes=1:ppn=12#PBS -l walltime=04:00:00…cd ~/workdir/echo “This is job number ${PBS_JOBID}”

Basics

• Parameters of jobs are defined by a bash script which contains “PBS commands” followed by script to execute

#PBS -q VisionLanguage#PBS -l nodes=1:ppn=12#PBS -l walltime=04:00:00…cd ~/workdir/echo “This is job number ${PBS_JOBID}”

Queue to use:VisionLanguage or secondary

Basics

• Parameters of jobs are defined by a bash script which contains “PBS commands” followed by script to execute

#PBS -q VisionLanguage#PBS -l nodes=1:ppn=12#PBS -l walltime=04:00:00…cd ~/workdir/echo “This is job number ${PBS_JOBID}”

• Number of nodes – 1, unless using MPI or other distributed programming

• Processors per node – Always 12, smallest computation unit is a physical node, which has 12 cores (with current hardware)*

*Some queues are configured to allow multiple concurrent jobs per node, but this is uncommon

Basics

• Parameters of jobs are defined by a bash script which contains “PBS commands” followed by script to execute

#PBS -q VisionLanguage#PBS -l nodes=1:ppn=12#PBS -l walltime=04:00:00…cd ~/workdir/echo “This is job number ${PBS_JOBID}”

• Maximum time job will run for – it is killed if it exceeds this

• 72:00:00 hours for primary queue• 04:00:00 hours for secondary queue

Basics

• Parameters of jobs are defined by a bash script which contains “PBS commands” followed by script to execute

#PBS -q VisionLanguage#PBS -l nodes=1:ppn=12#PBS -l walltime=04:00:00…cd ~/workdir/echo “This is job number ${PBS_JOBID}”

Bash comands are allowed anywhere in the script and will be executed on the scheduled worker node after all PBS commands are handled

Basics

• Parameters of jobs are defined by a bash script which contains “PBS commands” followed by script to execute

#PBS -q VisionLanguage#PBS -l nodes=1:ppn=12#PBS -l walltime=04:00:00…cd ~/workdir/echo “This is job number ${PBS_JOBID}”

There are some reserved variables that the scheduler will fill in once the job is scheduled (see `man qsub` for more variables)

BasicsScheduler variables (From manpage)

PBS_O_HOST the name of the host upon which the qsub command is running.

PBS_SERVER the hostname of the pbs_server which qsub submits the job to.

PBS_O_QUEUE the name of the original queue to which the job was submitted.

PBS_O_WORKDIR the absolute path of the current working directory of the qsub command.

PBS_ARRAYID each member of a job array is assigned a unique identifier (see -t)

PBS_ENVIRONMENT set to PBS_BATCH to indicate the job is a batch job, or to PBS_INTERACTIVE to indicate the job is a PBS interac- tive job, see -I option.

PBS_JOBID the job identifier assigned to the job by the batch system.

PBS_JOBNAME the job name supplied by the user.

PBS_NODEFILE the name of the file contain the list of nodes assigned to the job (for parallel and cluster systems).

PBS_QUEUE the name of the queue from which the job is executed.

There are some reserved variables that the scheduler will fill in once the job is scheduled (see `man qsub` for more variables)

Monitoring Jobs[iendres2 ~]$ qstat Sample output:JOBID JOBNAME USER WALLTIME STATE QUEUE333885[].taubm1 r-afm-average hzheng8 0 Q secondary333899.taubm1 test6 lee263 03:33:33 R secondary 333900.taubm1 cgfb-a dcyang2 09:22:44 R secondary 333901.taubm1 cgfb-b dcyang2 09:31:14 R secondary 333902.taubm1 cgfb-c dcyang2 09:28:28 R secondary 333903.taubm1 cgfb-d dcyang2 09:12:44 R secondary 333904.taubm1 cgfb-e dcyang2 09:27:45 R secondary 333905.taubm1 cgfb-f dcyang2 09:30:55 R secondary 333906.taubm1 cgfb-g dcyang2 09:06:51 R secondary 333907.taubm1 cgfb-h dcyang2 09:01:07 R secondary 333908.taubm1 ...conp5_38.namd harpole2 0 H cse 333914.taubm1 ktao3.kpt.12 chandini 03:05:36 C secondary 333915.taubm1 ktao3.kpt.14 chandini 03:32:26 R secondary 333916.taubm1 joblammps daoud2 03:57:06 R cse

States:Q – Queued, waiting to runR – RunningH – Held, by user or admin, won’t run until released (see qhold, qrls)C – Closed – finished runningE – Error – this usually doesn’t happen, indicates a problem with the cluster

grep is your friend for finding specific jobs(e.g. qstat –u iendres2 | grep “ R ” gives all of my running jobs)

Managing Jobs

qalter, qdel, qhold, qmove, qmsg, qrerun, qrls, qselect, qsig, qstat

Each takes a jobid + some arguments

Problem: I want to run the same job with multiple parameters

#PBS -q VisionLanguage#PBS -l nodes=1:ppn=12#PBS -l walltime=04:00:00

cd ~/workdir/./script <param1> <param2>

Solution: Create wrapper script to iterate over params

Where: param1 = {a, b, c}param2 = {1, 2, 3}

Problem 2: I can’t pass parameters into my job script

#PBS -q VisionLanguage#PBS -l nodes=1:ppn=12#PBS -l walltime=04:00:00

cd ~/workdir/./script <param1> <param2>

Solution 2: Hack it!

Where: param1 = {a, b, c}param2 = {1, 2, 3}

Problem 2: I can’t pass parameters into my job script

Where: param1 = {a, b, c}param2 = {1, 2, 3}

We can pass parameters via the jobname, and delimit them using the ‘-’ character (or whatever you want)

#PBS -q VisionLanguage#PBS -l nodes=1:ppn=12#PBS -l walltime=04:00:00

# Pass parameters via jobname:export IFS="-"i=1

for word in ${PBS_JOBNAME}; do echo $word arr[i]=$word ((i++))done

# Stuff to executeecho Jobname: ${arr[1]}cd ~/workdir/echo ${arr[2]} ${arr[3]}

Problem 2: I can’t pass parameters into my job script

Where: param1 = {a, b, c}param2 = {1, 2, 3}

qsub –N job-param1-param2 job_script

qsub’s -N parameter sets the job name

#PBS -q VisionLanguage#PBS -l nodes=1:ppn=12#PBS -l walltime=04:00:00

# Pass parameters via jobname:export IFS="-"i=1

for word in ${PBS_JOBNAME}; do echo $word arr[i]=$word ((i++))done

# Stuff to executeecho Jobname: ${arr[1]}cd ~/workdir/echo ${arr[2]} ${arr[3]}

Problem 2: I can’t pass parameters into my job script

#PBS -q VisionLanguage#PBS -l nodes=1:ppn=12#PBS -l walltime=04:00:00

# Pass parameters via jobname:export IFS="-"i=1

for word in ${PBS_JOBNAME}; do echo $word arr[i]=$word ((i++))done

# Stuff to executeecho Jobname: ${arr[1]}cd ~/workdir/echo ${arr[2]} ${arr[3]}

Where: param1 = {a, b, c}param2 = {1, 2, 3}

qsub –N job-param1-param2 job_script

Output would be:

Jobname: jobparam1 param2

Problem: I want to run the same job with multiple parameters

#PBS -q VisionLanguage#PBS -l nodes=1:ppn=12#PBS -l walltime=04:00:00

# Pass parameters via jobname:export IFS="-"i=1

for word in ${PBS_JOBNAME}; do echo $word arr[i]=$word ((i++))done

# Stuff to executeecho Jobname: ${arr[1]}cd ~/workdir/echo ${arr[2]} ${arr[3]}

Where: param1 = {a, b, c}param2 = {1, 2, 3}

#!/bin/bash

param1=({a,b,c})param2=({1,2,3}) # or {1..3} for p1 in ${param1[@]}; do for p2 in ${param2[@]}; do qsub –N job-${p1}-${p2} job_script donedone

Now Loop!

Problem 3: My job isn’t multithreaded, but needs to run many times

#PBS -q VisionLanguage#PBS -l nodes=1:ppn=12#PBS -l walltime=04:00:00

cd ~/workdir/./script ${idx} Solution: Run 12 independent

processes on the same node so 11 CPU’s don’t sit idle

Problem 3: My job isn’t multithreaded, but needs to run many times

#PBS -q VisionLanguage#PBS -l nodes=1:ppn=12#PBS -l walltime=04:00:00

cd ~/workdir/# Run 12 jobs in the backgroundfor idx in {1..12}; do ./script ${idx} & # Your job goes here (keep the ampersand) pid[idx]=$! # Record the PIDdone

# Wait for all the processes to finish for idx in {1..12}; do echo waiting on ${pid[idx]} wait ${pid[idx]}done

Solution: Run 12 independent processes on the same node so 11 CPU’s don’t sit idle

Matlab and The Cluster

Simple Matlab Sample#PBS -q VisionLanguage#PBS -l nodes=1:ppn=12#PBS -l walltime=04:00:00

cd ~/workdir/matlab -nodisplay -r “matlab_func(); exit;”

Matlab Sample: Passing Parameters

#PBS -q VisionLanguage#PBS -l nodes=1:ppn=12#PBS -l walltime=04:00:00

cd ~/workdir/param = 1param2 = \’string\’ # Escape string parametersmatlab -nodisplay -r “matlab_func(${param}); exit;”

#PBS -q VisionLanguage#PBS -l nodes=1:ppn=12#PBS -l walltime=04:00:00

cd ~/workdir/matlab -nodisplay -r “matlab_func(); exit;”X

Simple Matlab Sample

Running more than a few matlab jobs (thinking about using the secondary queue) ?

You may use too many licenses - especially Distributed Computing Toolbox (e.g. parfor)

Compiling Matlab CodeDoesn’t use any matlab licenses once compiledCompiles matlab code into a standalone executableConstraints:– Code can’t call addpath– Functions called by eval, str2func, or other implicit methods must be

explicitly identified• e.g. for eval(‘do_this’) to work, must also include %#function do_this

To compile (within matlab):>> addpath(‘everything that should be included’)>> mcc –m function_to_compile.m

isdeployed() is useful for modifying behavior for compiled applications(returns true if code is running the compiled version)

Running Compiled Matlab Code

• Requires Matlab compiler runtime>> mcrinstaller % This will point you to the installer and help install it% make note of the installed path MCRPATH (e.g. …/mcr/v716/)

• Compiled code generates two files:– function_to_compile and run_function_to_compile.sh

• To run:– [iendres2 ~]$ ./run_function_to_compile.sh MCRPATH param1 param2 … paramk– Params will be passed into matlab function as usual, except they will always be strings– Useful trick:

function function_to_compile(param1, param2, …, paramk)if(isdeployed) param1 = str2num(param1); %param2 expects a string paramk = str2num(paramk);end

Parallel For Loops on the Cluster

• Not designed for multiple nodes on shared filesystem:– Race condition from concurrent writes to:

~/.matlab/local_scheduler_data/

• Easy fix: redirect directory to /scratch.local

Parallel For Loops on the Cluster

1. Setup (done once, before submitting jobs): [iendres2 ~]$ ln –sv /scratch.local/tmp/USER/matlab/local_scheduler_data

~/.matlab/local_scheduler_data(Replace USER with your netid)

Parallel For Loops on the Cluster2. Wrap matlabpool function to make sure tmp data exists:

function matlabpool_robust(varargin)

if(matlabpool('size')>0)   matlabpool closeend

% make sure the directories exist and are empty for good measuresystem('rm -rf /scratch.local/tmp/USER/matlab/local_scheduler_data');

system(sprintf('mkdir -p /scratch.local/tmp/USER/matlab/local_scheduler_data/R%s', version('-release')));

% Run it:

matlabpool (varargin{:});Warning:/scratch.local may get filled up by other users, in which case this will fail.

Best Practices

• Interactive Sessions– Don’t leave idle sessions open, it ties up the nodes

• Job arrays– Still working on kinks in the scheduler, I managed

to kill the whole cluster• Disk I/O– Minimize I/O for best performance– Avoid small reads and writes due to metadata

overhead

Maintenance

• “Preventive maintenance (PM) on the cluster is generally scheduled on a monthly basis on the third Wednesday of each month from 8 a.m. to 8 p.m. Central Time. The cluster will be returned to service earlier if maintenance is completed before schedule.”

Resources

• Beginner’s guide:https://campuscluster.illinois.edu/user_info/doc/beginner.html

• More comprehensive user’s guide: http://campuscluster.illinois.edu/user_info/doc/index.html

• Cluster Monitor:http://clustat.ncsa.illinois.edu/taub/

• Simple sample job scripts/projects/consult/pbs/

• Forumhttps://campuscluster.illinois.edu/forum/