1 LINUX BASICS:

Linux traditionally operates in text mode or in graphical mode. Since CPU power and RAM are not the cost anymore these days, every Linux user can afford to work in graphical mode and will usually do so. This does not mean that you don't have to know about text mode: we will work in the text environment throughout this course, using a terminal window.

Linux encourages its users to acquire knowledge and to become independent. Inevitably, you will have to read a lot of documentation to achieve that goal; that is why, as you will notice, we refer to extra documentation for almost every command, tool and problem listed in this book. The more docs you read, the easier it will become and the faster you will leaf through manuals. Make reading documentation a habit as soon as possible. When you don't know the answer to a problem, refering to the documentation should become a second nature.

Linux is an implementation of UNIX. The Linux operating system is written in the C programming language.

"De gustibus et coloribus non disputandum est": there's a Linux for everyone.

Linux uses GNU tools, a set of freely available standard tools for handling the operating system

I found few basic commands that can be used on LINUX Operating system.

Quickstart commands

Command Meaning

ls Displays a list of files in the current working directory, like the dir command in DOS

cd directory change directoriespasswd change the password for the current userfile filename display file type of file with name filenamecat textfile throws content of textfile on the screenpwd display present working directoryexit or logout leave this sessionman command read man pages on commandinfo command read Info pages on commandapropos string search the whatis database for strings

1

Source(s):

http://tldp.org/LDP/intro-linux/html/

2 BASIC MPI

Overview

MPI (Message Passing Interface) is a standard interface for the message passing paradigm of parallel computing. This is a model of cooperative processes, working on separate data spaces and interchanging messages when they need to share or communicate data. The model implies an active role on both the part of the sending process and the receiving process.

The machine model underlying MPI is that of a multicomputer: a distributed memory machine with processors tied together via some kind of interconnection fabric. The multicomputer model ignores the underlying topology of the fabric, and presumes there are two types of memory accesses: local and distant. However, it is possible to impose a topology model on MPI, an advanced topic reserved for later.

Although MPI has an underlying distributed memory model, it can be used for

1. Distributed memory machines 2. Shared memory machines 3. Arrays of SMPs ("clusters") 4. Networks of workstations 5. Heterogeneous networks of machines

This is because the logical programming model does not have to match the physical machine architecture. Generally you can get more efficient code if your programming model matches the machine's physical hardware, but the advantage of MPI is that by aiming at the absolute minimal assumption about the hardware, your code will be portable across the wide variety of actual systems listed above.

Basic MPI Concepts

A MPI program consists of multiple processes, each with its own address space. Each process runs the same program (SPMD model), but has a unique number that identifies it. If there are p MPI processes participating in a single program, a process's identifying

2

number is an integer between 0 and p-1 and is called its "rank". In the statement of most SPMD algorithms, there are lines like if (myid == 0) { ... } else { ... }which is read "if my process identification number is zero, then do the following; otherwise, do something else". This is how MIMD programs are built on a SPMD model - each runs the same program, which branches dependent on the process's ID number. In MPI, that ID number is called its rank. Note that most distributed memory machines can be run directly in MIMD mode with each processor actually running a completely different program. However, SPMD is the model most often used to emulate MIMD actions. The reasons are psychological - it is easier to have a single source code to write and examine.

Each message in MPI consists of the data sent and a header . The header contains

1. The rank of the sender 2. The rank of the receiver 3. A message identifier number called its tag 4. A communicator identification

The MPI standard guarantees that the integers 0-32767 can be used as valid tag numbers, but most implementations allow far more. One basic concept in MPI is that of a communicator group : a set of MPI processes that are grouped together in working on a problem, which can send messages to each other. For the start, we will use the default communicator group MPI_COMM_WORLD , which sets a single context and involves all the processes running. This is a predefined communicator, of type MPI_Comm . Later we will cover more details about the concepts of communicators and contexts. But to get an idea of why different communicator groups may be needed in a single program, consider what happens if we are running an MPI program that calls a math library - which was also built to use parallelism via MPI. To keep process number 3 in our program getting confused with process number 3 as defined by the library, we need to have an additional identifier to distinguish them (and keep one from receiving a message intended for the other). This additional identification is the communicator group.

Basic MPI Functions

Although MPI has over 120 different functions that can be invoked, all parallel programs can be built using just six:

1. MPI_INIT() initializes MPI in a program. 2. MPI_COMM_SIZE() returns the number of cooperating processes. 3. MPI_COMM_RANK() returns the process identifier for the process that invokes

it. 4. MPI_SEND() sends a message. 5. MPI_RECV() receives a message. 6. MPI_FINALIZE() cleans up and terminates MPI

3

For our purposes, there are a few more that are useful right from the start: 1. MPI_BCAST(): send a message from one processor to all the others in the

specified communicator group. 2. MPI_ALLREDUCE(): perform a reduction operation , and make the reduced

scalar available to all participating processes.

The last one is useful for most dotproducts, since it is typically the case that the resulting scalar is needed by all the processors. For performance evaluation, we can also use:

1. MPI_WTIME(): returns a double that gives the number of seconds since either the beginning of the program, or 1 January 1970.

2. MPI_WTICK(): returns a float that gives the clock resolution.

We will not use all of them immediately. Here are some details about the ones we need right away; this is for the C language versions.

MPI_Init(&argc, &argv); This must be called at the beginning of the parallel program, after whatever global initializations need to be performed. Its two arguments are the two that a C function main() takes (so that they can be passed along to all the MPI processes).

MPI_Comm_size(MPIcomm comm, &p); This takes the communicator group as first argument (which will be MPI_COMM_WORLD for all our beginning programs). It returns in the second argument the number of processes participating in the communicator group.

MPI_Comm_rank(MPIcomm comm, &myrank); This takes the communicator group as first argument and returns the rank of the calling process.

MPI_Finalize(); Takes no arguments; it just cleans up things. Try leaving it off your program on the burrow to see what happens.

MPI_Send(void* message, int count, MPI_Datatype datatype, int dest, int tag, MPI_Comm comm);

MPI_Recv(void* message, int count, MPI_Datatype datatype, int src, int tag, MPI_Comm comm, MPI_Status *status);

The last two are the most basic send/receive pair, and their required arguments are void* message: The beginning address of the block of memory containing the

message. MPI_Datatype datatype: This is one of the allowed types, generally corresponding

to a C datatype. For example, these include (with C type in parentheses) o MPI_CHAR (signed char) o MPI_INT (signed int) o MPI_DOUBLE (double) o MPI_FLOAT (float)

int count: The message consists of a "count" number of the given datatype. int src : the source of the message (part of MPI_Recv calling sequence) int dest: the destination for the message (part of MPI_Send calling sequence) int tag: the tag for the message.

4

Note that there is no argument giving the "buffsize" as was mentioned earlier in generic send/receive functions. In MPI, if the sent message is too large to fit into the receiving buffer, it causes either segmentation fault (the best case) or weird corruption of your data.

The final argument of the MPI_Recv function gives information about the message as actually received. It is a C structure with at least three components, the source, the tag, and an error code of type MPI_ERROR. So if, for example, the receive used for the source field the wild-card MPI_ANY_SOURCE, then status->MPI_SOURCE will contain the rank of the process that sent the received message. Note that the MPI_Status variable does not necessarily have a field for the count of data items actually sent; you should use the function MPI_Get_count() for that.

Because it is often the case that one process needs to send or receive data to or from all other processes, MPI provides collective communication functions. If you are lucky, the vendor has optimized them for the machine topology; in MPICH they use a tree algorithm like the one above. Here are some of the functions:

1. int MPI_Bcast(void *msg, int count, MPI_Datatype, int root, MPI_Comm): sends a message from process root to all others in the communicator group. This function must be called by all participating processes. Also, count and datatype must match on all processes, unlike MPI_Send and MPI_Recv.

2. MPI_Reduce(void *operand, void *result, int count, MPI_Datatype, MPI_OP, int root, MPI_Comm): combines operands stored in operand and leaves the answer in result on process with rank "root". Here count, datatype, and operation MPI_OP must be the same on all processes. The types of operations are MAX, MIN, SUM, PROD, and various logical operations. Warning: although only root gets the result, all the participating processes must provide void *result, and all must provide the actual storage space it uses.

3. Often you want a global reduction operation with the result left on on every process, not just a root one. Instead of following MPI_Reduce with a MPI_Bcast, use instead MPI_Allreduce(). In general, the "All" word embedded in an MPI function means to have the operation result end up in all tasks in the communicator group.

Source(s): http://www.cs.indiana.edu/classes/b673/notes/mpi1.html

And also in addition to the above data , I found few links below where I can get information on MPITutorial material on MPI available on the Web

http://www-unix.mcs.anl.gov/mpi/tutorial/index.html

about parallel processing.http://linuxgazette.net/issue65/joshi.html

5

3 OPEN MPI

Open MPI provides a high performance platform for parallel applications on both homogeneous and heterogeneous platforms. The application developer is not unduly burdened by the cost of heterogeneous application development as Open MPI transparently handles process start-up,communication, and data conversion. Open MPI also determines all architecture and networking properties on a perpeer basis, and selects the most efficient mode of communicationwith the given peer, in order to maximize performance. our design to support multiple,disparate networks can offer a performance increase in some situations. Open MPI defines a job as the execution of either a single application,or multiple communicating applications, each potentially on a separate computing system. Within that context, the project generally breaks the definition of heterogeneity into four broad categories:

Processor heterogeneity Dealing with differences in processor speed, internal architecture (e.g., multicore), and communication issues caused by the transfer of data between processors with different data representations (different endian, floating point representation,etc.).Network heterogeneity Using different network protocols between processes in a job. This includes multiple network protocols between two processes and different network protocol to different processes in the job.Run-Time Environment heterogeneity Executing a job across multiple resources (e.g., multiple clusters) that are locally administered by possibly different scheduling systems, authentication realms, etc.Binary heterogeneity Coordinating execution of different binaries, either through the coupling of different applications or as part of a complex single application.

Bootstrapping StartupOpenRTE provides transparent, scalable support for high-performance applications in heterogeneous environments. OpenRTE originated within the Open MPI project but has since spun-off into its own effort, though the two projects remain closely coordinated. The ability of Open MPI to transparently operate in heterogeneous environments is largely due to the support services provided by OpenRTE, so it is useful to understand the OpenRTE architecture.The OpenRTE system is comprised of four major functional groups:

General Purpose Registry(GPR):A centralized publish/subscribe data repository used to store administrative information (e.g., communication contact information and process state) supporting OpenRTE and Open MPI operations.

6

Resource Management (RM) group A collection of four subsystems supporting the identification and allocation of resources to an application, mapping of processes to specific nodes, and launch of processes.

Support Services A suite of subsystems that provide general support for OpenRTE operations, including services to generate unique process names, I/O forwarding between application processes and the user, and OpenRTE’s communication subsystem.

Error Management (EM) group Several subsystems that collectively monitor the state of processes within an application and respond to changes in those states thatindicate errors have occurred.

Point-to-Point DesignOpen MPI provides point-to-point message transfer facilities via multiple MCA frameworks. The point-to-point architecture consists of four main layers: the Byte Transport Layer (BTL), BTL Management Layer (BML), Point-to-Point Messaging Layer (PML) and the MPI layer. Two additional frameworks are shown, the Memory Pool (MPool) and the Registration Cache (Rcache).

Message SchedulingTo effectively utilize multiple network interconnects Open MPI provides a mechanism to schedule a single message across these network resources. This mechanism is currently isolated at the BTL and BML levels in such a way as to allow other components to implement an effective scheduling policy. The BML also provides a simple round robin scheduling policy which other component may use as appropriate. For point-to-point communication the PML uses both round robin and custom scheduling based on a variety of factors. Interconnects may exhibit widely different performance characteristics which a scheduling policy should take into account. These performance characteristics are exported by each BTL and include both bandwidth and latency. During BTL initialization the BML prioritizes each BTL based on these characteristics allowing upper level components such as OB1 to choose the appropriate interconnect(s) to communicate with a peer. In addition to performance characteristics, the BML groups interconnects based on capabilities such as send/receive and RDMA. These groupings are cached on a data structure associated with each peer for efficient access. These groupings include Eager (Low Latency),Send/Receive and RDMA capable BTLs.

Source: http://www.open-mpi.org/papers/heteropar-2006/heteropar-2006-paper.pdf

4 TORQUE

TORQUE Resource Manager (Terascale Open-Source Resource and QUEue Manager)

TORQUE is an open source resource manager providing control over batch jobs and distributed compute nodes. It is a community effort based on the original *PBS project

7

and, with more than 1,200 patches, has incorporated significant advances in the areas of scalability, fault tolerance, and feature extensions contributed by NCSA, OSC, USC , the U.S. Dept of Energy, Sandia, PNNL, U of Buffalo, TeraGrid, and many other leading edge HPC organizations. This version may be freely modified and redistributed subject to the constraints of the included license.

TORQUE can integrate with Moab Workload Manager®  to improve overall utilization, scheduling and administration on a cluster. Customers who purchase Moab Workload Manager also receive free support for TORQUE.

Feature Set

TORQUE provides enhancements over standard OpenPBS in the following areas:

Fault Tolerance o Additional failure conditions checked/handled o Node health check script support

Scheduling Interface o Extended query interface providing the scheduler with additional and

more accurate information o Extended control interface allowing the scheduler increased control over

job behavior and attributes o Allows the collection of statistics for completed jobs

Scalability o Significantly improved server to MOM communication model o Ability to handle larger clusters (over 15 TF/2,500 processors) o Ability to handle larger jobs (over 2000 processors) o Ability to support larger server messages

Usability o Extensive logging additions o More human readable logging (ie no more 'error 15038 on command 42')

Source: http://www.clusterresources.com/pages/products/torque-resource-manager.php

5 OPEN PBS

OpenPBS is the original version of the Portable Batch System. It is a flexible batch queueing system developed for NASA in the early to mid-1990s. It operates on

8

networked, multi-platform UNIX environments. The menu on the right provides additional technical information.

Today's computing challenges often demand resources and skills not available within a single organization or at one location. Users need cost-effective services and tools that allow easy sharing of costly computing resources without knowing the specifics of each system environment.

Few of the features of OPEN PBS include

New comprehensive job-submission language that provides users a common syntax that is independent of hardware architectures.

Flexibility that enables IT managers to replace or add new hardware platforms without modifying job-submission mechanisms.

Automatic and efficient job placement on any hardware platform, including clusters, SMP, NUMA and new massively parallel architectures.

New node-virtualization functionality, which extends scalability and provides finer control over individual hardware components. This functionality results in higher levels of system availability, while maintaining excellent performance levels.

Tight integrations with five additional MPI libraries (IBM POE, Intel MPI, MPICH2, MPICH-GM/MX and SGI MPT), which enhance job process management.

Optimized backfilling capabilities that increase the throughput of smaller jobs without delaying the execution of larger jobs.

Guaranteed exclusivity to computational nodes, which enables maximum utilization and repeatability of the jobs.

Facilitation of the powerful SGI Altix and IBM Blue Gene feature-sets for faster, more consistent execution times, as well as better control, monitoring and cleanup of jobs.

Response to numerous customer requests for increased usability, reliability and enhanced scheduling algorithms.

Source : http://www.openpbs.org/about.html

Few useful links on OPEN PBS

User Manual : http://www.hrz.uni-dortmund.de/S2/lido_doc/openpbs.pdf

OpenPBs public home : http://www-unix.mcs.anl.gov/openpbs/

6 HOW TO DO SCRIPTS

9

Introduction

A Shell script is a program interpreted and executed by the shell, which is essentially a command line interpretor. So, think of a shell script as a list of commands that are run in sequence. This guide covers scripts created for the Borne shell and is based on Reg Quinton's Introduction to Shell Programing.

Creating a Script

Suppose you often type the command find . -name file -printand you'd rather type a simple command, say sfind fileCreate a shell script % cd ~/bin % emacs sfind % page sfind find . -name $1 -print % chmod a+x sfind % rehash % cd /usr/local/bin % sfind tcsh ./shells/tcsh

ObservationsThis quick example is far from adequate but some observations:

1. Shell scripts are simple text files created with an editor. 2. Shell scripts are marked as executeable 3. %chmod a+x sfind4. Should be located in your search path and ~/bin should be in your search path. 5. You likely need to rehash if you're a Csh (tcsh) user (but not again when you

login). 6. Arguments are passed from the command line and referenced. For example, as $1.

#!/bin/sh

All Bourne Shell scripts should begin with the sequence #!/bin/shFrom the man page for exec(2):

"On the first line of an interpreter script, following the "#!", is the name of a program which should be used to interpret the contents of the file. For instance, if the first line contains "#! /bin/sh", then the con- tents of the file are executed as a shell script."

10

You can get away without this, but you shouldn't. All good scripts state the interpretor explicitly. Long ago there was just one (the Bourne Shell) but these days there are many interpretors -- Csh, Ksh, Bash, and others.

Comments

Comments are any text beginning with the pound (#) sign. A comment can start anywhere on a line and continue until the end of the line.

Search Path

All shell scripts should include a search path specifica- tion: PATH=/usr/ucb:/usr/bin:/bin; export PATHA PATH specification is recommended -- often times a script will fail for some people because they have a different or incomplete search path.

The Bourne Shell does not export environment variables to children unless explicitly instructed to do so by using the export command.

Argument Checking

A good shell script should verify that the arguments sup- plied (if any) are correct.

if [ $# -ne 3 ]; then echo 1>&2 Usage: $0 19 Oct 91 exit 127 fiThis script requires three arguments and gripes accordingly.

Exit status

All Unix utilities should return an exit status. # is the year out of range for me?

if [ $year -lt 1901 -o $year -gt 2099 ]; then echo 1>&2 Year \"$year\" out of range exit 127 fi

etc...

# All done, exit ok

exit 0A non-zero exit status indicates an error condition of some sort while a zero exit status indicates things worked as expected.

11

On BSD systems there's been an attempt to categorize some of the more common exit status codes. See /usr/include/sysexits.h.

Using exit status

Exit codes are important for those who use your code. Many constructs test on the exit status of a command.

The conditional construct is:

if command; then command fiFor example, if tty -s; then echo Enter text end with \^D fiYour code should be written with the expectation that others will use it. Making sure you return a meaningful exit status will help.

Stdin, Stdout, Stderr

Standard input, output, and error are file descriptors 0, 1, and 2. Each has a particular role and should be used accordingly: # is the year out of range for me?

if [ $year -lt 1901 -o $year -gt 2099 ]; then echo 1>&2 Year \"$year\" out of my range exit 127 fi

etc...

# ok, you have the number of days since Jan 1, ...

case `expr $days % 7` in 0) echo Mon;; 1) echo Tue;;

etc...Error messages should appear on stderr not on stdout! Output should appear on stdout. As for input/output dialogue: # give the fellow a chance to quit

12

if tty -s ; then echo This will remove all files in $* since ... echo $n Ok to procede? $c; read ans case "$ans" in n*|N*) echo File purge abandoned; exit 0 ;; esac RM="rm -rfi" else RM="rm -rf" fiNote: this code behaves differently if there's a user to communicate with (ie. if the standard input is a tty rather than a pipe, or file, or etc. See tty(1)).

Language Constructs

For loop iteration

Substitute values for variable and perform task:

for variable in word ... do command doneFor example: for i in `cat $LOGS` do mv $i $i.$TODAY cp /dev/null $i chmod 664 $i doneAlternatively you may see: for variable in word ...; do command; done

Case

Switch to statements depending on pattern match

case word in [ pattern [ | pattern ... ] ) command ;; ] ... esac

For example:

13

case "$year" in

[0-9][0-9]) year=19${year} years=`expr $year - 1901` ;; [0-9][0-9][0-9][0-9]) years=`expr $year - 1901` ;; *) echo 1>&2 Year \"$year\" out of range ... exit 127 ;; esac

Conditional Execution

Test exit status of command and branch

if command then command [ else command ] fi

For example:

if [ $# -ne 3 ]; then echo 1>&2 Usage: $0 19 Oct 91 exit 127 fi

Alternatively you may see:

if command; then command; [ else command; ] fi While/Until Iteration

Repeat task while command returns good exit status.

{while | until} command do command done

For example:

14

# for each argument mentioned, purge that directory

while [ $# -ge 1 ]; do _purge $1 shift done

Alternatively you may see:

while command; do command; done Variables

Variables are sequences of letters, digits, or underscores beginning with a letter or underscore. To get the contents of a variable you must prepend the name with a $.

Numeric variables (eg. like $1, etc.) are positional vari- ables for argument communication.

o Variable Assignment

Assign a value to a variable by variable=value. For example:

PATH=/usr/ucb:/usr/bin:/bin; export PATH

or

TODAY=`(set \`date\`; echo $1)`o Exporting Variables

Variables are not exported to children unless explicitly marked.

# We MUST have a DISPLAY environment variable

if [ "$DISPLAY" = "" ]; then if tty -s ; then echo "DISPLAY (`hostname`:0.0)? \c"; read DISPLAY fi if [ "$DISPLAY" = "" ]; then DISPLAY=`hostname`:0.0 fi export DISPLAY fi

Likewise, for variables like the PRINTER which you want hon- ored by lpr(1). From a user's .profile:

15

PRINTER=PostScript; export PRINTER

Note: that the Cshell exports all environment variables.

o Referencing Variables

Use $variable (or, if necessary, ${variable}) to reference the value.

# Most user's have a /bin of their own

if [ "$USER" != "root" ]; then PATH=$HOME/bin:$PATH else PATH=/etc:/usr/etc:$PATH fi

The braces are required for concatenation constructs.

$p_01

The value of the variable "p_01".

${p}_01

The value of the variable "p" with "_01" pasted onto the end.

o Conditional Reference o ${variable-word}

If the variable has been set, use it's value, else use word.

POSTSCRIPT=${POSTSCRIPT-PostScript};export POSTSCRIPT

${variable:-word}

If the variable has been set and is not null, use it's value, else use word.

These are useful constructions for honoring the user envi- ronment. Ie. the user of the script can override variable assignments. Cf. programs like lpr(1) honor the PRINTER environment variable, you can do the same trick with your shell scripts.

${variable:?word}

16

If variable is set use it's value, else print out word and exit. Useful for bailing out.

o Arguments

Command line arguments to shell scripts are positional vari- ables:

$0, $1, ...

The command and arguments. With $0 the command and the rest the arguments.

$#

The number of arguments.

$*, $@

All the arguments as a blank separated string. Watch out for "$*" vs. "$@". And, some commands:

shift

Shift the postional variables down one and decrement number of arguments.

set arg arg ...

Set the positional variables to the argument list.

Command line parsing uses shift:

# parse argument list

while [ $# -ge 1 ]; do case $1 in process arguments... esac shift done

A use of the set command:

# figure out what day it is

17

TODAY=`(set \`date\`; echo $1)`

cd $SPOOL

for i in `cat $LOGS` do mv $i $i.$TODAY cp /dev/null $i chmod 664 $i done

o Special Variables o $$

Current process id. This is very useful for constructing temporary files.

tmp=/tmp/cal0$$ trap "rm -f $tmp /tmp/cal1$$ /tmp/cal2$$" trap exit 1 2 13 15 /usr/lib/calprog >$tmp

$?

The exit status of the last command.

$command # Run target file if no errors and ...

if [ $? -eq 0 ] then etc... fi

Quotes/Special Characters

Special characters to terminate words:

; & ( ) | ^ < > new-line space tab

These are for command sequences, background jobs, etc. To quote any of these use a backslash (\) or bracket with quote marks ("" or '').

Single Quotes

Within single quotes all characters are quoted -- including the backslash. The result is one word.

18

grep :${gid}: /etc/group | awk -F: '{print $1}'

Double Quotes

Within double quotes you have variable subsitution (ie. the dollar sign is interpreted) but no file name generation (ie. * and ? are quoted). The result is one word.

if [ ! "${parent}" ]; then parent=${people}/${group}/${user} fi

Back Quotes

Back quotes mean run the command and substitute the output.

if [ "`echo -n`" = "-n" ]; then n="" c="\c"

else n="-n" c=""

fi

and

TODAY=`(set \`date\`; echo $1)` Functions

Functions are a powerful feature that aren't used often enough. Syntax is

name () { commands }

For example:

# Purge a directory

_purge() { # there had better be a directory

19

if [ ! -d $1 ]; then echo $1: No such directory 1>&2 return fi

etc... }

Within a function the positional parmeters $0, $1, etc. are the arguments to the function (not the arguments to the script).

Within a function use return instead of exit.

Functions are good for encapsulations. You can pipe, redi- rect input, etc. to functions. For example:

# deal with a file, add people one at a time

do_file() { while parse_one

etc... }

etc...

# take standard input (or a specified file) and do it.

if [ "$1" != "" ]; then cat $1 | do_file else do_file fi

Sourcing commands

You can execute shell scripts from within shell scripts. A couple of choices:

sh command

This runs the shell script as a separate shell. For example, on Sun machines in /etc/rc:

sh /etc/rc.local

20

. command

This runs the shell script from within the current shell script. For example:

# Read in configuration information . /etc/hostconfig

What are the virtues of each? What's the difference? The second form is useful for configuration files where environment variable are set for the script. For example:

for HOST in $HOSTS; do

# is there a config file for this host?

if [ -r ${BACKUPHOME}/${HOST} ]; then. ${BACKUPHOME}/${HOST} fi etc...

Using configuration files in this manner makes it possible to write scripts that are automatically tailored for differ- ent situations.

Some Tricks

Test

The most powerful command is test(1).

if test expression; then

etc...

and (note the matching bracket argument)

if [ expression ]; then

etc...

On System V machines this is a builtin (check out the com- mand /bin/test).

On BSD systems (like the Suns) compare the command /usr/bin/test with /usr/bin/[.

Useful expressions are:

test { -w, -r, -x, -s, ... } filename

21

is file writeable, readable, executeable, empty, etc?

test n1 { -eq, -ne, -gt, ... } n2

are numbers equal, not equal, greater than, etc.?

test s1 { =, != } s2

Are strings the same or different?

test cond1 { -o, -a } cond2

Binary or; binary and; use ! for unary negation.

For example

if [ $year -lt 1901 -o $year -gt 2099 ]; then echo 1>&2 Year \"$year\" out of range exit 127 fi

Learn this command inside out! It does a lot for you.

String matching

The test command provides limited string matching tests. A more powerful trick is to match strings with the case switch.

# parse argument list

while [ $# -ge 1 ]; do case $1 in -c*) rate=`echo $1 | cut -c3-`;; -c) shift; rate=$1 ;; -p*) prefix=`echo $1 | cut -c3-`;; -p) shift; prefix=$1 ;; -*) echo $Usage; exit 1 ;; *) disks=$*; break ;; esac

shift

done

Of course getopt would work much better.

22

SysV vs BSD echo

On BSD systems to get a prompt you'd say:

echo -n Ok to procede?; read ans

On SysV systems you'd say:

echo Ok to procede? \c; read ans

In an effort to produce portable code we've been using:

# figure out what kind of echo to use

if [ "`echo -n`" = "-n" ]; then n=""; c="\c" else n="-n"; c="" fi

etc...

echo $n Ok to procede? $c; read ans Is there a person?

The Unix tradition is that programs should execute as qui- etly as possible. Especially for pipelines, cron jobs, etc.

User prompts aren't required if there's no user.

# If there's a person out there, prod him a bit.

if tty -s; then echo Enter text end with \^D fi

The tradition also extends to output.

# If the output is to a terminal, be verbose

if tty -s <&1; then verbose=true else verbose=false fi

23

Beware: just because stdin is a tty that doesn't mean that stdout is too. User prompts should be directed to the user terminal.

# If there's a person out there, prod him a bit.

if tty -s; then echo Enter text end with \^D >&0 fi

Have you ever had a program stop waiting for keyboard input when the output is directed elsewhere?

Creating Input

We're familiar with redirecting input. For example:

# take standard input (or a specified file) and do it.

if [ "$1" != "" ]; then cat $1 | do_file else do_file fi

alternatively, redirection from a file:

# take standard input (or a specified file) and do it.

if [ "$1" != "" ]; then do_file < $1 else do_file fi

You can also construct files on the fly.

rmail bsmtp < rcpt to: data from: <$1@newshost.uwo.ca> to: Subject: Signon $2

subscribe $2 Usenet Feeder at UWO . quit

24

EOF

Note: that variables are expanded in the input.

String Manipulations

One of the more common things you'll need to do is parse strings. Some tricks

TIME=`date | cut -c12-19`

TIME=`date | sed 's/.* .* .* \(.*\) .* .*/\1/'`

TIME=`date | awk '{print $4}'`

TIME=`set \`date\`; echo $4`

TIME=`date | (read u v w x y z; echo $x)`

With some care, redefining the input field separators can help.

#!/bin/sh # convert IP number to in-addr.arpa name

name() { set `IFS=".";echo $1` echo $4.$3.$2.$1.in-addr.arpa }

if [ $# -ne 1 ]; then echo 1>&2 Usage: bynum IP-address exit 127 fi

add=`name $1`

nslookup < < EOF | grep "$add" | sed 's/.*= //' set type=any $add EOF

Pattern Matching

There are two kinds of pattern matching available, matching from the left and matching from the right. The operators, with their functions.

25

Operator Function Example${foo#t*is} deletes the shortest

possible match from the left

export $foo="this is a test"echo ${foo#t*is}is a test

${foo##t*is} deletes the longest possible match from the left

export $foo="this is a test"echo ${foo#t*is}a test

${foo%t*st} deletes the shortest possible match from the right

export $foo="this is a test" echo ${foo%t*st}this is a

${foo%%t*st} deletes the longest possible match from the right

export $foo="this is a test"echo ${foo#t*is} 

Substitution

Another kind of variable mangling you might want to employ is substitution. There are four substitution operators in Bash.

Operator Function Example${foo:-bar} If $foo exists and is not null,

return $foo. If it doesn't exist, or is null, return bar.

export foo=""echo ${foo:-one}oneecho $foo 

${foo:=bar} If $foo exists and is not null, return $foo. If it doesn't exist, or is null, set $foo to bar and return bar

export foo=""echo ${foo:=one}oneecho $fooone

${foo:+bar} If $foo exists and is not null, return bar. If it doesn't exist, or is null, return a null.

export foo="this is a test" echo ${foo:+bar}bar

${foo:?"error message"}

If $foo exists and isn't null, return it's value. If it doesn't exist, or is null, print the error message. If no error message

export foo="one"for i in foo bar baz; do

26

is given, it prints parameter null or not set.Note: In a non-interactive shell, this will abort the current script. In an interactive shell, this will just print the error message.

eval echo \${$foo:?}onebash: bar: parameter null or not setbash: baz: parameter null or not set

Debugging

The shell has a number of flags that make debugging easier:

sh -n command

Read the shell script but don't execute the commands. IE. check syntax.

sh -x command

Display commands and arguments as they're executed. In a lot of my shell scripts you'll see

# Uncomment the next line for testing # set -x

Source: http://linux.dbw.org/shellscript_howto.html

7 JOBCONTROL

Job control, a feature standardized by POSIX.1 and mandated by many standards, allows a single terminal to run multiple jobs. Each job is a group of one or more processes, usually connected by pipes. Mechanisms are provided to move jobs between the foreground and the background and to prevent background jobs from accessing the terminal.

Jobs can either be in the foreground or in the background. There can only be one job in the foreground at a time. The foreground job is the job with which you interact--it receives input from the keyboard and sends output to your screen, unless, of course, you have redirected input or output, as described starting on page  ). On the other hand, jobs in the background do not receive input from the terminal--in general, they run along quietly without the need for interaction.

27

Some jobs take a long time to finish and don't do anything interesting while they are running. Compiling programs is one such job, as is compressing a large file. There's no reason why you should sit around being bored while these jobs complete their tasks; just run them in the background. While jobs run in the background, you are free to run other programs.

  Jobs may also be suspended. A suspended job is a job that is temporarily stopped. After you suspend a job, you can tell the job to continue in the foreground or the background as needed. Resuming a suspended job does not change the state of the job in any way--the job continues to run where it left off.

    Suspending a job is not equal to interrupting a job. When you interrupt a running process (by pressing the interrupt key, which is usually Ctrl-C) , the process is killed, for good. Once the job is killed, there's no hope of resuming it. You'll must run the command again. Also, some programs trap the interrupt, so that pressing Ctrl-C won't immediately kill the job. This is to let the program perform any necessary cleanup operations before exiting. In fact, some programs don't let you kill them with an interrupt at all.

Job Control Commands :

Nice & Renice :

The nice command is used to alter an initial job priority. On Linux systems this is fairly simple: the lower the nice command the higher the priority. The range on a Linux system is -20 (being the highest) to 19 (the lowest). Using nice is pretty simple. Let's say we want to make sure that a compile and install for fetchmail has a pretty high priority. We might do the following:

nice -n 5 make

We have lowered the nice number and raised the job priority initially for this task.

The renice command is used to alter the nice value of a job after it has been started. It is important to note that only root may alter the nice value of jobs it does not own, and non-root users may only alter their nice values between 0 and 20 (the former is obviously so users may not tamper with other users while the latter protects the privileged processes of the system). An example of using renice on a single process might look like so:

renice 5 -p 10023

where we lower the nice value to 5 of PID 10023. The renice command can also affect entire group of processes as well. For instance, let us say we wanted all of jdoe user's processes to have a nice value of 12. The syntax on Linux would be:

renice 12 -u jdoe

28

While running a command (job) you can pause/suspend it with ctrl-z and kill it with ctrl-c.

While running a job you can Shortcut

suspend a job ctrl -z

terminate a job ctrl -c

Running a job in background : The & is used to put the job in the background so my terminal is free for me to keep working. The system will inform me when the jobs are done. It is important not to logout while background jobs are running. On most systems you will see a warning message stating that there are running jobs. If this is ignored, the jobs will be terminated.

jobs

Lists the jobs running in the background, giving the job number. It is all too easy to confuse jobs and processes. Certain builtins, such as kill, disown, and wait accept either a job number or a process number as an argument. The fg, bg and jobs commands accept only a job number.

bash$ sleep 100 &[1] 1384

bash $ jobs[1]+ Running sleep 100 &

"1" is the job number (jobs are maintained by the current shell), and "1384" is the process number (processes are maintained by the system). To kill this job/process, either a kill %1 or a kill 1384 works.

disown

Remove job(s) from the shell's table of active jobs.

fg, bg

29

The fg command switches a job running in the background into the foreground. The bg command restarts a suspended job, and runs it in the background. If no job number is specified, then the fg or bg command acts upon the currently running job.

wait

Stop script execution until all jobs running in background have terminated, or until the job number or process ID specified as an option terminates. Returns the exit status of waited-for command.

You may use the wait command to prevent a script from exiting before a background job finishes executing

Optionally, wait can take a job identifier as an argument, for example, wait%1 or wait $PPID.

suspend

This has a similar effect to Control-Z, but it suspends the shell (the shell's parent process should resume it at an appropriate time).

logout

Exit a login shell, optionally specifying an exit status.

times

Gives statistics on the system time used in executing commands, in the following form:

0m0.020s 0m0.020s

This capability is of very limited value, since it is uncommon to profile and benchmark shell scripts.

kill

Forcibly terminate a process by sending it an appropriate terminate signal

builtin

Invoking builtin BUILTIN_COMMAND runs the command "BUILTIN_COMMAND" as a shell builtin, temporarily disabling both functions and external system commands with the same name.

enable

30

This either enables or disables a shell builtin command. As an example, enable -n kill disables the shell builtin kill, so that when Bash subsequently encounters kill, it invokes /bin/kill.

The -a option to enable lists all the shell builtins, indicating whether or not they are enabled. The -f filename option lets enable load a builtin as a shared library (DLL) module from a properly compiled object file.

autoload

This is a port to Bash of the ksh autoloader. With autoload in place, a function with an "autoload" declaration will load from an external file at its first invocation. This saves system resources.

Note that autoload is not a part of the core Bash installation. It needs to be loaded in with enable -f

Links : http://www.linuxplanet.com/linuxplanet/tutorials/2116/1/ http://www.linux.com/guides/abs-guide/x6689.shtml http://linuxreviews.org/beginner/jobs/

8 HOW TO MANAGE EXPERIMENTS

I found this link, I don’t know how far this will help you. Please let me know if this doesn’t work

http://mylinuxsaga.blogspot.com/search?q=LINUX

31