8/3/2019 Unix Mod 2
1/42
1
Problem-Solving Approaches in UNIX
When u use a computer, usually have a goal in mind ?
To learn more about the system, play a game,write a program ,edit a letter, communicate with e-mail, chat etc An operating system offers you a work environment for satisfying your goals.UNIX has dozens of command that are not standard equipment on most operating systems.
Example
There are commands that count words, search for strings, sorting etc. UNIX system offers fourapproaches to problem solving. They range from the very simple to the complex.
1. Simplest approach is to find a UNIX command that directly solves a problem
Eg:to sort a file use sort command
2. If u cant find a UNIX command that solves the problem directly look for an appropriate
combination of commands.
E.g.: If u wish to extract from the file lmcst those lines containing mca and havearrange those lines alphabetically ,
grep mca lmcst | sort
3. Shell programming (shell scripts) this is a file of unix commands that is used as a programto tell the shell what to do. Also UNIX allows you to use loops etc.
4. sometimes shell script wont solve a problem or else will solve too slowly. The obvious choiceon a UNIX system is the C language. Since UNIX is written in C and the UNIX system calls
and library functions are available to C programs as ordinary C functions.
4 -problem solving approaches.
Single command solutions
combined command solutions
shell scripts solutions
C program solutions
8/3/2019 Unix Mod 2
2/42
2
Using single UNIX command to solve problems
Find a command that solves your problem and use it.
You may not know about command and how to use it.
One solution to these problem is to browse through UNIX manual.
The UNIX commandline
ls -l -s /usr/black/flag
Command name options arguments
The simplest command lines consists just of a single command name,
such as date or who.
You will follow the command name with one or more arguments. Thesearguments provide additional information for the command.
These arguments are filenames or directories.
cd /usr/include
another type of argument is the option which modifies the behavior of a
command.
Sort lmcst // will sort the file contents alphabetically
sort -r lmcst // sort in reverse order
the options are marked with a hyphen .
User can use several options like ls -lsa
if you want to extend the command to next line use backslash and hitreturn.
rajesh@ubuntu:~$ ls -i ff \
> sort294680 ff 296300 sort
The symbol > indicates that the command is being continued to the nextline
8/3/2019 Unix Mod 2
3/42
3
Standard input, Standard output, Standard error
In a UNIX system, a program, including many UNIX commands, is normally connectedautomatically to three file called standard input, standard output, standard error.
If the program does not explicitly request other files, it gets its input from the
standard input, sends output to standard output and sends any error message to the
standard error. Unless other wise specified all three files are default to your terminal.(devices are treated as files).
ie, it takes input from your key board and sends output to your screen.
Unix allows you to change the standard input and output temporarily by using pipesand redirection.
Unixcommand
Standard input(S.I.)
Standard o/p (S.O)
Standard error (S.E)
Input comes
in here
Error message sent
out here
o/p sent out
here
COMPOUND UNIX COMMANDS
When a single UNIX command does not suffice to solve a problem or do a task, try joining
commands together. The chief tools are for this are redirection and the pipe.
Redirection
Redirection changes the assignments for standard input and standard output. The > operator
makes the filename following the operator become the new standard output, and the cat < f
lmcst
mcet
cet
mit
Both produce the same result.
cat like many file-reading programs can read the standard input and a file input during the execution of
the command.
The standard method is to use a hyphen instead of a filename to stand for standard input.
rajesh@linux-1dzg:~> cat -
Haihai
rajesh@linux-1dzg:~> cat f -
lmcst
mcet
cetmit
unix programming
unix programming
cat concatenated the two input. Since the file name came first, it was printed first. Then thekeyboard input, represented by the hyphen was printed.$ sort < f > f3
input taken from file f and the out put to be routed to f3
$ cat f3
cetlmcst
mcet
mit
Another redirection operator is append operator >>cat f f1 f2 f3 >> f4
8/3/2019 Unix Mod 2
5/42
5
Pipes
Redirection lets us connect programs to files.
Pipe facility lets us connect programs to other programs
A pipe | makes the output of one program the input of another.
grep fresnobillsdue | sort
the output of grep becomes the input to sort.
grep
sort And fresno
but fresno
billsdue
|
SI
Keyboard
SO
SI
SO
SE SE
Input from pipe can be combined with input from files. Use a hypen symbol , so that thecommand recognize that there will be a standard input.
Grep fresno billsdue | sort olddue > newdue
The output from grep becomes the standard input to sort, mean while sort opens the file olddue.The contents of this file are sorted together with the grep output and the sort output is redirectedto the file newdue. Redirection routes output to files while pipes route output to other programs.
The TEE
The tee commands reads the standard input and sends it onto the standard output. It alsoredirects a copy of what it has read into the file of your choice
cat part1 part2 | tee total | lpr
The o/p of cat becomes the standard input of tee It is sent on through another pipe to the lpr lineprinter command. Meanwhile a copy is stored in the file total.
Append to a file using tee (use -a option) : cat part3 part4 | tee -a total | lpr
Filters
A filter is a program that can take a flow of data form standard input, process (filter) it and sendthe result on to standard output.
Examples
8/3/2019 Unix Mod 2
6/42
6
cat, sort, grep
How can you tell if a command is a filter?
Read its description , if it can take input from the standard input and if it send its output to thestandard output, then it is filter.
SHELL SCRIPTS
If user (you) wants to use a sequence of command , he can commit them permanently to a file.Then the shell read the file and follow the script of commands in it, such a file is called a shellscript.
Example
echo Here is the date and time
date
To run this script, give the filename as argument to the sh (for shell) command.
$sh example
Here is the date and time
Tue Mar 3 19:43:52 PST 2009
Here sh command creates a new child shell process. This new shell reads the file example,
executes the command it finds and dies, returning control to the original shell.
User can make the example file executable, by giving executable (e) permission to the file
example.
$Chmod u+x example
$example
Here is the date and time
Tue Mar 3 19:43:52 PST 2009
Features of shell scripts
1. shell scripts can take arguments, just like most commands. These arguments (parameters) canbe symbolically represented within the script. So we can pass on values, filenames.
2. The Shell allows you to create shell variables and to assign values to them. They can be usedby shell scripts for many purpose.
3. the shell has a collection of metacharacters (?,*,[]), characters assigned special meanings thatgreatly expand the applicability of shell scripts.
8/3/2019 Unix Mod 2
7/42
7
4. the shell provides control structures such as if .....else constructs and for loops that let youimplement programming language approaches.
A script built up on standard UNIX commands is highly portable from one UNIX system toanother. Shell scripts take up a little storage. The drawback of shell script is that it is relativelyslow .
C programs for solving problems
Why C ?
Because it is the language that most (90%) of UNIX is written in, so the interface between C andUNIX is straightforward and natural. System calls, the subroutines used by kernel they are Cfunctions. UNIX has about 60 such system functions and they can be used in C programs justlike any other C functions.
C is a compiled language. Means you use one of the system editors to write a program. Then usubmit the program to the C compiler, which converts it to an executable program in machinelanguage. Then u can run the program.
Procedures
1. use an editor, such as vi, ex or ed to write the program. File name should end in .C (to identifyit is a c program to the compiler
example (mca.c)
main(){printf (mca @ lmcst \n);
}2. submit the file to cc, the C compiler
cc mca.c // if ur program is ok, the compiled version is placed in a file called a.out (bydefault)
3. to run the program, type a.out
$ a.out
mca @ lmcst
$
4. to change the name of the a.out file
mv a.out mca
now just type mca to run the program
8/3/2019 Unix Mod 2
8/42
8
Building your own command library of programs
.profile this file is created for you when ur UNIX account is set up.
When u login UNIX scans this file for instructions on how to handle your account.
(BSD versions use .login)
one line in the file is
PATH=.: /bin: /usr/bin (PATH is one of the buitl-in shell variable)
this provides the list of directories the shell uses to find command names.
: is used to separate one directory name from the next.
Change the PATH definition
PATH=.: /usr/raj/rvs: /bin: /usr/bin
once you created this setup you can use your commands the same as you use UNIX commands.
Any changes made to .profile do not go into effect until the system reads the file again.
Working with the Bourne shell
shell is the interface between you and the system. Acts as a command interpreter.
There are two shells in widespread use
the most widespread shell program is the Bourne shell, this is developed by Stephen bourne of
bell labs for bell release UNIX. Another shell is csh it is developed by berkely , developed forBSD version of UNIX. Applications developed on Bourne shell can be used for all UNIXsystems.
Shell's filename expansion capabilities
File name expansion is the ability to expand a shorthand notation for a group of filenames to acomplete set of explicit filenames.
Mr. Mister
Shell metacharacters
The shell uses a set of special symbols called metacharacters to let you produce pattern ortemplates to match various classes of filenames.
1.The ? - stands for any single character.
d?g would match three-character filename starting with d and
ending with g.
8/3/2019 Unix Mod 2
9/42
9
possible matches include dig,dog,d4g, d_8
2. The * - stands for any combination of zero or more characters.
(this will not match a period at the beginning of a filename , to protect .profile)
s*n match any filenames, regardless of length, that starts with s and ends with n.
possible matches sn,sun,soon,sudden,s747n,s1.bin
3. The [] - lets u represent a restricted range of single characters
enclosed within the brackets.
You can use [acdAC] or indicate a range by using a hyphen
[a-m].
d[io]g dig , dog (diog is worng)
each bracket pair represents just one character place in a name
Metacharacters can be used in conjunction with one another.
[A-Z]*[0-9] matches any filename starting with a capital letter and ending with a digit.
Example :
rajesh@ubuntu:~$ ls [A-Z]*[0-9]
B2 C8 CZ29 f1 f3 f4
rajesh@ubuntu:~$ ls [A-Z][0-9]
B2 C8 f1 f3 f4
*[0-9][0-9] matches any filename ending with two digits.
*[10-99] wont work because for a single bracket pair matches only one bracket.
Shell variable
A variable is a name associated with a value. Shell variable comes in two varieties
1. those created and maintained by the UNIX
2. those created by the user
User-Created shell variable
name=value (name of shell variable = value of shell variable)
8/3/2019 Unix Mod 2
10/42
10
pet=bulldog (pet is a shell variable name = its value is the string bulldog
shell variables are string variables
weight=156
the value of weight is a three-character string consisting of the digit characters 1 5 6.
the value of weight is not the number 156. the shell can if necessary interpret a digit-stringnumerically.
One special string called null string, it is a string consists of no characters.
Vacuous= ' '
notmuch=
idea=
No space are used when we assign a value to a variable.
You can use letters, digits and underscore character in variable names.
Example
figaro, HO, name2, go_where (LEGAL VARIABLES)
big-foot=squasher (illegal)
squasher=big-foot ( legal)
To find out the value of a shell variable, we can use echo command
$echo pet
pet
a variable name preceded by a $ as an argument to echo, the value of the variable is echoed.
$echo $pet
bulldog
// pet is a variable name
// $pet is the variable's value.
$echo the value of pet is $pet
the value of pet is bulldog.
rajesh@ubuntu:~$ pet=
rajesh@ubuntu:~$ echo the name of pet is $pet
8/3/2019 Unix Mod 2
11/42
11
the name of pet is
System shell variables
UNIX maintains its own set of shell variables.
Typesetto see what your system is using.
The set command makes the shell print out the variables it knows, (system and user variables).
HOME=/usr/mca08PATH=.:/bin: /usr/bin
Left of equal represents the variable name and to the right is the variables value. System shellvariables are in upper case.Some standard shell variable
EXINIT initialization instructions for the ex and vi editors
HOME this is set to the pathname of your home directory
IFS (internal field separator) this is set to a list of the characters that are used to separate wordsin a command line. This list consists of the space character, the tab character and new linecharacter.
LOGNAME The users login name
MAIL this variables value is the name of the directory in which electronic mail addressed toyou is placed. The shell checks the contents of the directory often.
PATH this names the directories which the shell will search to find commands that you use. Acolon is used to separate the directory names. The directories are searched in the order given.
PS1 (Prompt String 1) : this is the symbol used as your prompt ( normally it is set to $, we canredefine it by assigning a new value.
PS1=#
PS2 (Prompt String 2) : this prompt is used when UNIX thinks you have started a new line without finishing a command.
Example
you can continue a line by using a backslash before hitting the return key.
rajesh@rajesh-laptop:~$ echo hello ho\> w are youhello how are yourajesh@rajesh-laptop:~$
TERM this identifies the kind of terminal you use.
8/3/2019 Unix Mod 2
12/42
12
Many UNIX command use built-in shell variable.
cd $HOME
we can use system shell variables in programs also
Local and Global shell Variables : export
ordinarily a shell variable is a local variable. That is , it is known only to the shell that created it.When u start a new shell that shell is born ignorant of the old shell's variables.so each shellvariable are private (local) to itself.
rajesh@rajesh-laptop:~$ cat=katerajesh@rajesh-laptop:~$ echo $catkaterajesh@rajesh-laptop:~$ sh$ echo $cat$ rajesh@rajesh-laptop:~$ echo $cat
kate
If we want the new shell to know the old's shells variable use the export command. Any shellvariable used as an argument for this command will have copies of the variable and its valuepresented to all shells descending from it.this kind of variable is termed global.
rajesh@rajesh-laptop:~$ export catrajesh@rajesh-laptop:~$ cat=lmcstrajesh@rajesh-laptop:~$ echo $catlmcstrajesh@rajesh-laptop:~$ sh
$ echo $catlmcst$ rajesh@rajesh-laptop:~$ echo $catlmcstrajesh@rajesh-laptop:~$ sh$ export j$ j=john$ echo $jjohn$ rajesh@rajesh-laptop:~$ echo $j
Variables can be exported to subshells, but they cannot be exported back to parent shells.Exported shells are not completely global variables
SHELL SCRIPTS
two types of approaches
8/3/2019 Unix Mod 2
13/42
13
1. write an interactive script that requests the information it needs from the user. Prompts theuser fro input, perhaps asking a question, and then reads the users answer. (shell is an interactiveprogram it prompts with $ and reads the commands you type)
2. the script accepts arguments from from the command line.
cat sortfile
this method is more powerful and flexible.
Interactive Shell Scripts
this is based on using UNIX shell commands read and echo.
echo Dear User , what is ur name\?read nameecho ur name is $name
here we use the backslash to remove the special pattern-matching property of the question markcharacter. (this is called escaping or quoting the character)
place this program in a file glad and execute.
Echo enter three nosread a b c
the first variable gets the first value u type and so onif there are more words than variable,all the left over words get assigned to the last variable.
echo enter 3 nos
read a b cecho $aecho $becho $cecho $a - $c
enter 3 nos1 2 3 4 5 6123 4 5 6
1 - 3 4 5 6
If there are more variables than words, the unused variables are not assigned values. Read is abuilt in shell command., it is part of the sh program
program for copying a file
8/3/2019 Unix Mod 2
14/42
14
echo what file to copyread orgecho what name u desire for copyread copycp $org $copy
echo $org is copied into $copyShell scripts with argumentspositional parameters
convert pounds yen pesos
(shell script - $0) $1 $2 $3
refer the first argument as $1 , the second argument as $2, the special symbol $0 stands for thename of the command itself. $* stands for all the arguments from $1 on up ($1,$2,$3 ) thenumbered arguments are referred to as positional parameters $#, which represents the number ofarguments .
rajesh@rajesh-laptop:~/Desktop$ sh show a b c
I used show commandthses are my arguments a b cthe total arguments are 3the first arg is athe second arg is bthe third arg is c
cp $1 $HOME/Music/$1.svrajesh@rajesh-laptop:~/Desktop$ sh new\ file hh
rajesh@rajesh-laptop:~/Desktop$ ls /home/rajesh/Music/
hh.sv
we can use $1 more than once.
Write a shell script to make a shell script into an execuitable.
chmod u+x $1 // save this in a file execfile
rajesh@rajesh-laptop:~/Desktop$ sh execfile hh
copy this execfile in /usr/bin then u just type execfile hh
using set
set is used to assign values to positional parameters.
rajesh@rajesh-laptop:~/Desktop$ set hai how r u
rajesh@rajesh-laptop:~/Desktop$ echo $1 $2 $3 $4
8/3/2019 Unix Mod 2
15/42
15
hai how r u
Set is used in conjunction with the metacharacter ` (backquote) when a command is enclosed inbackquotes, the command is replaced by the output it produces, this process is called commandsubstitution.
rajesh@rajesh-laptop:~/Desktop$ echo date
date
rajesh@rajesh-laptop:~/Desktop$ echo `date`
Thu Mar 12 02:51:04 IST 2009
rajesh@rajesh-laptop:~/Desktop$ who am irajesh pts/0 2009-03-12 01:43 (:0.0)
rajesh@rajesh-laptop:~/Desktop$ set who am i
rajesh@rajesh-laptop:~/Desktop$ echo $1 $2 $3who am i
rajesh@rajesh-laptop:~/Desktop$ set `who am i`
rajesh@rajesh-laptop:~/Desktop$ echo $1 $2 $3 $4 $5rajesh pts/0 2009-03-12 01:43 (:0.0)
Write a shell script that print outs date information in this order : time , day of week, day
number, month, year.rajesh@rajesh-laptop:~/Desktop$ set `date`
rajesh@rajesh-laptop:~/Desktop$ echo `date`Thu Mar 12 03:28:25 IST 2009rajesh@rajesh-laptop:~/Desktop$ echo $4 $1 $3 $2 $603:28:32 Thu 12 Mar 2009
Eg)echo do not panic! All is wellname=$1echo you know $name, that are one of my favourite users.set `date`echo It is $1 $3 th $4 , $6echo relax $name
echo the time is now $4 $5
do not panic! All is wellyou know rajesh, that are one of my favourite users.It is Thu 12 th 03:38:07 , 2009relax rajeshthe time is now 03:38:07 IST
8/3/2019 Unix Mod 2
16/42
16
eg)set `who`who | grep -i $1echo $1
if [ $1 = rajesh ]then
echo the user $1 logged infi
rajesh@rajesh-laptop:~/Desktop$ sh userrajesh tty7 2009-03-12 03:59 (:0)rajesh pts/0 2009-03-12 04:03 (:0.0)rajeshthe user rajesh logged in
Looping and making choice
what truly make the shell a programming language is its ability to control the flow of a program
for loopcasewhileuntilif constructions
THE for LOOPa loop is a programming device that lets you cycle through the same steps
several times.
The general form of the Bourne shell for loop
for variable-name in value1 value2
do
commands
done
---------------------------------------------------------------------------------------------------
Eg)for i in 1 2 3 4doecho $i
done1234
8/3/2019 Unix Mod 2
17/42
17
Eg)for i in 1 2 3 4; do echo $i; done
Eg)for i in 1 2 3 4> do> echo $i> done
Eg)for i in fly spider frog> do> echo i know $i> echo swallowed a $i> donei know flyswallowed a flyi know spiderswallowed a spider
i know frogswallowed a frog
Printing file contents one word per line
for i in `cat glad`do
echo $idone
For loop with command line argument
for i in $*doecho $idonerajesh@rajesh-laptop:~$ sh hh 12 34 5 6123456
Eg)for idoecho $idone
eg)for i in $@doecho $i
8/3/2019 Unix Mod 2
18/42
18
done
Generating Values for aforLoop
1. for i in2. for i in $*
for i3. for file in *docat $filedone
4. take from shell variable // interactive program
echo Enter ur nameread name
for i in $namedo
echo $idone5. take value from the output of a command
rajesh@rajesh-laptop:~$ for i in `cat jj`doecho $idoneoutputecho
Enterurnamereadnameforecho$idone
Some times for loop is invaluable when you want to process several files in some manner.
To do a word count on several files
wc ex hh jj
for loop is invaluable at this time (wc command has a loop mechanism built in).
-m, --chars
8/3/2019 Unix Mod 2
19/42
19
print the character counts
-l, --lines
print the newline counts
-w, --words
print the word counts
To sort multiplte files
for i in $*do
sort $idone
Word-seeking program
#wordseek-searches files for words (to add comments)source=$1shift //drops the original $1 from the argument list, leaving $* equalto the rest of the argumentsfor i in cat $source` //getting words from source filedogrep $i $* // Si is the word , S* are the filenamesdonesh wordseek music ch[1-3]
CHOICE-MAKING : THE case STATEMENT
Syntax
case value inchoice1) commands;choice2) commands;....esac
Choice1, choice2 are labels that identify potential choices of action.One use for the case statement is to set up menus
Exampleecho " BASIC UNIX COMMANDS "echo "--------------------------"echo "command option"echo "__________________________"
8/3/2019 Unix Mod 2
20/42
20
echo "LIST 1"echo "DATE 2"echo "CURRENT DIRECTORY 3"echo "EXIT 4"echo "--------------------------"
echo "Enter your option.."read optioncase $option in
1)ls ;;2) who am i ;;
p|3) pwd ;;quit|4) exit ;;*) echo invalid choice ;;
esac
We can use command-line argument for case statement
echo " BASIC UNIX COMMANDS "echo "--------------------------"echo "command option"echo "__________________________"echo "LIST 1"echo "DATE 2"echo "CURRENT DIRECTORY 3"echo "EXIT 4"echo "--------------------------"echo "Enter your option.."read option
case $1 in1)ls ;;2) who am i ;;
p|3) pwd ;;quit|4) exit ;;*) echo invalid choice ;;
esac
PatternMatchingwe can use ? And [] metacharacters.
case $1 in[aeiouAEIOU]*) echo the given word starting with vowel ;;??) echo a two letter word ;;*) echo I dont know ;;esac
$ sh vowels earsthe given word starting with vowel
8/3/2019 Unix Mod 2
21/42
21
$ sh vowels EYEthe given word starting with vowel$ sh vowels ZZa two letter word$ sh vowels 1323
I dont know
Make the file execuitable and run$vowels eye
We can use case `pwd` in
using for and case together
for file in *
docase $file in*.c) echo it is a c file $file;;*.cc) echo it is a c++ file $file;;*.txt) echo it is a text file $file;;esacdone
# sh forcaseit is a c file a.cit is a c++ file a.cc
it is a text file a.txtit is a c file b.cit is a text file new file.txt
CONDITIONALLOOPING : while AND untilUNIX offers three loop structures : for, while and until.The general form of while loop is
while control command
do
commands
done
echo enter the choice //while loop like C or PASCAL iswrong cannt use < > == (conditions)
read iwhile [ $i -gt 10 ]doecho $i no greater than 10done
8/3/2019 Unix Mod 2
22/42
22
If the control command returns a zero exit status , the command between do and done areexecuted.
Word=$1 //whilegrepshift
while grep $word $1doshiftdoneecho first file not containing $word
First variable word gets the value arrahman. The shift command shfits ch1 to $1 the whilestatement become
while grep arrahman ch1do
shiftdone
If the grep command reports success, then the instructions between do and done are executed., ifit reports failure the loop is terminated.Suppose arrahman is found in ch1, then the grep reports success, and the following shiftcommand is executed , this shifts ch2 to $1 and the next loop becomes
while grep arrahman ch2doshiftdone
if the grep fails there is no arrahman in ch2 the loop terminatesthe script move onto next lineecho first file not containing $word
The Until loop
The until loop is executed until the test condition succeeds. A while loop runs until a controlcommand fails The until loop runs until a control command succeeds
until control commanddo
commandsdone
until ls | grep $1 > /dev/nulldo
sleep 30 //loop continues until the file is created.
8/3/2019 Unix Mod 2
23/42
23
doneecho file $1 is here
THEifSTATEMENTif statement is concerned with the status of commands
if control commandthen
commandsfi
first the command following the if is executed. If the command is successful (exit status 0), thenthe commands between the then and the fi are performed.
if cmp -s $1 $2 // -s suppresses the usual output only thestatus of the command (0 if the file match, 1 if they dont (fail)
thenrm $2fi
The if .....then.....else structureif control commandthen
commandselse
commandsfi
eg)if cmp -s $1 $2thenrm $2
echo the $2 was duplicate and removedelseecho the files $1 and $2 are not the samefi
If then .....elif then elseif control command 1then
commandselif control command2then
commandselse
commandsfi
8/3/2019 Unix Mod 2
24/42
24
eg)if [ $leap -eq 1 -a $date -gt 29 ]then
echo "date not valid"elif [ $leap -eq 0 -a $date -gt 28 ]
then echo "date not valid"echo "enter correct date and try"
elseecho "Date valid"
fi
THE test COMMAND$ dog=nikkit //set dog variable to nikkitest $dog=nikki // test for equality
the test command produces no o/p, but it produces a 0 exit status if the test is successful. And a 1exit status if it fails.
$? check status of test command
Eg)$dog=nikki$test $dog = nikki$echo $?0$test $dog = pluto$echo $?
1Test string1 = string2 (There is a space between two strings)
Eg)Echo what is the best operating system
if test $1 = UNIX
then
echo I agree
else
echo I never heard of $1
fi
eg) Echo i am thinking of a number between 1 and 50
read guessuntil test $guess = 33
do
echo nope guess again
read guess
done
echo well done // test
8/3/2019 Unix Mod 2
25/42
25
File checking
the test command has several options for checking the status of a file
-r file - true if the file exists and readable
-w file - true if the file exists and writable
-f file - true if the file exists and is not a directory
-d file - true if the file exists and is a directory
-s file - true if the file exists and has a size greater than zero
-x for identifying executable files
$ ls -l ex
-rw-r--r-- 1 rajesh rajesh 8 2009-03-15 20:17 ex
$ test -r ex
$ echo $?
0
$ test -x ex
$ echo $?
1
String Checkingcomparing two strings
string1 = string2 // true if the strings are samestring1 != string2 // true if the strings differ
the space are obligatorytest $ans = hoop //valid formtest $ans= hoop //invalid form
three options deal with the size of a string-n string // true if the string has nonzero lenght-z string // true if the string has zero lengthstring // true if the string is not the null string
An undefined variable , appears to be a null string$ frog=bumpy$ test $frog$ echo $?0
$ test $frag //frag not defined$ echo $?1
We can use test to make a while loop act like a foor loopwhile test $1do.....
8/3/2019 Unix Mod 2
26/42
26
shiftdone
Numerical Comparisonsthses options compare integers
n1 -eq n2 // true if the integers are equaln1 -ne n2 // true if the integers are not equaln1 -gt n2 // true if n1 is greater than n2n1 -lt n2 // true if n1 is less than n2n1 -ge n2 // true if n1 is greater than or equal to n2n1 -le n2 // true if n1 is less than or equal to n2
eg)if test $# -ne 2thenecho this pgm needs two args
$ sh numeric 1this pgm needs two args$ sh numeric 1 2
5 = 5 // true , the character string 5 is same as 55 = 05 // false , the character string '5' is different from '05'5 -eq 05 // true , 5 and 05 have the same numerical value
$ test 5 = 5$ echo $?0
$ test 5 = 05$ echo $?1
$ test 5 -eq 05$ echo $?0
Logical operations
test -rw flees is it possible ?
!-a binary and operator-o binary or operatortest ! -r film // true if the filem is not an existing, readable filetest -r film -a -w film // true if the file film is readable and writable
8/3/2019 Unix Mod 2
27/42
27
test -r film -o -w film // true if the file film is readable or writable
Using parentheses for groupingtest \( -r film -a -w film \) -o \( -r flam -a -w flam \)parentheses are shell metacharacters so use the backslash
Test with [ ]echo what is ur ageread ageif [ $age -le 6 ]thenecho child---------------------------------------------------if [ $age -gt 9 -a $age -lt 20 ]
// write a shell script to check whether the given file is exists or not
if test ! -f $1thenecho file $1 not foundelse echo file foundfi
UNIX Tools : grep, sed, tr, and awkgrep a pattern matching utilitysed a stream editortr a character translation programawk a pattern matching and processing labguage
Grepgrep stands for global regular expression printer
grep pattern filenameexamplegrep mca lmcstgrep seraches the string mca in the file lmcst and print the line containing the string
$ ls | grep aExamplesgladha
Templates
Common options for grep-v prints those lines that do not match-c prints only a count of the number of matching lines-l prints only the names of files with matching lines-n prints the line number with each output line-s prints no o/p, just return status
8/3/2019 Unix Mod 2
28/42
28
$ grep -v mca lmcstmc$ grep -c mca lmcst3
$ grep -l mca lmcstlmcst
$ cat lmcstmca is a pg coursemcamaster of computer applications - mcamc
Grep accepts multiple filenamegrep mca lmcst lmcst1 lmcst2
If the pattern contains a space or tab character in it, it is necessary toput them on in a single or double quotes.grep 'san francisco' hotelsorgrep san francisco hotels
Greps pattern-matching$ grep ar c?c1:haiarrahmanc2:arrahman oscar
grep 'm.a' lmcst // will count the space alsoa period represents any one character
eg)rajesh@rajesh-laptop:~$ cat > grepspatespotrespitesputterthe asp took
rajesh@rajesh-laptop:~$ grep 'sp.t' grepspatespotrespitesputterthe asp took
rajesh@rajesh-laptop:~$ grep 'sp..t' grep
8/3/2019 Unix Mod 2
29/42
29
sputter
Examples
h.t // matches hit, hst, h8th[ea]t // het , hat (heat is not matched)
[A-Z][^ A-Z] // two character first is a capital letter second is not a capitalletter (examples B4,Me,S!)^The // matches any line beginning with The , caret symbol is different
inside and outside the bracket\.$ // matches any line ending with periodho*t // matches ht(zero occurrences of the preceding
pattern),hot,hoot,hooot and so on
Some typical pattern-matching problem1. counting blank lines to count the number of blank lines in a file, first we use greps -coption, which counts the number of matche and Second we need a pattern to match a blank line
^$ // matches an empty line$ grep -c ^$ lmcst2when we write a file , we start the first line by hitting the space bar. In this case there will besome space character present but invisibleto find the spaces we can use *$a blank line can also contain a tab character , to look for lines containing just tabs, we can usethis pattern:*$ //matches blank line with any number of tabsIn C programs we can find the tab character by using the pattern^\t*$
we can combine these patterns to catch lines that may have blank lines,tabs or both^[ \t] * $to count the blank lines (with or without spaces or tabs) in the file text isgrep -c '^[ \t]*$' text
2. finding two patterns in proper sequenceto find all lines containing Mickey and Donald in that order, but which may have additionalcharacter in between them.
then Mickey tossed the gem to Donald
grep 'Mickey.*Donald' hero
rajesh@rajesh-laptop:~$ cat > spthe Mickey tossed the gem to DonaldMickey and Donald are friends
rajesh@rajesh-laptop:~$ grep 'Mickey.*Donald' spthe Mickey tossed the gem to Donald
8/3/2019 Unix Mod 2
30/42
8/3/2019 Unix Mod 2
31/42
31
Two other additions to egrep are the special + and ?, both are similiar to *, which stood for zero or more occurrences of the preceding character.
The + stands for 1 or more occurrences and the ? Stands for zero or one occurrence.
ho*t matches ht,hot,hoot etcho+t matches hot,hoot etcho?t matches ht and hotparentheses can be used with *, + , ?(ho)*t matches t,hot,hohot etcpattern using the egrep scheme of special characters are known asfull regualrexpressions.
CHARACTERTRANSLITERATION : tr
tr transliterates characters$cat philoI compute, therefore I am.
$ tr eiou ~#$% < philoI c$mp%t~, th~r~f$r~ I am.
tr is a pure filter, it takes input from the standard input. Small letters (i) and capital letters (I) aredifferent, above example I is not affected by I$ tr eiou EU < philoI cUmpUtE, thErEfUrE I am
Special Notations
you can indicate ranges with tr by using a hyphentr A-Z a-z //converts upper case to lower caseortr [a-z].[A-Z]
Optionstr command has three options-d uses just one list of characters, and it causes them to delete$tr -d eiou < philoI cmpt, thrfr I am-c option causes the substitutions to affect every character but those in the first list.
$ tr eiou * < philoI c*mp*t*, th*r*f*r* I am
$ tr -c eiou '*' < philo***o**u*e****e*e*o*e******$
-c option replaces all the space,punctuation with * andnew line character at the end
8/3/2019 Unix Mod 2
32/42
32
finally -s option compresses strings of repeated ouput characters at the end of the line, ie strings**** become *.$ tr -cs eiou '*' < philo*o*u*e*e*e*o*e*$
rajesh@rajesh-laptop:~$ cat > philoI compute , thereforerajesh@rajesh-laptop:~$ cat philo | tr "[a-z]" "[A-Z]" > ph1rajesh@rajesh-laptop:~$ cat ph1I COMPUTE , THEREFORE
$ tr eiou EU < philoI cUmpUtE, thErEfUrE I am
the last character from the second list is used for all the left over characters from the first list.
THE STREAMEDITOR: sed
to edit a file called philo u typeed philothe editor then creates a temporary copy of philo in the /tmp directory, all the changes made tothe copy, when we give the w command, the temporary file is copied into the original, the netresult is that the editor alters the original file.Sed is a UNIX filter, it can take input from a file, but its output is sent to the standard output, theoriginal file is not altered.The o/p from sed can be redirected to a new file if u wish to save it.Or it can be piped along to some other program as input.
What exactly does sed do?It takes a line from input, applies an editing command to it, and prints the result. Then it takes thenext line and repeats the process until the whole file has flowed through sed.Basic usage of sed issed 'editing command' filenameif no filename is given, then sed reads the standard input, this means it can receive input from apipe.If more than one filename is given, then they are processed in turn.
Editing commands for sed
it has two parts : a line specifier and an editing instructionif there are no line specifier then the command is applied to every line.There are two approaches to specifying lines.One way to specify a line is to give a line number or a range of line numbers.
3 // third line5,8 // lines 5 through 810, $ // lines 10 through the last line
8/3/2019 Unix Mod 2
33/42
33
second way to specify a line for sed is to provide a matching pattern enclosed in slashes.(patterns are the same as those used by grep)
Examples/huge/ all lines containing the string huge
/[Dd]o/ all lines containing strings Do or do/^[ \t]*$/ all blank lines.The two forms of line identification (line numbering and pattern matching) can be combined.1,/gossip/ From line 1 to the first mention of gossip! - to indicate the command is to be applied to lines that dont match the line specifier.
The editing instructions for seds substitutep printd deleteq quit
editing instructions immediately follows the line specifiersed '10q' text // quit after reading line 10sed prints each line as it is processed, this command prints the first 10 lines of a file.Sed '$p' text //prints the last linesed '/^[ \t]*$/d' text // delete all blank lines from textSed '/value/!d' text // delete every line in text not containing the string
valueThe sed substitution command (s)the general form is s/oldstring/newstringthis replaces the first occurrence of oldstring on a line with newstring.If u want to replace all occurrences on a given line we tack on a g (for global) at the end of thecommand.Sed 's/Michael/Michele/g' text
$ cat > textjohn is coming from tvmjohn and his friend john
$ sed 's/john/hary/' texthary is coming from tvmhary and his friend john
$ sed 's/john/hary/g' texthary is coming from tvmhary and his friend hary
Sed '1, /Jessup/s/Michael/Michele/g' text // replaces Michael with Michele up through thefirst line containing Jessup
8/3/2019 Unix Mod 2
34/42
34
special characters also can be used with sed
$ sed 's/john/"hary"/g' text"hary" is coming from tvm"hary" and his friend "hary"
$ sed 's/john/"john"/g' text"john" is coming from tvm"john" and his friend "john"
$ sed 's/john/"&"/g' text"john" is coming from tvm"john" and his friend "john"
$ sed 's/john/&! &!/g' textjohn! john! is coming from tvm
john! john! and his friend john! john!
Sed tags
you can enclose part of oldstring between \( and \)the enclosed part is said to be tagged.U can then refer in newstring to the tagged part as \1if u tag more than one part in old string, u can call the second tagged part \2, and so ons/ \ (john\) and \(hary\) / \2 and \1/ - replaces the phrase
john and hary with hary and john.\1 stands for john (first tagged string)\2 stands for hary (second tagged string)$ sed 's/ \(john\) is \(coming\) / \2 is \1/g' textcoming is johnfrom tvmjohn and his friend john
To change all occurrences ofoccurence and Occurence to occurence and Occurence s/ [Oo]ccurence/ [Oo]ccurence/g - this dont workthe pattern [Oo] can only be used in the left-hand pattern (oldstring)the only special symbols usable on the right side (new string) are & and \1 family. So patterns using [], . , * cant be used on the right side.S/ \ ([Oo]ccur \) ence / \ 1rence / g
$ sed 's/john/hary/g> s/hary/mary/g> s/mary/bony/g' textbony is coming from tvmbony and his friend bony-------------------------------------------------------$ sed 's/john/hary/gs/is/was/g' text
8/3/2019 Unix Mod 2
35/42
35
hary was coming from tvmhary and hwas friend hary
Options for sed
-n , -f , -e
-n option suppresses the automatic printing of each processed line,Sed -n '$p' listit print just the last line of the file list
the -f option followed by a space, then a filename, This option causessed to take its editing commands from the named filesed -f edit sermonsed to look in the file edit for commands to apply to the file sermon.
Sed -f name -f form memothis command first use the editing command from name and then from memo
the -e optionthe -e option lets you mix a command-line editing command with commands in a file.Sed -e 's/Boston/Wasco/g' -f form trav.txtBoston will replaced by Wasco then the editing command in form to be applied, and then on tothe next line.
A pattern-scanning and processing language : awk
The awk command (named after and by its creators : Aho, Weinberger, and Kernighan)awk command combines pattern matching, comparison making, numerical operations and c-like
programming features into one program.A basic command line looks likeawk awk-command filenameeach awk command has a line-specifier (a pattern) part followed by an instruction (an action).Awk lets you identify particular fields within a linea field is a squence of characters containing no blanks & separated by blanks from other fields.
Each word on this line is a separate field.Example of an awk command line.awk '/poultry/ { print $1, $3, $6}' suppliesthe pattern part is /poultry/.
The action part is enclosed in braces; it requests that the first, third and sixth fields of thematching line be printed.The $1 stands for first field, $3 stands for third fieldif there is no pattern portion the action is applied to every line of input.The following prints the first & second fields of the table supplies in reversed order.awk ' { print $2, $1}' supplies
8/3/2019 Unix Mod 2
36/42
36
If there is no action portion, the default action is to print the entire line which matches thepatternawk '/poultry/ ' suppliesmore than one command can be included by starting a new line for each command$ awk '/scandel/ { print $0}
/rumour/ { print $0} ' rag$0 stands for entire line
Eg)$ awk '/mca/ {print $0}> /john/ {print $0} ' lmcstmca hai haimcahai john mcahai john mca
eg)$ awk '/mca/ {print $1}
/john/ {print $0} ' lmcstmcamcahaihai john mca
If u have many commands, it is usually more convenient to place them in a file and the awk getits commands from the file.This is done by using the -f option.awk -f cawk ragthis command tells awk to apply the command in the cawk file to the input from the rag file.Fields and Recordsan awk program processes its input a record at a time.By default a record is just a line.The new line character is the default record separator.When ever awk encounters a newline character it starts a new record.
Within a record, the input is divided into fields.A field is a series of characters delineated by a fields separator, by default the field separator is ablank, that is a space or tab.
Examplemc |mcstBtech CSE lmcstlmcst is the second field in the first recordBtech is the first field in the second record.Awk has some built-in variables dealing with the fields and records.NR represents the current record no. NF represents the number of fields in the current record.
In the above example after reading first line NR =1 and NF = 2
8/3/2019 Unix Mod 2
37/42
37
the variable FS stands for field separator, by default this is blank (ie a space or tab character)we can reset this field separator to another value.This can be done by using -Fawk -F: '{print $1, $2}' lmcstthis prints the first two fields of the lmcst file, using a colon as a field separator for the input.
The variable RS represents the record separator
Pattern-Matching Metacharacters for awk
\ Match following character literally^ the beginning of a line$ the end of a line. any one character[list] any one character in list[^list] any one character not in list* zero or more occurrences of the preceding character
or group
+ one or more occurrences of the preceding characteror group? zero or more occurrences of the preceding character
or group(exp) is expexp1 \ exp2 means exp1 or exp2
Patterns in awk
the awk command accepts several forms of pattern-matchingFull Regular Expressionsawk recognizes the same full regular expressions that grep does.Regular Expressions and Tablesthe entire line is called $0.the ~ symbols indicate a matchthe !~ symbols indicate no matchto find all lines in which the second field matches flubbo$2 ~ /Flubbo/
$2 ~ /^flubbo$/this matches only lines in which the second field is flubbo.$2 !~ /^Noxin$/matches all lines for which the second field is not NoxinRelational Expressionsthis compare two expressionsawk command recognizes the same relational operators that C does.
$ cat > compare12 34 4412 55 23$ awk '$2 > $3' compare
8/3/2019 Unix Mod 2
38/42
38
12 55 23 // prints the line $2 greater than $3$ awk '$2 < $3' compare12 34 44 // // prints the line $2 less than $3awk also recognizes the arithmetic operators +, - , * , / and %.$ awk '$2 >=2 *$3' compare
12 55 23 // print line for which $2 greater than or equal to two times of $3
Relational operators for awk
Operator Meaning< less than greater than>= greater than or equal to== equal to!= not equal to
L
ogical operators for awk&& and|| or! not
$ awk '$2 > $1 && $3 > $1' compare12 33 4411 23 45!($2 > $1 && $3 > $1) // selects records for which the condition is not true.Rangesawk allows you to specify a range of lines.NR == 20, NR == 40 // the records 20 through 40 NR == 1, /[D]dismay/ // record 1 through the first appearance of
Dismay or dismay
Special Patternsawk has two special patterns BEGIN and ENDBEGIN is used to label actions to be done before any records are read.BEGIN {RS = :; FS = /}this command establishes the colon as the record separator and sets up slash as the fieldseparator.The END label is used to identify actions that are done after all the input has been read.
Actionsthe awk actions can print, it also has built in functions, ability to create variables , arrays,arithmetic operations and flow control statements.Printingto use print statement$ awk '{print $1}' lmcstmca
8/3/2019 Unix Mod 2
39/42
39
mcahaihai
eg)$ awk '{print $1, $2}' lmcst
mca haimcahaihai john
eg)$ awk '{print $1 + $2}' compare4534
Eg)$ awk '{print}' compare12 33 44
11 23 45
Eg)$ awk '{print $0, "total is: " , $1 + $2 + $3}' compare12 33 44 total is: 8911 23 45 total is: 79
Eg)$ awk '{printf "first val is: %6d second val is %d: total is: $%d\n", $1, $2, $1 + $2}'comparefirst val is: 12 second val is 33: total is: $45first val is: 11 second val is 23: total is: $34
Print Redirectionawk can redirect print output to files$ awk '{print $1 >> "col1"; print $2 >> "col2" }' comparethe first field in each line is appended to a file named col1the second field will go to the file named col2Variablesawk lets us create a variable, the value of a variable can be numeric or a string.We want to count the number of lines that begin with mca or with hai$ awk '$1 == "mca" || $1 == "hai" { count = count + 1 } END { print count, "lines"}' lmcst4 lines
We can use { count +=1} and {count++}+= operator adds whatever is to its right to whatever is to its left++ operator increments its variable by 1you have a table of numbers and wish to total the third column$ cat compare12 33 4411 23 45$ awk '{sum += $3}
8/3/2019 Unix Mod 2
40/42
40
> END {print sum}' compare89
Arrays$ awk '{sum[1] += $1; sum[2] += $2; sum[3] = sum[1] + sum[2]}
END {print sum[1], sum[2], sum[3]}' compare23 56 79this creates an array sum with three elements sim[1], sum[2], sum[3]the array index can be a non-null value (including a string)we can use tom and dick instead of 1 and 2to find how often the words philodox and snoach occur in a file/philodox/ { count[philodox]++}/snoach/ { count[snoach]++}END { print count[philidox], count[snoach]}
Flow Control
awk uses the if , if-else, for and while loops of CTheifstatement{if ( $1 < $2)print $1, $2elseprint $2, $1 }$ awk '{ if ($1 < $2)print $1, $2elseprint $2, $1 }' compare12 3411 552 3
Eg)$ awk '{ if ($1 < $2)print $1, $2}' compare12 34
2 3
The forstatementthe for statement has this form :for (initialize;test; update)
commandthe initialize expression is performed once, when the loop starts up.Then the test expression is examined, if it is true, the command is executed, if it is false the loopterminated.Each time loop is executed the update statement gets executed.
8/3/2019 Unix Mod 2
41/42
41
A non-C form of for loop
for ( i in array)
command
array is an array name , the command is performed for each array
element.
{ count[$1]++}
END { for (i in count)
print i, length(i), count[i] }
$ awk -f awkex lmcst
hai 3 2
mca 3 2
lmcst 5 1lm 2 1
awkex
$ cat lmcstmca hai haimca
haihai john mca
lmlmcst
The while loop
the while loop has the form:
while ( test )command // the command get executed as long as the test is true.
Prgm this prints out the fields in a line until it hits the end of the line orthe word private.
cat > samplejohns hobby is swimmingben's hobbies are fishingprivate sewing toe-pickingjillians hobbies are volleyball
{ field = 1while ( field
8/3/2019 Unix Mod 2
42/42
42
awk -f awkex sample
johns hobby is swimmingben's hobbies are fishingjillians hobbies are volleyball
the program uses the variable field to keep track of the field number.The while loop ends when it reaches the last field number (field == NF)or when the value of the field ($field) is private.
Jumps : continue, break, next and exit
break causes the loop to break off and terminatecontinue causes loop to skip to the beginning of the next iterationnext causes awk to skip to the next record, restarting the action there.exit takes you to the end of the input, but does not skip over an END statement.
Sprintf
sprintf function lets you format various expressions into a string.It uses the formatting conventions of C's printf()Ex:$1 has the value Dan and $3 has the value 65 thenlabel = sprintf(%s of the %d kg class, $1,$3)Results in the variable label having the valueDan of the 65 kg class