Tool Building 1: Shell CS 360 tool1. Page 2 tools1 CS 360, WSU Vancouver What is Systems Programming? (again) hardware drivers OS services system programs

Tool Building 1: Shell

CS 360

tool1

Page 2 tools1 CS 360, WSU Vancouver

What is Systems Programming? (again)

hardware

driversOS services

system programsapplications

interfaces

users

impl

emen

tatio

n la

yers

• implementing• building• Testing• Installing• deploying

System programs typically create environments

for applications


Some Tools That System Programmers Create For a software engineering team:

organize versions of product sources package designs & sources for peer review build the product test product ....

For a web site operator: update site per new look and prices ensure file security in place ...

For a LAN administrator add a new hire to all address books install new software in all workstations ...

Such tools manipulatefiles and directories


What Does the Shell Provide? Interactive interface to all system capabilities

navigate & manipulate file system invoke system commands & programs customize your environment

Programming environment for system programmers automate above interactions glue together smaller programs prototype solutions


Software Structure of Unix, Again

Programs:executable files numerous & specific

Shell:command line environment few & powerful

users

shell

programs

unixlibraries

unixkernel


Shell Examples Find words in a file

C program: 50+ lines of code Shell: 4 lines

Find most popular word C program: 75+ lines of code Shell: 2 lines

Outline a directory structure C program alone: 100+ lines of code Shell plus C program: 30 lines

Clone a directory structure C program: 150+ lines of code Shell: 1 line

Shell can reduce code bulk dramatically


Use C and Shell Together

C programming = data structures + algorithms

Shell programming = file system + utilities


The Shell is a Scripting Language

A professional uses both language types


Which Shell?

Unix architecture allows arbitrary shells The most popular have evolved and become quite powerful Always avoid religious wars and become productive!

Bourne shell "SH"

C shell "CSH"

Korn shell "KSH"

GNU Born Again shell "BASH"

editors "EMACS" & "VI"

our choice


Basic Shell Concepts

Command Lines Pipes Useful Utilities


Canonical Shell Command Format

command [ -switch ] … [ argument ] ...

• file name: ../bin/doit•built-in verb: cd ..

• simple tokens: grep -e abc ../src/foo.c ../src/bar.c

•wildcards: grep -e abc ../src/*.c•quoted text: grep -e "abc xyz” ../src/*.c•shell variables grep -e “$text” ../src/*.c•quoted text: grep -e ‘$text’ ../src/*.c

your program doesn'tsee the quotes

or wildcardsor variables

• single letters: ls -a -1•combined single letters: ls -a1•values: cc -o foobar foobar.c•strange and wonderful: find . -print


Wildcards Expand into a List of Pathnamesls ../src/*.c ls ../src/abc.c ../src/xyz.c

These are the most common pattern characters: * matches any string of characters, including none ? matches any single character [a-z] matches any character in range ~ your home directory ~bob Bob's home directory

Notes: invoked programs don't see the patterns

but, if no matches, pattern is passed to program * doesn't match names beginning with period when first in a pattern tilde interpreted only if first character in token


Echo Command Writes To Stdout% echo hello therehello there% echo

% echo "this is one argument"this is one argument% echo "*"*% echo *abc def xyz% echo $PWD/home/langd% echo "$PWD"/home/langd% echo '$PWD'$PWD% x="some value"% echo "$x" $x '$x'some value some value $x

$"'

note wellthe rules for:


Many Utilities are Filters (Text In/Text Out)

stdin stdoutprogram

Shell always provides your program with three open file descriptors: 0: "standard input" 1: "standard output" 2: "standard error" (= stdout usually)

Examples: translate lower to upper case: tr "a-z" "A-Z" find lines with "abc": grep -e "abc" delete lines with "abc": sed -e "/abc/d" extract lines starting with "xyz": sed -n -e "/^xyz/p" sort into ascending order: sort copy input to output: cat write to output: echo "hello, world!"


Use Redirection to Control Stdout and Stdin Stdout:

Your terminal unless redirected:

Stdin: Your terminal unless redirected:

Stderr: Your terminal unless redirected:

% grep -e "abc" > my-output-file% grep -e "abc" >> my-output-file

% grep -e "abc" < my-input-file

% grep -e "abc" 2> my-error-file% grep -e "abc" 2> /dev/null% grep -e "abc" 2>&1% echo "error!" >&2


Connect Filters into Pipelines

% tr " " "\n"

% tr " " "\n" | sort

% tr " " "\n" | sort | uniq

words

sorted words

unique words

tr " " "\n"stdin sort uniq stdout

Notes: thus stdout of one program is the stdin for the following program programs in pipe execute concurrently (no temp files used) can combine with redirection

% tr " " "\n" | sort | uniq < gettysburg > result

|


Use Grep & Sed to Find or Change Lines Grep finds lines:

grep [ -e pattern ] … display lines in stdin that contain any of the patterns

Sed rewrites lines: sed [ -e action ] … copy stdin to stdout, doing all actions on each line

Some Sed actions: /pattern/d delete line if contains pattern /pattern/r file read (copy to stdout) file after the line the pattern

matches s/pattern/replace/g substitute replace for all occurrences of pattern

Grep & Sed patterns: . match any character [a-z] match any character from "a" to "z" [^a-z] match any character except from "a" to "z" ^ match beginning of line $ match end of line (how match a dollar sign?) pattern* match zero or more occurrences of pattern

(match is greedy -- try some experiments)


Some Sed Examples Replace each occurrence of MYNAME with “Dick":

Delete all lines exactly 1 character long:

% sed -e "s/MYNAME/Dick/g"

% sed -e '/^.$/d'

why the single quotes?


Use Scripts to Package Commands for Reuse Count number of files in directory "foo" that have "doug" in name:

A script named "hunt" to do the same thing:

A new version that accepts two parameters:

Consider these invocations of the script: new-hunt foo doug betty new-hunt foo new-hunt

% ls -1 foo | grep -e "doug" | wc -l3

ls -1 foo | grep -e "doug" | wc -l

ls -1 $1 | grep -e $2 | wc -l

% new-hunt foo "doug"3

(notice the quietness!)% hunt3

new-hunt

hunt


Scripts & Paths Shell finds programs by looking for executable files in the path:

File is executable if has "x" mode:

A script is a text file that is executable:

The script is just a sequence of shell commands in sequence can have parameters supports if, while, and other programming constructs

% echo $PATH.:/home/langd/bin:/net/class/cs360/bin:/usr/bin

% ls -l my-program-rwxrwxr-x ~~~~~~~~~~ my-program

% vi my-script% chmod +x my-script

("whence foo"= where foowill be found)


Use the Shell Utility Toolkit Basic line oriented processing:

grep find patterns sed edit lines cut extract columns sort sort lines into specified order uniq display unique lines in a sorted file tail, head display last lines of a file, or first lines wc count words, lines, and characters

File system manipulation: cp, mv, rm, ls copy, move, delete, list files chmod, chown change file access modes, file owner find navigate directory structure testing files for attributes touch change last-modified time on a file

Miscellaneous: echo echo arguments on stdout cat copy a file or files to stdout tr translate characters date echo current date and time man see manual page on a command

Scripting Languages awk process files based on rich pattern matching & text manipulation perl do the same with more C-like facilities

documentation = man pages


Good Things to Know Editing last command:

Type ESC Then can edit recent commands using VI keystrokes (e.g.: k = go up, i = insert)

Using special keys: End terminal stdin with control-D Interrupt execution of a program with control-C

Continuing beyond one line: End with backslash to continue line Unbalanced quotes etc. also continue Prompt will change to indicate

% hunt foo "doug>


Whiteboard Exercises

Examples of scripts and pipelinesI love Unix


Some Word Scripts

This is a example docment, with some words.This is on line two (which follows line 1).This is on line three (which follows line 2.0).John siad, "Hello, world!".

test file "document1":

Note: following scripts are stored in the directory "/class/cs360/lab/tools1/words"

• extract words• find misspelled words• find most popular word


Words0:

tr " " "\n"

% words0 < document1Thisisaexampledocment,withsomewords.Thisisonlinetwo(whichfollowsline1).Thisis... more follows ...

script:

operation:

goal:• display words

preparation:

% vi words1% chmod +x words0

words0


Words1:

tr " " "\n" | sort | uniq

% words1 < document1"Hello,(which1).2.0).JohnThisadocment,examplefollowsislineonsiad,somethreetwowithwords.world!".

script:

operation:

goal:• display unique words

preparation:


words1


Words2:

tr " " "\n" | \tr "A-Z" "a-z" | \sort | \uniq

% words2 < document1"hello,(which1).2.0).adocment,examplefollowsisjohnlineonsiad,somethisthreetwowithwords.world!".

script:

operation:

goal:• display unique words• treat upper case as lower case

preparation:


words2


Words3:

tr "\$,;./!()'\"" " " | \tr -s " " "\n" | \tr "A-Z" "a-z" | \sort | \uniq

% words3 < document1

012adocmentexamplefollowshelloisjohnlineonsiadsomethisthreetwowhichwithwordsworld

script:

operation:

goal:• display unique words• treat upper case as lower case• treat punctuation as space• collapse spans of spaces

preparation:


words3


Spell:

words3 > /tmp/$$

comm -23 /tmp/$$ /usr/share/dict/words

rm /tmp/$$

% spell < document1012docmentfollowsjohnsiadwords

script:

operation:

goal:• display words that are not in dictionary

preparation:

% vi spell% chmod +x spell

spell


Popular

tr "\$,;./!()'\"" " " | \tr -s " " "\n" | \tr "A-Z" "a-z" | \sort | \uniq -c | \sort -nr | \head -1

script:

goal:• find words using words3 logic• sort words by number of occurrences• print most popular word

% popular < document14 lineoperation:

popular


Clone a Directory Structure

cp -mr $1 $2

script:

goal:• copy file or directory recursively• preserve ownership, modes, and times

% clone /home/langd /backup

operation:

preparation:

% vi clone% chmod +x clone

clone


Outline a Directory Structure

% find foo -type dfoofoo/binfoo/labfoo/lab/introfoo/lab/intro/my-directoryfoo/lab/dev1foo/lab/iofoo/demofoo/demo/afoo/demo/bfoo/hellofoo/examplefoo/example/onefoo/example/twofoo/example/two/three

"find" walks directories:

(lots of wonderful switches!)

% outline foofoo /bin /lab /intro /my-directory /dev1 /io /demo /a /b /hello /example /one /two /three

What we want:

We will create this using a small program that replaces text to left of slashes with blanks


Outline a Directory Structure

find $1 -type d | unslash

script:% outline foofoo /bin /lab /intro /my-directory /dev1 /io /demo /a /b /hello /example /one /two /three

goal:• outline a directory structure

preparation:

% vi outline% chmod +x outline% vi unslash.c% cc -o unslash unslash.c

operation:

outline


Implementation of Unslash Outline the logic:

read stdin lines– if not the first line:

overwrite with blanks all text to left of rightmost slash– write the line

return

Refine the logic: key variables:

– let "line" be an array that holds the line just read– let i index each character in line– let s index the rightmost slash or -1 if no slash

to do the overwriting:– set s by examining each character (left-to-right)– replace each line[0..s-1] with a blank

Translate logic into C next slide ...

0 1 2 3 4 5 6 7 8 9 10line before / a b / c / x y . z \0

s 5line after / x y . z \0


Code for Unslash is Simple#include "lineio.h"#define LINEMAX 1000

int main (int argc, char *argv[]) {

char line[LINEMAX]; /* input line & work area */int n; /* number of line 0, 1, ... */int i; /* examine line[i] next */int s; /* line[s] is a slash or s == -1 */

n = 0;while (getline (line)) {

/* overwrite text, if not first line */if (n > 0) {

/* set s to index of rightmost slash or to -1 */i = 0; s = -1;while (line[i] != '\0') {

if (line[i] == '/') s = i;++i;

}

/* replace line[0..s-1] with blanks */i = 0;while (i < s) {

line[i] = ' ';++i;

}}

/* write line and increment n */putline (line);++n;

}}

note how we find rightmost

slash

unslash.c


Discussion ... In this situation:

by using a Shell utility, we had an "almost" solution immediately by reformulating the problem as a filter, we could write a simple C program to

complete the solution

This is a general approach for tool building with the shell:

1. Use files and directories2. Use utilities whenever possible3. Fill gaps with small C programs4. Connect components with pipes5. Format data as text

Documents

Tool Building 1: Shell CS 360 tool1. Page 2 tools1 CS 360, WSU Vancouver What is Systems Programming? (again) hardware drivers OS services system programs