sed

Class Issues

• vSphere Issues– root only until lab 3

sed

• sed = stream editor• Allows you make changes to text files on

the fly on a line by line basis• Basic syntax is:sed ‘s/pattern/replacement/’ filename

• Works on files, default output is to the screen

Basic Examples

• Replace:– sed ‘s/1/ONE/g’ file.txt (would

replace the first 1 with ONE in file.txt)• Swap:– sed 'y/abc/ABC' file.txt (replace a

with A, b with B, c with C)• Show line numbers then lines:

– sed ‘=‘ file.txt• Lowercase: sed 's/[A-Z]/\l&/‘ file.txt

Special Replacements• \L - turn the replacement to lowercase until a \U

or \E is found,

• \l - turn the next character to lowercase

• \U - turn the replacement to uppercase until a \L or \E is found

• \u - turn the next character to uppercase,

• \E - stop case conversion started by \L or \U.

Remember the g?

• First Character Lower Case:– sed 's/[A-Z]/\l&/' addresses.txt

• All Characters Lower Case:– sed 's/[A-Z]/\l&/g' addresses.txt

• 5th Character Lower Case:– sed 's/[A-Z]/\l&/5' addresses.txt

Useful sed command line

• -f means read sed commands from file– sed –f mysed.txt <file list>

Gotchas

• Don’t do:– sed 's/[A-Z]/\l&/[3-5]' addresses.txt > addresses.txt

• Create a new file and mv it back.

awk

awk

• Programming language for data streams; usually files or subsets of files

• Developed at Bell Labs in the ’70’s by: Alfred Aho, Peter Weinberger, and Brian Kernighan

• Like sed but does the following as well:– specify particular fields of a matching line to

manipulate and/or output– use variables to store values temporarily– perform calculations on those variables.

How to run?

• Usually:awk ‘<awk commands>’ files to parse

Example (a cat replacement):

awk ‘{print $0}’ *

Fields in awk

• By default lines are delimited by white space: space characters or tabs

• $0 – whole line• $1 – is first field in line• $2 – is 2nd field in line• NF – number of fields on line

• No error reported if field doesn’t exist, simply outputs a blank line

Basic Pattern Matching

/pattern/ { do something }

Example:

/A/ { print “This line has an A”}

• No pattern means do the work on every line

Multiple patterns

• Just list them:awk ‘/A/ { print “this line has an A”}/B/ { print “this line has a B”}{ print “this gets printed always” }

‘ my_file

Don’t forget the closing ‘

Forming Patterns

• Can use most of the extended regular expressions, except {}

• Can do the relational expressions from other languages: <, >, ==, !=, <=, >=

Comparisons

• Numeric example:ls -l | awk '$5 >= 512 {print $0}'

• String example:ls -l | awk '$6 == “Sep” {print $0}'

Variables

• Variables are allowed:ls -l | awk '$5 >= 512 {size += $5}‘

• Variables don’t have to be initialized• However, we won’t ever see size output!

BEGIN/END

• BEGIN – run before any lines are read• END – run after every line has been read• You can have one, both, neither• Example:ls -l | awk '$5 >= 512 {size += $5}

END { print size}‘

Begin Example

awk ‘BEGIN { printf(“I’m pretending to be cat\n”) }

{ print $0}’ my_file• printf is C/Java like

Useful BEGIN command

• Remember BEGIN goes before reading of lines begins

• Want the first line of your file as a header? Use getline, but no pattern matching or processing of the lines read

awk ‘BEGIN { getline; print $0}’ my_file

Useful awk variables

• FILENAME – current file being parsed• NF – number of fields on the current line• FNR – current record in the file (usually

means current line)• NR – current record in the stream• There are many, many others as well,

read the man page

Standalone awk scripts• Add the following to top of file of awk commands: #!/usr/bin/awk –f• Don’t forget to chmod +x • Example:#!/usr/bin/awk -f/^.\// { dots += 1 }/^$/ { blank += 1 }/^l/ { sym += 1 }/^d/ { dir += 1 }/^-/ { file += 1 }{ total += 1 }END { printf("Dots=%d\nBlank Lines=%d\nSym Links=%d\

nDirectories=%d\nFiles=%d\nTotal=%d\n",dots, blank, sym, dir, file, total)}

Intro to Shell Scripting

Shell Script 101• List of commands interpreted by a shell

cd /var/tmpmkdir bokscd boksgzip -dc /var/tmp/boks_install.tar.gz | tar xf -

./install.jvss0001cd /var/tmprm -r boks*cd /etc/rc2.dmv S99boksm _S99boksm

Why need?

• Add users• List bad perms• Parsing log files• A menu system for users• Startup of system (.bashrc is a script)• etc• etc

Tests (if then else)

if [ $count –ne 0 ]thenavg=$((sum/count))

elseavg=0

fi

Looping (for)

for var in $@doecho $var

done

Variables in Scripts

I=0J=$IE=$(($C+$D))X=$(($X+1))

Notice the lack of spaces!

Input/Output/Assignment

• Input– read

• Output– echo– printf

Comparisions

• Numeric–-eq, -ne, -lt, -le, -gt, -ge

Strings==, !=• And is &&• Or is ||

Comparisions II

• File testing:– -d – is the item a directory?– -e – does the file already exist?– -f – does the file exist and is it a regular file?– -h (or –L) – is the item a symbolic link?

(current user)– -r – does the file exist and is it readable?– -w – does the file exist and is it writable?– -x – does the file exist and is it executable?

Sample Comparisons

• [ -f my_file ] (don’t forget spaces)• [ $x –ge 0 ]• [ $1 = start ]• [ “$1” = “” ] – empty string test!• [ “$1x” = “x” ] – better empty string test!

Command Line Parameters

• Accessed via $# parameters– $0 is script name– $1, $2 … $9 = parameters on the command

line– $@ or $* are all parameters– $# is total number passed

• Assignments say receive as parameter, assume it’s this one!

Accessing Parameters#!/bin/bashecho The command name is: $0.echo The number of command line arguments passed as parameters are $#.echo Yet another way to show total number: $*.echo The value of the command line arguments are: $1 $2 $3 $4 $5 $6 $7 $8 $9.echo Another way to display values of all of the arguments: $@.exit 0

Debugging?

• Set -vx shows info• -v = verbose• -x = lines after run