Introduction to Unix (CA263) File Processing (continued) By Tariq Ibn Aziz

Introduction to Unix (CA263)

File Processing (continued)

By

Tariq Ibn Aziz

Guide to UNIX Using Linux, Third Edition 2

Objectives

• Use the pipe operator to redirect the output of one command to another command

• Use the cut, paste, tr and grep command to search for a specified pattern in a file

• Use the uniq command to remove duplicate lines from a file

• Use the comm and diff commands to compare two files

• Use manipulation and transformation commands, which include sed, tr, and pr

• Design a new file-processing application by creating, testing, and running shell scripts


Advancing YourFile-Processing Skills

• Selection commands focus on extracting specific information from files


Advancing YourFile Processing Skills (continued)


Using the Selection Commands

• Using the Pipe Operator

– The pipe operator (|) redirects the output of one command to the input of another

– The pipe operator can connect several commands on the same command line


Cut Command

• To extract various column or fields of data file or the output of the command.– Cut –cchars file– cut –c5- data– It will extract characters 5 through the end of the line

of data and write the results to standard output.


Cut Example

$ whoroot console Feb 24 08:54taziz tty02 Feb 24 12:55dawn tty08 Feb 24 09:15amin tty10 Feb 24 15:35

$ who | cut –c1-8roottazizdawn amin


Cut Example


$ who | cut –c1-8 | sortamindawn roottaziz


Cut Example


$ who | cut –c10-16consoletty02tty08tty10


Cut Example


$ who | cut –c1-8,18-root Feb 24 08:54taziz Feb 24 12:55dawn Feb 24 09:15amin Feb 24 15:35


Cut CommandThe –d and –f Option

• The -d and –f option are used with cut when you have data that is delimited by a particular character. The format of the cut command is as follow:cut –ddchar –ffields file

• Where dchar is the character that delimits each field of data, and fields specifies the field to be extracted from file. Field numbers start at 1.


Cut CommandThe –d and –f Option

cut –ddchar –ffields file$cat phonebooktariq:905-3456:11 Driscoll Drkalim:205-3456:12 Driscoll Drimran:304-3456:13 Driscoll Drhasam:203-3456:14 Driscoll Dr$cut –d: –f1 phonebooktariqkalimimranHasam$


Cut Command

• Field are separated by tabs.$cat phonebookAlice Chebba 596-2015Barbara Swingle 205-9257Jeff Goldberg 295-3378Liz Stachiw 775-2298

$cut –c1-15 phonebookAlice Chebba 59Barbara SwingleJeff Goldberg 2Liz Stachiw 775$


Cut Command

• Field are separated by tabs, you should use the –f option to cut$cat phonebookAlice Chebba 596-2015Barbara Swingle 205-9257Jeff Goldberg 295-3378Liz Stachiw 775-2298

$cut –f1 phonebookAlice ChebbaBarbara SwingleJeff GoldbergLiz Stachiw$


Paste Command

• Its opposite of cut command, cut break lines apart, and paste command put lines together.$ cat namesTubsEmanuelLucyRalph$ cat numbers(307)542-5356(212)954-3456(212)MH6-9959(212)BE0-7741

$ paste names numbersTubs (307)542-5356Emanuel (212)954-3456Lucy (212)MH6-9959Ralph (212)BE0-7741


Paste Command The –d option

• Its opposite of cut command, cut break lines apart, and paste command put lines together.

$ paste –d'+‘ names numbersTubs+(307)542-5356Emanuel+(212)954-3456Lucy+(212)MH6-9959Ralph+(212)BE0-7741


Paste Command The –s option

• The –s option tells paste to paste together lines separated by tab from the same file, not from alternate files.

$ paste –s namesTubs Emanuel Lucy Ralph Fred

$ paste –d' ' –s -Tubs Emanuel Lucy Ralph Fred


tr command

• The tr filter is used to translate characters from standard output. The general form of the command is:tr from-char to-char$ cat introThe UNIX operating system was pioneered by Ken$ tr e x < introThx UNIX opxrating system was pionxxrxd by Kxn


tr command

• You can translate colon into tab character to produce more readable output, by simply tacking tr command to the end of the pipeline.$ cut –d: -f1,6 /etc/passwdroot:/cron:/bin:/uucp:/usr/spool/uucpasg:/$ cut –d: -f1,6 /etc/passwd |tr : ' 'root /cron /bin /uucp /usr/spool/uucpasg /


tr command

• Octal Values of some ASCII characters

Character Octal value

Bell 7

Backspace 10

Tab 11

NewLine 12

Linefeed 12

Formfeed 14

Carriage Return 15

Escape 33

$ date |tr ' ' '\12' SunMar1019:13:46EST1985


tr command

$ cat introThe UNIX operating system was pioneered by Ken

$tr '[a-z]' '[A-Z]' < introTHE UNIX OPERATING SYSTEM WAS PIONEERED BY KEN


tr commandThe –s Option

• You can use the –s option to tr to “squeeze” out multiple occurances of characters will be replaced by a single character.

$ cat lotsofspacesThis is an example of a

file that contains a lot of blank spaces.$ tr -s' ' ' ' < lotsofspacesThis is an example of a

file that contains a lot of blank spaces.


tr commandThe –d Option

• tr can also be used to delete single characters from a stream of input.

$ tr –d ' ' < introTheUNIXoperatingsystemwaspioneeredbyKen


Using the grep Command

• Used to search for a specific pattern in a file, such as a word or phrase

• grep’s options and wildcard support allow for powerful search operations

• You can increase grep’s usefulness by combining with other commands, such as head or tail command

Syntax: grep [-options] [pattern] [filename]Useful options includes-i ignore case-l lists only file names-c counts the number of line instead of showing them-r searches through files under all subdirectories


grep Example

$ grep shell ed.cmdfiles, and is independent of the shell.to the shell, just type in a q.


grep example

$ cat phone_bookAlice Chebba 596-2015Barbara Swingle 598-9257Jeff Goldberg 295-3378Liz Stachiw 775-2298Susan Goldberg 338-7776Tony Iannino 386-1295$$ grep Susan phone_bookSusan Goldberg 338-7776


grep –v options Example

• Print all lines that don’t contain Unix

$ cat intro

The Unix operating system was pioneered by Ken Thompson. Main goal of Unix was to create Unix environment for efficient program development

$$ grep –v 'Unix' introenvironment for efficient program development


grep –i option Example

• Print all lines by ignoring upper and lower case letters

$ cat intro

The unix operating system was pioneered by Ken Thompson. Main goal of UNIX was to create UNIX environment for efficient program development

$$ grep –i 'Unix' intro

The unix operating system was pioneered by Ken Thompson. Main goal of UNIX was to create UNIX


grep [tT] option Example

• grep search for either upper or lower case letters T

$ cat intro


$$ grep '[tT]he intro

The unix operating system was pioneered by Ken


grep -i option Example

• grep with –l just gives you a list of file that contains specified pattern

$ cat intro


$$ grep –l 'Main goal' *

intro$


grep -n option Example

• grep with –n gives you output of specified pattern with relative line number

$ cat intro


$$ grep –n 'environment' intro

3:environment for efficient program development $


Using the uniq Command

• Uniq command removes duplicate lines from a file• Compares only consecutive lines, therefore uniq

requires sorted input• uniq has an option that allows you to generate output

that contains a copy of each line that has a duplicateSyntax: uniq [-option] [file1 > file2]Useful options includes-u outputs only the lines of the source file that are not

duplicated-d outputs only the copy of each lines that has a duplicate, and

does not show unique line-i ignores case-c starts each line by showing the number of each instance


Uniq Command Example

$ cat partsmufflermufflershocksalternatorbatterybatteryradiatorradiatorcoilspark plugsspark plugscoil

$ uniq parts >inventory$ more inventorymufflershocksalternatorbatteryradiatorcoilspark plugscoil$Why coil is listed twice?


Uniq –u Command Example

$ cat partsmufflermufflershocksalternatorbatterybatteryradiatorradiatorcoilspark plugsspark plugscoil

$ uniq –u parts >inventory$ more inventoryshocksalternatorcoilcoil$Why coil is listed twice?Ans: Two occurrences

ofcoil are nottogether


Using the comm Command

• Used to identify duplicate lines in sorted files• Unlike uniq, it does not remove duplicates, and it works

with two files rather than one• It compares lines common to file1 and file2, and

produces three column output– Column one contains lines found only in file1– Column two contains lines found only in file2– Column three contains lines found in both files


Using the diff Command

• Attempts to determine the minimal changes needed to convert file1 to file2

• The output displays the line(s) that differ• Codes in the output indicate that in order for the files to

match, specific lines must be added or deleted


Using the pr Command toFormat Your Output

• pr prints specified files on the standard output in paginated form

• By default, pr formats the specified files into single-column pages of 66 lines

• Each page has a five-line header containing the file name, its latest modification date, and current page, and a five-line trailer consisting of blank lines


Using a Shell Script toImplement the Application

• Shell scripts should contain:– The commands to execute– Comments to identify and explain the script so that

users or programmers other than the author can understand how it works

• Use the pound (#) character to mark comments in a script file


Running a Shell Script

• You can run a shell script in virtually any shell that you have on your system

• The Bash shell accepts more variations in command structures that other shells

• Run the script by typing sh followed by the name of the script, or make the script executable and type ./ prior to the script name


Putting it All Together toProduce the Report

• An effective way to develop applications is to combine many small scripts in a larger script file

• Have the last script added to the larger script print a report indicating script functions and results

Documents

Introduction to Unix (CA263) File Processing (continued) By Tariq Ibn Aziz