Introduction to UNIX / Linux - 6

Summer 2015SILICON VALLEY UNIVERSITY

CONFIDENTIAL 1

Introduction to UNIX / Linux - 6

Dr. Jerry Shiao, Silicon Valley University

SILICON VALLEY UNIVERSITY CONFIDENTIAL 2Summer 2015

Introduction UNIX/Linux Course

Section 6 Advanced File Processing

Regular Expressions Compressing/Uncompressing Text and Binary Files

View Compressed Files Without Decompressing Execute Binary Files Without Decompressing

Searching For Commands and Files for Absolute Path Searching Text Files for Strings, Expressions, and Patterns

DataBase Operations Sorting contents of ASCII File. Cutting Fields (Column Wise) from Table. Pasting Fields (Row Wise) to Table.

Encoding and Decoding Binary to ASCII File File Encryption and Decryption

SILICON VALLEY UNIVERSITY CONFIDENTIAL 3

Copyright @2005 Pearson Addison-Wesley


Advanced File Processing Regular Expressions Used in UNIX Commands and Tools

Using set of nondigits and nonletter characters to define: Rules to represent one or more items.

Character string xy* means x, xy, xyy, xyyy, … Similar to shell metacharacters.

Prevent Shell from interpretting Regular Expression characters, use single quote (i.e. ‘x | y‘) or back slash (i.e. \*).

Common tools using Regular Expressions: awk – Pattern-scanning and action. ed – Line oriented editor. grep – Searches file(s) for pattern. egrep - Extended regular expression search pattern. sed – Stream editor used as batch (noninteractive) editor. vi – Full screen editor.



Introduction UNIX/Linux Course Advanced File Processing awk, ed, egrep, grep, sed, vi and Regular Expressions




Advanced File Processing awk, ed, egrep, grep, sed, vi and Regular Expressions




Advanced File Processing vi and Regular Expressions




Advanced File Processing vi commands and Regular Expressions



Introduction UNIX/Linux Course Advanced File Processing Compressing Files

Less disk space. Archived file saves considerable storage space.

Less time to copy. Compressed file is unreadable. Less time to transmit over network.

Transmit multiple times: Time spend compressing is fraction of time of transmitting uncompressed file.

UNIX commands for compressing and decompressing. Compress/uncompress: UNIX. gzexe: Compress executable files. gzip: Compress and uncompress. gunzip: Uncompressing files compressed with gzip. gzcat: Displaying compressed files. gzcmp: Comparing compressed files. gzmore: Displaying compress file by page. gzgrep: grep command for compressed file.




Advanced File Processing compress Command (UNIX).

Analyze for repeated patterns, substitute characters. Adaptive Lempel-Ziv coding. Compressed file contains nonprintable characters. Original file removed, new file with extension “.z”. Retains mod date, ownership, and access privileges.

compress [ options ] [ file-list ] Files-list: Files to compress. Resultant compressed to suffix .Z file. Options:

- c : Write compressed file to console (instead of .Z file). - f : Force compression. - v : Display compression percentage and names of compressed files.




Advanced File Processing compress Command (UNIX).




Advanced File Processing uncompress [ options ] [ file-list ]

- c : No files changed. Behavior similar to “gzcat”. - f : Force compression of file, regardless of no size reduction. - v : Verbose, write percentage reduction or expansion of file. Output goes into original file (without the .Z extension).




Advanced File Processing gzip [ options ] [ file-list ]

GNU tool for compressing files. Uses Lempel-Ziv coding. Compress each file in “file-list”, replaced with one with “.gz” extension. Retains mod time, ownership, and access privileges. - c : Send output to standard output, input file is not overwritten

with “.gz” file. - d : Uncompress a compressed “.gz” file. Same as ungzip. - f : Force compression when “.gz” file already exits. - l : File is gzip compressed file. Display sizes compressed size, uncompressed size, ratio, name. - r : Recursively compress files in the directory. - v : Display compression percentage and names. - N : Compression Speed. 1 fastest, 9 slowest. Default is 6.




Advanced File Processing gzip [ options ] [ file-list ]

$ gzip file1 $ gzip -l file1 compressed uncompressed ratio uncompressed_name 52 120 76.7% file1 $ ls file1* file1.gz file1_sort file1_sort_bak $ mv file1_sort file1 $ gzip file1 gzip: file1.gz already exists; do you wish to overwrite (y or n)? n not overwritten $ gzip -f file1 $ gzip -v file1_sort_bak file1_sort_bak: 77.5% -- replaced with file1_sort_bak.gz

“- l” option takes compressed file and provides statistics.

Gzip replaces the source file with the “.gz” compressed file.

“.gz” file exists. Ask to overwrite.

“- v” option displays compression ratio and compressed file name.




Advanced File Processing gunzip [ options ] [ file-list ]

GNU tool for decompressing files. Same as gzip –d <file-list>. Decompress each file in “file-list”, remove “.gz” extension. Retains mod time, ownership, and access privileges. - c : Send output to standard output. - f : Force decompression. - l : File is gzip compressed file. Display sizes compressed size, uncompressed size, ratio, name. - r : Recursively decompress files in the directory. - v : Display decompression percentage and names. - # : Decompression Speed. 1 fastest, 9 slowest. Default is 6.




Advanced File Processing gunzip [ options ] [ file-list ]

$ ls -l bash* -rw-r--r-- 1 sau users 285373 2012-10-17 15:58 bash2.man -rw-r--r-- 1 sau users 285373 2012-10-17 15:58 bash.man $ gzip -v bash.man bash2.man bash.man: 72.9% -- replaced with bash.man.gz bash2.man: 72.9% -- replaced with bash2.man.gz $ ls -l bash* -rw-r--r-- 1 sau users 77239 2012-10-17 15:58 bash2.man.gz -rw-r--r-- 1 sau users 77238 2012-10-17 15:58 bash.man.gz $ gzip -vd bash.man bash.man.gz: 72.9% -- replaced with bash.man $ gunzip -v bash2.man bash2.man.gz: 72.9% -- replaced with bash2.man $ ls -l bash* -rw-r--r-- 1 sau users 285373 2012-10-17 15:58 bash2.man -rw-r--r-- 1 sau users 285373 2012-10-17 15:58 bash.man

List two identical files.

Gzip replaces the source file with the “.gz” compressed file.

List compressed files.

Use “gzip –d” to uncompress the “.gz” file. Use gunzip to uncompress the “.gz” file.




Advanced File Processing gzexe [ options ] [ file-list ]

Compress executable files. File name does not change and compressed file will

automatically uncompress when executed. Gzip of executable file is NOT executable.

Creates backup file “~file-list”. Performance penalty when executed. Utility useful on embedded systems with limited resources. - d: Decompress compressed file “~file-list” is now the

compressed file.




Advanced File Processing gzexe [ options ] [ file-list ]

$ gzexe power power: 64.3% $ ls -lt total 32 -rwxr-xr-x 1 sau users 4408 2012-10-17 16:57 power -rwxr-xr-x 1 sau users 10042 2012-10-17 16:56 power~ $ ./power This program takes x and y values from stdin and displays x^y . . . $ gzexe -d power $ ls -lt total 32 -rwxr-xr-x 1 sau users 10042 2012-10-17 16:58 power -rwxr-xr-x 1 sau users 4408 2012-10-17 16:57 power~ $ ./power This program takes x and y values from stdin and displays x^y . . . $ ./power~ This program takes x and y values from stdin and displays x^y . . .

Decompress file to get original executable file.

Backup or original file, “~power” is created.

Compress executable file.

When compressed file execute, file is auto uncompressed before executed.

Execute original executable file.

Execute backup executable file.

Backup is the compressed file.




Advanced File Processing Convert Compressed File before displaying

Requires system resources: System Memory, Disk Storage, and Disk I/O Schedular.

zcat [ options ] [ file-list ] Expands and displays contents of “file-list” files. Does not rename expanded file. Supports files compressed with compress or gzip. Writes expanded output to standard output.

zmore [ file-list ] Allows viewing compressed or plain text files one screen at a

time. Supports files compressed with compress or gzip. Writes expanded output to standard output.




Advanced File Processing zcat and zmore.

$ man bash > man.bash $ gzip man.bash $ zcat man.bash | more BASH(1) NAME bash - GNU Bourne-Again SHell SYNOPSIS bash [options] [file] - - More - - <Cntrl> <C> $ zmore man.bash.gz BASH(1) NAME bash - GNU Bourne-Again SHell SYNOPSIS bash [options] [file] . . . lines 1-30 :q

Zcat uncompresses and sends output to stdout, but stdout need to be paged with more command.

Zmore similar to using zcat with more command.




Advanced File Processing Sorting.

Ordering a set of items in ascending or descending order. Sort Key: Using a field or portion of each item.

Determines the position of each item in the sorted list. Key depends on item to be sorted (i.e. personal records will

use last name, student ID, and social security numbers). Performed in a variety of software systems.

Words in a dictionary. People names in telephone directory. Arrival/Departure times in airport terminals. Student IDs in class list.



Introduction UNIX/Linux Course Advanced File Processing Sort items in text (ASCII) files. Sort items based on a field: Field separated by blank. sort [ options ] [ file-list ]

Sort lines in the ASCII files in “file-list”. Output goes to standard output. Default key starts at column 1. Fields are just words separated by blanks. - b : Ignore leading blanks. - d : Alphabetical order, ignoring all characters except letters, digits, and then blanks. - f : Case insensitive. +n [ -m ] : Field used as sort key. “n” specifies how many fields

to skip. Start at the first character of (n+1th) field and

end at the last character (mth) field (or end of line). - r : Sort in reverse order. - t : Field delimiter changed to <character>.




Advanced File Processing sort [ options ] [ file-list ]

$ cat donors.data Bay Ching 500000 China Jack Arta 250000 Indonesia Cruella Lumper 725000 Malaysia $ sort +1 -2 donors.data Jack Arta 250000 Indonesia Bay Ching 500000 China Cruella Lumper 725000 Malaysia $ cat filec File2: line2. : 111 File3: line4. : 222 File4: line3. : 333 $ sort -t: +1 -2 filec File2: line2. : 111 File4: line3. : 333 File3: line4. : 222

Field skips first word (+1) and end at second word (-2). Field delimiter is space.

Field delimiter is colon “:”. Field start at second word (+1) and end at second word (-2).




Advanced File Processing sort [ options ] [ file-list ]

$ cat filec File2: line4. : 111 File3: line4. : 100 File4: line3. : 333 $ sort -b -t: +1 -2 +2 -3 filec File4: line3. : 333 File3: line4. : 100 File2: line4. : 111 $

Primary Key –Field delimiter is colon “:”. Field skips first word (+1) and end at second word (-2).

Secondary Key –Field delimiter is colon “:”. Field start at third word (+2) and end at third word (-2).




Advanced File Processing Searching For Commands and Files find directory-list expression

Search “directory-list” to locate files meeting one or more criteria in “expression”.

Criteria in “expression”: -exec CMD : Files matching criteria executes CMD. -name <pattern> : Search files specified by <pattern>. -newer file : Search files modified after “file”. -perm <octal> : Search files with permission <octal>. -print : Display the pathnames of files found. -user <name> : Search for files owned by user name.

find ~ -name class* -print /home/sau/class3 /home/sau/kernel/linux-bcm-2.6.30/include/config/leds/class.h /home/sau/kernel/linux-bcm-2.6.30/include/config/classic . . .



Introduction UNIX/Linux Course Advanced File Processing Searching For Commands and Files whereis [ options ] [ file-list ]

Locate binaries (executables), source codes, and manual pages for the commands in file-list.

Searches ONLY standard locations: /usr/share/man/* Directories of manual files. /sbin, /etc, /usr/{lib, bin, ucb, lpp} Directories of binary files. /usr/src/* Directories of source code files.

Displays absolute pathnames for the located files. - b : Search for binaries (executables) only. - s : Search for source code only. $ whereis find compress tar find: /bin/find /usr/bin/find /usr/bin/X11/find /usr/share/man/mann/find.n.gz /usr/share/man/man1p/find.1p.gz /usr/share/man/man1/find.1.gz compress: /usr/share/man/man1p/compress.1p.gz tar: /bin/tar /usr/include/tar.h /usr/share/man/mann/tar.n.gz /usr/share/man/man5/tar.5.gz /usr/share/man/man1/tar.1.gz



Introduction UNIX/Linux Course Advanced File Processing Searching For Commands and Files which [ command-list ]

Takes command name in command-list and locates the file that contains it.

Searches all the directories in your PATH environment variable, in order, until it locates the command.

Stops at first occurrence: Finds where the shell is resolving the command.

Cannot locate aliases, functions, and shell builtins. type [ -tp ] name

Indicates how each name could be interpreted if used as a command name.

Prints string which is one of: alias, keyword, function, builtin, or file.

-t: Prints string alias, keyword, function, builtin, or file. -p: Returns name of the disk file that would be executed (“type –t

name” would return “file”) or nothing.



Introduction UNIX/Linux Course Advanced File Processing Searching For Commands and Files [student1@localhost ~]$ whereis grep

grep: /bin/grep /usr/share/man/man1p/grep.1p.gz /usr/share/man/man1/grep.1.gz[student1@localhost ~]$ which grep/bin/grep[student1@localhost ~]$ type grepgrep is /bin/grep[student1@localhost ~]$ type -p grep/bin/grep[student1@localhost ~]$ type -t grepfile[student1@localhost ~]$ whereis nicenice: /bin/nice /usr/share/man/man1p/nice.1p.gz /usr/share/man/man2/nice.2.gz /usr/share/man/man3p/nice.3p.gz /usr/share/man/man1/nice.1.gz[student1@localhost ~]$ whereis cdcd: /usr/share/man/man1p/cd.1p.gz /usr/share/man/man1/cd.1.gz[student1@localhost ~]$ which cd/usr/bin/which: no cd in (/usr/kerberos/bin:/usr/lib/ccache:/usr/local/bin:/usr/bin:/bin:/usr/X11R6/bin:/home/student1/bin)[student1@localhost ~]$ type cdcd is a shell builtin[student1@localhost ~]$




Advanced File Processing Searching Files Utilities to fine lines in Text Files that contain an

expression, string, or pattern. grep [ options ] pattern [ file-list ]

Search regular expression pattern. egrep [ options ] regexp [ file-list ]

Interpret document as extended regular expression supporting special characters (i.e. “|, ?, !, and {“).

fgrep [ options ] pattern [ file-list ] Search literal text string pattern.

Search files in “file-list” for given pattern, string, or expression. Lines matching options sent to standard output. - i : Ignore the case of letters during the matching process. - n : Print line numbers along with matched line. - v : Print nonmatching lines.




Advanced File Processing Searching Files

$ grep -n line1 file1 1:File2 line1. Change this line. 2:File2 line1.5 Insert this line. $ egrep "line1|line2" file1 File2 line1. Change this line. File2 line1.5 Insert this line. File2 line2. $ grep "^Line" filec Line : line5. : 099 $ grep "\<1" filec File2: line4. : 111 File3: line4. : 100

Preceed each line with the line number.

Using regular expression “|” to logically “OR” the two strings.

Using regular expression “^” to select pattern at beginning of line.

Using regular expression “<“ to select pattern at beginning of word.




Advanced File Processing Text Files can be processed as Tables:

Each line comprise a record. Each column is a field separated by field separated . ‘Tab’ is default field separated.

Cutting and Pasting Process files stored in table format. cut -blist [ -n ] [ file-list ] cut –clist [ file-list ] cut -flist [ -dchar ] [ -s ] - b list : Treat each byte as a column. - c list : Treat each character as a column. - d char : Use character “char” as field separator. Default <Tab>. - f < flist >: Cut fields specified in “flist”. - n : Do not split characters. - s : Do not output lines that do not have delimiter character. “blist”, “clist”, and “flist” can be comma-separated list or using “-” to specify

range of bytes, characters, or field.




Advanced File Processing Cutting and Pasting

Remove fields 1 and 2.

Fields: ‘Tab’ is the field separator.




Advanced File Processing Cutting and Pasting paste [ options ] file-list

Horizontally concatenates files in “file-list”. - d list : Use “list” as new line separator.




Advanced File Processing Cutting and Pasting paste [ options ] file-list

$ cut –f1-3 student_records > table1 $ cut -f4 student_addresses > table2 $paste table1 table2 John Doe ECE 312.111.9999 Pam Myer S 666.222.1212 Jim Davis CE 713. 999 .5555 Jason Kim ECE 434.000.8888 Amy Nash ECE 888.111.4444 $ rm table1 table2

NOTE: System overhead with separate ‘cut’ commands, ‘paste’, and ‘rm’.Use piping “ | “ to reduce system resources.

$ paste student_records student_addresses | cut –f1-3,7




Advanced File Processing Encoding and Decoding Used in mail utility that does not support attachments.

User want to send non-ASCII (multimedia) file, must send in body of the mail.

UNIX-to-UNIX encode convert a file to be mailed to a format that contains readable ASCII characters only.

uuencode [ source-file ] decode_label Encode “source-file” from binary to ASCII Default output binary file to standard output: Redirect standard

output to named file using “ > “. uudecode [ option ] [ encoded_file ]

Decode “encoded_file” from ASCII to binary. Default output binary file, decode_label, created. - p : Send binary version of the uuencoded file to standard

output.




Advanced File Processing Encoding and Decoding

- p

decode-label




Advanced File Processing Encoding and Decoding $ uuencode power2 decode_label > power2_uuencoded $ ls -lt total 140 -rw-r--r-- 1 sau users 13869 2012-10-18 04:13 power2_uuencoded -rwxr-xr-x 1 sau users 10042 2012-10-17 16:56 power2 $ head power2_uuencoded

begin 644 decode_labelM9&EF9B`M3F%U<B!D7S$O;&5C.5]F:6QE(&1?,B]L96,Y7V9I;&4*+2TM(&1?M,2]L96,Y7V9I;&4),C`Q-"TQ,"TP.2`Q.3HS,CHP."XR,3`R-3,X-3,@+3`W. . .

$ uudecode power2_uuencoded $ ls decode_label power2 power2_uuencoded $ ./decode_label This program takes x and y values from stdin and displays x^y . . .




Advanced File Processing File Encryption and Decryption Encryption is a process by which a file is converted to an

encrypted form, completely different from its original version. Prevent hackers from easily understanding text over LAN or WiFi. Original file remains intact, must be explicitly deleted.

Decryption is the reverse process by which a file is converted from a encrypted form to its original form.

mcrypt [ option ] -z : Use gzip to compress files before encryption. If specified at

decryption time, it will decompress these files. - d : Decrypt. - k : Enter keyword via command line, instead of prompting.

mdecrypt [ option ]




Advanced File Processing File Encryption and Decryption




Advanced File Processing File Encryption and Decryption

PassPhrasePassPhrase

Original file (Must be explicitly removed)




Advanced File Processing File Encryption and Decryption [student1]$ mcrypt -z students

Enter the passphrase (maximum of 512 characters)Please use a combination of upper and lower case letters and numbers.Enter passphrase:Enter passphrase: File students was encrypted.

[student1]$ lsstudents students.gz.nc

[student1]$ mdecrypt -z students.gz.ncEnter passphrase: File students.gz.nc was decrypted.

[student1]$ lsstudents students.gz students.gz.nc

[student1]$ diff students students.gz [student1]$

Documents

Introduction to UNIX / Linux - 6