Embed Size (px)
Citation preview
1
Plan for today and later
Little review
Little discussion of previous labp
Unix
Introduction
Utilities
(Bourn) Shell, shell command and shell script
System calls ?
struct integers arr[10]; // an array of ten structs
char record[40]; // space for structs also created
int i1,i2;
read from file and store in an array of structures
FILE * file;
file = fopen("data.txt", "r");
while (fgets(record, 50, file) != NULL)
{
sscanf(record, "%d %d", &i1, &i2);
arr[count].int1 = i1;
arr[count].int2 = i2;
count++;
}
2
struct integers arr[10]; // an array of structs
char record[40];
int i1,i2;
read from file and store in an array of structures
FILE * file;
file = fopen("data.txt", "r");
while (fscanf(file, "%d %d", &i1, &i2) == 2)
{ {
arr[count].int1 = int1;
arr[count].int2 = int2;
count++;
}
struct integers arr[10]; // an array of ten structs
char record[40];
read from file and store in an array of structures
FILE * f;
file = fopen("data.txt", "r");
while(fscanf(f,"%d %d",&arr[count].int1, &arr[count].int2) == 2)
{
count++;
}
3
/* Insert a new data element with key d into the end of the list. */
void insert(int d )
{
struct list * newP = malloc (sizeof(struct list));
newP -> data = d;
newP -> next = NULL;
if (head == NULL)
{t t S
{
head = newP;
}
struct newS;newS.data = d;newS.next =NULL
if (head == NULL)head = &newS;
setArr2
6
i is in stack --deallocatedwhen fun exits
4
void enter()
{
char name[LEN];
int age;
double cour1, cour2;
printf("name: "); scanf("%s", name);
printf("age: "); scanf("%d", &age);
printf(“course-1: "); scanf("%lf", &cour1);
printf(“course-2: "); scanf("%lf", &cour2);
record newStudent;
strcpy(newStudent.name, name);py( , );
newStudent.age = age;
strcpy(newStudent.course1, cour1);
strcpy(newStudent.course2, cour2);
database[count++] = &newPatient;}
5
Contents
Overview of UNIX Structure
File systems
Pipes
Shells
UNIX Philosophy
UNIX’s History
….
Utilities Basic
advanced
Application
Computer Systems A Computer System consists of hardware, system programs and application
programs.
Banking System Airline Reservation Adventure games
Compilers Editors Command Interpreter
Operating System
Machine language
Systemprograms
Application programs
Machine language
Microprogramming
Physical devices
Hardware
6
Layers and Views of a Computer System
•Computer Systems• Hardware• Operating System
Application Programs
End User ProgrammerOr Administrator
O ti
• Application program • Communication
Computer Hardware
Operating System
UtilitiesOperating System
Designer
Operating Systems
An operating system
a program that controls the execution of application programs and acts as an interface between the user (program) of a computer and the computer hardware.
An operating system has four major components: t process management,
input/output,
memory management, and
the file system.
7
Operating Systems
programs
operating system
hardware
A computer system cannot function without an operating system.
There are many different operating systems that are available for PCs, minicomputers, and mainframes;
the most common ones are Windows, VMS, MacOS, and UNIX. , , ,
MS Windows is only available for PCs
VMS is only available for mini‐computers and mainframes,
and UNIX is available for virtually all platforms.
Operating systems General architecture
Kernel: deal with hardware
Hw‐independent exposedepe de t e pose
to high level by system call
Utilities: small, come with os
Shell: textual c‐l interface
programs
operating system
hardware
8
Operating systems General architecture
Kernel: deal with hardware
Hw‐independent exposedepe de t e pose
to high level by system call
Utilities: small, come with os
Shell: textual c‐l interface
Unix System Structure
user c programsscripts
shell and utilities
kernel
scripts
lsksh
gccfind
open()fork()exec()
hardware
9
Unix System Structure Another picture
Kernel Manage resources
Storage
Memory
CPU
Display
Network
Sharing
user
shell and utilities
kernel
hardware Users
Tasks
Communication
10
Kernel Subsystems File system
Directory hierarchy, regular files, peripherals
Multiple file systems
Input/Output How processes access files, terminal I/O
Process management
How processes share CPU, memory and signals
SchedulingScheduling
Inter‐process Communication (IPC)
Memory management
UNIX variants have different implementations of different subsystems.
Kernel Subsystems
user
shell and utilities
kernel
hardware
11
Kernel Interactions Processes accesses kernel facilities via system calls
Peripherals communicate with the kernel via hardware pinterrupts
User Level
System Calls
Hardware Level
Kernel Level
Interrupts
Talking to Kernel
12
Shell and Utilities The rest of the operating system
Focus of the last 3 lectures of this course3
Cause of debate in Linux community
user
shell and utilities
kernelkernel
hardware
UNIX Shell (briefly) Shell is the (command line) user interface to the operating system
between you and the raw unix os
Functionality: Functionality:
Execute other programs
Manage files, processes
Interactively
When you log in, you interactively use the shell
Scripting
A set of shell commands that constitute an executable program: a script
13
Files and Processes A file is a collection of data that is usually stored on disk, although some files are stored on tape.
When a program is started, it is loaded from disk into RAM. When a program is running, it is called a process.
Most processes read and write data from files.
RAM DISK
inter-processcommunication
Unix file systems Unix expands the usual definition of file to include anything from which data can be taken, sent., including iod i (kb i )devices (kb, printer,)
In Unix, “Everything is a file”
Four kinds of files
Orindary files (regular files) hold info text/ binary files
Special files – (device files). Representing physical devices, Special files (device files). Representing physical devices, keyborards, terminals, printers and other peripherals.
Directory files (directories). Hold other files and directories
Link (pointer) to files hardlink, softlink
14
/incude
Fig. 2.2: Typical UNIX directories
Directory Typical Contents
/ The "root" directory
/bin Essential low‐level system utilities
/usr/binHigher‐level system utilities and application programs
/libProgram libraries (collections of system calls that can be included in programs by a compiler) for low‐level system utilities
l b l b f h h l l/usr/lib Program libraries for higher‐level user programs
/tmpTemporary file storage space (can be used by any user)
/home or /homesUser home directories containing personal file space for each user.
/etc UNIX system configuration and information files
/dev Hardware devices
15
Devices as Special Files Besides files, input and output can go from/to various hardware devices.
U i Phil h T h d i i l fil ! Unix Philosophy: Treat these devices as special files!
Terminals, printers, and other devices are accessible in the same way as disk‐based files.
/
home
…
bin usr … dev
lpr tty modem
/dev/lpr /dev/tty /dev/modem
Unix File System Files are just sequences of bytes
No file types (data vs. executable)
Example of UNIX philosophy (KISS: keep it simple ) Example of UNIX philosophy (KISS: keep it simple…)
Directories are lists of files and other directories, along with their status: creation date
permissions, etc.
Each directory entry links to a file on the disky y
moving a file does not actually move any data around.
creates link in new location
deletes link in old location
UNIX provides services for the creation, modification, and destruction of programs, processes, and files.
16
File Pathnames Two files in the same directory may not have the same name, but it’s
perfectly OK for several files in different directories to have the same name. We need pathnames to differentiate between files with the
l t d i diff t di t isame names located in different directories.
A pathname is a sequence of directory names that leads you through the hierarchy from a starting directory to a target file.
cp /cs/dept/course/2012‐13/W/2031/a2.out .
For example, here’s a small hierarchy that contains three files called “myFile” in three separate directories. /
myFile
myFile
binhome
timhuiwang
myFile
Absolute Pathnames A pathname relative to the root directory is often termed an absolute, or full, pathname.
/home/huiwang/myFile ~/myFile
/home/tim/myFile
/bin/myFile
/
myFile
myFile
binhome
timhuiwang
myFile
17
Absolute and Relative Pathnames A process may also unambiguously specify a file by using a pathname relative to its current working directory.
UNIX file system supports the following special fields that may be used when supplying a relative pathname:
Field Meaning
. current directory
.. parent directory Same in DOS
cd ..
Relative Pathnames Relative Pathnames (from /home/huiwang)
myFile or ./myFile
../tim/myFile
../../bin/myFile
/
myFile
myFile
binhome
timhuiwang
myFile
18
Links A link is a pointer to a file.
In fact, in UNIX all filenames (diectory entry) are just links to a file Most files only have one link links to a file. Most files only have one link.
-rw-r--r-- 1 jbond cs 154 Feb 4 15:00 letter3-rw-r--r-- 1 jbond cs 64 Feb 4 15:00 namesdrwxr-xr-x 2 jbond cs 512 Feb 4 15:00 secret/
Additional links to a file allow the file to be shared.
The ln command creates new links. $ ln names NAMES$ ls -ltotal 8-rw-r--r-- 2 jbond cs 64 Feb 6 18:36 NAMES-rw-r--r-- 1 jbond cs 154 Feb 4 15:00 letter3
-rw-r--r-- 2 jbond cs 64 Feb 4 15:00 namesdrwxr-xr-x 2 jbond cs 512 Feb 4 15:00 secret/
(a) Logical structure of current directory; (b) contents of current directory
19
Hard and Symbolic Links Symbolic links are different than regular links (often called hard links).
Can be thought of as a file that contains the name of another file Can be thought of as a file that contains the name of another file
Does not change link count for file
When original deleted, symbolic link remains
They exist because:
Hard links don’t work across file systemsy
Hard links only work for regular files, not directories
direntfile
dirent2
direct filesymlink
Hard link Symbolic Link
Unix Security Processes and files have an owner and may be protected against unauthorized access.
A set of users can form a group. A user can be a member of multiple groups.
A special user (id 0, name root) has complete control.
Each user has a primary (default) group.
20
File and Directory Permissions UNIX provides a way to protect files based on users and groups.
Three types of permissions:
read process may read contents of file
write process may write contents of file
execute process may execute file
Three sets of permissions:
permissions for owner
permissions for group
permissions for other
Same types and sets of permissions as for files apply to directories:
read process may read the directory contents (i.e., list files)
write process may add/remove files in the directory
execute process may open files in directory or subdirectories
Processes Unix can execute many processes simultaneously.
Program arguments
When a process is started, it is sent a list of strings: argv, argc
The process can use this list however it wants to
Ending a program
When a process ends, there is a return code associated with the process outcomewith the process outcome
This is a non‐negative integer 0 means success
anything larger than 0 represent various kinds of failure
The return value is passed to the parent process
21
Communication The components of a computer system cannot achieve very much when
they work in isolation;
For example:
A process may need to talk to a graphics card to display output.
A process may need to talk to a keyboard to get input.
A network mail system needs to talk to other computers to send and receive mail.
T d lk h h i d ll b Two processes need to talk to each other in order to collaborate on a single problem.
UNIX provides several different ways for processes and peripherals to talk to each other, depending on the type and the speed of the communication.
Communication Processes can communicate using a
number of means:
passing arguments, environment
read/ rite regular files read/write regular files
exit values
inter‐process communication with
shared queues, memory and semaphores
signals
pipes
sockets
A pipe is a one‐way medium‐speed data channel that allows two processes on the same machine to talk.
If the processes are on different machines connected by a network, then a mechanism called a “socket” may be used instead. A socket is a two‐way high‐speed data channel.
22
Unix Pipes
A special mechanism called a “pipe” built into the heart of A special mechanism called a pipe built into the heart of UNIX to support cascading utilities.
A pipe allows a user to specify that the output of one process is to be used as the input to another process.
Two or more processes may be connected in this fashion, resulting in a “pipeline” of data flowing from the first process through to the last.
Pipeline
The nice thing about pipelines is that many problems g p p y pcan be solved by such an arrangement of processes.
Each process in the pipeline performs a set of operationsupon the data and then passes the results on to the next process for further processing.
Process 1 Process 2 Process 3
A B C
23
Pipeline Example
A utility called who that outputs an unsorted list of the users, and another utility called sort that outputs a sorted version of its input.
These two utilities may be connected together with a pipe so that the output from who passes directly into sort, resulting in a sorted list of users.
who sort terminal
W S Twho | sort
Pipe‐Equivalent Communication Using a File
Could we use a file instead of a pipe? YES.
Run first program save output into file Run first program, save output into file Run second program, using file as input
Disadvantages:
Unnecessary use of the disk
process 1 process 2
who > tmp; sort tmpy
Slower Can take up a lot of space
Makes no use of multi‐tasking
Pipe is very similar, but does not involve the external device
all mechanisms stay in the realm of the operating system
p
24
Unix Utilities Standard UNIX comes complete with at least 200 small utility programs,
usually including:
shells,
editors,
a C compiler,
matching with regular expressions,
searching,
ti tilit a sorting utility,
software development tools,
text‐processing tools, etc.
Unix Versions UNIX is a fairly standard operating system, with two main versions that
are slowly merging into one.
UNIX was created in Bell Laboratories and evolved from that into what is currently known as “System V” UNIX.
The university of California at Berkeley obtained a copy of UNIX early on in its development and spawned another major version, known as BSD (Berkeley Standard Distribution) UNIX.
UNIX international UNIX international
AT&T, Sun Microsystems, ‐‐> System V Release 4.
Open Software Foundation
IBM, Digital Equipment Corporation, Hewlett‐Packard ‐‐> BSD UNIX, called OSF/1.
25
Unix Standards
Both groups tried to comply with a set of standards set by the POSIX (Portable Operating System Interface) committeeOperating System Interface) committee
Most of the best features of BSD UNIX have been rolled into most System V‐based versions of UNIX.
UNIX is mostly written in the C language, which makes it relatively easy to port to different platforms.
This feature is an important benefit and has contributed a great deal to the proliferation and success of UNIX.
Unix Yesterday A computer scientist named Ken
Thompson at Bell Laboratories built the first version of UNIX. It was built to support some gaming needs that could not be met by existing systems.
written by using assembly language,
only a single‐user system,
no network capability,
poor memory‐management system for sharing memory between processes.
In spite of the shortcomings,
‐‐> efficient, compact, and fast, which was exactly what he wanted.
26
Unix Yesterday A few years later, a colleague of Ken’s,
Dennis Ritchie, suggested that they rewrite UNIX using the C lanaguage.
the UNIX system suddenly had a huge advantage over other operating systems ‐its source code was understandable.
Only a small percentage of the original source code remained in assembly language, which meant the porting the operating system to a different machine was quite easy was quite easy.
AT&T, Bell Labs’ parent company, was not allowed to enter computer business, so Unix was not commercialized originally.
Unix Yesterday Bell Laboratories allowed universities to obtain a free copy of the UNIX source
code.
The University of California at Berkeley, made some huge improvements over the years, including the first good memory‐management system and the first real networking capability.
UC Berkeley offered a version of UNIX, called BSD (Berkeley Standard Distribution) to the general public.
27
Unix Today The currently commercially available versions of UNIX include, but are not limited to,
offerings from AT&T, DEC, IBM, HP, Silicon Graphics, Inc., and Sun Microsystems.
A more recent entry into the UNIX world is Linux, a free version of UNIX written by a y , ystudent in Finland and now marketed and supported by several different companies.
Embedded versions of UNIX are available for various hardware platforms.
Apple abandoned old Mac OS for UNIX
Purchased NeXT in December 1996
Unveiled in 2000
Based on 4.4BSD‐Lite
Aqua UI written over Darwin
Mac OS being phased out
Open Source
UNIX
System Vseries
BSDseriesseries series
V.2
V.3
V.4
4.2
4.3IBMAIX
OSF/1
Apollo
FIGURE 1.6 An abbreviated genealogy of UNIX
V.4.1Sun OS(Solaris)
HP/UX
OSF/1
28
What is the difference between Linux and Unix?
Ans 1: Linux is a unix‐like kernel. It is open source.
Ans2: To put it very generically, Linux is an operating system kernel, and UNIX is a certification for operating systems. The UNIX standard evolved from the original Unix system developed at Bell Labs. After Unix System V, it ceased to be developed as a single operating system, and was instead developed by various competing companies, such as Solaris (from Sun Microsystems), AIX (from IBM), HP‐UX (from Hewlett‐Packard), and IRIX (from Silicon Graphics). UNIX is a specification for baseline interoperability between these systems even specification for baseline interoperability between these systems, even though there are many major architectural differences between them. Linux has never been certified as being a version of UNIX, so it is described as being "Unix‐like." A comprehensive list of differences between Linux and "UNIX" isn't possible, because there are several completely different "UNIX" systems.
29
Linux as an unix‐like os
Kernel
Shells and GUI
sh bash, csh
KDE GNOME
System utilities
user
shell and utilitiesy
ls,cp,grep,awk,sed,wc,more
Application programs
Emacs, gcc, xfig,latex, soffice
kernel
hardware
30
Shells The $ or % prompt that you see when you first log in is displayed
by a special kind of program called a shell.
A Shell is a program that acts as a middleman between you and A Shell is a program that acts as a middleman between you and the raw UNIX operating system.
It lets you run programs, build pipelines of processes, save output to files, and run more that one program at the same time.
A shell executes all of the commands that you enter.
The four most popular shells are: the Bourne shell (sh) the Korn shell (ksh) the C shell (csh) the Bash Shell (bash)
59
user
shell and utilities
kernel
hardware
Basic utilities Introduces the following utilities, listed in alphabetical order:
cancel head mvcancelcatchgrpchmodchownclearcp
headlplprlprmlpqlpstatls
mvnewgrppagepasswd pwd rm rmdir
60
pdatefilegroups
lsmailmanmkdirmore
rmdirstty tail tset wc
31
Basic utilities Introduces the following utilities, listed in in groups:
Generalmanclearcaldatemail
Directorymkdirrmdircdpwd
Filecatmorecpmvrm
Filelplprlprmlpqlpstat
61
(quota –v) headtailchmodchgrpchownnewgrp
Running a utility To run a utility, simply enter its name at the prompt and press the Enter key.
Pressing the Enter key tells UNIX that you’ve entered the command an that you wish it to be executed.
E.g., 0ne utility that every system has is called date, which displays the current date and time:
$ date > run the date utility$ date ‐‐> run the date utility.Thu Aug 19 21:23:33 PDT 2004$ _
62
•Up/down arrow for history
•Tab key for auto complete
32
Running another shell
If you don’t like the shell just type in the name of the If you don t like the shell, just type in the name of the one that you like and press Enter.
%: echo $SHELL
/cs/local/bin/tcsh
%: sh ‐‐> tcsh ‐> sh%: sh > tcsh > sh
$ exit ^D
% ‐‐> back to tcsh
63
date cal
Without any arguments, date displays the current dateWithout any arguments, date displays the current dateand times. Lots formatting options
date +"%Z %y %Y"
cal: calenda (not calculator)
cal
cal 01 2011
bc
64
33
clear This utility clears your screen.
cls in MS‐DOS
echo print statement in unix
echo hello
mail / pine (not standard?)/ p ( )
Read and send mail
quota –v Check disk quota (not standard)
65
man: online help All UNIX systems have a utility called man ( short for manual page )
that puts this information at your fingertips.
man [section] word man ‐k keyword
The manual pages are on‐line copies of the original UNIXdocumentation, which is usually divided into eight sections. They contain information about utilities, system calls, file formats, and shells.
The first usage of man displays the manual entry associated with word. If no section number is specified, the first entry that it finds is displayed.
The second usage of man displays a list of all the manual entries that contain keyword.
66
34
Organization of the manual pages
The typical division of topics in manual pages (sections) is yp p p g ( )as follows:
1. Commands and Application Programs.
2. System Calls
3. Library Functions
4. Special Files
5. File Formats
6. Games
7. Miscellaneous
8. System Administration Utilities
67% man man
Using manHere’s an example of man in action:
$ man ‐k mode ‐‐‐> search for keyword “mode”chmod (1V) change the permissions mode of a file chmod (1V) ‐ change the permissions mode of a file chmod, fchmod(2V) ‐ change mode of file getty(8) ‐ set terminal mode ieeeflags(3M) ‐mode and status function umask(2V) ‐ set file creation mode mask
$ man chmod ‐‐‐> select the first manual entry. CHMOD(1V) USER COMMANDS CHMOD(1V) NAME
chmod ‐ change the permissions mode of a file chmod change the permissions mode of a file SYNOPSIS
chmod C ‐fR V mode filename … ‐‐> the description of chmod goes here.
SEE ALSO csh(1), ls(1V), sh(1), chmod(2V), chown(8)
68
35
Using man sections
$ man ‐s 2 chmod ‐‐‐> select the manual entry from section 2.CHMOD( V) SYSTEM CALLS CHMOD( V) CHMOD(2V) SYSTEM CALLS CHMOD(2V) NAME
chmod, fchmod ‐ change mode of file
SYNOPSIS #include <sys/stat.h> int chmod(path, mode) char *path;mode t mode; mode_t mode;
… the description of chmod() goes here. SEE ALSO
chown(2V), open(2V), stat(2V), sticky(8) $ _
69
Using special characters Some characters are interpreted specially when typed at a UNIX
terminal.
These characters are sometimes called metacharacters and may be These characters are sometimes called metacharacters and may be listed by using the stty utility with the ‐a (all)option.
Here’s an example of the use of the stty utility for listing metacharacters:
$ stty ‐a ‐‐> obtain terminal line settingsspeed 38400 baud; rows 50; columns 80; line = 0;intr = ^C; quit = ^\; erase = ^?; kill = ^U; eof = ^D; eol = <undef>;eol2 <undef>; start ^Q; stop ^S; susp ^Z; rprnt ^R; werase ^W;eol2 = <undef>; start = ^Q; stop = ^S; susp = ^Z; rprnt = ^R; werase = ^W;lnext = ^V; flush = ^O; min = 1; time = 0;‐parenb ‐parodd cs8 ‐hupcl ‐cstopb cread ‐clocal ‐crtscts‐ignbrk ‐brkint ‐ignpar ‐parmrk ‐inpck ‐istrip ‐inlcr ‐igncr icrnl ixon ‐ixoff‐iuclc ‐ixany ‐imaxbelopost ‐olcuc ‐ocrnl onlcr ‐onocr ‐onlret ‐ofill ‐ofdel nl0 cr0 tab0 bs0 vt0 ff0isig icanon iexten echo echoe echok ‐echonl ‐noflsh ‐xcase ‐tostop ‐echoprtechoctl echoke
70
36
Meta‐characters The carat(^) in front of each letter means that the Control key must be pressed at the
same time as the letter.
The default meaning of each option is as follows:
Meta‐character Meaning
erase Backspace one character
kill ^U Erase all of the current line.
werase Erase the last word.
rprnt Reprint the line.
flush Ignore any pending input and reprint the line.
lnext Don’t treat the next character speciallylnext Dont treat the next character specially.
susp ^Z Suspend the process for a future awakening.
intr ^C Terminate ( interruput ) the foreground job with
no core dump.
quit Terminate the foreground job and generate a core
dump.
stop Stop/restart terminal output.
eof ^D End of input.71
Terminating a Process: Control‐c There are often times when you run a program and then wish to stop it before
it’s finished.
The standard way to execute this action in UNIX is to press the keyboard sequence Control‐C.
Most processes are immediately killed and your shell prompt is returned.
Here’s an example of the use of Control‐C:
$ h d$ man chmodCHMOD(1V) USER COMMANDS CHMOD(1V) NAME
chmod ‐ change the permissions mode of a file ^C$ _
72
37
End of Input: Control‐d You must tell the utility when the input from the keyboard is finished. To do so, press Control‐D on a line of its own after the last line of input.
Control‐D signifies the end of input.
For example, the mail utility allows you to send mail from the keyboard to a named user:
$ mail tim ‐‐> send mail to my friend tim.Hi Tim, ‐‐> input is entered from the keyboard.
I hope you get this piece of mail. How about building a countryf h done of these days?
‐ with best wishes from Graham ^D ‐‐> tell the terminal that there’s no more input. $ _
73
Basic utilities Introduces the following utilities, listed in alphabetical order:
Generalmanclearcaldatebc
Directorymkdirrmdircdpwdls
Filecatmorecpmvrm
Filelplprlprmlpqlpstat
74
pinequota -v
headtailchmodchgrpchownnewgrp
38
pwd: Printing Working Directory To display your shell’s current working directory, use the pwd utility, which works like this:
login : huiwang
Password : ……
$pwd
/cs/home/huiwang
$ _
75
Making Directory: mkdir
mkdir ‐p newDirectoryName
Th kdi ili di Th i
Same in DOS
The mkdir utility creates a directory. The ‐p option creates any parent directories in the newDirectoryName pathname that do not already exist.
If newDirectoryName already exists, an error message is displayed and the existing file is not altered in any way.
$ mkdir lyrics ‐‐> creates a directory called “lyrics”. $ l l h k th di t li ti i d$ ls –l ‐‐> check the directory listing in order
‐‐> to confirm the existence of the‐‐> new directory.
‐rw‐r‐‐r‐‐ 1 huiwang faculty 106 Jan 30 23:28 heart.ver1drwxr‐xr‐x 2 huiwang faculty 512 Jan 30 19:49 lyrics/$ _
76
39
Listing Contents of a Directory: ls The ls utility, which lists information about a file or a directory.
ls ‐adglsFR { fileName }* {directoryName}* dir in dos
ls lists all of the files in the current working directory in alphabetical order, excluding files whose names start with a period.
The ‐l option generates a long listing, including permission flags, the file’s owner, and the last modification time.
The ‐a option causes . .. to be included in the listing.
The ‐d option causes the details of the directories themselves to be listed, rather than their contents.
The ‐S option sort the list in the size.
77dir in DOS
Listing Contents of a Directory The ‐g option list a file’s group.
h i h b f di k bl k h h fil he ‐s option causes the number of disk blocks that the file occupies to be included in the listing. ( A block is typically between 512 and 4K bytes. )
The ‐F option causes a character to be placed after the file’s name to indicate the type of the file: * means an executable file, / means a directory file, @ means a symbolic link, and = means a socket.
The ‐R option recursively lists the contents of a directory and its subdirectories.
78
40
Directory Listing an example Here’s an example of the use of ls :
$ ls ‐‐> list all files in current directory. heart
$ ls ‐l heart ‐‐> long listing of “heart.” ‐rw‐r‐‐r‐‐ 1 huiwang faculty 106 Jan 30 19:46 heart $ _
the name of the file the time that the file was last modified
the size of the file in bytesthe size of the file, in bytesthe group of the owner of the file
the username of the owner of the file the hard‐link count of the file
permission mode of the file
79
Changing Directories: cd cd [directoryName]
The following might be inconvenient; especially if we deal with large hierarchy:
l i /h h fil l$ more lyrics/heart.ver1 ‐‐> see the filel
Instead, change directory:
$ cd lyrics ‐‐> change directory
$ more heart.ver1 ‐‐> see the file
The cd shell command changes a shell’s current working directory to be
80
The cd shell command changes a shell s current working directory to be directoryName.
If the directoryName argument is omitted, the shell is moved to its owner’s home directory.
cd
cd .. one level parentSame in DOS
41
Deleting a Directory: rmdir
rmdir { directoryName }+
The rmdir utility removes all of the directories in the list of The rmdir utility removes all of the directories in the list of directory names provided in the command. A directory must be empty before it can be removed.
To recursively remove a directory and all of its contents, use the rm utility with the ‐r option (see next slide).
Here we try to remove the “lyrics draft” directory while it Here, we try to remove the lyrics.draft directory while it still contains the draft versions, so we receive the following error message:
$ rmdir lyrics.draftrmdir : lyrics.draft : Directory not empty.$ _ 81
Traversing Directories example$ cd
$ pwd ‐‐> display where I am
/cs/home/huiwang
d l i i h “l i ” di $ cd lyrics ‐‐> move into the “lyrics” directory
$ pwd
/cs/home/huiwang/lyrics
$ cd .. ‐‐> move up one level
$ pwd ‐‐> display new position
/cs/home/huiwang
$ cd downloads > move into the “lyrics” directory
82
$ cd downloads ‐‐> move into the lyrics directory
$ pwd
/cs/home/huiwang/downloads
$ ls ~/ ‐‐> “~/” refers to home directory
/cs/home/huiwang/
$ _
42
Basic utilities Introduces the following utilities, listed in alphabetical order:
Generalmanclearcaldatemail
Directorymkdirrmdircdpwdls
Filecatmorecpmvrm
Print Filelplprlprmlpq
83
quota -v headtailchmodchgrpchownnewgrp
pqlpstat
Creating a file with catcat ‐n {fileName}*
The cat utility takes its input from standard input or from a list of filesand displays them to standard output and displays them to standard output.
cat is short for “concatenate” which means “to connect in a series of links.”
By default, the standard input of a process is from the keyboard and the standard output is to the screen.
$ cat > heart ‐‐> store keyboard input into a file called “heart”.I hear her breathing I hear her breathing, I’m surrounded by the sound. Floating in this secret place, I never shall be found. ^D ‐‐> tell cat that the end of input has been reached.$ _
84
43
Displaying a File: cat cat with the name of the file that you wanted to display:
$ cat heart ‐‐> list the contents of the “heart” file.
I hear her breathing.
I’m surrounded by the sound.
Floating in this secret place,
I never shall be found.
$ _
cat is good for listing the contents of small files, but it doesn’t pause between full screens of output.
85
Concatenate files: cat cat with the name of the files that you wanted to concatenate (display together):
$ cat heart heart2 ‐‐> list the contents of both the files. I hear her breathing. I’m surrounded by the sound. Floating in this secret place, I never shall be found. This is a new fileThis is a new file$ _
usually use redirection to create a new file conconatingboth the input files
86
44
Displaying a File: moremore ‐f [+lineNumber] { fileName }*
The more utility allows you to scroll a list of files, one page at a time.
By default, each file is displayed starting at line 1, although the +optiony p y g g pmay be used to specify the starting line number.
The ‐f option tells more not to fold (or wrap) long lines.
After each page is displayed, more displays the message “‐‐more‐‐” to indicate that it’s waiting for a command.
To list the next page, press the space bar.
To list the next line, press the Enter key.
To quit from more, press the “q” key.
^B will display the previous page
H will display help page
Try:
$ ls –la /usr/bin > myLongFile
$ more myLongFile 87
Displaying a File: head and tailhead ‐n { fileName }*
The head utility displays the first n lines of a file. If n is not specified, it e ead ut ty d sp ays t e st es o a e. s ot spec ed, tdefaults to 10.
tail ‐n { fileName }*
The tail utility displays the last n lines of a file. If n is not specified, it defaults to 10.
The first two lines and last two lines of my “heart” file.
$ head ‐2 heart ‐‐> list the first two lines. I hear her breathing, I’m surrounded by the sound. $ tail ‐2 heart ‐‐> list the last two lines.Floating in this secret place, I never shall be found. $ head ‐15 myLongFile ‐‐> see what happens
88
45
Renaming/Moving a File: mvmv ‐i oldFileName newFileName // renamemv ‐i {fileName}* directoryName // move
mv ‐i oldDirectoryName newDirectoryName // rename
rename in DOS
move in DOS
mv ‐i oldDirectoryName existingDirectoryName // move
The first form of mv renames oldFileName as newFileName. The second form allows you to move a collection of files to (under) a
directory.
The third form allows you to rename an entire directory. The fourth form allows you to move the entire directory to (under ) The fourth form allows you to move the entire directory to (under )
another directory.
The ‐i option prompts you for confirmation if newFileName already exists so that you do not accidentally replace its contents. You should learn to use this option (or set a convenient shell alias that replaces “mv” with “mv –i”; we will come back to this later).
89
Renaming/Moving Files: mv
Here’s how to rename the file using the first form of the mv utility: the mv utility:
$ mv heart heart.ver1 ‐‐> rename to “heart.ver1”.
$ ls
heart.ver1
$ _
We will see other uses (moving files) shortly.
90move in DOS
46
Moving Files Once the “lyrics” directory is created, we can move the “heart.ver1” file into its new location. To do so, used mv and confirm the operation using ls:
No real data movment, just entry link switch
confirm the operation using ls:
$ mv heart.ver1 lyrics ‐‐> move into “lyrics”.$ ls ‐‐> list the current directory.lyrics/ ‐‐> “heart.ver1” has gonegone.$ ls lyrics ‐‐> list the “lyrics” directory. heart.ver1 ‐‐> “heart.ver1” has moved.$ _
91
Copying Files: cp To copy the file, I used the cp utility, which works as follows:
cp ‐i oldFileName newFileName
cp /cs/dept/course/2012-13/W/2031 .
cp ‐i oldFileName newFileNamecp ‐ir { fileName }* directoryName
The first form of cp copies the contents of oldFileName to newFileName.
If the label newFileName already exists, its contents are replaced by the contents of oldFileName. y
The ‐i option prompts you for confirmation if newFileNamealready exists so that you do not accidentally overwrite its contents. Like with mv, it is a good idea to use this option or create an alias.
92copy in DOS
47
Copying Files: cp The ‐r option causes any source files that are directories to be
recursively copied, thus copying the entire directory structure.
cp actually does two things It makes a physical copy of the original file’s contents.
It creates a new label in the directory hierarchy that points to the copied file.
$ cp heart.ver1 heart.ver2 ‐‐> copy to “heart.ver2”.p py
$ ls ‐l heart.ver1 heart.ver2 ‐‐> confirm the existence of both files.
‐rw‐r‐‐r‐‐ 1 glass 106 Jan 30 23:28 heart.ver1
‐rw‐r‐‐r‐‐ 1 glass 106 Jan 31 00:12 heart.ver2
$ cp ‐i heart.ver1 heart.ver2 ‐‐> what happens?
93
Reorganizing Directories example
$ cd ‐> move back to my home directory
$ mkdir lyrics final ‐> make the final lyrics directory$ mkdir lyrics.final > make the final lyrics directory
$ mv lyrics lyrics.draft ‐‐> rename the old lyrics dir
$ _
94
48
Deleting files: rm The rm utility allows you to remove a file’s label from the
hierarchy.
rm ‐fir {fileName}*
Th tilit fil ’ l b l f th di t hi h
del in DOS
The rm utility removes a file’s label from the directory hierarchy.
If the filename doesn’t exist, an error message is displayed.
The ‐i option prompts the user for confirmation before deleting a filename. It is a very good idea to use this option or create a shell alias that translates from “rm” to “rm –i”. If you don’t, you will loose some files one day – you have been warned!
The ‐f option inhibits all error messages and prompts. It overrides the –i option (also one coming from an alias). This is dangerous!
If fileName is a directory, the ‐r option causes all of its contents, including subdirectories, to be recursively deleted. Really used for deleting directories. 95
Removing Directories with Files The ‐r option of rm can be used to delete the “lyrics.draft” directory and all of its contents with just y y jone command:
$ cd ‐‐> move to my home directory.
$ rm ‐r lyrics.draft ‐‐> recursively delete directory.
$ $ _
96
49
Counting Lines, Words and Chars in Files:
wcwc ‐lwc {fileName}*
Th ili h b f li d d/ The wc utility counts the number of lines, words, and/or characters in a list of files.
If no files are specified, standard input is used instead.
‐l option requests a line count, ‐w option requests a word count, ‐c option requests a character count.
If no options are specified, then all three counts are displayed.
A word is defined by a sequence of characters surrounded by tabs, spaces, or new lines.
97
Counting Lines, Words and Characters in Files: wc
For example, to count lines, words and characters For example, to count lines, words and characters in the “heart.final” file, we used:
$ cd ~/lyrics.final
$ wc heart.final ‐‐> obtain a count of the number of li‐‐> words, and characters.
9 43 213 heart.final
$cat heart.final | wc
$cat CSE2031M | wc ‐l
98
50
File Attributes
We used ls to obtain a long listing of “heart.final” and got We used ls to obtain a long listing of heart.final and got the following output:
$ ls ‐lgsF heart.final
1 ‐rw‐r‐‐r‐‐ 1 huiwang faculty 213 Jan 31 00:12 heart.final
$ $ _
99
File Attributes
Field # Field value Meaning
1 1 the number of blocks of physical storage occupied by the file
1 -rw-r--r-- 1 huiwang faculty 213 Jan 31 00:12 heart.final
2 ‐ rw‐r‐‐r‐‐ the type and permission mode of the file, which indicates who can read,write, and execute the file
3 1 the hard‐link count
4 huiwang the username of the owner of the file
f lt th f th fil 5 faculty the group name of the file
6 213 the size of the file, in bytes
7 Jan 31 00:12 the time that the file was last modified
8 heart.final the name of the file
100
51
File Attributes File Types
Field 2 describes the file’s type and permission settings.
In ls –lgsF example:
1 ‐ rw‐r‐‐r‐‐ 1 huiwang faculty 213 Jan 31 00:12 heart.final
The first character of field 2 indicates the type of file, which is encoded as follows :
character File Type ‐ regular file d directory file yb buffered special file( such as a disk drive )c unbuffered special file( such as a terminal )l symbolic link p pipe s socket
101
Determining Type of a File: filefile { fileName }+
Th fil tilit tt t t d ib th t t f th fil N The file utility attempts to describe the contents of the fileNameargument(s), including the language in which any of the text is written.
not reliable; it may get confused.
When file is used on a symbolic‐link file, file reports on the file that the link is pointing to, rather than on, the link itself.
l For example,
$ file heart.final ‐‐> determine the file type.heart.final: ascii text $ _
102
52
File Permissions (Security) File permissions are the basis for file security. They are given in three
clusters. In the example, the permission settings are “rw‐r‐‐r‐‐”:
1 ‐ rw‐ r‐‐ r‐‐ 1 glass cs 213 Jan 31 00:12 heart final1 rw r r 1 glass cs 213 Jan 31 00:12 heart.final
User (owner) Group Others
rw‐ r‐‐ r‐‐ clusters
103
Each cluster of three letters has the same format:
Read permission Write permission Execute permissionr w x
File Permission The meaning of the read, write, and execute permissions depends
on the type of file:
R l fil Di fil S i l fil Regular file Directory file Special file
Read read the contents read the directory read from the file(list the names of files using the read() that it contains) system call.
Write change the contents Add or remove file write to the file to/from the directory using the write()
llsystem calls.
Execute execute the file access files in the No meaning. if the file is a program directory
104
53
Change File Permissions: chmodchmod ‐R change{, change}* {fileName }+
The chmod utility changes the modes (permissions) of the specified files according to the change parameters, which may take the following forms:
clusterSelection + newPermissions (add permissons)
clusterSelection ‐ newPermissions (subtract permissions)
clusterSelection = newPermissions (assign permissions absolutely)
where clusterSelection is any combination of:
u (user/owner)
g (group)
( h ) o (others)
a (all)
and newPermissions is any combination of
r (read) w (write)
x (execute) s (set user ID/set group ID)
105
Changing File Permissions The ‐R option recursively changes the modes of the files in directories.
Note that changing a directory’s permission settings doesn’t change the settings of the files that it contains. settings of the files that it contains.
To remove read permission from others, we used chmod as follows:
$ ls ‐l heart.final ‐> to view the settings before the change.‐rw‐r‐‐‐‐‐ 1 glass music 213 Jan 31 00:12 heart.final
$ chmod g‐r heart.final
$ ls ‐lg heart.final‐rw‐‐‐‐‐‐‐ 1 glass music 213 Jan 31 00:12 heart.final$ _
106
54
Changing File Permissions: examples
Requirement Change parameters
Add group write permission g+w
Remove user read and write permission u‐rw
Add execute permission for user, group, a+xand others. Give the group read permission only. g=r
Add writer permission for user, and u+w, g‐rremove group read permission.
107
Changing File Permission: examples Here’s an example of how to set these permissions:
$ cd ‐‐> change to home directory.
$ ls ‐ld . ‐> list attributes of home directory.
drwxr‐xr‐x 45 huiwang faculty 4096 Apr 29 14:35
$ chmod o‐rx ‐‐> update permissions. Other ‐rx
$ ls ‐ld . ‐‐> confirm. $
drwxr‐x‐‐‐ 45 huiwang faculty 4096 Apr 29 14:35
$ _
108
55
Changing File Permissions Using Octal Numbers
The chmod utility allows you to specify the new permission setting of a file as an octal number.
Each octal digit represents a permission triplet. g p p p
rwx 4 2 1
For example, if you wanted a file to have the permission settings of
rwxr‐x‐‐‐
then the octal permission setting would be 750, calculated as follows:
User Group Others
setting rwx r‐x ‐‐‐
binary 111 101 000
octal 7 5 0
109
Changing File Permissions Using Octal Numbers
The octal permission setting would be supplied to chmodp g ppas follows:
$ chmod 750 file ‐‐> update permissions.
$ ls ‐ld file ‐‐> confirm.
‐ rwx r‐x ‐‐‐ 45 huiwang faculty 4096 Apr 29 14:35 45 g y 4 9 p 9 4 35
$ _
For a web file, o musthas r permission.110
111 101 000
56
Basic utilities summary Introduces the following utilities, listed in alphabetical order:
Generalmanclearcaldatemail
Directorymkdirrmdircdpwd
Filecatmorecpmvrm
Filelplprlprmlpqlpstat
111
quota -v headtailchmodchgrpchownnewgrp
112
57
Utilities II – advanced utilitiesIntroduces utilities for power users, listed in alphabetical order:
t d atawkbiffcmpcryptdiffdump
od tartimetrulcompresscpio
grepgunzipperlsedsortsuumount
113
dumpegrepgziplnmount
cpiocroncrontabfgrepfind
uncompressuniqwhoami
We introduce about thirty useful utilities.
section Utilities
Filtering files egrep fgrep grep uniq
Introduces utilities for power users, grouped into logical sets
Utilities II – advanced utilities
Filtering files egrep, fgrep, grep, uniqSorting files sort Comparing files cmp, diff Archiving files tar, cpio, dump Searching for files find Scheduling commands at, cron, crontabProgrammable text processing awk, perlHard and soft links lnSwitching users suCh ki f il biff Checking for mail biff Transforming files compress, crypt, gunzip, gzip,
sed, tr, ul, uncompressLooking at raw file contents odMounting file systems mount, umountIdentifying shells whoamiDocument preparation nroff, spell, style, troffTiming execution of commands time
114
58
Filtering Files grep, uniq egrep, fgrep, and grep, which filter out, all lines that do not contain a
specified pattern. Gives you the line that contains the specified pattern
uniq, which filters out duplicate adjacent lines q, p j
$ cat inputFile ‐‐‐> list the file to be filtered line1 Well you know it’s your bedtime, line2 So turn off the light, line3 Say all your prayers and then, line4 Oh you sleepy young heads dream of wonderful things, line5 Beautiful mermaids will swim through the sea, l d ll b hline6 And you will be swimming there too.
$ grep the grepfile ‐‐‐> search for the word “the”line2 So turn off the light, line3 Say all your prayers and then, line5 Beautiful mermaids will swim through the sea, line6 And you will be swimming there too.
115
Searching for Regex: grep grep comes from the ed (Unix text editor) search command “global
regular expression print” or g/re/p This was such a useful command that it was written as a standalone
utilityy There are two other variants, egrep and fgrep that comprise the grep
family grep is the answer to the moments where you know you want a the file
that contains a specific phrase but you can’t remember it’s name
Family differences
grep ‐ uses regular expressions for pattern matchingg p g p p g egrep ‐ extended grep, uses a more powerful set of regular expressions
but does not support backreferencing, generally the fastest member of the grep family
fgrep ‐ file grep, does not use regular expressions, only matches fixed strings but can get search strings from a file
116
59
Searching for Regex: grep
$ grep ‐nw the grepfile ‐w: whole words only ‐n line number
i S ff h li h 2: Line 2 So turn off the light,
5: line 5 Beautiful mermaids will swim through the sea,
$ grep ‐wv the grepfile ‐‐> reverse the filter.
Line 1 Well you know it’s your bedtime,
line 3 Say all your prayers and then line 3 Say all your prayers and then,
Line 4 Oh you sleepy young heads dream of wonderful things,
Line 6 And you will be swimming there too.
117
-i ignore case CSE2031M /etc/passwd
Searching for Regex: grep$ grep ‐w x *.c ‐‐‐> search all files ending in “.c”.a.c:test ( int x ) fact2.c: long factorial(x) fact2.c: int x; fact2.c: if (( x==1 ) || ( x==0 )) fact2.c: result = x * factorial(x‐1);
$ grep ‐wl x *.c ‐‐‐> list names of files that contain matches.a.cfact2.c $ _
‐i ignore case
118
60
What is a Regular Expression? A regular expression (regex) describes a pattern to match
multiple input strings.
Regular expressions descend from a fundamental concept in Regular expressions descend from a fundamental concept in Computer Science called finite automata theory
Regular expressions are endemic to Unix Some utilities/programs that use them:
vi, ed, sed, and emacs awk, tcl, perl and Python grep, egrep, fgrep
120
Compilers scanf (“%[^/n]s”, str);
The simplest regular expression is a string of literal characters to match.
The string matches the regular expression if it contains the substring.
61
Regular Expressions: Exact Matches
regular expression cks
UNIX Tools rocks.
match
UNIX Tools sucks.
121
match
UNIX Tools is okay.no match
Regular Expressions: Multiple Matches A regular expression can match a string in more than A regular expression can match a string in more than one place.
Scrapple from the apple.
regular expression apple
122
match 1 match 2
62
Regular Expressions: Matching Any Character The . regular expression can be used to match any g p ycharacter.
For me to poop on.
regular expression o .
123
p p
match 1 match 2
Regular Expressions: Alternate Character Classes Character classes [] can be used to match any specific [] y pset of characters.
beat a brat on a boat
regular expression b [eor] a t
124
match 1 match 2 match 3
63
Regular Expressions: Negated Character Classes Character classes can be negated with the [^] syntax.g [ ] y
beat a brat on a boat
regular expression b [^eo] a t
125
match no match
scanf (“%[^/n]s”, str);
Regular Expressions: Other Character Classes Other examples of character classes:
[aeiou] will match any of the characters a, e, i, o, or u [kK]orn will match korn or Korn
Ranges can also be specified in character classes
[1‐9] is the same as [123456789] [abcde] is equivalent to [a‐e]
You can also combine multiple ranges
126
You can also combine multiple ranges
[abcde123456789] is equivalent to [a‐e1‐9]
Note that the ‐ character has a special meaning in a character class but only if it is used within a range
[‐123] would match the characters ‐, 1, 2, or 3
64
Regular Expressions: Named Character Classes
Commonly used character classes can be referred to by name alpha, lower, upper upper, alnum, digit, punct, cntl
Syntax [:name:]
127
[a‐zA‐Z] [[:alpha:]] [a‐zA‐Z0‐9] [[:alnum:]] [45a‐z] [45[:lower:]]
Important for portability across languages
Regular Expressions: Anchors Anchors are used to match at the beginning or end of a line (or both).
^means beginning of the line
$means end of the line$means end of the line
beat a brat on a boat
match
regular expression ^ b [eor] a t
128
regular expression b [eor] a t $
beat a brat on a boat
match
^$^word$
65
Regular Expression: Repetions The * is used to define zero or more occurrences of the single regular expression preceding it.
I got mail, yaaaaaaaaaay!
match
regular expression y a * y
129
For me to poop on.
match
regular expression o a * o
.*
Regular Expressions: Repetion Ranges, Subexpressions Ranges can also be specified
{n,m} notation can specify a range of repetitions for the immediately preceding regex
{n}means exactly n occurrences {n,}means at least n occurrences {n,m}means at least n occurrences but no more than m occurrences
Example: .{0,} same as .* a{2,} same as aaa*
If you want to group part of an expression so that * applies to more than just
130
If you want to group part of an expression so that * applies to more than just the previous character, use ( ) notation
Subexpresssions are treated like a single character a*matches 0 or more occurrences of a abc*matches ab, abc, abcc, abccc, … (abc)*matches abc, abcabc, abcabcabc, … (abc){2,3} matches abcabc or abcabcabc
66
Single Quoting Regex Since many of the special characters used in regexs also have special
meaning to the shell, it’s a good idea to get in the habit of single quoting your regexs
Thi ill i l h f b i d b h This will protect any special characters from being operated on by the shell
If you habitually do it, you won’t have to worry about when it is necessary
Even though we are single quoting our regexs so the shell won’t interpret the special characters, sometimes we still want to use an operator as itself
To do this, we escape the character with a \ (backslash)
131
To do this, we escape the character with a \ (backslash)
Suppose we want to search for the character sequence ‘a*b*’ Unless we do something special, this will match zero or more ‘a’s
followed by zero or more ‘b’s, not what we want! ‘a\*b\*’ will fix this ‐ now the asterisks are treated as regular characters
Extended Regular Expressions: Repetition Shorthands The * (star) has already been seen to specify zero or more
occurrences of the immediately preceding character
The + (plus) means one or moreb +d ill t h b d b d b d b t ill t t h abc+d will match abcd, abccd, or abccccccd but will not match
‘abd’ while abc?d will match abd and abcd but not ‘abccd’ Equivalent to {1,}
The ? (question mark) specifies an optional character, the single character that immediately precedes it July? will match Jul or July Equivalent to {0,1} Also equivalent to (Jul|July)
132
q ( | y)
The *, ?, and + are known as quantifiers because they specify the quantity of a match
Quantifiers can also be used with subexpressions (a*c)+ will match c, ac, aac or aacaacac but will not match ‘a’ or a blank line
67
Regular Expressions: Some Practical Examples Variable names in C
[a-zA-Z_][a-zA-Z_0-9]*
Dollar amount with optional cents
\$[0-9]+(\.[0-9][0-9])?
Time of day
(1[012]|[1 9]) [0 5][0 9] ( | )
133
(1[012]|[1-9]):[0-5][0-9] (am|pm)
HTML headers <h1> <H1> <h2> … <[hH][1-4]>
Regular Experessions: Quick Refrences
fgrep, grep, egrepx
xyz
Ordinary characters match themselves (NEWLINES and metacharacters excluded)Ordinary strings match themselves
\m^
Matches literal character mSt t f li
grep, egrep
^$.
[xy^$x][^xy^$z]
[a-z]r *
r1r2
Start of lineEnd of lineAny single characterAny of x, y, ^, $, or zAny one character other than x, y, ^, $, or zAny single character in given rangezero or more occurrences of regex rMatches r1 followed by r2
\(r\)\n
Tagged regular expression, matches rSet to what matched the nth tagged expression (n =
134
grep
egrep grep ‐E
\{n,m\}1-9)Repetition
r+r?
r1|r2(r1|r2)r3(r1|r2)*
{n,m}
One or more occurrences of rZero or one occurrences of rEither r1 or r2Either r1r3 or r2r3Zero or more occurrences of r1|r2, e.g., r1, r1r1, r2r1, r1r1r2r1,…)Repetition
68
grep and egrep RE
Pattern Maning Example
c Non-special, matches itself 'tom'
\c Turn off special meaning '\$'p g $
$ End of line 'ab$'
^ Start of line ^ab
. Any single character '..nodes'
[…] Any single character in [] '[tT]he'
[^…] Any single character not in [] '[^tT]he'
R* Zero or more occurrences of R 'e*'
135
R Zero or more occurrences of R e
R+ One or more occurrences of R (egrep) 'e+'
R? Zero or one occurrences of R (egrep) 'e?'
R1R2 R1 followed by R2 '[st][fe]'
R1|R2 R1 or R2 (egrep) 'the|The'
Regular expression and extended expression maybe Regular expression and extended expression maybe confusing.
Grep may behave differently in different shells.
So for this course
Use grep –E or egrepg p g p
Work on Burne sh (sh)
136
69
Do some more examplesWell you know it’s your bedtime,
So turn off the light,
Say all your prayers and then,
Oh you sleepy young heads dream of wonderful things,
Beautiful mermaids will swim through the sea,
And you will be swimming there too. y g
137
Removing Duplicate Lines: uniq The uniq utility displays a file with all of its identical adjacent
lines replaced by a single occurrence of the repeated line.
Here’s an example of the use of the uniq utility:
$ cat animals ‐‐‐> look at the test file. cat snake monkey snake dolphin elephant dolphin elephant goat elephant g ppig pigpig pigmonkey pig pig pig
138
70
Removing Duplicate Lines: uniq$ uniq animals ‐‐‐> filter out duplicate adjacent lines.cat snake monkey snake dolphin elephant dolphin elephant goat elephant pig pigmonkey pig pig pig$ uniq ‐c animals ‐‐‐> display a count with the lines.
1 cat snake 1 monkey elephant 1 monkey elephant 2 dolphin elephant 1 goat elephant 2 pig pig1 monkey pig 1 pig pig
139
How about un-adjacent lines?sort and then uniq
Sorting Files : sort The sort utility sorts a file in ascending or descending order
based on one or more soft fields.
$ cat sortfile ‐‐> list the file to be sorted. J St t h t th Jan Start chapter 1 30th Jan Start chapter 5 23rd Jan End chapter 3 23rd Mar Start chapter 7 27 Apr End Chapter 5 1 Feb End chapter 1 14
$ sort sortfile ‐‐> sort it. A E d Ch Apr End Chapter 5 1 Feb End chapter 1 14 Jan Start chapter 1 30th Jan End chapter 3 23rd Jan Start chapter 5 23rd Mar Start chapter 7 27
140
71
Sorting Files : sort
$ sort ‐r softfile ‐‐> sort it in reverse order.
Mar Start chapter 7 27 Mar Start chapter 7 27
Jan Start chapter 5 23rd
Jan End chapter 3 23rd
Jan Start chapter 1 30th
Feb End chapter 1 14
A E d Ch Apr End Chapter 5 1
141
Sort sorts a file in ascending or descending order based on sorts a file in ascending or descending order based on one or more fields.
Individual fields are ordered lexicographically, which means that corresponding characters are compared based on their ASCII value.
72
Sort sort ‐tc ‐r [+POS1 [‐POS2]] {sortField ‐bfMn}* {fileName}*{fileName}
‐tc separator is c instead of blank tab
‐r descending instead of ascending
‐b ignore leading blanks
‐f ignore case
‐M month sort (3 letter month abbreviation)(3 )
‐n numeric sort
+POS1 [‐POS2] key positions start [up to end]
Sort Examplessh-3.00$ cat ftoSortJohn Smith 1222 26 Apr 1956John Smith 1222 26 Apr 1956Tony Jones 1012 20 Mar 1950John Duncan 1111 20 Jan 1966Larry Jones 1223 20 Dec 1946Lisa Sue 1222 15 Jul 1980
sh-3.00$ sort ftoSortJohn Duncan 1111 20 Jan 1966John Duncan 1111 20 Jan 1966John Smith 1222 26 Apr 1956Larry Jones 1223 20 Dec 1946Lisa Sue 1222 15 Jul 1980Tony Jones 1012 20 Mar 1950
73
sh-3.00$ sort +2 -3 –n ftoSort # start 0Tony Jones 1012 20 Mar 1950John Duncan 1111 20 Jan 1966John Duncan 1111 20 Jan 1966John Smith 1222 26 Apr 1956Lisa Sue 1222 15 Jul 1980Larry Jones 1223 20 Dec 1946
sh-3.00$ sort +2 -3 +3 -4 -n ftoSortTony Jones 1012 20 Mar 1950Tony Jones 1012 20 Mar 1950John Duncan 1111 20 Jan 1966Lisa Sue 1222 15 Jul 1980John Smith 1222 26 Apr 1956Larry Jones 1223 20 Dec 1946
145
Sort Examplessh-3.00$ sort +4 -5 ftoSortJohn Smith 1222 26 Apr 1956Larry Jones 1223 20 Dec 1946Larry Jones 1223 20 Dec 1946John Duncan 1111 20 Jan 1966Lisa Sue 1222 15 Jul 1980Tony Jones 1012 20 Mar 1950
sh-3.00$ sort +4 -5 -M ftoSortJohn Duncan 1111 20 Jan 1966Tony Jones 1012 20 Mar 1950yJohn Smith 1222 26 Apr 1956Lisa Sue 1222 15 Jul 1980Larry Jones 1223 20 Dec 1946
74
sh-4.1$ sort +0 -1 sortfileApr End chapter 5 1Feb End chapter 1 14Jan End chapter 3 23rdJan Start chapter 1 30thJan Start chapter 5 23rdMar Start chapter 7 27May End chapter 7 17th
sh-4.1$ sort +0 -1 -M sortfileJan End chapter 3 23rdJan Start chapter 1 30thJan Start chapter 5 23rdpFeb End chapter 1 14Mar Start chapter 7 27Apr End Chapter 5 1May End chapter 7 17th
sh-4.1$
sort + uniq uniq is a little limited but we can combine it with sortuniq is a little limited but we can combine it with sort
sort | uniq -c
counts number of times line appears in file
output would now be:
3 abacus
1 bottle
148
1 bottle
75
sort + uniq To understand:
abacus
abacus
bottle
abacus
abacus
abacus
3 abacus
1 bottlesort uniq
To understand:
149
abacus bottle
Comparing Files: cmp, diff There are two utilities that allow you to compare the contents of two files:
cmp, which finds the first byte that differs between two files diff, which displays all of the differences and similarities between two files
Testing for sameness: cmp
The cmp utility determines whether two files are the same.
$ cat lady1 ‐‐> look at the first test file. Lady of the night, I hold you close to me, And all those loving words you say are right. $ t l d l k t th d t t fil $ cat lady2 ‐‐> look at the second test file. Lady of the night, I hold you close to me, And everything you say to me is right. $ cmp lady1 lady2 ‐‐ > files differ. lady1 lady2 differ: char 48, line 3$ _
150
76
File Differences: diff
The diff utility compares two files and displays a list of editing changes that would convert the first file into g gthe second file.
$ diff lady1 lady2 ‐‐> compare lady1 and lady2.
3c3
< And all those loving words you say are right. g y y g
…
> And everything you say to me is right.
$ _
151
Finding Files: find find startingDir searchOptions commandToPerform
$ find /code ‐name ‘*.c’ ‐print ‐‐> print C source files ‐‐> in the current directory or
f bd‐‐> any of its subdirectories../proj/fall.89/play.c./proj/fall.89/rerefee.c./proj/fall.89/player.c./rock/guess.c./rock/play.c./rock/player.c./rock/referee.c$ find /code mtime 14 ls > list modified files during the last 14 days$ find /code ‐mtime ‐14 –ls ‐‐> list modified files during the last 14 days…$ find . ‐name ‘*.txt’ ‐ print ‐‐> find all text files in the current directory…
152
77
Find Utility find pathList expression
finds files starting at pathList finds files starting at pathList
finds files descending from there
Allows you to perform certain actions
e.g. deleting the files
Find Utility ‐atime count
true if the file has been accessed within count daystrue if the file has been accessed within count days
‐ctime count
true if the contents of the file have been modified within count days or any of its file attributes have been modified
‐exec command
true if the exit code = 0 from executing the command.
command must be terminated by \; command must be terminated by \;
If {} is specified as a command line argument it is replaced by the file name currently matched
78
Find Utility ‐print
prints out the name of the current file and returns trueprints out the name of the current file and returns true
‐ls
displays the current file's attributes and returns true
!expression
negation of expression
expr1 [‐a] expr2
h t i it d short circuit and
expr1 ‐o expr2
short circuit or
Find Examples
$ find / ‐name x.c
searches for file x c in the entire file system searches for file x.c in the entire file system
$ find . ‐mtime 14 ‐ls
lists files modified in the last 14 days
$ find . ‐name '*.bak' ‐ls ‐exec rm {} \;
ls and then remove all files that end with .bak
$ find name ‘a2 c’ exec cp {} a2 c bak \; $ find . ‐name a2.c ‐exec cp {} a2.c.bak \;
$ find . ‐name ‘a2.c’ ‐exec cp {} {}.bak \;
Find a2.c and then cp it to a2.c.bak
79
Hard Links: ln The ln utility allows you to create both hard links and symbolic (soft)
links between files.
In the following example, we add a new label “hold” to the file referenced by the existing label “hold 3” referenced by the existing label hold.3 .
$ ls ‐l ‐‐> look at the current contents of the directory.total 3 ‐rw‐r‐‐r‐‐ 1 huiwang faculty 123 Jan 12 17:32 hold.1‐rw‐r‐‐r‐‐ 1 huiwang faculty 89 Jan 12 17:34 hold.2 ‐rw‐r‐‐r‐‐ 1 huiwang faculty 91 Jan 12 17:34 hold.3$ ln hold.3 hold ‐‐> create a new hard link.$ ls ‐l ‐‐> look at the new contents of the directory.total 4 ‐rw‐r‐‐r‐‐ 2 huiwang faculty 91 Jan 12 17:34 hold ‐rw‐r‐‐r‐‐ 1 huiwang faculty 124 Jan 12 17:32 hold.1 ‐rw‐r‐‐r‐‐ 1 huiwang faculty 89 Jan 12 17:34 hold.2 ‐rw‐r‐‐r‐‐ 2 huiwang faculty 91 Jan 12 17:34 hold.3
157
Hard Links: ln
$ rm hold ‐‐> remove one of the links.
$ ls ‐l ‐‐> look at the updated contents of the directory$ ls ‐l ‐‐> look at the updated contents of the directory.
total 3
‐rw‐r‐‐r‐‐ 1 huiwang faculty 123 Jan 12 17:32 hold.1
‐rw‐r‐‐r‐‐ 1 huiwang faculty 89 Jan 12 17:34 hold.2
‐rw‐r‐‐r‐‐ 1 huiwang faculty 91 Jan 12 17:34 hold.3
$ _
Note that the hard‐link count field was incremented from one to two when the hard link was added and then went back to one again when the hard link was deleted:
158
80
Soft Links: ln ‐sTo create a soft link, use ln -s
$ ls -ltotal 48-rw-r--r-- 1 user users 84 26 Sep 17:08 tmp
1 24 26 S 19 41 t 1-rw-r--r-- 1 user users 24 26 Sep 19:41 tmp1$ ln tmp tmp2$ ls -ltotal 56-rw-r--r-- 2 user users 84 26 Sep 17:08 tmp-rw-r--r-- 1 user users 24 26 Sep 19:41 tmp1-rw-r--r-- 2 user users 84 26 Sep 17:08 tmp2$ ln -s tmp tmp3$ ls -ltotal 64total 64-rw-r--r-- 2 user users 84 26 Sep 17:08 tmp-rw-r--r-- 1 user users 24 26 Sep 19:41 tmp1-rw-r--r-- 2 user users 84 26 Sep 17:08 tmp2lrwxr-xr-x 1 user users 3 27 Sep 09:08 tmp3 -> tmp$ _
159
Identifying Shells: whoami
You can use the whoami utility to display the owner of a shell:
Utility: whoami
Displays the owner of a shell.
For example, when user glass executes whoami at his terminal, he sees this: he sees this:
$ whoami
huiwang
$ _
160
81
Substituting a User: su stands for substitute user. This utility allows you to create a subshell owned by another user.
$ whoami ‐‐‐> find out my current user ID. $ yhuiwang$ su ‐‐‐> substitute user. Password: ‐‐‐> enter super‐user password here.$ whoami ‐‐> confirm my current user ID has changed.root $ … perform super‐user tasks here. $ ^D ‐‐> terminate the child shell.
You can also use sudo for “one shot” command execution
$ sudo rm ‐r / ‐‐‐> don’t try this! As a super‐user, you could do it.
Use su ‐u anotherUser for a particular user
161
cut deal with fields (columns)
Used to split lines of a fileUsed to split lines of a file
A line is split into fields
Fields are separated by delimiters
A common case where a delimiter is a space:
162
field
delimiter
hello there world
82
cut Syntax
John Smith 1222 26 Apr 1956Tony Jones 1012 20 Mar 1950
field2 field1 field3
Syntax
cut [-ffields] [-ccolumns][-dcharacter] [filename …]
If filenames are given on command line, input is taken from those files
If no filenames are given, input comes from stdin
163
g p
This approach to input is very common
cut
Two main forms ‐ extracting fields
t f3 d “ ” cut -f3 –d “,” extract field 3 from each line (string 1)
fields are separated by ',’
Default is tab, (not “ ”) cut –f3 ‐d” ”
e g with an input of
164
e.g. with an input of
hello,there,world,!
output would be just "world"
83
sh-3.00$ cat ftoSortJohn Smith 1222 26 Apr 1956Tony Jones 1012 20 Mar 1950John Duncan 1111 20 Jan 1966Larry Jones 1223 20 Dec 1946Lisa Sue 1222 15 Jul 1980
indigo 397 % cut -f1 ftoSortJohnTonyJohnLarryLisaindigo 398 % cut f1 3 ftoSort indigo 399 % cut f 1 3 ftoSortindigo 398 % cut -f1,3 ftoSortJohn 1222Tony 101John 1111Larry 1223Lisa 1222
indigo 399 % cut -f 1-3 ftoSortJohn Smith 1222Tony Jones 101John Duncan 1111Larry Jones 1223Lisa Sue 1222
Transforming Files: tr tr copies the standard input to the standard output with
substitution or deletion of selected characters input characters from string1 are replaced with the
corresponding characters in string2 (the last character in string2 i d f ddi if )is used for padding if necessary)
tr –cds string1 string2
the –d option results in deletion from the input string of all characters specified in string1
the –s option condenses all repeated characters in the input string to a single character
with the –c option, string1 is substituted with its complement; i.e., all characters that are not in the original string1
166
84
Transforming Files with tr: Examples
$ tr xyz abcthis is a x world y zthi i ld bthis is a a world b c^D
$ cat aFilea lot of space
$tr alo xyz < aFile --> transfer a to x, l to y--> and o to z
x yzt zf spxce
167
Transforming Files with tr: Examples--> transfer all lower case charcters
--> to upper case$ tr [a-z] [A-Z] [:lower:] [:upper:]A LOT OF SPACEA LOT OF SPACE
--> transfer all spaces into dots$ tr " " .a...lot.....of......space
--> transfer all spaces into dots and--> compact the dotsp
$ tr " " . | tr -s "."a.lot.of.space
$ tr 0-9 “*” < input.txt //replace digits with *
168
85
AWK
Largely manipulate fields columns. l Alternative to cut
awk gets it’s input from files
directly from standard input
169
Pattern Scanning and Processing
awk: utility that scans one or more files and performs an action on all lines that match a particular conditionan action on all lines that match a particular condition
The conditions and actions are specified in an awk program.
awk reads a line
breaks it into fields separated by tabs/spaces
or other separators specified by ‐F optionor other separators specified by F option
86
awk [condition] [ \{ action \} ]
action is one of the following kinds of C‐like statements
if‐else; while; for; break; continue
i t t t t i assignment statement: var=expression
print; printf;
next (skip remaining patterns on current line)
exit (skips the rest of the current line)
list of statements
$awk ‘{print}’
$awk ‘/eric/’ data.txt
$awk ‘ /eric/ {print}’ data.txt
$awk ‘/eric/ {print $0}’ data.txt
awk Command accessing individual fields:
$1 $n refer to fields 1 thru n $1, ..., $n refer to fields 1 thru n
$0 refers to entire line
built‐in variable NF means number of fields % awk -F: '{ print NF, $1 }' /etc/passwd
prints the number of fields and the first field in the /etc/passwd file/etc/passwd file
‐F: means to use : as the field separator
87
awk: Fields Each input line is split into fields.
Special variable FS: field separator: default is whitespace (1 b )or more spaces or tabs)
awk –F: sets FS to the character : can also be changed in BEGIN
$0 is the entire line
173
$1 is the first field, $2 is the second field, …., $NF is the last field
Only fields begin with $, variables are unadorned
awk: Simple Output From AWK Printing Every Line
If an action has no pattern, the action is performed to all input lines
{ print }{ p }will print all input lines to standard out
{ print $0 }will do the same thing
Printing Certain Fields
multiple items can be printed on the same output line with a single print statement
174
statement
{ print $1, $3 }
expressions separated by a comma are, by default, separated by a single space when output
88
sh‐3.00$ more data.txt
Lisa 28 30.5 A
Zhao 44 17 b
eric 20 15.2 C
jeason 3 3.4 d
jin 3 4.3 e
tien 3 2.1 c
awk '{print $1, $3}' data.txtlisa 30.5
zhao 17
eric 15.2
jeason 3.4
jin 4.3
tien 2.1
175
sh‐3.00$ more data.txt
Lisa 28 30.5 A
Zhao 44 17 b
eric 20 15.2 C
jeason 3 3.4 d
jin 3 4.3 e
tien 3 2.1 c
awk ‘\eric\ {print $1, $3}' data.txteric 15.2
\ \ $ $awk ‘\eric\ {print $1, “--”, $3}' data.txteric -- 15.2
176
89
Another simple example
Problem : Get the userid of user “eric” from the /etc/passwd file.
Suppose /etc/passwd file contains the following entries
eric:x:504:504::/home/eric:/bin/bash
try:x:500:500::/home/try:/bin/bash
optima:x:501:501::/home/optima:/bin/bash
optimal:x:502:502::/home/optimal:/bin/bash
awk will see this file as follows
li d (b d f l ) i l h d i h fil 1 line = 1 record (by default) so in total there are 4 records in the file.
1 record = 7 fields separated by “:” (Not by default)
Note : Default field separator is space.
A simple example (cont..)
$ awk –F”:” ‘/eric/ {print $1 “ “ $3}’ /etc/passwd
Awk executable
pattern
Action to perform on line
If pattern matchesField
Separator
pattern to search The file
to operate upon
eric 504
90
Utilities II – advanced utilitiesIntroduces utilities for power users, listed in alphabetical order:
t d atawkbiffcmpcryptdiffdump
od tartimetrulcompresscpio
grepgunzipperlsedsortsuumount
179
egrepgziplnmount
croncrontabfgrepfind
uncompressuniqwhoami
We introduce about thirty useful utilities.
section Utilities
Filtering files egrep fgrep grep uniq
Introduces utilities for power users, grouped into logical sets
Utilities II – advanced utilities
Filtering files egrep, fgrep, grep, uniqSorting files sort Comparing files cmp, diff Archiving files tar, cpio, dump Searching for files find Scheduling commands at, cron, crontabProgrammable text processing awk, perlHard and soft links lnSwitching users su Ch ki f il biff Checking for mail biff Transforming files compress, crypt, gunzip, gzip,
sed, tr, ul, uncompressLooking at raw file contents odMounting file systems mount, umountIdentifying shells whoamiDocument preparation nroff, spell, style, troffTiming execution of commands time
180
91
Transforming Files: sed Stream‐oriented, non‐interactive, text editor
Alternative to tr
Look for patterns one line at a time and change lines accordingly
like awk.
182
sed vs. awk
sed is a pattern‐action language, like awk
awk processes fields while sed only processes lines
tr process word by word
92
sed Syntax sed [‐n] [‐e] [‘command’] [file…]
sed [‐n] [‐f scriptfile] [file…][ ] [ p ] [ ]
‐n ‐ only print lines specified with the print command (or the ‘p’ flag of the substitute (‘s’) command)
‐f scriptfile ‐ next argument is a filename containing editing commands
183
‐e command ‐ the next argument is an editing command rather than a filename, useful if multiple commands are specified
If the first line of a scriptfile is “#n”, sed acts as though ‐n had been specified
Scripts A script is nothing more than a file of commands
Each command consists of an address and an action, ,where the address can be a regular expression or line number.
address action command
address action
address action
184
address action
address action
script
93
Scripts (continued)
As each line of the input file is read, sed reads the first p ,command of the script and checks the address against the current input line: if there is a match, the command is executed
if there is no match, the command is ignored
sed then repeats this action for every command in the script file
185
p
When it has reached the end of the script, sed outputs the current line (pattern space) unless the ‐n option has been set
sed Commands sed commands have the general form
[address[, address]][!]command [arguments]
sed copies each input line into a pattern space
if the address of the command matches the line in the pattern space, the command is applied to that line
if the command has no address, it is applied to each line as it enters pattern space
186
if a command changes the line in pattern space, subsequent commands operate on the modified line
When all commands have been read, the line in pattern space is written to standard output and a new line is read into pattern space
94
Addressing An address can be either a line number or a pattern, enclosed in slashes
( /pattern/ ) A pattern is described using regular expressions (BREs, as in grep)
If no pattern is specified, the command will be applied to all lines of the input file
To refer to the last line: $
Most commands will accept two addresses If only one address is given, the command operates only on that line If two comma separated addresses are given, then the command
187
If two comma separated addresses are given, then the command operates on a range of lines between the first and second address, inclusively
The ! operator can be used to negate an address; i.e., address!commandcauses command to be applied to all lines that do not match address
Commands command is a single letter
Example: Example:
Deletion: d
[address1][,address2]d
D l t th dd d li ( ) f th tt li ( )
188
Delete the addressed line(s) from the pattern space; line(s) not passed to standard output.
A new line of input is read and editing resumes with the first command of the script.
95
Address and Command Examples: deleted deletes all lines
6d deletes line 6
/^$/d deletes all blank lines
1,10d deletes lines 1 through 10
1,/^$/d deletes from line 1 through the first blank line
/^$/,$d deletes from the first blank line through the last line of the file
189
/^$/,10d deletes from the first blank line through line 10
/^ya*y/,/[0‐9]$/d deletes from the first line that beginswith yay, yaay, yaaay, etc throughthe first line that ends with a digit
Sed CommandsAlthough sed contains many editing commands, we are only going to cover the following subset:
b is ‐ substitutea ‐ appendi ‐ insertc ‐ changed – deletep ‐ printr ‐ readw ‐ write
190
w writey ‐ transform= ‐ display line numberN ‐ append the next line to the current oneq ‐ quit
96
Print The print command (p) can be used to force the pattern space to be
output, useful if the ‐n option has been specified
Syntax: Syntax:
[address1[,address2]]p
Note: if the ‐n or #n option has not been specified, p will cause the line to be output twice!
Examples:
191
1,5p will display lines 1 through 5
/^$/,$p will display the lines from the firstblank line through the last line of the file
Substitute Syntax:
[address(es)]s/pattern/replacement/[flags]
pattern ‐ search pattern replacement ‐ replacement string for pattern flags ‐ optionally any of the following
n a number from 1 to 512 indicating which occurrence of pattern should be replaced
g global replace all occurrences of pattern
192
g global, replace all occurrences of pattern in pattern space
p print contents of pattern space
97
Substitute Examples
$ sed ‘s/Puff Daddy/P. Diddy/’ data.txt
S b tit t P Didd f th fi t Substitute P. Diddy for the first occurrence
of Puff Daddy in pattern space
$ sed ‘s/Tom/Dick/2’ data.txt
Substitutes Dick for the second occurrence
of Tom in the pattern space
193
$ ‘s/wood/plastic/p’ data.txt
Substitutes plastic for the first occurrence
of wood and outputs (prints) pattern space
Substitute Examples$ sed ‘s/[1‐9]/*/’ data.txt
Substitute digits with *g
tr [1‐9] ‘*’ < data.txt
$ sed ‘1,3 s/[1‐9]/*/’ data.txt
Substitute digits in line 1‐3 with *
$ sed ‘s/[a‐z]//’ data txt
194
$ sed s/[a z]// data.txt
Substitute lower case letter with blank (remove)
98
Substituting Text $ sed 's/^/ /' data.txt
indents each line in the file by 2 spacesindents each line in the file by 2 spaces
$ sed 's/^ *//' data.txt
removes all leading spaces from each line of the file
$ sed '/a/d' file > file.new
deletes all lines containing 'a'
Example: add two lines in the beginning of the file
% cat sed11i\abcd\efg% sed -f sed1 file > file.new
