CS2204: Introduction to Unix Operating Systemcourses.cs.vt.edu/~cs2204/fall2004/resources/CS2204_notes_Spring_2004.pdfSystem calls kernel user mode Christian Allgood, Virginia Tech,

1

CS2204: Introduction to Unix

January 19th, 2004Class Meeting 1

* Notes adapted by Christian Allgood from previous work by other members of the CS faculty at Virginia Tech

Christian Allgood, Virginia Tech, 2004 2

What is Unix?

A modern computer operating systemOperating System

“a program that acts as an intermediary between a user of the computer and the computer hardware”Software that manages your computer’s resources (files, programs, disks, network)Examples: Windows, MacOS, Solaris, BSD, Linux (e.g. Mandrake, Red Hat, Slackware)

ModernStable, flexible, configurable, allows multiple users and programs


Why Unix?

Used in many scientific and industrial settingsOpen-source operating system (OS)Huge number of free and well-writtensoftware programsExcellent programming environmentInternet servers and services run on Unix

Roughly 65% of the world’s web servers are Linux/Unix machines running Apache


Brief History of Unix

Ken Thompson and Dennis Richieoriginally developed the earliest versions of Unix at Bell Labs for internal use in the 1970s

Simple and elegantWritten in a high-level language instead of assembly language

Small portion written in assembly language (kernel)Remaining code written in C on top of the kernel


Unix Variants

Two main threads of developmentBerkeley software distribution (http://www.bsd.org)Unix System Laboratories (http://www.unix.org)

Sun: SunOS, SolarisSGI: IrixFreeBSD, OpenBSD, NetBSDHewlett-Packard: HP-UXApple: OSX (Darwin)Linux (many flavors)


Brief History of Linux

Andrew Tanenbaum, a Dutch professor developed MINIX to teach the inner workings of operating systems to his studentsIn 1991 at the University of Helsinki, Linus Torvalds, inspired by Richard Stallman’s GNU free software project and the knowledge presented in Tanenbaum’soperating system, created Linux, an open-source, Unix-based operating systemOver the last decade, the effort of thousands of open-source developers has resulted in the establishment of Linux as a stable, functional operating system

2


Layers in a Unix-based System

Hardware(CPU, memory, disks, terminals, etc.)

Unix Operating System(process/memory management, file system, I/O)

Standard Library(open, close, read, write, etc.)

Standard Utility Programs(shells, editors, compilers, etc.)

UsersUser Interface

Library Interface

System calls

kernel

user mode


Unix Structure

The kernel is the core of the Unix operating system, controlling the system hardware and performing various low-level functions. Other parts of a Unix system (including user programs) call on the kernel to perform services for them.The shell accepts user commands and is responsible for seeing that they are carried out.


Unix Structure (cont.)

Over two hundred utility programs or toolsare supplied with the Unix system. These utilities (or commands) support a variety of tasks such as copying files, editing text, performing calculations, and developing software.This course will introduce a limited number of these utilities and tools, focusing on those that aid in software development.


Getting Started

Logging in to a Unix machine requires an account on that systemAfter logging in, some information about the system will be displayed, followed by a shell prompt, where commands may be entered

$%#username@hostname>hostname%


The Shell

The shell is the program you use to send commands to the Unix systemSome commands are a single wordwhodatels

Others use additional informationcat textfilels –l


Command Syntax

Commands must be entered exactlycommand options argument(s)

Options modify a command’s executionArguments often indicate upon what a command should act (often filenames)

3


Example Command: ls

ls –lls –als –lals –a; ls –lls –al textfile1ls –al textfile1 textfile2ls –al directory


Logging Out

Always log out when you are doneUse the exit command to log out of a shellNote: If you are running a window system, logging out of a shell only ends that shell. You must also log out of the window system, typically using a graphical menu.

1

Unix Filesystem

January 26th, 2004Class Meeting 2

* Notes adapted by Christian Allgood from previous work by other members of the CS faculty at Virginia Tech

Unix Filesystem

The filesystem is your interface to:physical storage (disks) on your machinestorage on other machinesoutput devices

Everything in Unix is a file (programs, textfiles, peripheral devices, terminals)There are no drive letters in Unix – the filesystem is a logical view of the storage devices

Working Directory

Working Directory – the current directory in which you are locatedpwd (print working directory) command outputs the absolute path (more later) of your working directoryUnless you specify another directory, commands will assume that you want to operate on the working directory

Home Directory

Directory for users to store personal filesAt login, your working directory will be set to your home directoryThe path (more later) to your home directory can be referred to by the ~(tilde) symbolThe home directory of “user1” is represented by ~user1

Unix File Hierarchy

Root Directory: /Directories may contain plain files or other directoriesResult is a tree structure for the filesystemUnix does not recognize any special filename extensions

/

bin home etc

user1 user2

cs2204textfile

lab1txt lab2txt

Unix Paths

Separate directories by /Absolute Path

start at root and follow the tree

Example:/home/user1/textfile~users1/textfile~/textfile

/

bin home etc

user1 user2

cs2204textfile

lab1txt lab2txt

2

Unix Paths (cont)

Relative Pathstart at working directory. . – level above. – working directory

/

bin home etc

user1 user2

cs2204textfile

lab1txt lab2txt

Some Standard Directories

/ - root directory/bin – standard commands and utilities/dev – block and character device directory/etc – host-specific configuration files and directories/home – users directory/lib – library directory/sbin – system commands and utilities (needed to boot)/tmp – temporary files/usr – most user utilities and applications/var – files that vary as the system runs (logs, spools)

Changing Directories

cd – changes the working directorycd <directory_path>can use absolute or relative path namescd without any arguments is the same as: cd ~Examples:

cd /home/user1cd ../../user1

Output of ls -lFlrwxrwxrwx 1 callgood Grads 20 Jan 24 20:16 home ->

/home/grads/callgood/-rw-r--r-- 1 callgood Grads 392419 Sep 22 10:07

atoll.jpgdrwxr--r-- 2 callgood Grads 64 Jan 24 18:33 cs2204/

We’ll keep coming back to this slide

permissions user group modified date filename

file type size

Types of Files

Plain ( - )Most files, binary or text

Directory ( d )Directory is actually a filePoints to another set of files

Link ( l )Pointer to another file or directory

Specialb – block device (disks, CD-ROM)c – character device (keyboard, joystick)

Manipulating Files

touch <file>create a new file or change last modified date

mv <file1> <file2> rename file1 as file2

mv <file1> <dir> move file1 into the dir directory

mv <file1> <dir/file2>move file1 into dir and rename as file2

cp <file1> [<file2>|<dir>|<dir/file2>]copy file with new name, into directory, or both

rm [-i] <file(s)>remove file or list of files

3

Creating and Removing Directories

mkdir <directory_name>create a subdirectory of the current directory

rmdir <directory_name>remove directoryonly works for empty directories

rm -r <directory_name>remove directory and all of its contents, including subdirectories

Creating Links

ln -s <existing_file> <link_name>creates a symbolic linklink_name will be a pointer to existing_file which may be in another directory or even on another physical machine

File Ownership

Each file has a single ownerchown command can be used to change the owner; usually only root user can use this commandUsers can also belong to various groupsGroups may have different permissions than everyone else

File Permissions

Permissions are used to allow or disallow access to files or directoriesThree types of permission:

Read ( r )Write ( w )Execute ( x )

Permission exists on three levels:User ( u )Group ( g )World ( o )

File Permissions (cont)

chmod <mode> <file(s)>chmod 700 textfile

chmod g+rw textfile

r w xuser

r w xgroup

r w xworld

ugo rwx+/-

File Modification Date

Last time a file was changedUseful when . . .

there are many copies of a filemany users are working on a file

touch command can be used to update the modification date to the current date (or to create a file if it doesn’t exist)

4

Looking at File Contents

cat <filename(s)>short for concatenateoutput the contents of the file all at once

more <filename(s)>output the contents of the file one screen at a timeallows forward and backward search

Getting Help on Unix Commands

man <command_name>shows all of the documentation for a command (more-style output)

apropos <keyword>shows you all of the commands with the specified keyword in their description

1

Text Editing

February 2nd, 2004Class Meeting 3

Text Editing

Last week, you learned to manipulate files in the file system (cp, mv, rm, ln) and to view their contents (cat, more)How do you modify the contents of files?Unix editors work with plain ASCII text files: vi, emacs, picoWindows systems use their own window-based editors: Notepad, Wordpad

Why vi?

Availabilityincluded on practically all Unix/Linux systems

Command-line editor, allowing use with remote loginSimple and powerful

vi Basics

Invoke with: vi [filename(s)]Editor takes over screenVarious modes:

command mode – characters send commands to editor (saving, quitting, searching, replacing)insert modeappend modechange mode

typing changes the text

Command Mode

vi starts in this modeFrom other modes, typing Esc will enter command modeCommands for:

Cursor movementEditingFile operationsSearchingEntering other modes

Cursor Movement

Up: kDown: jLeft: hRight: lLarger Movements

n{j|k|l|h} – move n characters/lines up/down/left/rightCtrl-F, Ctrl-B – page down, page upw – move to the beginning of the word:n – move to line n0, $ – move to beginning or end of file

2

Editing Commands

u – undo last typingx – delete current characterdd – delete current linedw – delete current wordrx – replace current character with xyy – copy current linep – paste copied/deleted itemsJ – join two lines

can be preceded by a number to perform operation multiple times

File Operations

ZZ, :wq – save and quit:w – save:w <filename> – save as <filename>:q – quit :q! – quit without saving:e <filename> – load another file:n – load next file

Search

/string – search for string?string – search backwards for stringn – repeat previous searchN – repeat search in opposite direction% – find match of current (,[, or {

Entering Modes from Command Mode

i – insert text before current characterI – insert text at beginning of linea – append text after current characterA – append text at end of the lineo – open line above current lineO – open line below current linecw – change (overwrite) current wordC – change text after cursor

Additional Info

Unix in a Nutshell – Chapter 8 lists all vicommandsvi uses all lowercase letters except ‘v’ (and many uppercase characters and punctuation) for commands

Be careful!Learn undo commands

emacs is covered in Chapter 7

More Shell Commands

ps – list current processestop – dynamic display of system’s utilization by processeskill – terminate a processtime – keep timing information for a process

1

I/O Redirection and Regular Expressions

February 9th, 2004Class Meeting 4

Redirecting stdout

Instead of displaying output to the terminal, you can redirect the output to another file using the > operator>> operator is used to append output to an existing fileExamples:man ls > ls_help.txtecho $PWD > current_directorycat file1 >> file2

Redirecting stdin

Instead of reading from a terminal, you can tell a program to read from another file using the < operatorExamples:

mail [email protected] < messageinteractive_program < command_list

Pipes and Filters

Pipe – a way to send the output of one command to the input of anotherFilter – a program that takes input and transforms it in some way

wc – gives a count of words/lines/charactersgrep – searches a line with a given string (regular expression)moresort – sorts lines alphabetically or numerically

Examples of Filtering

ls -la | morecat file | wcman ksh | grep “history”ps aux | grep “user1” | wc -lwho | sort > current_users

What is a Regular Expression?

A regular expression (RE) is a pattern of metacharacters that define a set of stringsEach of these strings is said to match the regular expressionPattern matching is useful in many real-world situations:

searching for a file on the filesystemfinding and replacing text in a fileextracting data elements from a database

2

Unix programs that use REs

grep (search within files)egrep (grep with extended REs)vi/emacs (text editors)sed (stream editor)awk (pattern scanning language)perl (scripting language)

Basic vs. Extended REs

In basic regular expressions the metacharacters ?, +, { }, ( ), and | have no special meaning (grep)

To give them special meaning, use the escaped versions: \?, \+, \{ \}, , and \|

When using extended regular expressions, these metacharacters have special meaninggrep -E = egrep

Using egrep

egrep pattern filename(s)To be safe, put quotation marks around your patternExamples:egrep “abc” textfileegrep -i “abc” textfileegrep -v “abc” textfileegrep -n “abc” textfile

Metacharacters

Period (.): matches any single character“a.c” matches abc, adc, a&c, a;c, …“u..x” matches unix, uvax, u3(x, …

Asterisk (*): matches zero or more occurences of the previous RE

not the same as wildcards in the shell“ab*c” matches ac, abc, abbc, abbbc, …“.*” matches any string

Metacharacters (cont)

Plus (+): matches one or more occurences of the preceding RE

“ab+c” matches abc, abbc, abbc, but not acQuestion Mark (?): matches zero or one occurences of the preceding RE

“ab?c” matches ac or abc, but not abbcLogical Or (|): matches RE before | orRE after |

“abc|def” matches abc or def


Caret (^): beginning of line“^D” matches all strings starting with D

Dollar Sign ($): end of line“d$” matches all strings ending with d

Backslash (\): escapes other metacharacters

“file\.txt” matches file.txt, but not fileatxt

3


Square Brackets ([ ]): indicates a set/range of characters

any characters in the set will match^ before the set negates the set- indicates rangeExamples:

“[fF]un” matches fun or Fun“b[aeiou]g” matches bag, beg, big, bog, bug“[A-Z].*” matches any string beginning with a capital letter“[^abc].*” matches any string not containing an a, b, or c


Parentheses ( ): used to group REs when using other metacharacters

“a(bc)*” matches a, abc, abcbc, abcbcbc, …“(foot|base)ball” matches football or baseball

Braces ({ }): specify the number of repetitions of an RE

“[a-z]{3}” matches three lowercase letter“m.{2,4}” matches strings with m followed by between 2 and 4 characters

What do these mean?

egrep “^B.*s$” fileegrep “ [0-9]{3} “ fileegrep “num(ber)? [0-9]+” fileegrep “word” file | wc -legrep “[A-Z].*\?” fileWhat if grep was used instead?Remember, RE matches largest string--color option illustrates the largest match

1

Unix Window System

February 16th, 2004Class Meeting 5

Why Window Systems?

Increased usability due toVisibility

Graphical representation of programsSeeing multiple environments at once

Direct ManipulationEnables powerful graphics programs

Pixar: http://www.pixar.comhttp://news.com.com/2100-1001-983898.htmlhttp://slashdot.org/articles/03/02/09/1637254.shtml?tid=106

Window Systems and Unix

Most Unix users can be considered experts, and are fiercely protective of the command lineAll current Unix systems have a built-in window system, due to the obvious advantages of a graphical user interface

X Windows

Practically all Unix window systems are based on X Windows (XFree86)Standard Version: X11R6Complex system with many partsX11:

Manages the screen spaceDraws simple graphicsAssigns rectangular regions to various programs

X’s Client-Server Architecture

X is actually meant to work over the networkX server: software that runs on the machine where the program’s output will be displayedX client: program running on the same or another machineClient sends drawing and other X commands to the server, which displays the results

Historical Use of X

Users sat at “X terminals” – graphical terminals that only knew how to run an X serverThey logged in to other Unix machines remotely and ran X clients thereThis gave users the benefits of a window system without the need for a full-featured computer on every desk

2

Features of X

Transparent remote executionGives programs their own virtual screenIncludes important windowing concepts

Window damageWindow reveal eventsBacking store

X11 programs are highly portable

Desktop Environments

Purpose is to integrate the various interface components into a cohesive, consistent wholeProvides file managers (Nautilus), handles desktop icons, and overall desktop operationsCan handle window operations or use a window managerPopular desktop environments

GNOME (GTK+)KDE (Qt)XFce (GTK+)

Window Managers

Not part of X11 itself; run on top of X11Place borders and decorations on windowsHandle input from usersThere are many, many choices with different “look and feel”http://xwinman.org/

1

Unix Shell Environments

February 23rd, 2004Class Meeting 6

Shell Characteristics

Command-line interface between the user and the systemAutomatically starts when you log in, waits for user to type in commandsA Unix shell is both a command interpreter, which provides the user interface to the rich set of utilities, and a programming language, allowing these utilities to be combined.

Main Shell Features

Interactivity AliasesFile-name completion

Scripting languageAllows programming (shell scripting) within the shell environmentUses variables, loops, conditionals, etc.Next lecture

Various Unix Shells

sh (Bourne shell, original Unix shell)ksh (Korn shell)csh (C shell, developed at Berkeley)tcshbash (Bourne again SHell)Differences mostly in level of interactivity support and scripting details

http://www.faqs.org/faqs/unix-faq/shell/shell-differences/

Bourne Again SHell

We will be using bash as the standard shells for this classSuperset of the Bourne shell (sh)Borrows features from sh, csh, tcsh, and kshCreated by the Free Software Foundation

Changing Your Shell

On most Unix machines (including the lab) . . .which bash

chsh

On some machines . . . Ypchsh

2

Environment Variables

A set of variables the shell uses for certain operationsVariables have a name and a valueCurrent list can be displayed with the env commandA particular variable’s value can be displayed with echo $<var_name>

Environment Variable Examples

Some interesting environment variables:$HOME /home/grads/callgood$PATH /usr/local/bin:/bin:/usr/bin:/usr/X11R6/bin$PS1 \u@\h:\w\$$USER callgood$HOSTNAME mango.cslab.vt.edu$PWD /home/grads/callgood/cs2204

Setting Environment Variables

Set a variable with <name>=<value>Examples:PS1=myprompt>

PS1=$USER@$HOSTNAME:PS1=“multiple word prompt> ”PATH=$PATH:$HOME/bin

PATH=$PATH:~DATE=`date`

Aliases

Aliases are used as shorthand for frequently-used commandsSyntax: alias <shortcut>=<command>Examples:alias ll=“ls –lF”alias la=“ls –la”alias m=morealias up=“cd ..”alias prompt=“echo $PS1”

Repeating Commands

Use history command to list previously entered commandsUse fc –l <m> <n> to list previously typed commands from m through n

Editing on the Command Line

bash provides a number of line editing commands; many are the same as emacs editing commandsM-b Move back one wordM-f Move forward one wordC-a Move to beginning of lineC-e Move to end of lineC-k Kill text from cursor to end of line

3

Login Scripts

You don’t want to enter aliases, set environment variables, etc., each time you log inAll of these things can be done in a script that is run each time the shell is started

Login Scripts (cont)

For bash, order is . . . /etc/profile~/.bash_profile

~/.bash_login (if no .bash_profile)~/.profile (if neither are present)

~/.bashrc

After logout . . . ~/.bash_logout

Example .bash_profile (partial)

# .bash_profile

# include .bashrc if it existsif [ -f ~/.bashrc ]; then

. ~/.bashrcfi

# Set variables for a warm fuzzy environment

export CVSROOT=~/.cvsrootexport EDITOR=/usr/local/bin/emacsexport PAGER=/usr/local/bin/less

Example .bashrc (partial)

# .bashrc

# abbreviations for some common commandsalias f=finger alias h=history alias j=jobs alias l='ls -lF' alias la='ls -alF' alias lo=logout

alias ls='ls -F'

Login Shell

login shell

interactive shell

interactive shellinteractive shell

/etc/profile~/.bash_profile~/.bashrc ~/.bashrc

~/.bashrc~/.bashrc

Background Processing

Allows you to run your programs in the background

callgood@mango:~/$ emacs textfile&callgood@mango:~/$

4

stdin, stdout, and stderr

Each shell (and in fact all programs) automatically open three “files” when they start up

Standard input (stdin): Usually from the keyboardStandard output (stdout): Usually to the terminalStandard error (stderr): Usually to the terminal

Programs use these three files when reading (e.g. cin), writing (e.g. cout), or reporting errors/diagnostics

1

Shell Scripting

March 1st, 2004Class Meeting 7

Shell Script (Program)

What is a shell script?A series of shell commands placed in an ASCII text file

Commands include . . . anything you can type on the command lineshell variablescontrol statements (if, while, for, . . . )

Script Execution

Provide script as an argument to the shell program (i.e. bash my_script)Or specify which shell to use within the script

First line of the script: #!/bin/bashMake sure that the script is executableRun directly from the command line

No compilation is necessary

Simple Script

#!/bin/bash

echo “Hello, World!”cd ~pwd

Output:

Hello, World!/home/grads/callgood

Shell Variables

NumericStringsArraysvar refers to the name of the variable, $varrefers to the value

var=100 # sets the value to 100(( var2=1+$var ))

Variable names begin with an alphabetic character and can include alpha, numeric, and the underscore

Numeric Variables

Integer variables are the only pure numeric variables that can be used in bashDeclaration and setting value

declare –i var=100

Numeric expressions are enclosed in double parentheses

(( var+=1 ))

Operators are the same as in C/C+++, -, *, /, %, &, |, <, >, <=, >=, ==, !=, &&, ||

2

String Variables

Unless variables are explicitly declared as another type, they are considered to be stringsvar=100 makes the var the string 100

However, placing the variable within double parentheses will treat it as an integer(( var2=1+$var ))

String Variables (cont)

Using substrings${string:n}

${string:5} # first five chars${string:-2} # last two chars

${string:n:m}${string:0:4} # first to fifth${string:1:3} # second to fourth

${#string} # length of string

Concatenating stringsstring_var1=“$string_var1 $string_var2”

Array Variables

Array is a list of values – do not have to declare sizeReference a value by ${a[index]}

${a[3]} # value in fourth position$a # same as ${a[0]}

Use the declare –a command to declare an array

declare –a sportssports=(basketball football soccer)sports[3]=hockey

Array Variables (cont)

Array initializationsports=(football basketball)moresports=($sports tennis)

${array[@]} or ${array[*]} refers to the entire contents of the arrayecho ${moresports[*]}

Output: football basketball tennis

Command Line Arguments

If arguments are passed to a script, they can be referenced by $1, $2, $3, . . . $0 refers to the name of the script$@ refers to all of the arguments – notan array – space separated group of strings$# refers to the number of arguments

Output and Quoting

echo command is used to output information to standard out

echo –n does not print a newlineShell interprets $ and `` within double quotes

$ - variable substitution` - command substitutionecho “`date +%D`” # 03/01/04

Shell does not interpret special characteswithin single quotes

echo ‘`date +%D`” # `date +%D`

3

Redirecting Output to File

Redirecting output to a file is the same as on the command lineExamples

echo “$var” > $OUTFILE # overwriteecho “$var” >> $OUTFILE # append

String expansionecho $’\n\n\n’echo $’\t’

Conditions

If using integers: (( condition ))If using strings: [[ condition ]]Examples

(( a == 10 ))(( b >= 3 ))[[ $1 = -n ]][[ ($v != fun) && ($v != games) ]]

Special conditions for file existence, file permissions, ownership, file type, . . .

Conditions (cont)

[[ -e $file ]] – File exists?[[ -f $file ]] – Regular file?[[ -d $file ]] – Directory?[[ -L $file ]] – Symbolic link?[[ -r $file ]] – Read permission?[[ -w $file ]] – Write permission?[[ -x $file ]] – Execute permission?

If Statement

Syntaxif conditionthenstatements

elifthenstatements

elsestatements

fi

optional

If Statement (cont)

Exampleif [[ -r $file ]]thenecho “$file is readable”

elif [[ (-w $file) && (-x $file) ]]thenecho “$file is writeable and exectuable”

fi

For Loops

Syntaxfor var [in list]dostatements

done

If list is omitted, $@ is assumedOtherwise ${list[*]}, where list is an array variable

4

For Loops (cont)

Example

colors=(yellow red blue green orange)for color in ${colors[*]}doecho $color

done

While Loops

Syntax

while conditiondostatements

done

The keywords break, continue, and return have the same meaning as in C/C++

Case Statements

Syntax

case expression inpattern1)

statements ;;pattern2)statements ;;

...*)statements ;;

esac

Case Statements (cont)

Example

case $1 in-a)# statements related to option a;;

-b)# statements related to option b;;

*)# all other options;;

esac

Command Substitution

Use the command of a command in a variable, condition, . . .

`command`$(command)

Example

files=(`ls`)for file in ${files[*]}doecho $file

done

Functions

Syntax

function function_name {statements . . .

}function_name() {

statements . . . }

Functions can take arguments, in the same way that the script takes arguments (i.e. $1 represents first argument, $2 the second, . . . )

5

Functions (cont)

Examplefunction add_two{

(( $sum=$1+$2 ))return $sum

}. . . # calling a functionadd_two 1 3echo $?

$? represents the return value of the last function call or commandFunction definition needs to occur before the function is called in the script

Assignment Suggestions

One approach to the script:Handle arguments, setting flags for each

i.e., if recursive option is given, set a variable that indicates that it was set

Write part of script that generates the makefile in a function; call this function recursively, if necessary (i.e. if recursive option was set)

Assignment Suggestions (cont)

Within function, store the output of ls command in an array; loop through this array and test filename of each file in the current directory for valid filenames and/or extensions

files=(`ls`)for file in ${files[*]}do# no need to check directoriesif [[ -f $file ]]then# test for valid filename and/or extensions

fidone

Assignment Suggestions (cont)

Remember that you need to know all of the valid filenames before you can begin writing any of the makefile; you may need to store parts of the output in temporary files, and then concatenate these files into your makefile; be sure that you remove and temporary files that you create

1

Development: g++ and make

March 15th, 2004Class Meeting 8

Software Development Process

Creation of source filesRCS

Compilation and linkingg++ and make

Running and testing programsgdb

Development Tools

Creation of source filesText editors

viemacs

Revision control systemsrcscvs

Compilation and linkingCompilers

gccg++

Automatic building toolsmake

Running and testing programs (gdb)

Compilation Process

Preprocessor

Compiler

Optimizer

Assembler

Linker-Loader

#include, #define, #ifdef

Translates code into assembly language

Optional: Language-neutral code optimization

Translates assembly code into machinelanguage code kept in object files

Links object files and libraries to form one executable file

Basic g++ Examples

g++ hello.cccompile hello.ccproduce executable a.out

g++ -o hello hello.cccompile hello.ccproduce executable hello

g++ -o hello hello.cc util.cccompile hello.cc and util.ccproduce exectuable hello

Using Intermediate Files

From any source file you can produce an object file to be linked in later to an executable

g++ -c hello.ccg++ -c util.ccg++ -o hello hello.o util.o

2

g++ Options

-ccompile source files, but do not linkoutput is an object file corresponding to source file

-o <file>puts output in file called <file>

g++ Options (cont)

-ginclude debugging symbols in the outputto be used later by debugging program (gdb)

-Walldisplay all warnings – program may still compile

-D<macro>defines macro with the string ‘1’

g++ Options (cont)

-l<name>include library called lib<name>.a

-I<path>look for include files in the directory provided

-L<path>look for libraries in the directory provided

There are default directories in which g++looks for include files and libraries

Using make to Compile

With medium to large software projects containing many files, it is difficult to:

Type commands to compile all the files correctly each timeKeep track of which files have been changedKeep track of dependencies among files

make automates this process

Basic Operation of make

Reads a file called [Mm]akefile that contains rules for building a program

if program is dependent on another file, then that file is builtall dependencies are built, working backward through the chain of dependenciesprograms are only built if they are older than the files they depend u pon

Basic Makefile Example

program : main.o iodat.o dorun.og++ -o program main.o iodat.o dorun.o

main.o : main.ccg++ -c main.cc

iodat.o : iodat.ccg++ -c iodat.cc

dorun.o : dorun.ccg++ -c dorun.cc

3

Parts of a Makefile

Dependency Linescontain target names and dependencies (optional)dependencies

filestargets

Commandsbelow dependency linealways begin with a tabany commands that you can run on a command line

program : main.o iodat.o dorun.og++ -o program main.o iodat.o dorun.o

main.o : main.ccg++ -c main.cc

Parts of a Makefile (cont)

commandtab

dependenciestarget

Macros and Special Variables

Use macros to represent text in a makefilesaves typingallows easy modification of the makefileAssignment

MACRONAME = macro valueUsage: ${MACRONAME}

Special variables are used in commands$@ represents the target$? represents the dependencies

Simplifying the Example

OBJS = main.o iodat.o dorun.oCC = /usr/bin/g++

program : ${OBJS}${CC} -o $@ $?

main.o : main.cc${CC} -c $?

iodat.o : iodat.cc${CC} -c $?

dorun.o : dorun.cc${CC} -c $?

Invoking makeBe sure that the description file

is called makefile or Makefileis in the directory where the source files are

makebuilds the first target in the file

make target(s)builds target(s)

Other options-n: don’t run the commands, just list them-f <file>: use <file> instead of [Mm]akefile

Suffix Rules

Still tedious to specifically tell make how to build each .o file from .c/.cc fileSuffix rules can be used to generalize such situationsA default suffix rule turns .c/.cc files into .o files by running the command

${CC} ${CFLAGS} –c $<

4

Simplest Makefile Example

OBJS = main.o iodat.o dorun.oCC = /usr/bin/g++

program : ${OBJS}${CC} -o $@ $?

Makefile Tips

Include a way to clean up your mess

clean :rm –f *.o core

Include a target to build multiple programs

all : program1 program2 program3

1

Development: Revision Control

March 22nd, 2004Class Meeting 9





What is revision control?

A way to ensure that edits on files are:logged and archivedconsistent

Especially applicable when projects include:

multiple filesmultiple developersmaintenance (corrective and enhancing)

Why do you need revision control?

Software development is normally done in teamsMultiple people may be responsible for code in a single fileYou may want to freeze a working version and create a new revision branchYou may want to roll back development to a previous point in time

Overview of RCS

Maintains complete revision information for a set of files (not limited to source code)Uses major and minor revision numbers (e.g. 1.1, 2.3)Supports locking files so that only one user can edit at a timeSupports merging two edits of the same fileCan automatically place revision information within the files themselves

Key RCS Commands

rcs (administer project)ci (check in file)co (check out file)rlog (view the log)rcsdiff (see changes since last revision)

2

Basic RCS Usage

Create a subdirectory RCS of the directory containing the files in questionCheck in the files to create an initial revision (1.1)Check out and lock files to edit themCheck files back in to create a new revision and allow others to edit them

Checking Out Files

Assume we start with the file main.cchecked in: RCS/main.c,v

co main.ccreates read-only file main.c in current directory

co –l main.ccreates writable file main.c in current directorylocks files so others cannot check it out

Checking In Files

Assume we have locked and edited main.cci main.c

creates a new revision (e.g. 1.2)removes main.c from current directory

ci –u main.ccreates a new revisionleaves read-only main.c in current directory

ci –l main.ccreates a new revision (checkpoint)allows you to keep editing

Setting the Revision Number

Perhaps we have done major revisions to revision 1.4 of main.c and we want to start a new branch (2.x) of the revision treeci –r2 main.cco –rR file

retrieves revision number R of file instead of most recent version

Using Keyword Substitution

Each time you check in a new revision, RCS keeps information about the author, date/time, a log message, etc.You can show this information in files by placing special tags within the files when setting up RCS

Keyword Example

/** Last Revision:* $Revision$ by* $Author$ on * $Date$*/

int main(){

. . .

}

/** Last Revision:$Revision: 1.1 $ by$Author: callgood $$Date: 2004/03/12 $

*/

int main(){

. . .

}

1

Development: Revision Control

March 28th, 2004Class Meeting 10





Why use a debugger?

No one writes perfect code the first time, every timeDesk checking code can be tedious and error-pronePutting print statements in the code requires re-compilation and a guess as to the source of the problemDebuggers are powerful and flexible

Common Debugger Functions

Run programStop program at breakpointsExecute program one line of code at a timeDisplay values of variablesShow sequence of function callsCatch signals

The GNU Debugger (gdb)

Free command-line debuggerCommon utilization:

gdb executablegdb executable coregdb executable process_id

Execution Commands

list or llists source codelistlist function_namelist line_number

run or rrun program from beginningrunrun argument1 argument2

next or nexecute next line, stepping over function calls

step or sexecute next line, stepping into function calls

2

Breakpoint Commands

break or bset a breakpointbreak function_namebreak line_number

delete or ddelete a breakpointdelete breakpoint_number

continue or ccontinue execution when stopped

Program Information Commands

print or pprints value of a variable or expressionprint xprint x*yprint function(x)

displaycontinuously display value of a variable

undisplayStop displaying value of a variable

Miscellaneous Commands

setchange a variable’s valueset n=3

help or hdisplay help texthelphelp command_namehelp keyword

quit or qquit gdb

1

System Programming: File Management

March 22nd, 2004Class Meeting 11

Layers in a Unix-based System

Hardware(CPU, memory, disks, terminals, etc.)

Unix Operating System(process/memory management, file system, I/O)

Standard Library(open, close, read, write, etc.)

Standard Utility Programs(shells, editors, compilers, etc.)

UsersUser Interface

Library Interface

System calls

kernel

user mode

Unix System Programming

Programming that uses special features of the Unix system (kernel)Programs make system calls via librariesTypes of system calls

File IOProcess ManagementInter-Process Communication (IPC)Signal Handling

C versus C++

No string data typesUse character arrays insteadUse strcpy(), strncpy(), strcmp(), strncmp() to “assign” and compare character arrays

No embedded declarationsMust declare all variables at the beginning of a code block

Very different File and Standard IO functionsprintf() versus coutscanf() and fgets() versus cin

Basic File IO

Remember everything in Unix is a fileProcesses keep a list of open filesFiles can be opened for reading, writingMust include <stdio.h> to use IO system calls

Basic File IO (cont)

Each file is referenced by a file descriptor (integer)Three files are opened automatically

FD 0: standard inputFD 1: standard outputFD 2: standard error

When new files are opened, it is assigned the lowest available FD

2

open()

fd = open(path, flags, mode)path: char*, absolute or relative pathflags:

O_RDONLY – open for readingO_WRONLY – open for writingO_RDWR – open for reading and writingO_CREAT – create the file if it doesn’t existO_TRUNC – truncate the file it exists (overwrite)O_APPEND – only write at the end of the file

mode: specify permissions if using O_CREATEReturns newly assigned file descriptorfd = open(“myFile”, O_CREAT, 00644)

read()

bytes = read(fd, buffer, count)Read from file associated with fd; place count bytes into bufferfd: file descriptor to read frombuffer: pointer to an arraycount: number of bytes to readReturns number of bytes read or -1 if an error occurred

int fd = open(“someFile”, O_RDONLY);char buffer[4];int bytes =

read(fd, buffer, 4*sizeof(char));

write()

bytes = write(fd, buffer, count)Write contents of buffer to file associated with fdfd: file descriptorbuffer: pointer to an arraycount: number of bytes to writeReturns the number of bytes written or -1 if an error occurred

int fd = open(“someFile”, O_WRONLY);char buffer[4];int bytes =

write(fd, buffer, 4*sizeof(char));

close()

return_val = close(fd)Closes an open file descriptorReturns 0 on success, -1 on error

File IO using FILEs

Most Unix programs use higher-level IO functionsfopen()

printf()scanf()fclose()

These use the FILE data type instead of file descriptors

fopen()

FILE *file_stream = fopen(path, mode)path: char*, absolute or relative pathmode:

r – open file for readingr+ – open file for reading and writingw – overwrite file or create file for writingw+ – open for reading and writing; overwrites filea – open file for appending (writing at end of file)a+ – open file for appending and reading

fclose(file_stream)Closes open file stream

3

printf()

printf(formatted_string,...)

formatted_string: string that describes the output information

variable types are escaped with % (see next slide)

string is followed by as many expressions as are referenced in the formatted string

printf() (cont)

int term = 15;

printf(“Twice is \n”, term, 2*term);%d %d

formatted string expressions

Escaping Variable Types

%d, %i – decimal integer%u – unsigned decimal integer%o – unsigned octal integer%x, %X – unsigned hexadecimal integer%c - character%s – string or character array%f – float%e, %E – double (scientific notation)%g, %G – double or float%% - outputs a % character

printf() Examples

printf("The sum of %d, %d, and %d is %d\n", 65, 87, 33, 65+87+33);

Output: The sum of 65, 87, and 33 is 185printf("Error %s occurred at line %d \n", emsg, lno);

emsg and lno are variablesOutput: Error invalid variable occurred at line 27

printf("Hexadecimal form of %d is %x \n", 59, 59);

Output: Hexadecimal form of 59 is 3B

scanf()

scanf(formatted_string, ...)

Similar syntax as printf, only the formatted string represents the data that you are reading inMust pass variables by referenceExamplescanf(“%d %c %s”, &int_var, &char_var, string_var);

printf() and scanf() Families

fprintf(file_stream, formatted_string, ...)

Prints to a file stream instead of stdoutsprintf(char_array, formatted_string, ...)

Prints to a character array instead of stdout

fscanf(file_stream, formatted_string, ...)

Reads from a file stream instead of stdinsscanf(char_array, formatted_string, ...)

Reads from a string instead of stdin

1

System Programming: Process Management

April 12th, 2004Class Meeting 12

Processes in Unix

Process – basic unit of executionExecuting instance of a programHas a process IDOccurs in a hierarchy of processes (parents and children)Has its own state/context/memory

Process Management

System calls dealing with:Process creationSetting the program a process executesWaiting for a process to terminateTerminating a processSending signals to a process

Process Creation

pid = fork();

Creates a new child process that is an exact copy of the current process

Same program running at the same locationSame variable valuesSame open files

Only difference: child has a new process ID

fork (cont)

.

.

.pid = fork()if(pid == 0){

// child}else{

// parent}

DATA

Process 1

DATA

.

.

.pid = fork()if(pid == 0){

// child}else{

// parent}

Process 2

Setting the Program of Execution

exec family of functionsexecvp(executable_name, args)Others: execl, execlp, execle, execv, execve

Replaces the current process with a new process image running the executable specified with the arguments listedNew process retains the old process ID and any open files

2

Waiting for a Process to Terminate

pid = wait(&status);Suspends the execution of the calling process until any child process terminatesReturns the process ID of the terminating childPuts exit status (int) of child in status

pid = waitpid(pid, &status, options)

Suspends execution of the calling process until a specific child terminates

Using fork, exec, wait together

pid = fork();

if(pid == 0){execl(“./program”, “program”,

arg1, NULL);}else{

pid = wait(&status);}

Process Termination

exit(status);

Terminates the processCloses all open file descriptorsReturns status to the parent process

Sending Signals to a Process

retval = kill(pid, signal);

Some common signals a user program might send:

SIGINT: interrupt (Ctrl-C)SIGKILL: killSIGUSR1, SIGUSR2: user-defined

Inter-Process Communication (IPC)

Information passing between processesTwo basic paradigms

Message passing: processes send information back and forth in message/packetsShared memory: processes share a chunk of physical memory and read/write data there to share that information

IPC with pipes

Example of message passingint fds[2];retval = pipe(fds);Creates two file descriptors (a pipe is a file), the first for reading and the second for writingHow does another process connect to this pipe?

3

Basic Pipe Example

int fds[2];char s[100];int pid;

retval = pipe(fds);pid = fork();if(pid == 0){

read(fds[0], s, 100);printf(“Read: %s\n”, s);

}else{

write(fds[1], “hello”, 6);}

Notes:

• Data is written/read in order (FIFO)

• Reads block until there is data to read

• Writes block if the pipe is full

• Closing the writing fdcauses EOF to be read on the other end

1

Intro to sed and awk

April 19th, 2004Class Meeting 13

sed

Stream editorOriginally derived from the ed line editorUsed primarily for non-interactive operations

Operates on data streamsUsage

sed options `address action` file(s)Example: sed `1,$s/^bold/BOLD/g` foo

Line Addressing

Using line numbersRange separated by commas n,m

Example: sed ‘1,3’ foo.txtSpecify first line and step f~s

Example: sed ‘2~3’ foo.txtsed ‘3,6p’ foo.txt

Print lines 3-6 from foo.txtIs this the output you get?

sed automatically prints inputUse –n option to suppress printing every linesed –n ‘3,6’ foo.txt

Line Addressing (cont)

sed –n ‘$p’ foo.txt$ represents the last linePrints only the last line of the file

Reversing line criteriased –n ‘3,$!p’ foo.txtPrints every line except 3-$ (last line)

Context Addressing

Use patterns/regular expressions rather than explicitly specifying line numberssed –n ‘/^From: /p’ $HOME/mbox

Retrieve all the sender lines from the mailbox fileNote that / / mark the beginning and end of the pattern to match

ls –l | sed –n ‘/^...w/p’Prints the line if the fourth character is a ‘w’

Substitution

Strongest feature of sedSyntax[address]s/expression1/subs/flags

Examplesed ‘s/|/:/’ data.txt

Substitute the ‘|’ character with ‘:’First occurrence in a particular line

sed ‘s/|/:/g’ data.txtAll occurrences in the line

2

Using Files

Tedious to type in commands at the prompt, especially if commands are repetitiveCan put commands in a file and send can use them

Do not need to enclose commands in quotes

sed –f cmds.sed data.txt

awk

Powerful pattern scanning and processing languageNamed after creators Aho, Weinberger, and KernighanMost commands operate on entire line; awkoperates on fields within each lineUsage:

awk options [scriptfile] files(s)Example: awk –f script.awk foo.txt

Processing Model

BEGIN{# Commands executed before input

}{# Main execution loop for each line of input

}END{# Commands executed after input

}

Other Features

Formatted printing using C-style printfConditional statements (if-else)Looping constructs

forwhiledo-while

Associative Arrays

Normal arrays use integers as their indicesAssociative arrays use strings as their indicesExample:age[“John”] = 23

Special Variables

FSField separatorDefault value is space

RSRecord separatorDefault value is newline (‘\n’)

FNRRecord number in the current input file

OFSOutput field separatorDefault value is space

3

Special Variables (cont)

ORSOutput record separatorDefault value is newline (‘\n’)

NFNumber of fields in the current input record

NRNumber of records seen so far

1

Basic UNIX system administration

CS 2204Class meeting 14

*Notes by Doug Bowman and other members of the CS faculty at Virginia Tech. Copyright 2001-2003.

(C) Doug Bowman, Virginia Tech, 2001 2

System administrationThus far, we’ve only discussed:

the use of UNIX from an end user point of viewSystem programming - accesses the core OS but doesn’t change the way it operates

System administration is another level: changing the way the system is set up and operates for end usersStrongly related to security issues


The superuserMost sys. admin. tasks can only be done by the superuser (also called the root user)Superuser

has access to all files/directories on the systemcan override permissionsowner of most system files

Shell command: su <username>Set current user to superuser or another user with proper password access


Administration through filesAs you would expect, system settings are stored in filesMost of these are stored in /etcWe’ll look at files related to:

Users and groupsFile systemsSystem initializationSystem upkeep

In section 5 of the man pages


/etc/passwdInformation about system users

bowman:x:65:20:D.Bowman:/home/bowman:/bin/ksh

login name[encrypted password]

user IDgroup ID

“real” name

command interpreterhome directory


/etc/groupInformation about system groups

faculty:x:23:bowman,ribbens,mcquain

group name[encrypted group password]

group ID

list of group members

2


/etc/fstabInformation about file systems

/dev/cdrom /cdrom iso9660 defaults,ro,user,noauto 0 0

mount pointfile system type

file system (local device or remote dir.)

mount optionsother flags


/etc/inittab and /etc/init.d/inittab: configuration file for the init process

Defines “run levels”init.d: Directory containing system initialization scripts

Script rc <n> is run at run level nStarts and stops various services


Configuring the systemOften by directly editing files with a text editorIn some cases there are programs that modify the files for youMany systems also have nice graphical user interfaces that let you manipulate these files indirectly


File System Implementation

A possible file system layout


Internal View of a File SystemBoot Block:

The first block in a UNIX file system, contains the boot program and other initialization information or unused.

Super BlockAlways the second block, contains the complete “catalog” of specific information about the file system, including lists of free memory


Internal View of a File SystemInode list blocks

List of inodes for the file system, contiguous and always follows the super block. The number ofinodes is specified by the system administrator

File access and type information, collectively known as the mode. File ownership information. Time stamps for last modification, last access and last mode modification. Link count. File size in bytes. Addresses of physical blocks.

3


Internal View of a File SystemData blocks

OS files, user data and program files etc.


File system Commandsmount

mount a file systemumount

unmount a file systemfsck

check and repair a Linux file systemsync

flush filesystem buffers


crontabUseful to have a script or command executed without human intervention

a script to verify that the networks are working correctly

cron daemon reads cron configuration files called “crontab” short for “cron table”

parse crontabsfind the soonest command to be rungo to sleep until the command’s execution time has arrived


What’s cron and crontabs?Under UNIX, periodic execution is handled by the cron daemon

read one or more configuration files containing as following

command linestimes at which they are to be invoked(on some systems) login names under which they are to run

crontabs/etc/crontab


Format of Crontab filesSeven or six fields

minute hour day month weekday [username] command

an asterisk matches all possible values, a single integer matches that exact value, a list of integers separated by commas (no spaces) used to match any one of the values two integers separated by a dash (a range)used to match any value within the range


Example crontab

SHELL=/bin/bashPATH=/sbin:/bin:/usr/sbin:/usr/binMAILTO=rootHOME=/

01 * * * * root nice - n 19 run- parts /etc/cron.hourly02 4 * * * root nice - n 19 run- parts /etc/cron.daily22 4 * * 0 root nice - n 19 run- parts /etc/cron.weekly42 4 1 * * root nice - n 19 run- parts /etc/cron.monthly

1

Final Exam Review

May 3rd, 2004Class Meeting 15

Current Grades

Posted on BlackboardClose to half of the class has an A/A-¾ of the class is currently passingIncludes P5 gradesDoes not include some make-up grades

P6 should be graded by Thursday

Final Exam

Saturday, May 8th

1:05pm – 3:05pmWhittemore 300

16% of final grade (160 points)40 questions, 4 points per questionMultiple choice, scantron

Unix Basics

History of Unix/LinuxWho are Ken Thompson and Dennis Richie?Who is Linus Torvalds?What is the kernel?

Basic commandsBasic commands that you should have become familiar with over the semesterExample: ls, chmod, rm, mv, cp, cd, egrep, etc.Do not need to memorize syntax or options; just need to know what command does

Unix Filesystem

Hierarchical organization of filesystemabsolute versus relative pathsHidden files

File ownership and permissionsuser, group, otherread, write, executeBe able to read the output of ls -la

File editingvi

Basic commands (saving, quitting, editing)insert versus command mode

Redirection and Regular Expressions

Pipes, filters, and redirection<, >, >>, |What is a pipe?What is a filter?

Regular ExpressionsWhat are regular expressions?basic versus extended regular expressionsMetacharacters: { }, ^, $, [ ], ?, *, +, |, ., ( )

2

Shell Scripting and X Windows

X Windowsclient versus server

Shell ScriptingHigh-level understanding of shell scripting

Given example script, be able to describe what it would doWhat is a shell script?What kinds of variables can be used?Single quotes versus double quotes versus back ticsSpecial variables

$0, $1, $2, $#, $?

System Programming

MakefilesWhat is a dependency?What is a target?Special variables

$@, $?Macros

gcc/g++How do you specify an include directory?How do you specify a library directory?How do you link a library?Compilation processKnow all options covered in notes

System Programming (cont)

File I/OWhat are file descriptors?Related system calls:

open, close, fopen, fclose, read, write, printf-family, scanf-family

Process ManagementWhat is a process?Related system calls:

fork, execvp, wait/waitpid, kill

Miscellaneous

sed and awkGiven an example sed or awk script, be able to describe what it would do

System administrationKnow what files do

/etc/passwd/etc/groups/etc/inittab

Be able to read a crontab file

Documents

CS2204: Introduction to Unix Operating Systemcourses.cs.vt.edu/~cs2204/fall2004/resources/CS2204_notes_Spring_2004.pdfSystem calls kernel user mode Christian Allgood, Virginia Tech,