1

Introduction to UNIXUNIX & LINUX Fundamentals

2

Purpose of This Lecture

● Fundamentals of using UNIX and UNIX-like systems

● History of UNIX & Linux● Basic system commands● Data Management● Constructing basic shell scripts

3

History of UNIX

● "...the number of UNIX installations has grown to 10, with more expected..." – Dennis Ritchie and Ken Thompson, June 1972

● "... When BTL withdrew from the project, they needed to rewrite an operating system (OS) in order to play space war on another smaller machine (a DEC PDP-7 [Programmed Data Processor] with 4K memory for user programs). The result was a system which a punning colleague called UNICS (UNiplexed Information and Computing Service)--an 'emasculated Multics'; no one recalls whose idea the change to UNIX was"

4

Early Movers and Shakers

● Dennis Ritchie & Ken Thompson● PDP11

5

And then there was C…

● in 1972, Ritchie rewrote B and called the new language C; Thompson created the pipe--a "Write programs that do one thing and do it well. Write programs to work together. Write programs that handle text streams, because that is a universal interface."

6

Bringing UNIX to the desktop

● Unix was very expensive● Microsoft DOS was the mainstream OS● MINIX, tried but was not a full port

● An Open Source solution was needed!

7

Linux 0.02 – October 5, 1991

● “Do you pine for the nice days of minix-1.1, when men were men and wrote their own device drivers? Are you without a nice project and just dying to cut your teeth on a OS you can try to modify for your needs? Are you finding it frustrating when everything works on minix? No more all-nighters to get a nifty program working? Then this post might be just for you :-)” - Linus Benedict Torvalds

● "I still maintain the point that designing a monolithic kernel in 1991 is a fundamental error. Be thankful you are not my student. You would not get a high grade for such a design :-)" (Andrew Tanenbaum to Linus Torvalds)

8

1990’s Movers and Shakers

● Richard Stallman, father of the GNU Project● Linus Torvalds

9

Why UNIX/LINUX is Still Used

● 30+ years of development● Many academic, scientific, and system tools● Open Source ● System Stability● Lightweight● Easy Development

10

The Basics

● Command-Line– Interaction with UNIX & LINUX is based on

entering commands to a text terminal– Oftentimes there are no ‘warnings’ with commands,

no ‘undo’ ● The Shell

– The user environment that enables interaction with the kernel, or lower-system OS.

– Windows Explorer would be a shell for Microsoft Windows.

11

Common Shells

● sh – the original UNIX shell; still located in /bin/sh● Bash is a Unix shell written for the GNU Project and is

installed on most Linux systems.● csh – C Shell, modeled after the C programming language

used by UNIX systems● tcsh – C Shell with modern improvements such as

filename completion

● echo $SHELL – displays what shell your account is using● chsh – change your shell

12

Before we go further…

● Read the Manual. – man command– man [section] command– man –k keyword (search all manuals based on keyword)

● Most commands have a built-in UNIX manual, even the man command!

● Commands without manuals have help too, with –h, --help, or /? option.

13

The Manual

● The manual pages are divided into eight sections depending on type of command.

– 1 commands and applications

– 2 system calls

– 3 C library functions

– 4 special files

– 5 file formats

– 6 games

– 7 misc.

– 8 system administration utilities

14

Conventions for this lecture

● This lecture is too short to give you all of the options, so look at the manual for specific syntax for commands.

● Commands will be in bold, options will be in italics.– command -arguments

● Output will be shown in its own bordered table

15

Command Conventions

● In help files and manuals, commands will have required input and option input

● cp [OPTION] source destination– Optional arguments are in brackets, required

arguments are not.● cp –R or cp - -recursive

– Short options ‘-’, long options ‘- -’

16

Who’s Path is it Anyway?

● UNIX treats the directory structure as a hierarchy of individual paths

/

usrhomedev

bhecker bin

17

Directories

● What is a working directory?– The directory your shell is currently associated with.

At anytime in the system your login is associated with a directory

● pwd – view the path of your working directory● ls – view your working directory

18

Finding your home

● Each user has a home directory which can be found with:– cd– cd ~bhecker– cd $HOME

● The tilde character ‘~’ will tell the shell to auto-complete the path statement for the cd command

● $HOME refers to an environment variable which contains the path for home.

19

More file commands

● cd directory

- change your current working directory to the new path● ls –a

– Show hidden files – Hidden files are files that begin with a period in the filename ‘.’

● mv – moves one file to another

● cp – copies files or directories

● rm – remove files & directories– rm –rf

● remove everything with no warnings– rm –rf *

● most dangerous command you can run!

20

Recursive Directories

● Often times a manual will refer to ‘recursive’ actions on directories. This means to perform an action on the given directory and recursively to all subdirectories. – cp –R source destination

● Copy recursively all directories under source to destination

21

Relative vs. Absolute Path

● Commands expect you to give them a path to a file. Most commands will let you provide a file with a relative path, or a path relative to your working directory.

– ../directory - the ‘..’ refers to looking at our previous directory first

– ./executable - ‘.’ says this directory, or our working directory

● Absolute, or Full paths are complete. An easy way to know if a path is absolute is does it contain the ‘/’ character at the beginning?

– /home/user/directory/executable - a full path to file executable

22

Poking around in $home

● How much space do I have?– Quota

● command to see all quotas for your directories are, if any.

● How much space am I taking up?– du

● command to find out how much space a folder or directory uses.

– df ● display space information for the entire system

23

Helpful Hints on Space

● Almost all commands that deal with file space will display information in Kilobytes, or Bytes. Nobody finds this useful.

● Many commands will support a ‘-h’ option for “Human Readable” formatting. – ls –lh

● displays the working directory files with a long listing format, using “human readable” notation for space

24

Representing Space

● Bit – either a 1 or 0● Byte – 8 bits “0000 1111”, or “x0F”, or 16● KB, Kilobyte - 1024 Bytes● MB, Megabyte – 1024KB● GB, Gigabyte – 1024MB● TB, Terabyte – 1024 GB● PB, Petabyte – 1024 TB

25

Permissions

● The *NIX systems are multi-user environments where many users run programs and share data. Files and directories have three levels of permissions: World, Group, and User.

● The types of permissions a file can contain are:

Read permissions Write Permissions Execute Permissions

r w x

26

Permissions Cont.

User (owner) Group Others (everyone else)

rw- r-- r--

● File permissions are arranged in three groups of three characters.

● In this example the owner can read & write a file, while others have read access

27

Changing Permissions

● chmod – change permissions on a file or directory

● chown – change file ownership to another user

– Both options support ‘-R’ for recursion.

28

All About Me

● Every user id corresponds to a unique user or system process

– whoami – returns the use rid of the current user– passwd – change password

● What is my group? – support!

lslogin2$ ls -l ~eturner

total 391632

drwx------ 2 eturner support 4096 Sep 12 15:41 anothertest

-rwx------ 1 eturner support 8487 Jun 7 13:12 a.out

drwx------ 5 eturner support 4096 Apr 7 2006 automake-test

29

What everyone else is up to

● top – show a detailed, refreshed, description of running processes on a system.

● uptime – show the system load and how long the system has been up.– ‘load’ is a number based on utility of the cpu’s of the system. A

load of 1 indicates full load for one cpu.

lslogin2$ uptime

15:43:52 up 3 days, 20:45, 154 users, load average: 2.04, 2.00, 2.36

30

Working With Programs

● Commands or programs on the system are identified by their filename and by a process ID which is a unique identifier.– ps

● Display process information on the system– kill pid

● Terminates the process id– ^c (control+c) terminates the running program– ^d (control+d) terminates your session.

● Only you and the superuser (root) has permissions to kill processes you own.

31

Advanced Program Options

● Oftentimes we must run a command in the background with the ampersand ‘&’ character– command –options &

● Runs command in background, prompt returns immediately

● Match zero or more characters wildcard ‘*’– cp * destination

● Copy everything to destination– This option can get you into trouble if misused

32

Editing Files

● emacs vs vi– Among the largest ‘nerd battle’ in history. emacs relies heavily

on key-cords (multiple key strokes), while vi is mode based. (editor mode vs command mode)

– vi users tend to enter and exit the editor repeatedly, and use the Unix shell for complex tasks, whereas emacs users usually remain within the editor and use emacs itself for complex tasks

● pico (nano) – editor originally used for the email client pine, simple no-frills

editor which resembles notepad for windows.

33

Input and Output

● Programs and commands can contain an input and output. These are called ‘streams’. UNIX programming is oftentimes stream based. – Programs also have an error output. We will see later how to catch

the error output.

● STDIN – ‘standard input,’ or input from the keyboard● SDTOUT – ‘standard output,’ or output to the screen● STDERR – ‘standard error,’ error output which is sent to

the screen.

34

File Redirection

● Oftentimes we want to save output (stdout) from a program to a file. This can be done with the ‘redirection’ operator.– myprogram > myfile – using the ‘>’ operator we redirect

the output from myprogram to file myfile

● Similarly, we can append the output to a file instead of rewriting it with a double ‘>>’– myprogram >> myfile – using the ‘>’ operator we

append the output from myprogram to file myfile

35

Input Redirection

● Input can also be given to a command from a file instead of typing it to the screen, which would be impractical.

– cat programinput > mycommand - This command series starts with the command ‘cat’ which prints a file to a screen. programinput is printed to stdout, which is redirected to a command mycommand.

36

Redirecting stderr

● Performing a normal redirection will not redirect sdterr. In Bash, this can be accomplished with ‘2>’– command 2> file1

● Or, one can merge stderr to stdout (most popular) with ‘2>&1’– command > file 2>&1

37

Pipes

● Using a pipe operator ‘|’ commands can be linked together. The pipe will link the standard output from one command to the standard input of another.

● Very helpful for searching files

38

Searching

● A large majority of activity on UNIX systems involve searching for files and information.– find – utility to find files– grep – the best utility ever written for UNIX, searches for

patterns inside files and will return the line, if found

lslogin2$ find ./ -name output-evansapp-n0.out534086

./test/run_0/output-evansapp-n0.out534086

lslogin2$ cat test/run_0/output-evansapp-n0.out534086 | grep CPU

CPU time : 0.03 sec.

39

Packing Files

● When creating backups of files, or transferring to other hosts, files must be packed into larger files. This is needed for ease of manipulation, transfer speeds, and file management.

● tar – create or extract a packed file. tar stands for ‘tape archive’.

40

Compressing Files

● Compressing files can gain file space at the expense of cpu time to compress and decompress files.

● Compression works well for text files, but not as well for binary files with random data such as float values.

● Compression algorithms & commands– gzip, gunzip, bzip2, bunzip2

41

Using tar to create compressed files

● Tar will create compressed files for you– tar –czvf mytarfile.tar.gz directory

● creates a compressed file named mytarfile.tar.gz containing all of the files in the directory directory

– tar –xzvf mytarfile.tar.gz ● uncompresses all directories and files inside the file

mytarfile.tar.gz into the working directory

42

Connecting to Another Machine

● Secure Shell vs. Restricted Shell– ssh is an encrypted remote login program that is ‘secure’ to trust

across non secure networks.

● ssh userid@hostname

43

Copying Files to Remote Hosts

● copy local file lfile to rfile on remote machine rsys – scp lfile rsys:rfile

● -p preserves modification time, access time and mode from original

– scp -p lfile rsys:rfile

● copy rfile from remote machine rsys to local file lfile – scp -p rsys:rfile lfile

44

Running Commands on a Remote Host

● Commands can be executed on a remote host with ssh

● ssh userid@hostname “ls” – Run ls on remote host hostname

45

My Environment

● View all system variables by the command ‘env’

● Depending on shell, startup commands can be managed with the files .profile for bash and .cshrc with c shell

46

Basic Shell Scripts

● Many times it is helpful to create a ‘script’ of commands to run instead of typing them in individually. Scripts can be made to aid in post-processing, system administration, and automate menial tasks

● #!/bin/bash– First statement inside a script, will list which shell to run this

script in

● # - says what will follow is a comment and not to execute

47

Variables

● By convention system variables are capitalized– HOME – location of the home directory– OLDPWD – location of the previous working directory– PATH – locations to look inside for executable files

● Setting system variables differs by shell. bash uses export, csh uses setenv

● User defined variables in scripts are lower-case by convention– myvariable=10

● sets myvariable to 10– echo $myvariable

● prints myvariable

48

Conditionals

if condition then

condition is zero (true - 0) execute all commands up to else statement

else if condition is not true then execute all

commands up to fi fi

49

Performing Loops● Loops are statements that are repeated until the conditions are met.

for { variable name } in { list }

do

execute one for each item in the list until the list is not finished (And repeat all statement between do and done)

done

for i in 1 2 3 4 5

doecho "Welcome $i times“

done

50

Putting it Together

#!/bin/bash

#my first script

#scp replacement

remotefile=labresults

myserver=saguaro.fulton.asu.edu

mylsinfo=`ssh $myserver ls $remotefile 2>&1`

ismissing=`echo $mylsinfo | grep ERROR`

if [ "$ismissing" ]

then

echo "$remotefile not found! Exiting!"

else

ssh -n "cat < $remotefile" > $localfile

Fi

51

More…

● mylsinfo=`ssh $myserver ls $remotefile 2>&1`– `Backticks` are used to place output from a command

into a variable● if [ "$ismissing" ]

– Is $ismissing set (has a value)? If so then the expression is true, otherwise false

52

References

The textbook for this course:

● Graham Glass & King Ables UNIX for Programmers and Users