1
Introduction to UNIXUNIX & LINUX Fundamentals
2
Purpose of This Lecture
● Fundamentals of using UNIX and UNIX-like systems
● History of UNIX & Linux● Basic system commands● Data Management● Constructing basic shell scripts
3
History of UNIX
● "...the number of UNIX installations has grown to 10, with more expected..." – Dennis Ritchie and Ken Thompson, June 1972
● "... When BTL withdrew from the project, they needed to rewrite an operating system (OS) in order to play space war on another smaller machine (a DEC PDP-7 [Programmed Data Processor] with 4K memory for user programs). The result was a system which a punning colleague called UNICS (UNiplexed Information and Computing Service)--an 'emasculated Multics'; no one recalls whose idea the change to UNIX was"
4
Early Movers and Shakers
● Dennis Ritchie & Ken Thompson● PDP11
5
And then there was C…
● in 1972, Ritchie rewrote B and called the new language C; Thompson created the pipe--a "Write programs that do one thing and do it well. Write programs to work together. Write programs that handle text streams, because that is a universal interface."
6
Bringing UNIX to the desktop
● Unix was very expensive● Microsoft DOS was the mainstream OS● MINIX, tried but was not a full port
● An Open Source solution was needed!
7
Linux 0.02 – October 5, 1991
● “Do you pine for the nice days of minix-1.1, when men were men and wrote their own device drivers? Are you without a nice project and just dying to cut your teeth on a OS you can try to modify for your needs? Are you finding it frustrating when everything works on minix? No more all-nighters to get a nifty program working? Then this post might be just for you :-)” - Linus Benedict Torvalds
● "I still maintain the point that designing a monolithic kernel in 1991 is a fundamental error. Be thankful you are not my student. You would not get a high grade for such a design :-)" (Andrew Tanenbaum to Linus Torvalds)
8
1990’s Movers and Shakers
● Richard Stallman, father of the GNU Project● Linus Torvalds
9
Why UNIX/LINUX is Still Used
● 30+ years of development● Many academic, scientific, and system tools● Open Source ● System Stability● Lightweight● Easy Development
10
The Basics
● Command-Line– Interaction with UNIX & LINUX is based on
entering commands to a text terminal– Oftentimes there are no ‘warnings’ with commands,
no ‘undo’ ● The Shell
– The user environment that enables interaction with the kernel, or lower-system OS.
– Windows Explorer would be a shell for Microsoft Windows.
11
Common Shells
● sh – the original UNIX shell; still located in /bin/sh● Bash is a Unix shell written for the GNU Project and is
installed on most Linux systems.● csh – C Shell, modeled after the C programming language
used by UNIX systems● tcsh – C Shell with modern improvements such as
filename completion
● echo $SHELL – displays what shell your account is using● chsh – change your shell
12
Before we go further…
● Read the Manual. – man command– man [section] command– man –k keyword (search all manuals based on keyword)
● Most commands have a built-in UNIX manual, even the man command!
● Commands without manuals have help too, with –h, --help, or /? option.
13
The Manual
● The manual pages are divided into eight sections depending on type of command.
– 1 commands and applications
– 2 system calls
– 3 C library functions
– 4 special files
– 5 file formats
– 6 games
– 7 misc.
– 8 system administration utilities
14
Conventions for this lecture
● This lecture is too short to give you all of the options, so look at the manual for specific syntax for commands.
● Commands will be in bold, options will be in italics.– command -arguments
● Output will be shown in its own bordered table
15
Command Conventions
● In help files and manuals, commands will have required input and option input
● cp [OPTION] source destination– Optional arguments are in brackets, required
arguments are not.● cp –R or cp - -recursive
– Short options ‘-’, long options ‘- -’
16
Who’s Path is it Anyway?
● UNIX treats the directory structure as a hierarchy of individual paths
/
usrhomedev
bhecker bin
17
Directories
● What is a working directory?– The directory your shell is currently associated with.
At anytime in the system your login is associated with a directory
● pwd – view the path of your working directory● ls – view your working directory
18
Finding your home
● Each user has a home directory which can be found with:– cd– cd ~bhecker– cd $HOME
● The tilde character ‘~’ will tell the shell to auto-complete the path statement for the cd command
● $HOME refers to an environment variable which contains the path for home.
19
More file commands
● cd directory
- change your current working directory to the new path● ls –a
– Show hidden files – Hidden files are files that begin with a period in the filename ‘.’
● mv – moves one file to another
● cp – copies files or directories
● rm – remove files & directories– rm –rf
● remove everything with no warnings– rm –rf *
● most dangerous command you can run!
20
Recursive Directories
● Often times a manual will refer to ‘recursive’ actions on directories. This means to perform an action on the given directory and recursively to all subdirectories. – cp –R source destination
● Copy recursively all directories under source to destination
21
Relative vs. Absolute Path
● Commands expect you to give them a path to a file. Most commands will let you provide a file with a relative path, or a path relative to your working directory.
– ../directory - the ‘..’ refers to looking at our previous directory first
– ./executable - ‘.’ says this directory, or our working directory
● Absolute, or Full paths are complete. An easy way to know if a path is absolute is does it contain the ‘/’ character at the beginning?
– /home/user/directory/executable - a full path to file executable
22
Poking around in $home
● How much space do I have?– Quota
● command to see all quotas for your directories are, if any.
● How much space am I taking up?– du
● command to find out how much space a folder or directory uses.
– df ● display space information for the entire system
23
Helpful Hints on Space
● Almost all commands that deal with file space will display information in Kilobytes, or Bytes. Nobody finds this useful.
● Many commands will support a ‘-h’ option for “Human Readable” formatting. – ls –lh
● displays the working directory files with a long listing format, using “human readable” notation for space
24
Representing Space
● Bit – either a 1 or 0● Byte – 8 bits “0000 1111”, or “x0F”, or 16● KB, Kilobyte - 1024 Bytes● MB, Megabyte – 1024KB● GB, Gigabyte – 1024MB● TB, Terabyte – 1024 GB● PB, Petabyte – 1024 TB
25
Permissions
● The *NIX systems are multi-user environments where many users run programs and share data. Files and directories have three levels of permissions: World, Group, and User.
● The types of permissions a file can contain are:
Read permissions Write Permissions Execute Permissions
r w x
26
Permissions Cont.
User (owner) Group Others (everyone else)
rw- r-- r--
● File permissions are arranged in three groups of three characters.
● In this example the owner can read & write a file, while others have read access
27
Changing Permissions
● chmod – change permissions on a file or directory
● chown – change file ownership to another user
– Both options support ‘-R’ for recursion.
28
All About Me
● Every user id corresponds to a unique user or system process
– whoami – returns the use rid of the current user– passwd – change password
● What is my group? – support!
lslogin2$ ls -l ~eturner
total 391632
drwx------ 2 eturner support 4096 Sep 12 15:41 anothertest
-rwx------ 1 eturner support 8487 Jun 7 13:12 a.out
drwx------ 5 eturner support 4096 Apr 7 2006 automake-test
29
What everyone else is up to
● top – show a detailed, refreshed, description of running processes on a system.
● uptime – show the system load and how long the system has been up.– ‘load’ is a number based on utility of the cpu’s of the system. A
load of 1 indicates full load for one cpu.
lslogin2$ uptime
15:43:52 up 3 days, 20:45, 154 users, load average: 2.04, 2.00, 2.36
30
Working With Programs
● Commands or programs on the system are identified by their filename and by a process ID which is a unique identifier.– ps
● Display process information on the system– kill pid
● Terminates the process id– ^c (control+c) terminates the running program– ^d (control+d) terminates your session.
● Only you and the superuser (root) has permissions to kill processes you own.
31
Advanced Program Options
● Oftentimes we must run a command in the background with the ampersand ‘&’ character– command –options &
● Runs command in background, prompt returns immediately
● Match zero or more characters wildcard ‘*’– cp * destination
● Copy everything to destination– This option can get you into trouble if misused
32
Editing Files
● emacs vs vi– Among the largest ‘nerd battle’ in history. emacs relies heavily
on key-cords (multiple key strokes), while vi is mode based. (editor mode vs command mode)
– vi users tend to enter and exit the editor repeatedly, and use the Unix shell for complex tasks, whereas emacs users usually remain within the editor and use emacs itself for complex tasks
● pico (nano) – editor originally used for the email client pine, simple no-frills
editor which resembles notepad for windows.
33
Input and Output
● Programs and commands can contain an input and output. These are called ‘streams’. UNIX programming is oftentimes stream based. – Programs also have an error output. We will see later how to catch
the error output.
● STDIN – ‘standard input,’ or input from the keyboard● SDTOUT – ‘standard output,’ or output to the screen● STDERR – ‘standard error,’ error output which is sent to
the screen.
34
File Redirection
● Oftentimes we want to save output (stdout) from a program to a file. This can be done with the ‘redirection’ operator.– myprogram > myfile – using the ‘>’ operator we redirect
the output from myprogram to file myfile
● Similarly, we can append the output to a file instead of rewriting it with a double ‘>>’– myprogram >> myfile – using the ‘>’ operator we
append the output from myprogram to file myfile
35
Input Redirection
● Input can also be given to a command from a file instead of typing it to the screen, which would be impractical.
– cat programinput > mycommand - This command series starts with the command ‘cat’ which prints a file to a screen. programinput is printed to stdout, which is redirected to a command mycommand.
36
Redirecting stderr
● Performing a normal redirection will not redirect sdterr. In Bash, this can be accomplished with ‘2>’– command 2> file1
● Or, one can merge stderr to stdout (most popular) with ‘2>&1’– command > file 2>&1
37
Pipes
● Using a pipe operator ‘|’ commands can be linked together. The pipe will link the standard output from one command to the standard input of another.
● Very helpful for searching files
38
Searching
● A large majority of activity on UNIX systems involve searching for files and information.– find – utility to find files– grep – the best utility ever written for UNIX, searches for
patterns inside files and will return the line, if found
lslogin2$ find ./ -name output-evansapp-n0.out534086
./test/run_0/output-evansapp-n0.out534086
lslogin2$ cat test/run_0/output-evansapp-n0.out534086 | grep CPU
CPU time : 0.03 sec.
39
Packing Files
● When creating backups of files, or transferring to other hosts, files must be packed into larger files. This is needed for ease of manipulation, transfer speeds, and file management.
● tar – create or extract a packed file. tar stands for ‘tape archive’.
40
Compressing Files
● Compressing files can gain file space at the expense of cpu time to compress and decompress files.
● Compression works well for text files, but not as well for binary files with random data such as float values.
● Compression algorithms & commands– gzip, gunzip, bzip2, bunzip2
41
Using tar to create compressed files
● Tar will create compressed files for you– tar –czvf mytarfile.tar.gz directory
● creates a compressed file named mytarfile.tar.gz containing all of the files in the directory directory
– tar –xzvf mytarfile.tar.gz ● uncompresses all directories and files inside the file
mytarfile.tar.gz into the working directory
42
Connecting to Another Machine
● Secure Shell vs. Restricted Shell– ssh is an encrypted remote login program that is ‘secure’ to trust
across non secure networks.
● ssh userid@hostname
43
Copying Files to Remote Hosts
● copy local file lfile to rfile on remote machine rsys – scp lfile rsys:rfile
● -p preserves modification time, access time and mode from original
– scp -p lfile rsys:rfile
● copy rfile from remote machine rsys to local file lfile – scp -p rsys:rfile lfile
44
Running Commands on a Remote Host
● Commands can be executed on a remote host with ssh
● ssh userid@hostname “ls” – Run ls on remote host hostname
45
My Environment
● View all system variables by the command ‘env’
● Depending on shell, startup commands can be managed with the files .profile for bash and .cshrc with c shell
46
Basic Shell Scripts
● Many times it is helpful to create a ‘script’ of commands to run instead of typing them in individually. Scripts can be made to aid in post-processing, system administration, and automate menial tasks
● #!/bin/bash– First statement inside a script, will list which shell to run this
script in
● # - says what will follow is a comment and not to execute
47
Variables
● By convention system variables are capitalized– HOME – location of the home directory– OLDPWD – location of the previous working directory– PATH – locations to look inside for executable files
● Setting system variables differs by shell. bash uses export, csh uses setenv
● User defined variables in scripts are lower-case by convention– myvariable=10
● sets myvariable to 10– echo $myvariable
● prints myvariable
48
Conditionals
if condition then
condition is zero (true - 0) execute all commands up to else statement
else if condition is not true then execute all
commands up to fi fi
49
Performing Loops● Loops are statements that are repeated until the conditions are met.
for { variable name } in { list }
do
execute one for each item in the list until the list is not finished (And repeat all statement between do and done)
done
for i in 1 2 3 4 5
doecho "Welcome $i times“
done
50
Putting it Together
#!/bin/bash
#my first script
#scp replacement
remotefile=labresults
myserver=saguaro.fulton.asu.edu
mylsinfo=`ssh $myserver ls $remotefile 2>&1`
ismissing=`echo $mylsinfo | grep ERROR`
if [ "$ismissing" ]
then
echo "$remotefile not found! Exiting!"
else
ssh -n "cat < $remotefile" > $localfile
Fi
51
More…
● mylsinfo=`ssh $myserver ls $remotefile 2>&1`– `Backticks` are used to place output from a command
into a variable● if [ "$ismissing" ]
– Is $ismissing set (has a value)? If so then the expression is true, otherwise false
52
References
The textbook for this course:
● Graham Glass & King Ables UNIX for Programmers and Users