Using the Linux Command Line - University of Arizonamath.arizona.edu/~swig/documentation/unix_command_line/love_swig... · Basic Commands Linux Filesystem 3 More Advanced Commands

Using the Linux Command Line

David Love

Software Interest GroupUniversity of Arizona

October 28, 2011

1 Introduction

2 BasicsConnecting to UNIX/LinuxBasic CommandsLinux Filesystem

3 More Advanced CommandsA List of Useful CommandsFile Transfer with LinuxPermissions and Executables

4 Your Environment, Shell Scripting & Variables

5 Making Things Better & Easier

Intro Basics Commands Scripting Improvements

What Are UNIX and Linux?

UNIX is an operating system originally developed in 1969at Bell Labs.

Has been continually modified since then.

Currently, UNIX is a set of standards that an OS mightabide by

Apple OSX is UNIX.Technically, Linux is not UNIX.Sometimes see “*nix” to denote both UNIX and “Unix-like”systems.

Linux is an open source operating system, originallydeveloped in 1992.

Has been under continual development, is still free andopen source.

David Love Command Line Linux

Intro Basics Commands Scripting Improvements Connection Basic Commands Filesystem

Why use UNIX/Linux?

It is the OS of high-performance computing.

As of June 2011, 471 of the 500 fastest supercomputers ranLinux.1

23 run some other UNIX system.

Which leaves 6 that run Windows.

Lots of academic software is written for Linux.

Run code without using up your computer.

Easy file transfer

Backups!Always have homework available.

1http://top500.org/stats/list/37/osDavid Love Command Line Linux

http://top500.org/stats/list/37/os


Why use UNIX/Linux?

It is the OS of high-performance computing.

As of June 2011, 471 of the 500 fastest supercomputers ranLinux.1

23 run some other UNIX system.Which leaves 6 that run Windows.

Lots of academic software is written for Linux.

Run code without using up your computer.

Easy file transfer

Backups!Always have homework available.

1http://top500.org/stats/list/37/osDavid Love Command Line Linux

http://top500.org/stats/list/37/os


Connecting to Math Linux

Windows Three options:

UA Software https:

//sitelicense.arizona.edu/ssh/

PuTTY http://www.chiark.greenend.org.

uk/~sgtatham/putty/

Cygwin http://www.cygwin.com/

Apple Comes with a terminal.

Linux Comes with a terminal.


https://sitelicense.arizona.edu/ssh/

https://sitelicense.arizona.edu/ssh/

http://www.chiark.greenend.org.uk/~sgtatham/putty/

http://www.chiark.greenend.org.uk/~sgtatham/putty/

http://www.cygwin.com/


Where do we Connect?

Math & Applied Math (& Statistics?) grad students willconnect to:

gila.math.arizona.edu Main server for doing mosteverything.

iguana.math.arizona.edu Server for working with remotedesktop.

chivo1,chivo2,chivo3,chivo4 For high performancecomputing. MUST first connect through anotherserver (gila).



From Windows with PuTTY

Open PuTTY.

When ConfigurationWindow opens,change:

Host NamePort

Set to SSH

and open theconnection



From Cygwin/Apple/Linux with aTerminal

Open a terminal.

Connect with the command

ssh -p PORT USER@SERVER

USER Your username. Mine is dlove

SERVER The server to connect to, e.g.,gila.math.arizona.edu

PORT Port number to connect to. Math departmentuses 31415. Other servers may use the defaultport, 22, which needs no -p tag.

To connect to gila, I use

ssh -p 31415 [email protected]



The Structure of a Linux Command

Linux Command Structore

commandname [OPTIONS] ARGUMENTS

The basic command:

1 Begins with the name of the command.

2 Might include some options. As the name suggests, optionsare generally optional.

1 In Linux’s documentation, optional things are in brackets.2 Most options are denoted with a dash and a letter, -r.3 They can be strung together, -rfp. Some require

arguments.4 Word length options are generally denoted with two dashes,

--recursive.

3 May include one or more arguments.






The basic command:



1 In Linux’s documentation, optional things are in brackets.2 Most options are denoted with a dash and a letter, -r.

3 They can be strung together, -rfp. Some requirearguments.

4 Word length options are generally denoted with two dashes,--recursive.







The basic command:




arguments.

4 Word length options are generally denoted with two dashes,--recursive.







The basic command:




arguments.4 Word length options are generally denoted with two dashes,

--recursive.




Basic Unix Commands

pwd - print the name of your current/workingdirectory.

cd [directoryname] - change your workingdirectory.

ls [directoryname] - list the contents of directories.

mkdir directoryname - make a new directory.

rmdir directoryname - remove/delete a directory.

cp source destination - copy files and directories.

mv source destination - move files and directories.

rm filenamelist - remove/delete files.

cat filename - concatenate file, i.e., display thecontents.

logout - log out of a terminal.



Examples & Useful Options

If I log into gila, I can type some commands:

[ d love@gi la ˜ ] $ pwd/u1/ dlove

[ d love@gi la ˜ ] $ l sbin / P i c tu r e s /pub ht tp in t e rne t / Desktop/Publ ic / Documents/makevnc∗ Templates/Music/ Videos /

pwd told me the folder I was in.

ls gave a list of the files and subdirectories in /u1/dlove



More Examples

[ d love@gi la ˜ ] $ l sbin / P i c tu r e s /pub ht tp in t e rne t / Desktop/Publ ic / Documents/makevnc∗ Templates/Music/ Videos /

The / at the end indicates a subdirectory.

The * indicates an executable file.

Notice the prompt, showing:

Username: dloveMachine name: gilaCurrent directory: ~



Useful Options & Other Details

cp source file destination

-r Recursive, copy directory and allcontents.

-i Interactive, ask before copying overanother file.

-f Force, override interactive.-u Update, only copy when source is

newer than destination.

Example cp file newfile

Copy file to newfile.

Example cp file new directory/

Copy file the the subdirectorynew directory/



Usefule Options & Other Details

ls [directory or files]

-a Dislpay all files, even hidden files.-l Display long form with more

information.-R Recursive, display contents of

subdirectories as well.

rm [file]

-r Recursive, remove files fromsubdirectories.

-i Interactive, confirm before deleting.-f Force, override interactive.



Useful Linux Shortcuts

Tab Completion

Pressing the Tab key will complete a keyword or filename inUNIX/Linux. If the completion is not unique, pressing Tab

twice will give a list of possible completions.



Files in Linux

Filenames in Linux can contain any character except / andthe null character.

You should avoid special characters like $, ?, *, \.When using files with special characters or spaces incommands, they need to be escaped with a \.rm file\ with\ spaces\*

Files don’t need extensions, but some like .c, .tex arestandard.

Files can have multiple extensions, like hw.tex.backup.

Filenames beginning with a . are “hidden,” and notdisplayed by ls without using the -a option.



The UNIX Filesystem

The UNIX/Linux filesystem is quite different thanWindows. Some things that might be helpful as a Linuxuser:

/ The root directory, contains everything else inthe filesystem.

/bin bin is a subdirectory of /, and contains someof the most basic executable files.

/usr Contains most other executables, libraries,and documentation files.

/etc “Editable Text Configuration files.”Configuration files for the entire system. Canprovide examples for your own config. files.

/tmp Space for temporary files./home Contains home directories for all users.



Math Department Filesystem

The department computers have some nonstandarddirectories under /:

/scratch Space to put temporary files generated by anycomputations you do.

/u1,/u2,...,/u6 Home directories on the departmentcomputers are located in one of thesedirectories, instead of /home. My directory isin /u1. Yours might be somewhere different.



File System Abbreviations

UNIX and Linux have abbreviations for some directories:

~ Home directory.

. Current directory.

.. Parent directory.

If I’m in my home directory ~, or /u1/dlove, then:

. is /u1/dlove, and

.. is /u1.


Intro Basics Commands Scripting Improvements Useful List File Transfer Executables

Wildcards

Linux uses some characters to indicate patterns of text.

? Fill in any single character.

ls hw?.tex lists all filenames like hw1.tex,hwQ.tex, etc.

* Fill in any number of characters

ls hw* lists all files starting with hw, likehw1.tex, hw583.pdf, etc.

cat hw*.tex concatenates all files that start withhw and end with .tex

[] Specify a range for a single character

cp hw[1-3].tex .. Copy files hw1.tex, hw2.tex,hw3.tex to the parent directory.



More Wildcards

Of course, wildcards can be combined.

rm -f hw[1-4]?.[a,p]* removes, without confirmation,files that:

Begin with hw,Followed by a digit between 1 and 4,Followed by another single character, andHave an extension that begins with an a or p



Output Redirection

Linux prints to the screen a lot, and lets you work with that.

> Write or overwrite output to a file.

cat hw*.tex > allhw.tex Concatenate all fileshw*.tex into allhw.tex.

>> Append output to end of a file.

cat newhw*.tex >> allhw.tex Concatenate thefiles, add to the end of allhw.tex.

| “Pipe” output into a new command. Example:

tee FILE prints output both to FILE and thescreen.COMMAND | tee FILE prints the output ofCOMMAND to both FILE and the screen.



man pages

UNIX & Linux systems have a built-in manual: the man

pages.

The structure of a man page:

Name A short description of the command.Synopsis The basic form of the command:

Optional arguments enclosed in “[]”.Options with finite choices enclosed in “{}”.

Description A longer description of the command,including all options.

Other things in the man page might be:

Examples,Bug reporting,List of similar commands.



screen

screen creates virtual terminals that:

Can continue to run when you log out.Allow you to come back whenever you want.

Typing screen creates a new terminal.

Â-d (CTRL-a then d) detaches (leaves) a session.

The -r option (screen -r) reattaches (continues) asession.

Other commands:

Â-c Create another terminal window.Â-n Cycle to next terminal window.Â-p Cycle to previous terminal window.

logout Close the terminal window.



grep

grep PATTERN [FILE...] prints lines in a file that containPATTERN.

grep for prog.c prints all lines in the file prog.c

containing the pattern for.

Some options:

-n Print line number.-H Print file name.-i Ignore case.-v Print non-matching lines.-w Only match full words.

Example: grep -nHiw rand01 * in a code I used.



diff

diff FILE1 FILE2 intelligently compares files line by lineand prints the differences.

If given directories, it compares files of the same name ineach.

Output:

< Indicates a line in the first file.> Indicates a line in the second file.= Indicates common lines.

Options:

-i Ignore case.-w Ignore white-space.-q Print only whether files differ.

gvimdiff is an editor that incorporates diff, but youshould know vim first.



quota, ps and top

quota Displays how much space your account is allotted.Use -s option.

ps Lists running processes.

Without arguments, it shows processesrunning from the current terminal.Use -u to specify a user name, e.g., ps -u

dlove.

top Real time list of processes using system resources.

Quit by typing q.Type u to enter a username.



kill & killall

Closing programs from the terminal:

^C Kills the program currently running in theterminal.

kill PID Kills process with id number PID

-9 Signal that can’t be blocked.

killall NAME Kill a process by name.

-9 Signal that can’t be blocked.



tar & zip

tar, short for “tape archive” creates archives.

Options:

-c,-r,-x Create, append to, extract archive.-j,-z Compress with bzip2, gzip.

-v Verbose.-f Specify filename for archive

Examples:

tar -cvjf archive.tar.bz2 folder Compress folder

into archive archive.tar.bz2.

zip and unzip work with .zip archives.



find

find recursively searches through directories to find fileswith given characteristics.

find / -name passwd will find all files named “passwd”on the computer.

find -iname resume.tex will find all files named”resume.tex,” but is case insensitive.

find -name "*.sh" -executable will find allexecutable files with name *.sh.

When using wildcards, enclose in quotes to prevent the shellfrom expanding them.

find has a lot of possible tests and options. Seehttp://www.thegeekstuff.com/2009/03/

15-practical-linux-find-command-examples/ for atutorial of some useful ones.


http://www.thegeekstuff.com/2009/03/15-practical-linux-find-command-examples/

http://www.thegeekstuff.com/2009/03/15-practical-linux-find-command-examples/


locate

locate also finds files based on a name or path pattern.

It searches through a database–much faster than find!

But database is only updated periodically.

Many fewer options than find.

locate [OPTIONS] PATTERN is the basic command.



xargs

xargs puts the output from one command into argumentsfor another command.

Very useful with find!

To deal with filenames that contain spaces or other difficultcharacters, use print0 (zero) option with find, and -0

(zero) option with xargs.

Command: find OPTIONS -print0 | xargs -0 -I fileCOMMAND file OTHER ARGUMENTS.

-I option replaces file with the output of find.

Examples

find . -name "*.cpp" -print0 | xargs -0I file rm

file

find . -name "*.tex" -print0 | xargs -0I file mv

file file.backup



Secure Shell Copy

Basic method is to use scp

Open a terminal.

Connect with the command

scp -P PORT SOURCE DESTINATION

SOURCE Where the file currently is.DESTINATION Where you want to file to be.

PORT Port number to connect to. Math departmentuses 31415. Other servers may use the defaultport (no -P tag).



scp Continued

Suppose I want to copy ∼/file.txt from gila to Documents

on my computer:

scp -P 31415 [email protected]:∼/file.txt∼/Documents/



rsync: File Synchronization

rsync is a much more powerful tool:

Checks files on local machine and server, only transfers filesthat have changed.

Only transfers differences between changed files–Very Fast!

Suppose I want to copy ∼/file.txt from gila to Documents

on my computer:

rsync --rsh=’ssh -p 31415’

[email protected]:∼/file.txt ∼/Documents/

Servers using the default port don’t need the --rsh=’ssh -p

31415’ information.



Permissions and chmod

Three types of user can have permissions for each file:

user You.group Various subsets of the logins. On math

systems, only group is “users.”others Everyone else.

And three types of permissions:

read Can read files.write Can write or change files.

execute Can run executables or list contents ofdirectories.



Changing Permissions

Permissions are changed with the chmod command.

chmod MODE FILE

Most intuitive use of MODE is:

TYPE±PERM Adds/removes permissions PERM to users ofTYPE.

TYPE=PERM Gives TYPE exactly permissions PERM

Examples:

chmod u+x file Allow file to be executed.chmod a+X file Allows anyone to execute if file is a

directory or if someone else has executepermission.

chmod go-rwx file No one from groups or others canread, write or execute file.



Running Code on Linux

To run a program, must specify where the program is.

Run program cep1, which is in /home/dlove:/home/dlove/cep1

Can use shortcuts.

Run program nv, in the current directory, type:./nv



Bash

When you’re using the command line, you’re actually usinga shell.

The shell provides the user interface, and can function likea programming language.

Back in the day, some guy named Bourne wrote theBourne shell, with an executable named sh.

You’ll still see shell scripts with extension .sh.

Then other people developed other shells.

In 1989, some guy named Fox wanted to update theBourne shell.

Logically, he named it:

The Bourne-Again Shell (Bash).

Bash the default shell in most UNIX/Linux systems.



Bash












Bash












Files in Your Home Directory

Many “hidden” files, beginning with a “.” are in yourhome directory.

Can see them with ls -a option.

Many programs use these files to store configurationinformation.

Hidden files can almost always be edited with only a texteditor.

The most important hidden files:

.bashrc Contains the setup information for the shell..profile Similar to .bashrc.

There are often basic files in /etc.



Introduction to Variables

Bash uses variables to store important information.

Variables can be used like in a programming language.

By convention, variables in the shell are named withcapital letters.

Variables value is accessed with $.

For example, the PATH variable stores the folders thatinclude executables.

Using the echo command:

[ d love@gi la ˜ ] $ echo $PATH/usr / l o c a l / sb in : / usr / l o c a l / bin : / usr / sb in :/ usr / bin : / sb in : / bin : / usr /games

Can add more by PATH=$PATH:DIRECTORY. E.g., in.bashrc.



Introduction to Variables

Bash uses variables to store important information.

Variables can be used like in a programming language.

By convention, variables in the shell are named withcapital letters.

Variables value is accessed with $.

For example, the PATH variable stores the folders thatinclude executables.

Using the echo command:

[ d love@gi la ˜ ] $ echo $PATH/usr / l o c a l / sb in : / usr / l o c a l / bin : / usr / sb in :/ usr / bin : / sb in : / bin : / usr /games

Can add more by PATH=$PATH:DIRECTORY. E.g., in.bashrc.



Scripts

Scripts in Bash are a lot like programs in C, but havehigher level coding.

Still things like:

For loops,If tests,Changing values of variables.

Next, I have an example code to run multiple iterations ofa code for my research.



An Example Script

1 #! /bin/bash

2 # $1=problem, $2=m

3 for G in 0100 0200 1000

4 do

5 if [ $2 -ge $gamma ];

6 then

7 RUN="./rd $1 -s $2 -g $G | tee $1 m$2 $G.dat | grep Rep"

8 echo ${RUN}9 eval ${RUN}10 fi

11 done

12 ./parser $1 m$2 *.dat -o total $1 m$2.dat

13 echo "Done!"



An Example Script

1 #! /bin/bash


3 for G in 0100 0200 1000

4 do

5 if [ $2 -ge $gamma ];

6 then



11 done


13 echo "Done!"

Program to use to interpret



An Example Script

1 #! /bin/bash


3 for G in 0100 0200 1000

4 do

5 if [ $2 -ge $gamma ];

6 then



11 done


13 echo "Done!"

A comment. $1 is first command line argument. $2 is second.



An Example Script

1 #! /bin/bash


3 for G in 0100 0200 1000

4 do

5 if [ $2 -ge $gamma ];

6 then



11 done


13 echo "Done!"

Basic structure of a for loop



An Example Script

1 #! /bin/bash


3 for G in 0100 0200 1000

4 do

5 if [ $2 -ge $gamma ];

6 then



11 done


13 echo "Done!"

Basic structure of an if test. -ge means “≥“



An Example Script

1 #! /bin/bash


3 for G in 0100 0200 1000

4 do

5 if [ $2 -ge $gamma ];

6 then



11 done


13 echo "Done!"

Load a string containing the command into RUN. Variables areevaluated inside quotes.



An Example Script

1 #! /bin/bash


3 for G in 0100 0200 1000

4 do

5 if [ $2 -ge $gamma ];

6 then



11 done


13 echo "Done!"

Print the string to the screen.



An Example Script

1 #! /bin/bash


3 for G in 0100 0200 1000

4 do

5 if [ $2 -ge $gamma ];

6 then



11 done


13 echo "Done!"

Execute the code using eval.



An Example Script

1 #! /bin/bash


3 for G in 0100 0200 1000

4 do

5 if [ $2 -ge $gamma ];

6 then



11 done


13 echo "Done!"

Execute another program to consolidate all the output files.



An Example Script

1 #! /bin/bash


3 for G in 0100 0200 1000

4 do

5 if [ $2 -ge $gamma ];

6 then



11 done


13 echo "Done!"



My Solution to File Transfer

I have three rsync-based scripts to transfer files from mycomputers to gila.

I make sure that the ∼/Documents folder is identical on mydesktop, laptop, and on gila:

ups (“Upstream Backup”) Copies ∼/Documents frommy computer to gila. If it finds files on gila thatare not on my computer, it deletes them.

dow (“Downstream Backup”) Copies ∼/Documentsfrom gila to my computer. If it finds files on mycopy of ∼/Documents that are not on gila, itdeletes them.

ms (“Math Sync”) Synchronizes in both direction, butwill not delete files anywhere.



My Code

My code has 4 lines that need to be edited:

Change the user name.

MATHSERVER="[email protected]"

The folder on your computer to sync.

LOCALDOCS=∼/DocumentsThe folder on gila to sync.

MATHDOCS="∼/Documents"The options. The ’n’ tells rsync not to change any files.Remove this when you’re confident the code is correct.

OPTS="-vrtun"



Aliases

Aliases make complicated commands simple.

Put them in .bashrc or .profile.

Format is alias NEWCOMMAND=OLDCOMMAND.

For example, the file /etc/skel/.bashrc contains:

alias ll=’ls -alF’

alias la=’ls -A’

alias l=’ls -CF’

alias grep=’grep --color=auto’



Networking Aliases

I have some aliases set up specifically for networking:

alias math=’ssh -p 31415

[email protected]’

alias mathi=’ssh -p 31415

[email protected]’

alias mathvnc=’vncviewer

iguana.math.arizona.edu:4 -passwd

$HOME/.vnc/passwd’



Passwordless Connection & Privacy

Most connection methods allow you to connect withoutpasswords.

Make sure that only you have permission to read therequired files!



SSH Without Password

To login from a terminal without giving a password

Run ssh-keygen -t rsa

Accept the default location for the key.

Enter your password.

Make a .ssh folder on the server (e.g., gila):

ssh -p 31415 [email protected] mkdir -p

.ssh

Append id rsa.pub to your list of authorized keys on gila

cat .ssh/id rsa.pub | ssh

[email protected] ’cat >>

.ssh/authorized keys2’



VNC Without Password

To use VNC without a password, you need only modify yourown computer.

Run vncpasswd

Accept the default location for the password file.

Enter your password.

There’s no need for a view-only password.

Call your VNC viewer with the -passwd option, and thelocation of your password, e.g.,

vncviewer iguana.math.arizona.edu:12 -passwd

$HOME/.vnc/passwd

(Note: $HOME is the same as ~.)


Resources

Linux Documentation Project http://tldp.org/guides.html

Introduction to Linux, a Hands On Guide http:

//tldp.org/LDP/intro-linux/html/index.html

SSH Login Without Passwordhttp://linuxproblem.org/art_9.html

VNC Without Passwordhttp://www.dotkam.com/2009/03/22/

vnc-into-remote-server-without-typing-a-password/

http://tldp.org/guides.html

http://tldp.org/LDP/intro-linux/html/index.html

http://tldp.org/LDP/intro-linux/html/index.html

http://linuxproblem.org/art_9.html

http://www.dotkam.com/2009/03/22/vnc-into-remote-server-without-typing-a-password/

http://www.dotkam.com/2009/03/22/vnc-into-remote-server-without-typing-a-password/

The End