122
Introduction to Unix Shell Script Programming Summer course, Institute of Bioinformatics National Yang-Ming University

Introduction to Unix Shell Script Programming Summer course, Institute of Bioinformatics National Yang-Ming University

  • View
    222

  • Download
    5

Embed Size (px)

Citation preview

Page 1: Introduction to Unix Shell Script Programming Summer course, Institute of Bioinformatics National Yang-Ming University

Introduction to Unix Shell Script Programming

Summer course, Institute of Bioinformatics

National Yang-Ming University

Page 2: Introduction to Unix Shell Script Programming Summer course, Institute of Bioinformatics National Yang-Ming University

Menu Today!

Part X: Are You Ready? The essentials for Unix-like systems (review of previous courses) Basic editor - vi Principles of programming - C language

Part I: Shell Script in Practice Basics about Shell and Exercises of system Shell Scripts Perl Shell Script

Part II: The applications of Shell Script Massive routing jobs Scheduling Backup

Page 3: Introduction to Unix Shell Script Programming Summer course, Institute of Bioinformatics National Yang-Ming University

Before the Course

Checking list and examine yourself

Text editors: Vi, Vim or Emacs

Basic commands for Unix-like systems Login your system Manipulation of files and directories

Basic concept of C programming language

Page 4: Introduction to Unix Shell Script Programming Summer course, Institute of Bioinformatics National Yang-Ming University

Some Essentials for Unix-like systems

Part X-I

Page 5: Introduction to Unix Shell Script Programming Summer course, Institute of Bioinformatics National Yang-Ming University

Basic Unix Commands

Commands communicate with kernel

How to use them? the “man” command…man <command> keyword search for commands

apropos topic whatis topic man –k topic

directory list: ls … how many things in your current directory change to dir: cd … jump to another place print dir: pwd … if you are lost, it tells you where you are now

Page 6: Introduction to Unix Shell Script Programming Summer course, Institute of Bioinformatics National Yang-Ming University

Directory and File Commands

the copy command: cp [-r] A B to copy entire directory and contents, use –r recursive copy (-r) copy dir contents to destination

rename or move files: mv

create dir: mkdir Adir

remove file: rm [-r]...or dir: rmdir Adir, must be empty caution ! recursive delete of sub-directories: no “undo”!!

Page 7: Introduction to Unix Shell Script Programming Summer course, Institute of Bioinformatics National Yang-Ming University

Organising and Managing Your Files

View the file content use more (or less!) head or tail

the cat command: print file or “catenates” files cat file cat file1 file2 > file3

file archiving (tar) tar cvf archive.tar f1 .. fN (or a directory) … Create an tar archive tar xvf archive.tar … eXtract an tar archive tar tvf archive.tar … lisT the content of tar archive without extracting it if wanna compress it with gZip simultaneously, add z (zcvf or zxvf)

compress files: gzip/gunzip (GNU) or compress/uncompress

cat also could be a simple editor cat > file (press RETURN) enter data (press RETURN) ^D (exit)

Page 8: Introduction to Unix Shell Script Programming Summer course, Institute of Bioinformatics National Yang-Ming University

File permissionProtecting Your Files

Change file permission chmod: chmod [-R] newperm file symbolic form and (absolute) octal form

chmod og+w myfile or chmod 777 myfile

Change file owner chown

Change file group access chgrp (usually done by root)

user group others“-” for file“d” for directory“l” for link“b” for storage device“c” for mouse/keyboard

read(4) write(2) execute(1)

user group other all

= (set) + (add) – (remove)

Page 9: Introduction to Unix Shell Script Programming Summer course, Institute of Bioinformatics National Yang-Ming University

Hard link, Soft link, inodes

Compare item hard link symbolic (soft) link

pointer to … program, file program, file, directory

If pointed program, file or directory is renamed, moved or deleted …

Not brokenBroken

(like MS-windows short-cut)

Does it span disk drives?

(cross disk?)NO

(within one partition)YES

How to create? ln urfile linkname ln –s urfile linkname

size same as targeted file smaller than targeted file

inodes (file series number) same as targeted file different from target

Page 10: Introduction to Unix Shell Script Programming Summer course, Institute of Bioinformatics National Yang-Ming University

Example: build links

view the information about target file

create soft and hard links

check size of links

check series number of links

Page 11: Introduction to Unix Shell Script Programming Summer course, Institute of Bioinformatics National Yang-Ming University

Some Other useful Utilities

most write to stdout read from stdin

sort lines in file lexically sort myfile > my-sorted-file

removes adjacent duplicates in input uniq myfile outfile TIPS: sort before uniq

search pattern in files grep "foobar" f1 f2 ... fN

count lines, words, and bytes in file wc [-clmw] myfile

content of file

line (l) word (w) byte (c)

character (m)

Page 12: Introduction to Unix Shell Script Programming Summer course, Institute of Bioinformatics National Yang-Ming University

Review: Linux command (I)

Purpose SyntaxSee date $ date

See who’s using system $ who

Print current work directory $ pwd

List name of file in current directory $ ls or dirs (like dos)

Create text file $ cat > [filename]

Move or rename file/directory $ mv [file1] [file2]

Create multiple file copies via links $ ln [source] [link]

Remove file $ rm [filename]

Remove all files in given directory $ rm –rf [directory]

Page 13: Introduction to Unix Shell Script Programming Summer course, Institute of Bioinformatics National Yang-Ming University

Review: Linux command (II)

Purpose SyntaxChange file access permission $ chmod {u|g|o|a} {+|-} {r|w|x} [filename]

Read mail, send mail mail / mail [username]

See more about yourself who am I

Log out logout, exit, ctrl + D

Count file contents wc [filename]

Search pattern in file grep [pattern] [filename]

Sort file sort -{r|n|nr} [filename], reverse, numerical

Compare file cmp/diff [file1] [file2]

Page 14: Introduction to Unix Shell Script Programming Summer course, Institute of Bioinformatics National Yang-Ming University

Finding Out Who's About

list of users logged onto host who

list users on local machines users

list of processes ps –al (a=all users, l=long

listing)

get info on user with login X finger X

start a line-based chat with X talk X

Page 15: Introduction to Unix Shell Script Programming Summer course, Institute of Bioinformatics National Yang-Ming University

Redirection

Mostly all command gives output on your screen (we call the standard output, stdout for short) or take input from your keyboard (standard input, stdin for short)

Instead for efficiency, it is possible to send output to file or read input from file ☺

There are 3 main redirection symbols (operators) >, >> and <

Page 16: Introduction to Unix Shell Script Programming Summer course, Institute of Bioinformatics National Yang-Ming University

(1) Output result to file

syntax: command > filename

note if the file you specify already exists, doing this will

overwritten it, else a new file is created

Check whether you can do $ls > r1.txt cat r1.txt or use vi to check the content

Page 17: Introduction to Unix Shell Script Programming Summer course, Institute of Bioinformatics National Yang-Ming University

(2) Output result appended to file

syntax: command >> filename

note if the file you specify already exists, doing this will

append the result to file without overwrite it, else a new file is created

Check whether you can do $ls >> r1.txt cat r1.txt or use vi to check the content

Page 18: Introduction to Unix Shell Script Programming Summer course, Institute of Bioinformatics National Yang-Ming University

(3) Input data from file

syntax: command < filename

Check whether you can do edit a file r3.txt $cat < r3.txt check what shows in your screen

Page 19: Introduction to Unix Shell Script Programming Summer course, Institute of Bioinformatics National Yang-Ming University

Pipe

Sometime we need to build a pipeline composite from many different programs for our tasks, but temporary files are always verbose ><

There is a smart way to connect these programs without temporary files

Output of 1st command

take input from 1st command for next command

||

Page 20: Introduction to Unix Shell Script Programming Summer course, Institute of Bioinformatics National Yang-Ming University

Pipe (cont’d)

pipe is a temporary storage where the output of one command can be stored and passed as the input for next command

One can use pipe to run more than 2 commands at once

Syntax: command 1 | command 2

Page 21: Introduction to Unix Shell Script Programming Summer course, Institute of Bioinformatics National Yang-Ming University

Example of using Pipe

command using pipe meaning$ who | sort sort the list of current users

$ who | grep userA check whether userA has logged in

$ ls -l | grep fileA check whether fileA in current directory

more example?

Try to think one example of yours!

Page 22: Introduction to Unix Shell Script Programming Summer course, Institute of Bioinformatics National Yang-Ming University

Processes

It is any kind of program or task carried out by your machine, a program in execution

When task is running, O.S. will give it a number (PID, process-id, 32bits)

Purpose Syntax

See currently running process ps

Get information about all running process ps –ag

Stop process kill

Stop all process except your shell kill 0

Background processing command &

Page 23: Introduction to Unix Shell Script Programming Summer course, Institute of Bioinformatics National Yang-Ming University

Working on remote machines

Telnet/SSH – logon to a remote machine Starts a session on specified host machine Can run in terminal window over slow connection

FTP – File Transfer Protocol Move files between machines

ftp [host] Use ls, pwd, cd to navigate directories Use put file/get file to upload/download file bin to switch to binary transfer mode (safest) Type close or exit to close connection

anonymous login (public archive - e-mail as password)

Page 24: Introduction to Unix Shell Script Programming Summer course, Institute of Bioinformatics National Yang-Ming University

Help on Getting Started

Read the manual pages! man [command]

Web pages by google http://www.linux.org/ (English site) http://linux.vbird.org/ (Chinese site)

Page 25: Introduction to Unix Shell Script Programming Summer course, Institute of Bioinformatics National Yang-Ming University

Text Editors, Vi

Part X-II

Page 26: Introduction to Unix Shell Script Programming Summer course, Institute of Bioinformatics National Yang-Ming University

Vi - Old Faithful

has a command mode and edit mode available everywhere loads very quickly

Common mode1. cursor moving2. search, replace3. delete char or line4. copy and paste line

Edit modeyou can enter any char

Command modeyou can enter any char

i, o, a, R

[ESC]

:, /, ?

Page 27: Introduction to Unix Shell Script Programming Summer course, Institute of Bioinformatics National Yang-Ming University

New and Open a file

1. enter vi command 2. new file (common mode)

3. press “i” into edit mode “Insert”4. press “R” into edit mode “Replace”5. press “o” into edit mode “Insert” in new line6. press “a” into edit mode “Insert” (same as “i”)

Page 28: Introduction to Unix Shell Script Programming Summer course, Institute of Bioinformatics National Yang-Ming University

Common mode (I)

type command function

cursor moving

↑/k/shift + “–”

↓/j/shift + “+”up and down one line

ctrl + d, ctrl + u up and down half page

ctrl + b, ctrl + f up and down one page

n + space/enter (n is an integer) down n lines

0, $ move to begin/end of line

H, M, L move in current screen

G (end of file)

nG (n-th line of file)move in entire file

Page 29: Introduction to Unix Shell Script Programming Summer course, Institute of Bioinformatics National Yang-Ming University

Common mode (II)

type command function

search and replace

/pattern, ?pattern search pattern after/before cursor

:n1,n2s/word1/word2/g

replace word1 as word2 between line n1 and n2

1 stands for the 1st line, $s stands for the last line

add “c”, it replaces with user confirmation

delete

x, X delete a char at current cursor/before cursor

nx delete n chars begin with current cursor

dd delete current line

ndd, ncj delete n lines begin with current line

d1G delete all lines between the 1st line and current line

dG delete all lines between current line and the last line

Page 30: Introduction to Unix Shell Script Programming Summer course, Institute of Bioinformatics National Yang-Ming University

Common mode (III)

type command function

copy

yy copy current line

nyy copy n lines begin with current line

y1G copy lines between the 1st line and current line

yG copy lines between current line and the last line

pasteP, p paste at last line/next line w.r.t the current line

J join the content of current line and next line shown in current line

undo u back to last action

Page 31: Introduction to Unix Shell Script Programming Summer course, Institute of Bioinformatics National Yang-Ming University

Command mode

command function:w save current edited file

:w [filename] save file as [filename]

:n1, n2 w [filename]

save n1~n2 lines as [filename]

:w! forcedly save if file is read only

:wq save and quit

:q quit vi

:q! forcedly quit if you have modified the file but not wanna save it

:e! restore file before modification

ZZ save file if modified and quit (fast quit)

:r [filename] add content of [filename] to current cursor

:set nu/:set nonu enable/disable line number display

:! [command] execute command line commands without leaving vi

Page 32: Introduction to Unix Shell Script Programming Summer course, Institute of Bioinformatics National Yang-Ming University

Edit mode

your cursor

line number

your current cursor position

(line, position in line)

percentage

Page 33: Introduction to Unix Shell Script Programming Summer course, Institute of Bioinformatics National Yang-Ming University

Your show time

exercise instructions objective

Introduce yourself in 100 words in a text fileOf course, in English

NoteTry to copy one line or multiple lines

Page 34: Introduction to Unix Shell Script Programming Summer course, Institute of Bioinformatics National Yang-Ming University

System Shell Script

Part I-I

Page 35: Introduction to Unix Shell Script Programming Summer course, Institute of Bioinformatics National Yang-Ming University

Shell !?

This?

This?

or this?

Page 36: Introduction to Unix Shell Script Programming Summer course, Institute of Bioinformatics National Yang-Ming University

First glance — Linux Shell

Why Shell? Computer only realize the command in binary form

which is difficult for most of human So OS provides a special program call ‘shell’ accepts

human’s command in ‘readable’ form and translates them into 1 and 0 stream

Your commands

Linux shell Converted binary commands

OS kernel

$ ls$ man$ date

BASH000100010101001011000011100110100

Linux kernel

Page 37: Introduction to Unix Shell Script Programming Summer course, Institute of Bioinformatics National Yang-Ming University

Text Shells

Page 38: Introduction to Unix Shell Script Programming Summer course, Institute of Bioinformatics National Yang-Ming University

Kernel

Definition It is heart of Linux OS It manages all resources of OS

What it charges I/O (Input and Output) Process Devices File Memory

Page 39: Introduction to Unix Shell Script Programming Summer course, Institute of Bioinformatics National Yang-Ming University

What is Shell?

Shell is an command language interpreter that executes commands read from the standard input device (your keyboard) or from a file

In Linux OS, it may use one of the following most popular shells (BASH, CSH and KSH)

In Microsoft DOS, the name of shell is COMMAND.COM, but it is NOT as powerful as Linux shell

Page 40: Introduction to Unix Shell Script Programming Summer course, Institute of Bioinformatics National Yang-Ming University

Operating System Shell

Shell of an operating system it is a program that presents an interface to

various operating system functions and services

Why named “shell”? it is an outer layer of interface between the user

and the innards of the OS (kernel)

Page 41: Introduction to Unix Shell Script Programming Summer course, Institute of Bioinformatics National Yang-Ming University

Shell, an interface

Main categories CLI (Command Line Interface)

it makes things clear text shell (what we are going to learn now)

GUI (Graphical Use interface) it makes things look easygraphic shell (what people always use nowadays)

Page 42: Introduction to Unix Shell Script Programming Summer course, Institute of Bioinformatics National Yang-Ming University

CLI

Unix shells Bourne shell (sh)

Almquist shell (ash) Bourne-Again shell (bash)

C shell (csh) TENEX C shell (tcsh)

Korn shell (ksh) Scheme shell (scsh) Z shell (zsh)

Plan 9 and Unix rc shell (rc)

DOS: command.com

OS/2 and windows NT: cmd.exe

DOS, OS/2 and NT 4DOS, 4OS2, 4NT

Page 43: Introduction to Unix Shell Script Programming Summer course, Institute of Bioinformatics National Yang-Ming University

GUI

MS windows windows explorer litestep Geoshell BB4Win Emerge Desktop

Mac OS: Machitosh Finder

X-window system (Unix) KDE, GNOME Blackbox, CDE

DOSSHELL

KDEGNOME

MS Mac

DOSSHELL

Page 44: Introduction to Unix Shell Script Programming Summer course, Institute of Bioinformatics National Yang-Ming University

Popular Linux Shells

Shell NameDeveloped by …

RemarkWho Where

*BASH

(Bourne-Again SHell)Brian Fox and Chet Ramey

Free Software Foundation

Most common shell in Linux. It is Freeware shell and usually be the default shell.

CSH (C Shell) Bill JoyUniversity of California (Fro

BSD)

The C shell’s syntax and usage are very similar to the C programming language

KSH (Korn Shell)David G.

KornAT & T Bell Labs

claim that it has combined all advantages of 2 shells above

Page 45: Introduction to Unix Shell Script Programming Summer course, Institute of Bioinformatics National Yang-Ming University

Bell Labs: The beginning

The research and development center of Lucent Technologies, formerly AT&T.

Bell labs is one of the most renowned scientific laboratories in the world

Alexander Graham Bell (1847~1922)

Alexander Graham Bell founds the company that becomes AT&T with 2 financial backers at 1876. He is also the inventor of the telephone.

His famous sentence “Mr. Watson. Come here! I want you!” were the first words to travel over a wire, ringing in the birth of electronic communication

Page 46: Introduction to Unix Shell Script Programming Summer course, Institute of Bioinformatics National Yang-Ming University

Bell Labs: Brief

Its official name is Bell Telephone Laboratories or Bell Labs which was originally the research and development arm of the United states Bell System, and it was also the premier corporate facility of its type, developing a range of revolutionary technologies from telephone switches to specialized coverings for telephone cables, to transistor.

The work done by Bell Labs are Research

theoretical underpinnings for communications it includes math, physics, material science, behavioral sciences, computer

programming System engineering

concerning itself with conceiving the highly complex systems that make up the telecommunication networks

Development hardware and software for Bell System’s communication networks

Page 47: Introduction to Unix Shell Script Programming Summer course, Institute of Bioinformatics National Yang-Ming University

Bell Labs: big events

1933, discovered radio waves emitted from center of galaxy

1947, invention of transistor (Nobel Prize in Physics in 1956)

1948, Claude Shannon published “A Mathematical Theory of Communication”

1954, the development of photovoltaic cell

1957, electronic music by Max Mathews

1970s, Unix and C language

1971, computerized switching system for telephone traffic

1980, the realization of the world’s first single chip 32-bit microprocessor, the BELLMAC-32A

1980, C++ language

late 1980s and early 1990s, developed Plan9 as a replacement for Unix

1990s, inferno OS

Page 48: Introduction to Unix Shell Script Programming Summer course, Institute of Bioinformatics National Yang-Ming University

Variables in Linux

In the machine you are using, it has memory for storing your data. These memory are divided into smaller locations (address) and one can give them a name called memory variable or variable

There are 2 kinds of variables System variables

created and maintained by O.S. itself, all their name are capital. User-defined variable (UDV)

created and maintained by user, all their name are lower-case.

Check variables set: check all variables env: check system variables only

Page 49: Introduction to Unix Shell Script Programming Summer course, Institute of Bioinformatics National Yang-Ming University

The shell you use now

What is your current shell? grep username /etc/passwd echo $SHELL chsh (you can change the default shell here)

How many shell you can use cat /etc/shells

How to change current shell temporarily? just type the name of new shell

Page 50: Introduction to Unix Shell Script Programming Summer course, Institute of Bioinformatics National Yang-Ming University

Script Components

designator line Tell the O.S. where is the correct interpreter it begins with #!

comments it begins with #

shell commands correct separator: semicolon (;) or new line (\n)

Page 51: Introduction to Unix Shell Script Programming Summer course, Institute of Bioinformatics National Yang-Ming University

Tea Time

Take a break!

Review what you have learned

Page 52: Introduction to Unix Shell Script Programming Summer course, Institute of Bioinformatics National Yang-Ming University

Your first Bash shell script: How are you doing?

locate the shell you use, start with #!

comments, start with #

set variables (bash style)

output the results

Page 53: Introduction to Unix Shell Script Programming Summer course, Institute of Bioinformatics National Yang-Ming University

Discussion:the difference between apostrophes and quotation marks

Page 54: Introduction to Unix Shell Script Programming Summer course, Institute of Bioinformatics National Yang-Ming University

Your 2nd Bash shell script: some arithmetic computations

locate the shell you use, start with #!

comments, start with #

No variable declaration

Declare as Integers

Page 55: Introduction to Unix Shell Script Programming Summer course, Institute of Bioinformatics National Yang-Ming University

Discussion: what is the difference?

Page 56: Introduction to Unix Shell Script Programming Summer course, Institute of Bioinformatics National Yang-Ming University

Declare

Syntax: declare [-afir] variable[=value]

Options: -a: declared as array -f: declared as function -i: declared as integer -r: declared as read-only variable

Page 57: Introduction to Unix Shell Script Programming Summer course, Institute of Bioinformatics National Yang-Ming University

Interactive shell script

Page 58: Introduction to Unix Shell Script Programming Summer course, Institute of Bioinformatics National Yang-Ming University

Multiple Arguments:How to access?

Page 59: Introduction to Unix Shell Script Programming Summer course, Institute of Bioinformatics National Yang-Ming University

Logical operators

operator meaning= -eq equal

!= -ne Not equal

< -lt Less than

> -gl Greater than

-le Less than or equal (<=)

-ge Greater than or equal (>=)

-a AND

-o OR

arithmetic only, not for text Both arithmetic and text

Page 60: Introduction to Unix Shell Script Programming Summer course, Institute of Bioinformatics National Yang-Ming University

Flow control

Branches If then, else Case

Loops (for, while, until) it is a block of code that iterates a list of

commands as long as the loop control condition is true

Page 61: Introduction to Unix Shell Script Programming Summer course, Institute of Bioinformatics National Yang-Ming University

Branches

Page 62: Introduction to Unix Shell Script Programming Summer course, Institute of Bioinformatics National Yang-Ming University

(1) If then, else

Syntax and example: If [condition 1]; then [statement 1] elif [condition 2]; then [statement 2] else [statement 3] fi

cond 2

cond 1

Start

stat 1stat 2stat 3

then

then

else

else

Page 63: Introduction to Unix Shell Script Programming Summer course, Institute of Bioinformatics National Yang-Ming University

Simple form

yn=y

script is runningSTOP!

thenelse

User enter … (stored as yn)

Page 64: Introduction to Unix Shell Script Programming Summer course, Institute of Bioinformatics National Yang-Ming University

Complex conditions

yn=y || yn=Y

script is runningSTOP!

thenelse

User enter … (stored as yn)

Page 65: Introduction to Unix Shell Script Programming Summer course, Institute of Bioinformatics National Yang-Ming University

else if

Page 66: Introduction to Unix Shell Script Programming Summer course, Institute of Bioinformatics National Yang-Ming University

Check your file

Page 67: Introduction to Unix Shell Script Programming Summer course, Institute of Bioinformatics National Yang-Ming University

logic tags for file identification

tag stands for …

-e whether it exists or not

-f whether file exists or not

-d whether directory exists or not

-L whether file is link file

-r whether it can be read

-w whether it can be written

-x whether it can be executed

-s whether file is empty (true is not empty)

Page 68: Introduction to Unix Shell Script Programming Summer course, Institute of Bioinformatics National Yang-Ming University

Check and Create file

check properties while it exists

ask user whether wanna create it while it does not exist

Page 69: Introduction to Unix Shell Script Programming Summer course, Institute of Bioinformatics National Yang-Ming University

Remark: if then, else statements

You can link several conditions by || or && Each condition must be between [ ]

Caution! the space between condition statement and [ or ] is necessary

Page 70: Introduction to Unix Shell Script Programming Summer course, Institute of Bioinformatics National Yang-Ming University

Advanced Example

Analyze your host ~

Page 71: Introduction to Unix Shell Script Programming Summer course, Institute of Bioinformatics National Yang-Ming University

(2) Case

syntax and example case variable in

value_1) [statements_1] ;;value_2) [statements_2] ;;value_3) [statements_3] ;;*) [statements_for_exception]

esac

Page 72: Introduction to Unix Shell Script Programming Summer course, Institute of Bioinformatics National Yang-Ming University

Fortune Teller

Page 73: Introduction to Unix Shell Script Programming Summer course, Institute of Bioinformatics National Yang-Ming University

Loops

Page 74: Introduction to Unix Shell Script Programming Summer course, Institute of Bioinformatics National Yang-Ming University

for loop

syntax: for (( initial value; stop criteria; increment/decrement ))

do [statements] done

simple test The sum from 1 to 100

Page 75: Introduction to Unix Shell Script Programming Summer course, Institute of Bioinformatics National Yang-Ming University

Do what gauss has doneby shell script

Page 76: Introduction to Unix Shell Script Programming Summer course, Institute of Bioinformatics National Yang-Ming University

while loop and until loop

syntax while [ condition ]

do [statement] done

syntax until [ condition ]

do [statements] done

Page 77: Introduction to Unix Shell Script Programming Summer course, Institute of Bioinformatics National Yang-Ming University

Rewritten in while loop

Page 78: Introduction to Unix Shell Script Programming Summer course, Institute of Bioinformatics National Yang-Ming University

Rewritten in until loop

Page 79: Introduction to Unix Shell Script Programming Summer course, Institute of Bioinformatics National Yang-Ming University

Compare while and until

while it holds, continuewhile it reaches, stop

><=

Page 80: Introduction to Unix Shell Script Programming Summer course, Institute of Bioinformatics National Yang-Ming University

for loop, revisit

alternative syntax for variable in variable_list

do [statement] done

it differs significantly from its C counterpart

Page 81: Introduction to Unix Shell Script Programming Summer course, Institute of Bioinformatics National Yang-Ming University

Solar system

Page 82: Introduction to Unix Shell Script Programming Summer course, Institute of Bioinformatics National Yang-Ming University

How you login?

HOSTSSH/Telnet

enter username and password

check 1

User

read UID amd GIDread setting of home directory and shell

found

search the username you entered in /etc/passwd

check 2

check the password you entered in /etc/shadow

Login successfully

Login failed

pass

NOT found

NOT pass

Page 83: Introduction to Unix Shell Script Programming Summer course, Institute of Bioinformatics National Yang-Ming University

inside /etc/passwd

Notes for each field password has been moved to /etc/shadow UID (32bits)

0 (administrator), 1~500 (system UID), rest for others

Page 84: Introduction to Unix Shell Script Programming Summer course, Institute of Bioinformatics National Yang-Ming University

/etc/passwd and cut command

cut function: select portions of each lines of a file syntax: cut –f list [-d delim] [file] example: cut –f 1 –d : myfile the example above uses “:” to separate each line of myfile into

fields, then select the first field for each line

part of /etc/passwd of Mandrake 9.0

Page 85: Introduction to Unix Shell Script Programming Summer course, Institute of Bioinformatics National Yang-Ming University

Print all accounts in your host

Page 86: Introduction to Unix Shell Script Programming Summer course, Institute of Bioinformatics National Yang-Ming University

Summary: for, while and until

For loop: for (( triple condition))

do [statements] done

While while [ condition ]

do [statement] done

Until until [ condition ]

do [statements] done

alternative For loop for variable in variable_list

do [statement] done

Page 87: Introduction to Unix Shell Script Programming Summer course, Institute of Bioinformatics National Yang-Ming University

select

syntax: select variable [in list]

do [statements] break … it make script stop after selection

done

Note: if in list is not given, it will read list from arguments (shown in ex15.sh) compare to alternative for loop

Page 88: Introduction to Unix Shell Script Programming Summer course, Institute of Bioinformatics National Yang-Ming University

create Menu using select

Page 89: Introduction to Unix Shell Script Programming Summer course, Institute of Bioinformatics National Yang-Ming University

Function

syntax function function_name {statements} function function_name () {statements}

Page 90: Introduction to Unix Shell Script Programming Summer course, Institute of Bioinformatics National Yang-Ming University

Create menu with function

omitted list here, it will be read from arguments

call function here, call by name

Page 91: Introduction to Unix Shell Script Programming Summer course, Institute of Bioinformatics National Yang-Ming University

How to debug?

please man your shell first man bash man sh

syntax check bash --debugger script_name sh –n script_name

Page 92: Introduction to Unix Shell Script Programming Summer course, Institute of Bioinformatics National Yang-Ming University

References

Advanced Bash Scripting Guide http://www.tldp.org/LDP/abs/html/

Bash Reference Manual http://www.gnu.org/software/bash/manual/bashref.html

Learn from examples!

Page 93: Introduction to Unix Shell Script Programming Summer course, Institute of Bioinformatics National Yang-Ming University

Perl Shell Script

Part I-II

Page 94: Introduction to Unix Shell Script Programming Summer course, Institute of Bioinformatics National Yang-Ming University

Programming Languages

Imperative (Procedural) language C, C++, Java, Pascal

Functional language ML, Lisp

Scripting language Perl, PHP, Python

Page 95: Introduction to Unix Shell Script Programming Summer course, Institute of Bioinformatics National Yang-Ming University

Perl

Practical Extraction and Report Language

Designed by Larry Wall in 1987

It mixed the features from C, shell scripting, Lisp, and other programming languages

Page 96: Introduction to Unix Shell Script Programming Summer course, Institute of Bioinformatics National Yang-Ming University

Perl you are using (I)

version

location

author

http://en.wikipedia.org/wiki/Larry_Wall

http://www.wall.org/~larry/

Page 97: Introduction to Unix Shell Script Programming Summer course, Institute of Bioinformatics National Yang-Ming University

Perl you are using (II)

most module are installed here

Perl search path

Page 98: Introduction to Unix Shell Script Programming Summer course, Institute of Bioinformatics National Yang-Ming University

Perl basic data types

Scalar express numerical values and character strings e.g. $number, $string

Array number-indexed values (characters or numbers) e.g. @array

Hash similar to array, but the indexes could be non-numerical values e.g. %hash

Function (subroutine) call a function : & plus the function name, e.g. &func

Reference similar to pointer in C e.g. $add_1 = \$number; $add_2 = \@array; $add_3 = \%hash; $add_4 = \&func

Page 99: Introduction to Unix Shell Script Programming Summer course, Institute of Bioinformatics National Yang-Ming University

Your First Perl Script

tell system which interpreter to use

single line comments after #

code content

Page 100: Introduction to Unix Shell Script Programming Summer course, Institute of Bioinformatics National Yang-Ming University

Perl sub routine

$catch = &func_name;&func_name;

sub func_name { … return $value (optional)}

catch the return valueno return value

declare sub routine

Page 101: Introduction to Unix Shell Script Programming Summer course, Institute of Bioinformatics National Yang-Ming University

Global and Local variable

#! /usr/bin/perl

$sum = 10;

sub spoof { #my $sum; $sum = 7;}

spoof();print "$sum\n";

sum is 7

#! /usr/bin/perl

$sum = 10;

sub spoof { my $sum; $sum = 7;}

spoof();print "$sum\n";

sum is 10

refer to the same variable (the same mem add)

Page 102: Introduction to Unix Shell Script Programming Summer course, Institute of Bioinformatics National Yang-Ming University

Check Address

#! /usr/bin/perl

$sum = 10;

sub spoof { my $sum; $sum = 7;

my $sum_add = \$sum; print "add inside $sum_add\n";}

spoof();print "sum is $sum\n";

$sum_add = \$sum;print "add outside $sum_add\n";

#! /usr/bin/perl

$sum = 10;

sub spoof { #my $sum; $sum = 7;

my $sum_add = \$sum; print "add inside $sum_add\n";}

spoof();print "sum is $sum\n";

$sum_add = \$sum;print "add outside $sum_add\n";

Page 103: Introduction to Unix Shell Script Programming Summer course, Institute of Bioinformatics National Yang-Ming University

Scalar Numerical

a is 2, b is 3, and c is 5

2 + 3 = 52 - 3 = -12 * 3 = 62 / 3 = 0.6666666666666672 ^ 3 = 8

#! /usr/bin/perl

# Filename: math.pl# Function: math operators# Comments:

# declare 3 scalars$a = 2; $b = 3; $c = 5;

print "\na is $a, b is $b, and c is $c\n\n";

print "$a + $b = ".($a+$b)."\n";print "$a - $b = ".($a-$b)."\n";print "$a * $b = ".($a*$b)."\n";print "$a / $b = ".($a/$b)."\n";print "$a ^ $b = ".($a**$b)."\n";

Page 104: Introduction to Unix Shell Script Programming Summer course, Institute of Bioinformatics National Yang-Ming University

Scalar String

perlperlperl is easy$a\tis easyperl is easy$b\tis easy

#! /usr/bin/perl

$a = "perl";$b = 'perl';

$c = "$a\tis easy";$d = '$a\tis easy';

$e = "$b\tis easy";$f = '$b\tis easy';

print "$a\n";print "$b\n";print "$c\n";print "$d\n";print "$e\n";print "$f\n";

Page 105: Introduction to Unix Shell Script Programming Summer course, Institute of Bioinformatics National Yang-Ming University

Array and Hash

my @array;$array[0] = 'first';$array[1] = 'second';$array[2] = 'third';

my %hash;$hash{key} = 'value'; #assignmentprint $hash{key};

Page 106: Introduction to Unix Shell Script Programming Summer course, Institute of Bioinformatics National Yang-Ming University

Check the modules installed

Page 107: Introduction to Unix Shell Script Programming Summer course, Institute of Bioinformatics National Yang-Ming University

Use Perl module to check newly installed modules

#! /usr/bin/perl

# Filename: check_module.pl# Function: check all installed perl modules# Comments:

use strict;use ExtUtils::Installed;

my $inst = ExtUtils::Installed->new();my @modules = $inst->modules();my $module_num = @modules;

print "\nnumber of installed modules: $module_num\n\n";

foreach my $module (@modules) { my $version = $inst->version($module) || "???"; print "$module -- $version\n";}print "\n";

Page 108: Introduction to Unix Shell Script Programming Summer course, Institute of Bioinformatics National Yang-Ming University

“Man” your Perl

in most of Linux distribution, Perl is a built-in function

Page 109: Introduction to Unix Shell Script Programming Summer course, Institute of Bioinformatics National Yang-Ming University

Multiple Line Comments

use POD start with = string, string is NON-empty end with =cut, tell where to stop comment

use HERE-docs <<'#'; and << << '*/'; and */

use quote operators q{ comments };

http://www.perl.com/lpt/a/663

Page 110: Introduction to Unix Shell Script Programming Summer course, Institute of Bioinformatics National Yang-Ming University

POD style comment

#! /usr/bin/perl

# Filename: ex2.pl# Function: multiple comment I# Comments: POD style multiple comment

=commentprint "line within POD\n";=cut

print "line outside POD\n";

Page 111: Introduction to Unix Shell Script Programming Summer course, Institute of Bioinformatics National Yang-Ming University

HERE-docs style comment

#! /usr/bin/perl

# Filename: ex3.pl# Function: multiple comment II# Comments: a little bit verbose ><

<<'#';print "line1 within\n";#print "line1 outside\n";

<< '*/';print "line2 within\n";*/print "line2 outside\n";

method 1

method 2

Page 112: Introduction to Unix Shell Script Programming Summer course, Institute of Bioinformatics National Yang-Ming University

quote style comment

#! /usr/bin/perl

# Filename: ex4.pl# Function: multiple line comment III# Comments: use quote

q{ print "comment within quote\n";};

print "comment outside quote\n";

Page 113: Introduction to Unix Shell Script Programming Summer course, Institute of Bioinformatics National Yang-Ming University

Another choice

Acme::Comment It is a source filter It works before the Perl interpreter Namely, it modifies the code before Perl Explicitly, it removes the comment from source

http://search.cpan.org/~kane/Acme-Comment-1.02/lib/Acme/Comment.pm

Page 114: Introduction to Unix Shell Script Programming Summer course, Institute of Bioinformatics National Yang-Ming University

Acme style

use Acme::Comment type=>'C++', own_line=>1;

/* if (ref $mod) { $bar->{do}->blat(msg => 'blarg'); eval {

i'm sooo sick of this time for some coffee

*/

// I prefer beer. --sqrnhttp://search.cpan.org/~kane/Acme-Comment-1.02/lib/Acme/Comment.pm

Page 115: Introduction to Unix Shell Script Programming Summer course, Institute of Bioinformatics National Yang-Ming University

Install Perl modules by CPAN

You need to have root privilege command: cpan or perl –MCPAN –e shell

cpan shell -- CPAN exploration and modules installation (v1.7602)ReadLine support available (try 'install Bundle::CPAN')

cpan>

Please try “help”

Page 116: Introduction to Unix Shell Script Programming Summer course, Institute of Bioinformatics National Yang-Ming University

Perl Regular ExpressionAn example

% perl pattern3.plFrist ip = 255Second ip = 255Third ip = 255Fourth ip = 0

Frist ip = 255Second ip = 255Third ip = 255Fourth ip = 0

Page 117: Introduction to Unix Shell Script Programming Summer course, Institute of Bioinformatics National Yang-Ming University

Metacharacter

. 代表任意字元,但不包括換行字元 \n

^ 代表字串開頭

$ 代表字串結尾

\ 將其後的字元跳脫,使其回歸原字元的涵義,如: \\ , \. , \@ , \? , \*

| 替代樣式

[0-9] 一個數字字元[^0-9] 非數字[a-z] 一個英文小寫字母[^a-z] 非小寫[A-Z] 一個英文大寫字母[^A-Z] 非大寫[a-zA-Z] 英文字母[^a-zA-Z] 非英文字母

Page 118: Introduction to Unix Shell Script Programming Summer course, Institute of Bioinformatics National Yang-Ming University

Pattern Quantifier

* 代表比對成功的次數是 0 或 0 次以上 + 代表比對成功的次數是 1 或 1 次以上 ? 代表比對成功的次數是 0 或 1 次 {n} 代表比對成功的次數是 n 次{n,} 代表比對成功的次數至少是 n 次{n,m} 代表比對成功的次數至少是 n 次,但不超過 m 次

Page 119: Introduction to Unix Shell Script Programming Summer course, Institute of Bioinformatics National Yang-Ming University

Character Patterns

\r Carriage return (歸位字元 )\n New line (換行字元 )\t Tab (跳位字元 ) \w 比對到文字以及數字 . 即 [A-Za-z0-9_]. \W 比對到非文字以及數字 . 即 [^A-Za-z0-9_]. \s 比對到空白字元 . 即 [\ \t\f\r\n]. \S 比對到非空白字元 . 即 [^\ \t\f\r\n]. \d 比對到數字字元 . 即 [0-9] \D 比對到非數字字元 . 即 [^0-9]\b 比對到單字的邊界\B 比對到非單字的邊界\033 8 進位數 \x1B 16 進位數

Page 120: Introduction to Unix Shell Script Programming Summer course, Institute of Bioinformatics National Yang-Ming University

Examples 1

/abc/ 找到含有 abc 的字串

/^abc/ 找到開頭是 abc 的字串

/abc$/ 找到結尾是 abc 的字串

/a|b/ 找到有 a 或 b 的字串

/ab{2,4}c/ 找 到 a 後 面 跟 著 2-4 個 b , 再 跟 著 c

的字串,若只有 /ab{2,}c/則會找二個以上的 b /ab*c/

找 到 a 後 面 跟 著 0 個 或 多 個 b,再跟著 c的字串,如同 /ab{0,}c/

Page 121: Introduction to Unix Shell Script Programming Summer course, Institute of Bioinformatics National Yang-Ming University

Examples 2

/ab+c/ 找 到 a 後 面 跟 著 一 個 以 上 的 b , 再 跟 著 c

的字串,如同 /ab{1,}c/ /a.c/

.可以代表任何字元,除了 new line字元 (\n)外。 /\d/

找到含有數字的字串,如同 /[0-9]/ /\w/

找到含有字母的字串,如同 /[a-zA-Z0-9_]/ /\s/

找到含有 white space的字串,如同 /[ \t\r\n\f]/ /\*/

找到含有字元 *的字串,在反斜線“ \”後面的字元 Perl 會把它當作普通字元看待

Page 122: Introduction to Unix Shell Script Programming Summer course, Institute of Bioinformatics National Yang-Ming University

The End