Upload
ashley
View
47
Download
0
Embed Size (px)
DESCRIPTION
2.0.0.3.3 Introduction to the Shell – Session 3. Job control Start, stop and monitor running processes Monitoring disk usage Command line tools grep, cut, sort, uniq, paste, fold, tr, head, tail… Command line Perl When command line tools don’t quite do it. Seeing Running Processes: top. - PowerPoint PPT Presentation
Citation preview
04/24/23 Introduction to the Shell – Session 3 1
2.0.0.3 – Introduction to the Shell
2.0.0.3.3Introduction to the Shell – Session 3
· Job control· Start, stop and monitor running
processes· Monitoring disk usage
· Command line tools· grep, cut, sort, uniq, paste, fold,
tr, head, tail…· Command line Perl
· When command line tools don’t quite do it
04/24/23 Introduction to the Shell – Session 3 2
2.0.0.3 – Introduction to the Shell
Seeing Running Processes: top· See a constantly updated list of what’s running with
· Sorted by CPU use
· Also shows available memory and processors
· Processes can be controlled based on the PID (process ID)
top
Overall memory use
Overall CPU use
Process memory use
Process CPU use
I’m not running all that!Many processes are run by the OS (as user “root”) in the background. These processes generally have very low PIDs.
04/24/23 Introduction to the Shell – Session 3 3
2.0.0.3 – Introduction to the Shell
Seeing Running Processes: ps· See a complete list of what’s running with
· By default only shows what you are running (no root or other user’s processes)
· Can select which processes to show: eg.U by userp by PIDr only running processes
· See full list of everything running with ps auxw
ps
04/24/23 Introduction to the Shell – Session 3 4
2.0.0.3 – Introduction to the Shell
Seeing Running Jobs: jobs· See currently running jobs with
· These are processes that the current user is running in this terminal (does not show other user’s processes, or jobs started from other terminal windows)
· Useful for job control….
jobs
04/24/23 Introduction to the Shell – Session 3 5
2.0.0.3 – Introduction to the Shell
Putting Processes in the Background: &, bg and fg· What if you’re running a long job (eg. BLAST), but you want to use your prompt at the same time?
· Put the job in the background by starting it with <command> &
· Or, once the job is running, stop it with Cntl-Z (interrupts but does not kill), then use
· To put the job back in the foreground (and lose your prompt), use
· To specify which job to fg or bg, use the PID or %<job id>
fg
bg
Job ID
04/24/23 Introduction to the Shell – Session 3 6
2.0.0.3 – Introduction to the Shell
Ending Jobs: Cntl-C, kill· To end (without saving anything) a job currently running in the terminal, use Cntl-C
· To kill a job that is running in the background or from another terminal, use
· If it really won’t die, try kill -9 but only as a last resort!
Killing applicationsIf you are running applications (eg. word processors, web browsers) from the command line, do NOT kill them unless you cannot exit the application normally – this can corrupt files or cause application errors.
kill
04/24/23 Introduction to the Shell – Session 3 7
2.0.0.3 – Introduction to the Shell
Sharing the CPU: nice· Jobs are assigned priority (the lower the number, the higher the priority) which determines sharing of CPU
· Set the priority of your job lower when running intensive jobs so that others can use the CPU too with
· Or, for running jobs, with
nice
renice
Priority
Niced
04/24/23 Introduction to the Shell – Session 3 8
2.0.0.3 – Introduction to the Shell
Memory and Swap· Nice does not control memory use· When running memory-intensive jobs, avoid requiring swap (virtual memory – actually disk space) as this will slow all jobs on that computer
04/24/23 Introduction to the Shell – Session 3 9
2.0.0.3 – Introduction to the Shell
Checking Disk Location and Usage: df· To confirm that the disk you are on has enough space for you to create large files:
-h human-readable
df
04/24/23 Introduction to the Shell – Session 3 10
2.0.0.3 – Introduction to the Shell
Determining Disk Space Usage: du· To check total space taken by a directory structure, use
· Find out what is taking up all your disk space!
-h human-readable-c produce total
--max-depth only print totals for
this deep in directory tree
du
Saving spaceCut down on disk usage by using du and find to find large, old files, and use gzip to compress them.
04/24/23 Introduction to the Shell – Session 3 11
2.0.0.3 – Introduction to the Shell
Working with Text· Simple text processing is a large part of bioinformatics
· Converting between program formats· Data processing
· Command line tools are a fast and easy way to do a lot of text processing· Usually faster than Perl· Don’t require any coding!
04/24/23 Introduction to the Shell – Session 3 12
2.0.0.3 – Introduction to the Shell
Searching for Text: grep· Search for text in a file with
· Prints matching lines· Can do regular expression matching (slightly different from Perl)
· Useful options:-i case-insensitive-w text must appear as a “word” (surrounded by white space)
-v find non-matching lines-c count matching lines-B <#> print # lines before matching lines
-A <#> print # lines after matching lines
-f <file> find lines matching patterns listed in <file>
grep
04/24/23 Introduction to the Shell – Session 3 13
2.0.0.3 – Introduction to the Shell
Sorting Text: sort· Sort a file using
· Useful options:-f case-insensitive-g sort numerically (i.e. 11<100)-r reverse sort-k <#> sort on column #-u only print one copy of unique lines
sort
04/24/23 Introduction to the Shell – Session 3 14
2.0.0.3 – Introduction to the Shell
Working with Columns: cut and paste· Get particular columns from a file with
· Have to specify which field (column) to get· Useful options:-f <#> fields to output-d field delimiter
· Put columns back together with
cut
paste
04/24/23 Introduction to the Shell – Session 3 15
2.0.0.3 – Introduction to the Shell
Finding Unique or Repeated Lines: uniq· Determine unique lines with
· Collapses repeated lines
· Useful options:-i case insensitive-c count copies of each line-d print only duplicated lines-u print only unique lines-f <#> do not compare the first # fields
uniq
Pipe it!The UNIX text tools are particularly powerful in combination – use pipes to do a series of processing commands on your text.
04/24/23 Introduction to the Shell – Session 3 16
2.0.0.3 – Introduction to the Shell
Counting: wc· Count lines, words, and characters with
· Good for checking if files have correct number of records
· Useful in combination with other commands
wc
lines words
bytes (characters)
How many SAGE tags match more than one gene
04/24/23 Introduction to the Shell – Session 3 17
2.0.0.3 – Introduction to the Shell
Replacing Characters: tr· Replace characters with
· Like Perl’s tr///
· Useful options:-d delete-s replace repeats with single occurrence
-c use complement (replace everything EXCEPT what is specified)
tr
Input to trInstead of reading from a file, tr reads from standard input. If you want it to read from a file, redirect that file to standard input with “<“, or use cat and pipe the output to tr.
Format comma-separated list
Capitalize
04/24/23 Introduction to the Shell – Session 3 18
2.0.0.3 – Introduction to the Shell
Formatting Text: fold· To wrap long lines in a file, use
· Good for formatting before printing text files
· Useful options:-w width to wrap to-s break at spaces
fold
04/24/23 Introduction to the Shell – Session 3 19
2.0.0.3 – Introduction to the Shell
Getting Parts of Files: head and tail· Print the first 10 lines with
· Print the last 10 lines with
· Specify how many lines to get with head <#>eg. head -100 -> gets first 100 lines
· Good for making smaller “test” files
head
tail
Divide a file into three
Make a test file
04/24/23 Introduction to the Shell – Session 3 20
2.0.0.3 – Introduction to the Shell
Comparing files: diff· Find differences between lines in files with
· Can also compare directories
· Useful options:-b ignore differences in white space
-i ignore changes in case
diff
Confirm file copying
04/24/23 Introduction to the Shell – Session 3 21
2.0.0.3 – Introduction to the Shell
More advance processing: sed and awk· Simple programming languages· Designed for command-line use· Good for more advanced text processing
· But I prefer….
04/24/23 Introduction to the Shell – Session 3 22
2.0.0.3 – Introduction to the Shell
Command Line Perl· Perl can be run on a script, or directly on text input at the command line (a very short Perl script that does not have to be saved to disk)
· Input file with cat or <
· Options to use command line:-e find code on command line not in file
-n run command line script on each line of input
· Check out other Perl command line options with man perlrun
“What?” ‘Quotes?’When running Perl on the command line, use single quotes (‘) so that the shell does not try to interpret the Perl commands. Within these quotes, you can then use double quotes (“) for eg. print statements.
04/24/23 Introduction to the Shell – Session 3 23
2.0.0.3 – Introduction to the Shell
2.0.0.3.3Introduction to the Shell – Session 3
· Now you know….· How to control your processes· How to keep up on system usage· How to play with text on the
command line· How to run Perl from the
command line
04/24/23 Introduction to the Shell – Session 3 24
2.0.0.3 – Introduction to the Shell
2.0.0.3.3Further Readings
· Job control· Linux Cookbook Ch. 4.3· Learning the UNIX Operating System Ch. 6
· Command line goodies· Linux Cookbook Ch.13.1, 15.1-15.4
· Command line Perl· Linux Cookbook Ch. 13.5, 15.5