Upload
others
View
9
Download
0
Embed Size (px)
Citation preview
Am
elG
houi
la &
Em
naA
chou
riC
3BI H
and
s-on
NG
S co
urse
–IP
P -2
1stN
ov 2
016 Creating directoriesandnavigating
through thefilestructure
Part3
1
Am
elG
houi
la &
Em
naA
chou
riC
3BI H
and
s-on
NG
S co
urse
–IP
P -2
1stN
ov 2
016
mkdir Make directory:createsanewdirectory
rmdir Removesanemptydirectory
pwd Displaystheabsolutepathofthecurrentworkingdirectory
cd Changedirectory:allows movingfromonedirectorytoanother
ls Listsadirectorycontent
2
Commandsformanipulatingdirectories
Am
elG
houi
la &
Em
naA
chou
riC
3BI H
and
s-on
NG
S co
urse
–IP
P -2
1stN
ov 2
016
• ls liststhecontentofthecurrentdirectorybydefault• Syntax:ls [OPTION][dirname]
Someusefuloptions:• -l:showssizes,modifieddateandtime,fileorfolder
nameandowneroffileandpermissions
• - a:Listallfilesincludinghiddenfilestartingwith‘.’• -lh:showssizesineasierreadableformat• -R:recursivelylistssub-directories• -lS:sortingbyfilesizes
3
ls command
Am
elG
houi
la &
Em
naA
chou
riC
3BI H
and
s-on
NG
S co
urse
–IP
P -2
1stN
ov 2
016
• mkdir:makesadirectory• Syntax:mkdir dirname [path]
• mkdir dirname:wouldcreateadirectorywiththespecifieddirname
• Thenewcreateddirectorywillbecreatedinyourcurrentworkingdirectory
• Ifyouwanttocreateitelsewhere,youhavetospecifythepath:mkdir dirname path
4
Create directories
Am
elG
houi
la &
Em
naA
chou
riC
3BI H
and
s-on
NG
S co
urse
–IP
P -2
1stN
ov 2
016
• Norealdistinctionbetweenthenamesofordinaryfilesandthenamesofdirectoryfiles.
• Notwofilesinthesamedirectorycanhavethesamename.
• Filesindifferentdirectoriescanhavethesamename.
• Linuxiscase-sensitive:Sanger,sangerandSANGERaredifferentandwouldrepresentthreedistinctfiles.
• Inmostcases,fileextensionsareoptional(.txt,.exe,etc.)
5
WhatyoushouldknowaboutfilenamesinLinux
Am
elG
houi
la &
Em
naA
chou
riC
3BI H
and
s-on
NG
S co
urse
–IP
P -2
1stN
ov 2
016
• cd:changetheworkingdirectory
• Commandsyntax:cd <path>
• Thepathnameofthedirectoryyouwanttomovetoshouldbespecified
• Youcanspecifyeithertheabsolutepathortherelativepath
6
Moveinthefilessystem
Am
elG
houi
la &
Em
naA
chou
riC
3BI H
and
s-on
NG
S co
urse
–IP
P -2
1stN
ov 2
016
/
home
Watson
INDA2016Currentworking
directory
• MovetoWatsondirectory
1. cd/home/Watson2. cd../..3. cd..+cd..
• MovetoGenomics directory
1. cd/home/Watson/INDA2016/Genomics
2. cd../Genomics3. cd..+cdGenomicsLinuxGenomics
7
Moveinthefilessystem:example
Am
elG
houi
la &
Em
naA
chou
riC
3BI H
and
s-on
NG
S co
urse
–IP
P -2
1stN
ov 2
016
command [–options] [arguments]Examples:
ls –lh /home/Watson/INDA2016pwdmkdir Test1
8
Commandsbasicsyntax
Am
elG
houi
la &
Em
naA
chou
riC
3BI H
and
s-on
NG
S co
urse
–IP
P -2
1stN
ov 2
016
• mancommandname
9
Howtogethelpforacommandfromtheterminal?
Am
elG
houi
la &
Em
naA
chou
riC
3BI H
and
s-on
NG
S co
urse
–IP
P -2
1stN
ov 2
016 Creating newfilesandusing text
editors
Part4
10
Am
elG
houi
la &
Em
naA
chou
riC
3BI H
and
s-on
NG
S co
urse
–IP
P -2
1stN
ov 2
016
• nano: a simpleandeasy-to-usetexteditor• IsinstalledbydefaultinUbuntuandmanyotherLinuxdistros
• It’saWYSIWYGeditor:“whatyouseeiswhatyouget.”Whatyoutypedirectlygoesintothetextinput
• vim,emacs,gedit,Geany:excellentprogramsbutdorequiresomelearning
11
Texteditors
Am
elG
houi
la &
Em
naA
chou
riC
3BI H
and
s-on
NG
S co
urse
–IP
P -2
1stN
ov 2
016
nameoftheprogramandversionnumber thenameofthefileyou
areediting
shortcuts
12
Getstartedwithnano
Am
elG
houi
la &
Em
naA
chou
riC
3BI H
and
s-on
NG
S co
urse
–IP
P -2
1stN
ov 2
016
• Tosearchforatextstring,hitCtrl+W,andenteryoursearchterm
• Thissearchcanthenbecancelledmid-executionbyhittingCtrl+C withoutdestroyingyourbuffer
• Ctrl+X:finishtypingandcloseanopenfile
Remember:nano pathname
- Opensthefileifit’sexistingalready,youcanmodifyandsavechanges
- Createsanewfileinthespecifiedpathifitdoesnotexist
13
Somenano shortcuts
Am
elG
houi
la &
Em
naA
chou
riC
3BI H
and
s-on
NG
S co
urse
–IP
P -2
1stN
ov 2
016
Let’splaywithnano
• Createexamplefile undersession1
• Open theterminal• Log in(/home/Username)• Typepwd tocheck• MovetotheSession1directoryunder theLinuxdirectory
cdSession1?cd./INDA2016/Linux/Session1
Username
Coursera
Genomics
/
home
INDA2016
Linux
Session1
Session3Session2
14
Am
elG
houi
la &
Em
naA
chou
riC
3BI H
and
s-on
NG
S co
urse
–IP
P -2
1stN
ov 2
016 Manipulating
filesanddirectories
Part5
15
Am
elG
houi
la &
Em
naA
chou
riC
3BI H
and
s-on
NG
S co
urse
–IP
P -2
1stN
ov 2
016
16
• Displayingwholecontentofafileorpartsofit(default+options)
• cat:viewthecontentofashortfilesyntax cat <filename>
more:viewthecontentofalongfileandnavigatethroughitsyntax more <filename>
• less:viewthecontentofalongfile,byportionssyntax less <filename>
• head:viewthefirstlinesofalongfilesyntax head <filename>
• tail:viewthelastlinesofalongfilesyntax tail <filename>
Basicviewingfilecommands
Am
elG
houi
la &
Em
naA
chou
riC
3BI H
and
s-on
NG
S co
urse
–IP
P -2
1stN
ov 2
016
17
• head commanddisplaysatextfilecontent,bydefault:10firstlinesatatime
• Syntax:head <options> <filename>
• tail commanddisplaysatextfilecontent,bydefault:10lastlinesatatime
• Syntax:tail <options> <filename>
Viewfilecontents:headandtailcommands
Am
elG
houi
la &
Em
naA
chou
riC
3BI H
and
s-on
NG
S co
urse
–IP
P -2
1stN
ov 2
016
• cp:copyfilesanddirectoriessyntax cp <pathfrom><pathto>
• mv:moveorrenamefilesanddirectoriessyntax mv <pathfrom><pathto>
• rm:removefilesanddirectoriessyntax rm pathname
18
Copy,moveandremove
Am
elG
houi
la &
Em
naA
chou
riC
3BI H
and
s-on
NG
S co
urse
–IP
P -2
1stN
ov 2
016
• Simplestform:cp file1file2
è Copythecontentsoffile1intofile2.Iffile2doesnotexist,itiscreated.Otherwise,file2issilentlyoverwrittenwiththecontentsoffile1.
• cp filenamedirpath
èMakeacopyofthefile(ordirectory)intothespecifieddestinationdirectory
19
Copyingcommand
Am
elG
houi
la &
Em
naA
chou
riC
3BI H
and
s-on
NG
S co
urse
–IP
P -2
1stN
ov 2
016
Themvcommandmovesorrenamesfilesanddirectoriesdependingonhowitisused• Torenameafile:
mv filename1filename2Iffile2exists,itscontentsaresilentlyreplacedwiththecontentsoffile1.Toavoidoverwriting,usetheinteractivemode:
mv-i filename1filename2
• Tomoveafile(oradirectory)toanotherdirectory:mv filedirpath
• Tomovedifferentfiles(oradirectory)toanotherdirectory:mv file1file2file3dirpath
• Tomovedirectorytoanotherdirectory:mv dir1dir2
Ifdir2doesnotexist,thendir1isrenameddir2.Ifdir2exists,thedirectorydir1ismovedwithindirectorydir2
20
Copyingcommand:mv
Am
elG
houi
la &
Em
naA
chou
riC
3BI H
and
s-on
NG
S co
urse
–IP
P -2
1stN
ov 2
016
Therm commanddeletes filesanddirectories
Toremove afile:rm filename
Toremove manyfiles:rm filename1filename2
Addtheinteractivemodetopromptuserbeforedeletingwith–irm -i filename1filename2
Deletedirectorieswithalltheircontentsrm -r dir1dir2
21
Thermcommand
Am
elG
houi
la &
Em
naA
chou
riC
3BI H
and
s-on
NG
S co
urse
–IP
P -2
1stN
ov 2
016 Extracting informationfrom files
Part6
22
Am
elG
houi
la &
Em
naA
chou
riC
3BI H
and
s-on
NG
S co
urse
–IP
P -2
1stN
ov 2
016
23
• wc printsnewline,word,andbytecountsforeachfile
• syntax:wc <options> <filename>• Someusefuloptions:• -c:printthebytecounts• -m:printthecharactercounts• -l:printthenewlinecounts• Formoreinfoaboutthedifferentcommandsuseman commandname
Somestatisctics aboutyourfilecontent:wc command
Am
elG
houi
la &
Em
naA
chou
riC
3BI H
and
s-on
NG
S co
urse
–IP
P -2
1stN
ov 2
016
24
• sort:reorderthecontentofafile“alphabetically”syntax: sort <filename>uniq:removesduplicatedlinessyntax: uniq <filename>join:comparethecontentsof2files,outputsthecommonentriessyntax: join <filename1><filename2>
• diff:comparethecontentsof2files,outputsthedifferencessyntax: diff <filename1><filename2>
Basicscommandstomanipulatefiles
Am
elG
houi
la &
Em
naA
chou
riC
3BI H
and
s-on
NG
S co
urse
–IP
P -2
1stN
ov 2
016
25
• sort outputsasortedorderofthefilecontentbasedonaspecifiedsortkey(default:takesentireinput)
• Syntax:sort <options><filename>• Defaultfieldseparator:Blank• Sortedfilesareusedasaninputforseveralothercommandssosortisoftenusedincombinationtoothercommands
• For<options>see man
Sortingdata
Am
elG
houi
la &
Em
naA
chou
riC
3BI H
and
s-on
NG
S co
urse
–IP
P -2
1stN
ov 2
016
26
• sort examples
w Sortalphabetically(defaultoption):sort <filename>w Sortnumerically:sort-n<filename>w Sortonaspecificcolumn(n°4):sort–k4<filename>w Sortbasedonatabseparator:sort-t$'\t’<filename>w ...
Sortexamples
Am
elG
houi
la &
Em
naA
chou
riC
3BI H
and
s-on
NG
S co
urse
–IP
P -2
1stN
ov 2
016
27
• grep:tosearchfortheoccurrenceofaspecificpattern(regularexpressionusingthewildcards…)inafileSyntax:grep <pattern><filename>cut:isusedtoextractspecificfieldsfromafile
Syntax:cut <options><filename>
Extractingdatafromfiles
Am
elG
houi
la &
Em
naA
chou
riC
3BI H
and
s-on
NG
S co
urse
–IP
P -2
1stN
ov 2
016
28
• grep (“globalregularexpressionprofile”)isusedtosearchfortheoccurrenceofaspecificpattern(regularexpression…)inafile
• Grep outputthewholelinecontainingthatpattern• Syntax:grep <options><pattern><filename>• For<options>see man
Example:Extract lines containing thepatternxxxfrom afile:
grep xxx <filename>Extract lines that donotcontain patternxxxfrom afile:
grep –vxxx <filename>
grep command
Am
elG
houi
la &
Em
naA
chou
riC
3BI H
and
s-on
NG
S co
urse
–IP
P -2
1stN
ov 2
016
29
• uniq outputsafilewithnoduplicatedlines• Uniq requiresasortedfileasaninput• Syntax:uniq <options><sorted_filename>• For<options>see man• Usefuloptionis-ctooutputeachlinewithitsnumberofrepeats
uniqcommand
Am
elG
houi
la &
Em
naA
chou
riC
3BI H
and
s-on
NG
S co
urse
–IP
P -2
1stN
ov 2
016
30
• join isusedtocompare2inputfilesbasedontheentriesinacommonfield(called“joinfield”)andoutputsamergedfile
• joinrequiressortedfilesasaninput• Lineswithidentitical“joinfield”willbepresentonlyonceintheoutput
• Structure:join <options><filename1><filename2>
• For<options>see man
Joincommand
Am
elG
houi
la &
Em
naA
chou
riC
3BI H
and
s-on
NG
S co
urse
–IP
P -2
1stN
ov 2
016
31
• diff isusedtocompare2inputfilesanddisplaysthedifferententries
• Canbeusedtohighlightdifferencesbetween2versionsofthesamefile
• Defaultoutput:common lines notshowed,onlydifferent lines areindicated andshowswhat hasbeenadded (a),deleted (d)orchanged (c)
• Structure:diff <options><filename1><filename2>• For<options>see man
diffcommand
Am
elG
houi
la &
Em
naA
chou
riC
3BI H
and
s-on
NG
S co
urse
–IP
P -2
1stN
ov 2
016
32
• grep:tosearchfortheoccurrenceofaspecificpattern(regularexpressionusingthewildcards…)inafileSyntax:grep <pattern><filename>cut:isusedtoextractspecificfieldsfromafile
Syntax:cut <options><filename>
Extractingdatafromfiles
Am
elG
houi
la &
Em
naA
chou
riC
3BI H
and
s-on
NG
S co
urse
–IP
P -2
1stN
ov 2
016
33
• grep (“globalregularexpressionprofile”)isusedtosearchfortheoccurrenceofaspecificpattern(regularexpression…)inafile
• Grep outputthewholelinecontainingthatpattern• Syntax:grep <options><pattern><filename>• For<options>see man
Example:Extract lines containing thepatternxxxfrom afile:
grep xxx <filename>Extract lines that donotcontain patternxxxfrom afile:
grep –vxxx <filename>
grep command
Am
elG
houi
la &
Em
naA
chou
riC
3BI H
and
s-on
NG
S co
urse
–IP
P -2
1stN
ov 2
016
34
• grep example
Let’sconsiderafilenamed“ghandi.txt”$ cat ghandi.txtThedifferencebetweenwhatwedoandwhatwearecapableofdoingwouldsufficetosolvemostoftheworld'sproblems
$ grep what ghandi.txtThedifferencebetweenwhat wedoandwhat wearecapableofdoing
$ grep-vwhat ghandi.txtwouldsufficetosolvemostoftheworld'sproblems
grep command(2)
Am
elG
houi
la &
Em
naA
chou
riC
3BI H
and
s-on
NG
S co
urse
–IP
P -2
1stN
ov 2
016
35
• cut isusedtoextractspecificfieldsfromafile• Structure:cut <options><filename>• For<options>see man• Importantoptionsare
w -d(fielddelimiter)w -f(fieldspecifier)
Example:extractfields2and3fromafilehaving‘space’asaseparatorcut-d’‘-f2,3<filename>
cut command
Am
elG
houi
la &
Em
naA
chou
riC
3BI H
and
s-on
NG
S co
urse
–IP
P -2
1stN
ov 2
016
Outputsredirectionandcombiningdifferent commands
Part7
36
Am
elG
houi
la &
Em
naA
chou
riC
3BI H
and
s-on
NG
S co
urse
–IP
P -2
1stN
ov 2
016
37
• Bydefault,thestandardoutputofany commandwillappear totheterminalscreen.
• Redirectionoftheoutputresult toafileis possible.• Thisis particularly useful forbig files• Syntax:command optionsfilename.in > filename.out
Commands outputs
Am
elG
houi
la &
Em
naA
chou
riC
3BI H
and
s-on
NG
S co
urse
–IP
P -2
1stN
ov 2
016
38
• Ifthefileexists,theresultwill be redirected toit$ cat ghandi.txtThedifference between what we doandwhat we arecapableofdoingwould suffice tosolvemost oftheworld's problems$ cut -d’‘-f2,3ghandi.txtdifference betweenwhat wesuffice toofthe
$ cut -d’‘-f2,3ghandi.txt > ghandi.txt.out$ cat ghandi.txt.outdifference betweenwhat wesuffice toofthe
• Ifthefiledoes notexist,it will beautomatically created andtheresultredirected toit.
Outputsredirection
Am
elG
houi
la &
Em
naA
chou
riC
3BI H
and
s-on
NG
S co
urse
–IP
P -2
1stN
ov 2
016
39
• The standard output of any command will be oneunique output
• As seen previously, this output can be printed in thescreen or redirected to a file
• However, the output result of a command can also beredirected to another command
• This is particularly useful when several operations areneeded for a file, with no need to store theintermediate outputs
Commands combination
Am
elG
houi
la &
Em
naA
chou
riC
3BI H
and
s-on
NG
S co
urse
–IP
P -2
1stN
ov 2
016
40
• Combiningseveralcommandsisdonethankstotheuseofa“|”character
• Structure:command1 options1filename1.in |command2 options2> filename.out
• Thiscanbedoneforasmanycommandsasneeded
Commands combination:example
Am
elG
houi
la &
Em
naA
chou
riC
3BI H
and
s-on
NG
S co
urse
–IP
P -2
1stN
ov 2
016Wildcards
41
• Sincetheshellusesfilenamessomuch,itprovidesspecialcharacterstohelprapidlyspecifyinggroupsoffilenames
• Agroupofspecialcharacters arecalledwildcardsallowselectingfilenamesbasedonpatternofcharacters
Am
elG
houi
la &
Em
naA
chou
riC
3BI H
and
s-on
NG
S co
urse
–IP
P -2
1stN
ov 2
016Wildcards
42
Wildcard Meaning* Matches anycharacters
^ Matchesanysetofcharacters startingbythecharacterfollowing thesign
? Matchesanysingle character
[!characters] Matchesanycharacterthatisnotamemberofthesetcharacters
[characters] Matchesanycharacterthatisamemberofthesetcharacters.ThesetofcharactersmayalsobeexpressedasaPOSIXcharacterclass suchasoneofthefollowing:[:alnum:]Alphanumericcharacters[:alpha:]Alphabeticcharacters[:digit:]Numerals[:upper:]Uppercasealphabeticcharacters[:lower:] Lowercasealphabeticcharacters
Am
elG
houi
la &
Em
naA
chou
riC
3BI H
and
s-on
NG
S co
urse
–IP
P -2
1stN
ov 2
016
Wildcards:examples
43
Wildcard Meaninga* Any filename starting with a
* Allpossible filenames
^a Anyfilenamestartingbya
A*.fasta AllfilenamesthatbeginwithAandendwith.fasta
????.vcf Anyfilenamesthatcontainexactly4charactersandendwith.vcf
[abc]* Anyfilenamethatbeginswith"a"or"b"or"c"followedbyanyothercharacters
[[:upper:]]* Anyfilenamethatbeginswithanuppercaseletter.Thisisanexampleofacharacterclass