Upload
buinguyet
View
214
Download
0
Embed Size (px)
Citation preview
Introduc)ontoI/O
CSE421/521:Opera)ngSystemsKarthikDantu
SlidesadoptedfromCS162classatBerkeley,CSE451atU-WashingtonandCSE421byProfKosaratUB
Assignment#1-Clarifica)ons• Grading– 5%-Design– 10%-Implementa)on/tests
• Git– Workingwithgithubtosetuprepos– Shouldbeupbyearlynextweek
• Detailsofscheduling– Incomingclassesandrecita)ons– Assignmentgivenearly
• Deadlines– 9/26–Designdocument– 10/3–Implementa)onviaAutograder
HowDoestheKernelProvideServices?
• Yousaidthatapplica)onsrequestservicesfromtheopera)ngsystemviasyscall,but…
• I’vebeenwri)ngallsortofusefulapplica)onsandInevereversawa“syscall”!!!
• That’sright.• Itwasburiedintheprogramminglanguagerun)melibrary(e.g.,libc.a)
• …Layering
OSRun-TimeLibrary
OS
Proc 1
Proc 2
Proc n…
OS
Appln login Window Manager
…OS library OS library OS library
AKindofNarrowWaist
Compilers
Web Servers
Web Browsers
Databases
Word Processing
Portable OS Library
System Call Interface
Portable OS Kernel
Platform support, Device Drivers
x86 ARMPowerPC
Ethernet (1Gbs/10Gbs) 802.11 a/g/n/ac SCSI ThunderboltGraphics
PCIHardware
Software
System
UserOS
Application / Service
KeyUnixI/ODesignConcepts• Uniformity– fileopera)ons,deviceI/O,andinterprocesscommunica)onthroughopen,read/write,close
– Allowssimplecomposi)onofprograms• find|grep|wc…
• Openbeforeuse– Providesopportunityforaccesscontrolandarbitra)on– Setsuptheunderlyingmachinery,i.e.,datastructures
• Byte-oriented– Evenifblocksaretransferred,addressingisinbytes
• Kernelbufferedreads– Streamingandblockdeviceslooksthesame– readblocksprocess,yieldingprocessortoothertask
Puhngittogether:webserver
Server
Kernel
Hardware
requestbuffer
replybuffer
11. kernel copy from user buffer to network buffer
Network interface Disk interface
12. format outgoing packet and DMA
6. disk request
10. network socket write
1. network socket read
2. copy arriving packet (DMA)
syscall
wait
interrupt
3. kernel copy
RTU
5. file readsyscall
8. kernel copy
RTU
7. disk data (DMA)
interrupt
4. parse request 9. format reply
Request Reply
Kernel buffer reads
KeyUnixI/ODesignConcepts• Uniformity
– fileopera)ons,deviceI/O,andinterprocesscommunica)onthroughopen,read/write,close
– Allowssimplecomposi)onofprograms• find|grep|wc…
• Openbeforeuse– Providesopportunityforaccesscontrolandarbitra)on– Setsuptheunderlyingmachinery,i.e.,datastructures
• Byte-oriented– Evenifblocksaretransferred,addressingisinbytes
• Kernelbufferedreads– Streamingandblockdeviceslooksthesame– readblocksprocess,yieldingprocessortoothertask
• Kernelbufferedwrites– Comple)onofout-goingtransferdecoupledfromtheapplica)on,allowingittocon)nue
Puhngittogether:webserver
Server
Kernel
Hardware
requestbuffer
replybuffer
11. kernel copy from user buffer to network buffer
Network interface Disk interface
12. format outgoing packet and DMA
6. disk request
10. network socket write
1. network socket read
2. copy arriving packet (DMA)
syscall
wait
interrupt
3. kernel copy
RTU
5. file readsyscall
8. kernel copy
RTU
7. disk data (DMA)
interrupt
4. parse request 9. format reply
Request Reply
Kernel buffer writes
KeyUnixI/ODesignConcepts• Uniformity
– fileopera)ons,deviceI/O,andinterprocesscommunica)onthroughopen,read/write,close
– Allowssimplecomposi)onofprograms• find|grep|wc…
• Openbeforeuse– Providesopportunityforaccesscontrolandarbitra)on– Setsuptheunderlyingmachinery,i.e.,datastructures
• Byte-oriented– Evenifblocksaretransferred,addressingisinbytes
• Kernelbufferedreads– Streamingandblockdeviceslooksthesame– readblocksprocess,yieldingprocessortoothertask
• Kernelbufferedwrites– Comple)onofout-goingtransferdecoupledfromtheapplica)on,
allowingittocon)nue• Explicitclose
I/O&StorageLayers
High Level I/O
Low Level I/O Syscall
File System
I/O Driver
Application / Servicestreams
handles
registers
descriptors
Commands and Data Transfers
Disks, Flash, Controllers, DMA
TheFileSystemAbstrac)on• High-levelidea
– Filesliveinhierarchicalnamespaceoffilenames• File
– Namedcollec)onofdatainafilesystem– Filedata
• Text,binary,linearizedobjects– FileMetadata:informa)onaboutthefile
• Size,Modifica)onTime,Owner,Securityinfo• Basisforaccesscontrol
• Directory– “Folder”containingfiles&Directories– Hierachical(graphical)naming
• Paththroughthedirectorygraph• Uniquelyiden)fiesafileordirectory
– /home/ff/cs162/public_html/fa17/index.html– LinksandVolumes(later)
CHigh-LevelFileAPI–Streams(review)
• Operateon“streams”-sequenceofbytes,whethertextordata,withaposi)on
#include<stdio.h>FILE*fopen(constchar*filename,constchar*mode);intfclose(FILE*fp);
Mode Text Binary Descriptions
r rb Open existing file for readingw wb Open for writing; created if does not exista ab Open for appending; created if does not existr+ rb+ Open existing file for reading & writing.w+ wb+ Open for reading & writing; truncated to zero if exists, create otherwisea+ ab+ Open for reading & writing. Created if does not exist. Read from beginning,
write as append
Connec)ngProcesses,Filesystem,andUsers
• Processhasa‘currentworkingdirectory’
• AbsolutePaths– /home/kdantu/cs421
• Rela)vepaths– index.html,./index.html-currentWD– ../index.html-parentofcurrentWD– ~,~kdantu-homedirectory
CAPIStandardStreams• Threepredefinedstreamsareopenedimplicitlywhenaprogramisexecuted– FILE*stdin–normalsourceofinput,canberedirected– FILE*stdout–normalsourceofoutput,canberedirected– FILE*stderr–diagnos)csanderrors,canberedirected
• STDIN/STDOUTenablecomposi)oninUnix– Recall:UseofpipesymbolsconnectsSTDOUTandSTDIN
• find|grep|wc…
ChighlevelFileAPI–StreamOps#include<stdio.h>//characterorientedintfputc(intc,FILE*fp);//rtncorEOFonerrintfputs(constchar*s,FILE*fp);//rtn>0orEOFintfgetc(FILE*fp);char*fgets(char*buf,intn,FILE*fp);
#include<stdio.h>//characterorientedintfputc(intc,FILE*fp);//rtncorEOFonerrintfputs(constchar*s,FILE*fp);//rtn>0orEOFintfgetc(FILE*fp);char*fgets(char*buf,intn,FILE*fp);//blockorientedsize_tfread(void*ptr,size_tsize_of_elements,size_tnumber_of_elements,FILE*a_file);size_tfwrite(constvoid*ptr,size_tsize_of_elements,size_tnumber_of_elements,FILE*a_file);
ChighlevelFileAPI–StreamOps
ChighlevelFileAPI–StreamOps#include<stdio.h>//characterorientedintfputc(intc,FILE*fp); //rtncorEOFonerrintfputs(constchar*s,FILE*fp); //rtn>0orEOFintfgetc(FILE*fp);char*fgets(char*buf,intn,FILE*fp);//blockorientedsize_tfread(void*ptr,size_tsize_of_elements,size_tnumber_of_elements,FILE*a_file);size_tfwrite(constvoid*ptr,size_tsize_of_elements,size_tnumber_of_elements,FILE*a_file);//formattedintfprintf(FILE*restrictstream,constchar*restrictformat,
...);intfscanf(FILE*restrictstream,constchar*restrictformat,
...);
ExampleCode#include<stdio.h>
#defineBUFLEN256FILE*outfile;charmybuf[BUFLEN];
intstoretofile(){char*instring;outfile=fopen("/usr/homes/testing/tokens","w+");if(!outfile)return(-1);//Error!while(1){instring=fgets(mybuf,BUFLEN,stdin);//catchesoverrun!//Checkforerrororendoffile(^D)if(!instring||strlen(instring)==0)break;//Writestringtooutputfile,exitonerrorif(fputs(instring,outfile)<0)break;}fclose(outfile);//Flushesfromuserspace}
CStreamAPIposi)oning
• Preserveshighlevelabstrac)onofuniformstreamofobjects• Addsbufferingforperformance
intfseek(FILE*stream,longintoffset,intwhence);longintftell(FILE*stream)voidrewind(FILE*stream) High%Level%I/O%%
Low%Level%I/O%%Syscall%
File%System%
Upper%I/O%Driver%
Lower%I/O%Driver%
offset (SEEK_CUR)
offset (SEEK_SET) offset (SEEK_END)
What’sbelowthesurface??Application / Service
High Level I/O
Low Level I/O Syscall
File System
I/O Driver
streamshandlesregisters
descriptors
Commands and Data Transfers
Disks, Flash, Controllers, DMA
CLowlevelI/O• Opera)onsonFileDescriptors–asOSobjectrepresen)ngthestateofafile– Userhasa“handle”onthedescriptor
#include<fcntl.h>#include<unistd.h>#include<sys/types.h>intopen(constchar*filename,intflags[,mode_tmode])intcreat(constchar*filename,mode_tmode)intclose(intfiledes)
Bit vector of:• Access modes (Rd, Wr, …)• Open Flags (Create, …)• Operating modes (Appends, …)
Bit vector of Permission Bits:• User|Group|Other X R|W|X
http://www.gnu.org/software/libc/manual/html_node/Opening-and-Closing-Files.html
CLowLevel:standarddescriptors
• Crossinglevels:Filedescriptorsvs.streams• Don’tmixthem!
#include<unistd.h>STDIN_FILENO-macrohasvalue0STDOUT_FILENO-macrohasvalue1STDERR_FILENO-macrohasvalue2intfileno(FILE*stream)FILE*fdopen(intfiledes,constchar*opentype)
CLowLevelOpera)ons
• Whenwritereturns,dataisonitswaytodiskandcanberead,butitmaynotactuallybepermanent!
ssize_tread(intfiledes,void*buffer,size_tmaxsize)-returnsbytesread,0=>EOF,-1=>errorssize_twrite(intfiledes,constvoid*buffer,size_tsize)-returnsbyteswrittenoff_tlseek(intfiledes,off_toffset,intwhence)intfsync(intfildes)–waitfori/otofinishvoidsync(void)–waitforALLtofinish
Andlotsmore!• TTYsversusfiles• Memorymappedfiles• FileLocking• AsynchronousI/O• GenericI/OControlOpera)ons• Duplica)ngdescriptors
intdup2(intold,intnew)intdup(intold)
Anotherexample:lowio-std.c#include<stdlib.h>#include<stdio.h>#include<string.h>#include<unistd.h>#include<sys/types.h>#defineBUFSIZE1024intmain(intargc,char*argv[]){charbuf[BUFSIZE];ssize_twritelen=write(STDOUT_FILENO,"Iamaprocess.\n",16);ssize_treadlen=read(STDIN_FILENO,buf,BUFSIZE);ssize_tstrlen=snprintf(buf,BUFSIZE,"Got%zdchars\n",readlen);writelen=strlen<BUFSIZE?strlen:BUFSIZE;write(STDOUT_FILENO,buf,writelen);exit(0);}
What’sbelowthesurface??Application / Service
High Level I/O
Low Level I/O Syscall
File System
I/O Driver
streamshandlesregisters
descriptors
Commands and Data Transfers
Disks, Flash, Controllers, DMA
Recall:SYSCALL
• Lowlevellibparametersaresetupinregistersandsyscallinstruc)onisissued– Atypeofsynchronousexcep)onthatenterswell-definedentrypointsintokernel
What’sbelowthesurface??
High Level I/O
Low Level I/O Syscall
File System
I/O Driver
Application / Servicestreams
handlesregisters
descriptors
Commands and Data Transfers
Disks, Flash, Controllers, DMA
File descriptor number - an int
File Descriptors - a struct with all the info about the files
InternalOSFileDescriptor• InternalDataStructuredescribingeverythingaboutthe
file– Whereitresides– Itsstatus– Howtoaccessit
• Pointer:structfile*file
FileSystem:fromsyscalltodriver
ssize_tvfs_read(structfile*file,char__user*buf,size_tcount,loff_t*pos){ssize_tret;if(!(file->f_mode&FMODE_READ))return-EBADF;if(!file->f_op||(!file->f_op->read&&!file->f_op->aio_read))return-EINVAL;if(unlikely(!access_ok(VERIFY_WRITE,buf,count)))return-EFAULT;ret=rw_verify_area(READ,file,pos,count);if(ret>=0){count=ret;if(file->f_op->read)ret=file->f_op->read(file,buf,count,pos);elseret=do_sync_read(file,buf,count,pos);if(ret>0){fsnotify_access(file->f_path.dentry);add_rchar(current,ret);}inc_syscr(current);}returnret;}
In fs/read_write.c
LowerLevelDriver• Associatedwithpar)cularhardwaredevice• Registers/Unregistersitselfwiththekernel• Handlerfunc)onsforeachofthefileopera)ons
Recall:DeviceDrivers• DeviceDriver:Device-specificcodeinthekernelthatinteractsdirectlywiththedevicehardware– Supportsastandard,internalinterface– SamekernelI/Osystemcaninteracteasilywithdifferentdevicedrivers– Specialdevice-specificconfigura)onsupportedwiththeioctl()system
call
• DeviceDriverstypicallydividedintotwopieces:– Tophalf:accessedincallpathfromsystemcalls
• implementsasetofstandard,cross-devicecallslikeopen(), close(), read(), write(), ioctl(), strategy()
• Thisisthekernel’sinterfacetothedevicedriver• TophalfwillstartI/Otodevice,mayputthreadtosleepun)lfinished
– Boqomhalf:runasinterruptrou)ne• Getsinputortransfersnextblockofoutput• MaywakesleepingthreadsifI/Onowcomplete
LifeCycleofAnI/ORequest
Device DriverTop Half
Device DriverBottom Half
DeviceHardware
Kernel I/OSubsystem
UserProgram
Sowhathappenswhenyoufgetc?
High Level I/O
Low Level I/O Syscall
File System
I/O Driver
Application / Service
streamshandlesregisters
descriptors
Commands and Data Transfers
Disks, Flash, Controllers, DMA
Communica)onbetweenprocesses• Canweviewfilesascommunica)onchannels?
• ProducerandConsumerofafilemaybedis)nctprocesses– Maybeseparatedin)me(ornot)
• However,whatifdatawriqenonceandconsumedonce?– Don’twewantsomethingmorelikeaqueue?– Cans)lllooklikeFileI/O!
write(wfd,wbuf,wlen);
n=read(rfd,rbuf,rmax);
Communica)onAcrosstheworldlookslikefileIO
• ConnectedqueuesovertheInternet– Butwhat’stheanalogofopen?– Whatisthenamespace?– Howaretheyconnectedin)me?
write(wfd,wbuf,wlen);
n=read(rfd,rbuf,rmax);
RequestResponseProtocol
write(rqfd,rqbuf,buflen);
n=read(rfd,rbuf,rmax);
Client (issues requests) Server (performs operations)
requests
responses
write(wfd,respbuf,len);
n=read(resfd,resbuf,resmax);
service requestwait
RequestResponseProtocol
write(rqfd,rqbuf,buflen);
n=read(rfd,rbuf,rmax);
Client (issues requests) Server (performs operations)
write(wfd,respbuf,len);
n=read(resfd,resbuf,resmax);
service requestwait
requests
responses
Client-ServerModels
• Fileservers,web,FTP,Databases,…• Manyclientsaccessingacommonserver
Server
Client 1
Client 2
Client n
***
Conclusion(I)• SystemCallInterfaceis“narrowwaist”betweenuserprogramsandkernel
• StreamingIO:modeledasastreamofbytes– MoststreamingI/Ofunc)onsstartwith“f”(like“fread”)– Databufferedautoma)callybyC-libraryfunc)ons
• Low-levelI/O:– Filedescriptorsareintegers– Low-levelI/Osupporteddirectlyatsystemcalllevel
• STDIN/STDOUTenablecomposi)oninUnix– UseofpipesymbolsconnectsSTDOUTandSTDIN
• find|grep|wc…