Unix Programming Environment Part 6-2 – Standard I/O Library, File and Directory

Unix Programming Unix Programming EnvironmentEnvironment

Part 6-2 – Standard I/O Library, File and Directory

Prepared by Xu Zhenya( [email protected] )

Draft – Xu Zhenya( 2002/10/01 )

mailto:[email protected]







Unix Programming EnvironmentUnix Programming Environment Dept. of CSE, BUAADept. of CSE, BUAA

AgendaAgenda 1. Overview1. Overview

2. Standard I/O Routines 2. Standard I/O Routines

Chapter 6Chapter 6

3. Files and Directories3. Files and Directories

Chapter 7( 1, 2, 3 )Chapter 7( 1, 2, 3 )


Overview (1)Overview (1)

O.S. Kernnel

Standard I/O Lib

Async I/OSystem Callsread, write, open, close

Memory-mapping files

pread/pwrite


Overview (2)Overview (2) 1. POSIX I/O: 1. POSIX I/O:

ssize_t pread( int fd, void * buf, size_t nbytes, off_t offset );ssize_t pread( int fd, void * buf, size_t nbytes, off_t offset ); ssize_t pwrite( int fd, const void * buf, size_t nbytes, off_t offset );ssize_t pwrite( int fd, const void * buf, size_t nbytes, off_t offset );

2. scatter /gather I/O:2. scatter /gather I/O: ssize_t readv / writev ( int fd, const struct iovec *iov, int iovcnt );ssize_t readv / writev ( int fd, const struct iovec *iov, int iovcnt );

struct iovec { caddr_t iov_base; int iov_len; }struct iovec { caddr_t iov_base; int iov_len; }

3. Nonblocking I/O3. Nonblocking I/O O_NONBLOCK & O_NDELAYO_NONBLOCK & O_NDELAY

4. I/O Multiplexing4. I/O Multiplexing select() & poll()select() & poll()

5. Async I/O5. Async I/O Performance: Commercial RDMS, concurrency modelPerformance: Commercial RDMS, concurrency model

SIGIOSIGIO

Implementation: user-level, kernel-levelImplementation: user-level, kernel-level

6. Memory-mapping file I/O6. Memory-mapping file I/O void *mmap(void *addr, size_t len, in prot, int flags, int fd, off_t void *mmap(void *addr, size_t len, in prot, int flags, int fd, off_t

off );off );

7. 64-bit files: off_t => offset_t, llseek7. 64-bit files: off_t => offset_t, llseek


Standard I/O (1)Standard I/O (1)

1. History1. History

The standard I/O library was rewritten by Dennis Ritchie around 1975. The standard I/O library was rewritten by Dennis Ritchie around 1975. Surprisingly, little has changed in the standard I/O library after more Surprisingly, little has changed in the standard I/O library after more than 27 years. than 27 years.

2. Streams and 2. Streams and FILEFILE objects objects

Linux: /usr/include/libio.h – struct _IO_FILE;Linux: /usr/include/libio.h – struct _IO_FILE;

On some UNIX systems, < 256 /1024On some UNIX systems, < 256 /1024

STDIN_FILENO, STDOUT_FILENO, STDERR_FILENOSTDIN_FILENO, STDOUT_FILENO, STDERR_FILENO

3. Buffering3. Buffering

Fully buffered: Fully buffered:

files that reside on disksfiles that reside on disks

fflush()fflush(): forces a write of all user-space buffered data for the : forces a write of all user-space buffered data for the given output or update streamgiven output or update stream


Standard I/O (2)Standard I/O (2) Line buffered: Line buffered:

perform I/O when a new line charater is encountered on input or perform I/O when a new line charater is encountered on input or output. output.

the stream refers to a terminal. the stream refers to a terminal.

Unbuffered:Unbuffered:

The standard I/O library does not buffer the characters. The standard I/O library does not buffer the characters.

stderrstderr

Notes:Notes:

Default setting: Default setting: Standard error is always unbuffered. Standard error is always unbuffered.

All other streams are line buffered if they refer to a terminal device; otherwise All other streams are line buffered if they refer to a terminal device; otherwise they are fully buffered. they are fully buffered.

The size of line buffers are 128bytes, and the size of full buffers are 1K bytes.The size of line buffers are 128bytes, and the size of full buffers are 1K bytes.

setbuf(), setvbuf()setbuf(), setvbuf() make sure that the buffer space still exists…make sure that the buffer space still exists…


Standard I/O (3)Standard I/O (3)

4. Reading the textbook: 4. Reading the textbook:

P131: table6-3P131: table6-3

P141: table6-4P141: table6-4

5. Misc.5. Misc.

FILE * freopen(const char *pathname, const char *type, FILE * freopen(const char *pathname, const char *type, FILE *fp );FILE *fp );

Typically used to open a specified file as one of the predefined Typically used to open a specified file as one of the predefined streams: standard input, output or error.streams: standard input, output or error.

FILE * fdopen( int fd, const char * type );FILE * fdopen( int fd, const char * type );

Often used with descriptors that are returned by the functions that Often used with descriptors that are returned by the functions that create pipes and network communication channels.create pipes and network communication channels.


Standard I/O (4)Standard I/O (4) int fileno( FILE * stream );int fileno( FILE * stream );

ferror(), feof(), clearerr()ferror(), feof(), clearerr()

Positioning a stream: Positioning a stream:

ftell(), fseek(), rewind()ftell(), fseek(), rewind()

fgetpos(), fsetpos()fgetpos(), fsetpos()

Formatted output:Formatted output:

vprintf, vfprintf, vsprintf,vprintf, vfprintf, vsprintf,

va_list; man vprintf; man 3 stdargva_list; man vprintf; man 3 stdarg

Binary I/OBinary I/O

size_t fread( void *p_buf, size_t size, size_t nobj, FILE size_t fread( void *p_buf, size_t size, size_t nobj, FILE *fp );*fp );

Read and write a C structure? Read and write a C structure? Compilers: the binary layout of a structure: alignment, (Compilers: the binary layout of a structure: alignment, (#pragma pack(1)#pragma pack(1)))

CPUs’ architecture: Floating-point valuesCPUs’ architecture: Floating-point values

To exchange binary data : Network protocolsTo exchange binary data : Network protocols


Standard I/O (5)Standard I/O (5) 6. Temporary files6. Temporary files

char * tmpnam( char * ptr );char * tmpnam( char * ptr );

If If ptrptr is NULL, then return a pointer to a static area. => copy is NULL, then return a pointer to a static area. => copy

If ptr is not NULL, then lengthof( ptr ) >= L_tmpnam ( <stdio.h> )If ptr is not NULL, then lengthof( ptr ) >= L_tmpnam ( <stdio.h> )

char *tempnam(const char *directory, const char *prefix );char *tempnam(const char *directory, const char *prefix );

to specify both the directory and a prefix of the generated pathname.to specify both the directory and a prefix of the generated pathname.

FILE * tmpfile( void );FILE * tmpfile( void );

A temporary file( wb+ ) that is automatically removed when it is close or on A temporary file( wb+ ) that is automatically removed when it is close or on program termination.program termination.

7. Cautions for Using Standard I/O7. Cautions for Using Standard I/O

Don’t mix standard I/O and system-level I/O. Don’t mix standard I/O and system-level I/O.

If we write to the file at system-level, but then read at the standard I/O level If we write to the file at system-level, but then read at the standard I/O level => we might lose the changes.=> we might lose the changes.


File and Directory (1)File and Directory (1)

ProcessEnvironment

f_vnodef_offsetf_credf_count

v_flagsv_typev_countv_data

uf_ofile uf_pofile uf_refcnt



inode

rnode

The filestructure inthe kernel

vnode inthe kernel

Process: the file descriptor flags: CLOSE_ON_EXEC a pointer to a file table entryThe file structure in the kernel: the file status flags( read, write, append, sync, nonblocking, etc ) the current file offset, reference count and a pointer to the v-node table entry for this filev-node structure:

On Solaris, we can use “crash” to trace all data structures in the diagram.

File Descriptor 0


inode->i_op => ramfs.c::ramfs_get_inodeOO ::Object Model


File and Directory (2)File and Directory (2)


File Operations (1)File Operations (1)

1. open & create1. open & create

open, createopen, create

2. read & write, synchronize the file to disk, change the 2. read & write, synchronize the file to disk, change the offset location, change the size of a fileoffset location, change the size of a file

read, writeread, write

sync, fsyncsync, fsync

lseek, llseeklseek, llseek

ftruncate, truncateftruncate, truncate

3. close, delete3. close, delete

close, unlink, removeclose, unlink, remove


File Operations (2)File Operations (2)

4. check accessiblility, files’ attributes4. check accessiblility, files’ attributes

stat, fstat, lstatstat, fstat, lstat

accessaccess

umask, chmod, fchmod, chown, fchown, lchown, utimeumask, chmod, fchmod, chown, fchown, lchown, utime

5. fcntl, ioctl5. fcntl, ioctl

6. processes6. processes

chdir, fchdir, getcwd, chrootchdir, fchdir, getcwd, chroot

umask, dup, dup2umask, dup, dup2


open and creatopen and creat int open( const char *pathname, int flags, mode_t mode );int open( const char *pathname, int flags, mode_t mode );

O_RDONLY, O_WRONLY, O_RDWRO_RDONLY, O_WRONLY, O_RDWR

O_APPEND, O_TRUNCO_APPEND, O_TRUNC

O_CREAT, O_EXCLO_CREAT, O_EXCL

O_SYNC, O_SYNC,

O_NONBLOCK, O_NDELAYO_NONBLOCK, O_NDELAY

int creat( const char *pathname, mode_t mode );int creat( const char *pathname, mode_t mode );

open( pathname, O_WRONLY | O_CREAT | O_TRUNC, mode );open( pathname, O_WRONLY | O_CREAT | O_TRUNC, mode );

Temporary files: Temporary files:

open( pathname, O_RDWR | O_CREAT | O_TRUNC, mode );open( pathname, O_RDWR | O_CREAT | O_TRUNC, mode );


close, unlink, lseek, truncateclose, unlink, lseek, truncate close, unlink, removeclose, unlink, remove

DS in the kernel memory, links of an inode in the file systemDS in the kernel memory, links of an inode in the file system

When a process terminates, all open files are automatically closed by the kernel.When a process terminates, all open files are automatically closed by the kernel.

unlink: links– unlink: links–

If the name was not the last link to a fileIf the name was not the last link to a file

If the name was the last link to a file and no process have the file openIf the name was the last link to a file and no process have the file open

If the name was the last link to a file and any process have the file openIf the name was the last link to a file and any process have the file open

removeremove deletes a name from the file system. It calls deletes a name from the file system. It calls unlinkunlink for files, and for files, and rmdirrmdir for directories. for directories.

lseeklseek

Symbol const: SEEK_SET(0), SEEK_CUR(1), SEEK_END(2)Symbol const: SEEK_SET(0), SEEK_CUR(1), SEEK_END(2)

Only recoding the current file offset within the kernel => NO I/O operations to take Only recoding the current file offset within the kernel => NO I/O operations to take place.place.

A hole in the fileA hole in the file

int truncate( const char *path, off_t length );int truncate( const char *path, off_t length );

int ftruncate( int fd, off_t length );int ftruncate( int fd, off_t length );


read and writeread and write

readread

The number of bytes actually read is less than the requested The number of bytes actually read is less than the requested amount: amount:

A regular file: the end of fileA regular file: the end of file

A terminal device: line A terminal device: line

From a network endpoint: From a network endpoint:

Record-oriented devices like a magnetic tapeRecord-oriented devices like a magnetic tape

writewrite

The return value is usually equal to the nbytes argument, The return value is usually equal to the nbytes argument, otherwise an error has occurred: exceeding the size limit, etcotherwise an error has occurred: exceeding the size limit, etc

I/O EfficiencyI/O Efficiency

Textbook, p148Textbook, p148


fcntlfcntl int fcntl( int fd, int cmd, … /* int arg */ );int fcntl( int fd, int cmd, … /* int arg */ );

Change the properties of a file that is already opened. Change the properties of a file that is already opened.

Example: Example:

flag = fcntl( fd, F_GETFL, 0 );flag = fcntl( fd, F_GETFL, 0 );

fcntl( fd, F_SETFL, flag | O_NONBLOCK ); fcntl( fd, F_SETFL, flag | O_NONBLOCK );

The fcntl function is used for five different purposes: The fcntl function is used for five different purposes:

Duplicate an existing descriptorDuplicate an existing descriptor

Get/set file descriptor flagsGet/set file descriptor flags

Get/set file status flagsGet/set file status flags

Get/set asynchronous I/O ownershipGet/set asynchronous I/O ownership

Get/set record locksGet/set record locks


ioctlioctl

int ioctl( int fd, int request, … );int ioctl( int fd, int request, … );

manipulates the underlying device parameters of special files: manipulates the underlying device parameters of special files: device drivers, etcdevice drivers, etc

Terminal I/O => POSIX.1 standardTerminal I/O => POSIX.1 standard


stat, fstat and lstatstat, fstat and lstat int stat( const char *file_name, struct stat *buf );int stat( const char *file_name, struct stat *buf ); int fstat( int filedes, struct stat *buf );int fstat( int filedes, struct stat *buf ); int lstat( const char *file_name, struct stat *buf );int lstat( const char *file_name, struct stat *buf );

struct stat {struct stat {

1.1. dev_t st_dev; /* device */dev_t st_dev; /* device */

2.2. ino_t st_ino; /* inode */ino_t st_ino; /* inode */

3.3. mode_t st_mode; /* protection */mode_t st_mode; /* protection */

4.4. nlink_t st_nlink; /* number of hard links */nlink_t st_nlink; /* number of hard links */

5.5. uid_t st_uid; /* user ID of owner */uid_t st_uid; /* user ID of owner */

6.6. gid_t st_gid; /* group ID of owner */gid_t st_gid; /* group ID of owner */

7.7. dev_t st_rdev; /* device type (if inode device) dev_t st_rdev; /* device type (if inode device) */*/

8.8. off_t st_size; /* total size, in bytes */off_t st_size; /* total size, in bytes */

9.9. unsigned long st_blksize; /* blocksize for filesystem I/O unsigned long st_blksize; /* blocksize for filesystem I/O */*/

10.10. unsigned long st_blocks; /* number of blocks allocated */unsigned long st_blocks; /* number of blocks allocated */

11.11. time_t st_atime; /* time of last access */time_t st_atime; /* time of last access */

12.12. time_t st_mtime; /* time of last modification */time_t st_mtime; /* time of last modification */

13.13. time_t st_ctime; /* time of last change */time_t st_ctime; /* time of last change */

};};

Textbook: p156, example: checkmail.cTextbook: p156, example: checkmail.c


access, umask and chmodaccess, umask and chmod

int access( const char *pathname, int mode);int access( const char *pathname, int mode);

checks whether the process would be allowed to read, write or test checks whether the process would be allowed to read, write or test for existence of the file (or other file system object). for existence of the file (or other file system object).

a symbolic link: the file referred to by this symbolic linka symbolic link: the file referred to by this symbolic link

R_OK, W_OK, X_OK, F_OKR_OK, W_OK, X_OK, F_OK

umask, chmod, fchmod, chown, fchown, lchown, umask, chmod, fchmod, chown, fchown, lchown, utimeutime


File SharingFile Sharing

In one processIn one process

int dup( int oldfd );int dup( int oldfd );

int dup2( int oldfd, int newfd );int dup2( int oldfd, int newfd );

Textbook, p163, system()Textbook, p163, system()

I/O redirection, some Servers like WWWI/O redirection, some Servers like WWW

Among processesAmong processes

O_CREAT & O_EXCLO_CREAT & O_EXCL

Atomic Operations: Atomic Operations:

lseek and write => pread & pwritelseek and write => pread & pwrite

the offset of a open file in the kernel the offset of a open file in the kernel


DirectoryDirectory 1. directory format ( textbook, p153 )1. directory format ( textbook, p153 )

struct direntstruct dirent

{{

#ifndef __USE_FILE_OFFSET64#ifndef __USE_FILE_OFFSET64

__ino_t d_ino;__ino_t d_ino;

__off_t d_off;__off_t d_off;

#else#else

__ino64_t d_ino;__ino64_t d_ino;

__off64_t d_off;__off64_t d_off;

#endif#endif

unsigned short int d_reclen;unsigned short int d_reclen;

unsigned char d_type;unsigned char d_type;

char d_name[256]; /* We must not include limits.h! */char d_name[256]; /* We must not include limits.h! */

};};

2. accessing a directory2. accessing a directory

DIR *opendir(const char *name);DIR *opendir(const char *name); struct dirent *readdir(DIR *dir);struct dirent *readdir(DIR *dir); void rewinddir(DIR *dir);void rewinddir(DIR *dir); int closedir(DIR *dir);int closedir(DIR *dir); exec(): close all directory streamexec(): close all directory stream

3. creating and deleting, renaming a directory3. creating and deleting, renaming a directory mkdir, rmdir, renamemkdir, rmdir, rename


Symbol LinkSymbol Link

readlink, symlinkreadlink, symlink


mmap() – (1)mmap() – (1) void * mmap(void *start, size_t void * mmap(void *start, size_t

length, int prot , int flags, int length, int prot , int flags, int fd, off_t offset);fd, off_t offset);

1. Map a file or a POSIX shared 1. Map a file or a POSIX shared memory object into the calling memory object into the calling process’s address space. The process’s address space. The underlying objects may be:underlying objects may be:

a regular filea regular file

a special device(/dev/zero): a special device(/dev/zero): anonymous mappinganonymous mapping

shm_open(): unrelated process shm_open(): unrelated process ( POSIX )( POSIX )

2. function list: 2. function list:

mmap, munmap, msync, mmap, munmap, msync, mprotectmprotect

3. Linux: shared library, 3. Linux: shared library, executable binary files, executable binary files, read/write operations.read/write operations.

Thespace

mappedto FD

ObjectPointed by

FD Offset &Length

Process’s AS

Thespace

mappedto FD

Process’s AS


1. vm_area, vm_area driver = filemap1. vm_area, vm_area driver = filemap

2. vm_area is linked to the inode, its page too. 2. vm_area is linked to the inode, its page too.

3. when page fault during accessing the memory, the fault handler will call the driver, which should invoke 3. when page fault during accessing the memory, the fault handler will call the driver, which should invoke the inode’ mmap() to load from the file.the inode’ mmap() to load from the file.


mmap() – (2)mmap() – (2)

Notes for mmap():Notes for mmap():

1. flag == MAP_PRIVATE: don’t change the underlying object; 1. flag == MAP_PRIVATE: don’t change the underlying object; during the first writing, the kernel would dup a private copy. It during the first writing, the kernel would dup a private copy. It imply that we can see the change before the first time writing. imply that we can see the change before the first time writing.

2. flag == MAP_SHARED: the underlying object will be changed. 2. flag == MAP_SHARED: the underlying object will be changed.

3. the mapped length & the size of the underlying object3. the mapped length & the size of the underlying object

PAGESIZE: sysconf( _SC_PAGESIZE );PAGESIZE: sysconf( _SC_PAGESIZE );

SIGBUS: out of the length of the underlying objectSIGBUS: out of the length of the underlying object

SIGSEGV: out of the mapped segmentSIGSEGV: out of the mapped segment

Use truncate & ftruncate frist to resize the underlying objectUse truncate & ftruncate frist to resize the underlying object

4. Fork(): if flag == MAP_SHARED, the child process inherits the 4. Fork(): if flag == MAP_SHARED, the child process inherits the mapped memory. mapped memory.


mmap() – (3)mmap() – (3) 5. how to use mmap()5. how to use mmap()

Directly access the memory, not using read() & write() Directly access the memory, not using read() & write()

!!!! But the system calls read() & write are atomic operations!!!! But the system calls read() & write are atomic operations

Not all FD can be mapped into memory. Like a terminal or socketNot all FD can be mapped into memory. Like a terminal or socket

Implement the shared memory among unrelated processesImplement the shared memory among unrelated processes

The underlying object provides the initial values for the mapped memory. The underlying object provides the initial values for the mapped memory.

Any change will be write back to the underlying object.Any change will be write back to the underlying object.

6. how to use anonymous mapping? 6. how to use anonymous mapping?

On SYSV, we can use /dev/zero to map: ZFOD( zero-fill-on-demand)On SYSV, we can use /dev/zero to map: ZFOD( zero-fill-on-demand)

After loading the program, BSS, heap and stack will use anonymous After loading the program, BSS, heap and stack will use anonymous mapping. mapping.

In the applications, a In the applications, a parent and children parent and children processes can use processes can use anonymous mapping to share memory without creating or opening a anonymous mapping to share memory without creating or opening a real file. real file.

7. truncate & ftruncate to resize the underlying object7. truncate & ftruncate to resize the underlying object

Documents

Unix Programming Environment Part 6-2 – Standard I/O Library, File and Directory