TDC368 UNIX and Network Programming

TDC368UNIX and Network Programming

Camelia Zlatea, PhD

Email: [email protected]

Week 3-4: UNIX File System

– File Representation

– File Handles

– System Calls (APIs)

System Calls: dup dup2 pipe Library functions: popen(); pclose();

mailto:[email protected]







UNIX Network Programming – TDC368-901 Page 2Spring 2003

Outline

UNIX File System: files, catalogs (inodes; types,size, permissions).

UNIX File I/O: unbuffered I/O functions (open ,close,seek, umask,read,write).

Standard I/O buffering (fopen, fclose, fflush).

File Sharing

Communication between related processes via pipe (pipe, dup,dup2)

Appendix:

UNIX system calls: C library functions, C++ classes, POSIX API's.


UNIX File System

Hierarchical organization Device independence

– compatibility among files or catalogs, devices and IPC

Types of files– regular files (byte-stream, no format)

– directory files (groups of files mapped on user’s projects)

– special files (to access peripheral devices, /dev/...)

– pipe files (inter process communication mechanisms)


UNIX Type of Files

Regular Files– Shell commands: cp, rm, mv, cat, ls

– System Calls: <fcntl.h>, <unistd.h>

open, creat, read, write, close, link, unkink

Directory Files– Shell commands: mkdir, rmdir, mv

– System Calls: <dirent.h>

Special Files– mknod ( root privileges)

Pipe Files– mkfifo


File Attributes

File Type (-,d, b,c,p)

ls -l Access Permission for owner, group and others (r,w,x) File Owner ID and GID Number of hard links of a file File Size Last access, modify and change permission time INODE number File system ID


File Attributes

Constant Attributes– file type

– file inode number

– File System ID

Changeable Attributes– chmod (command and system call)

– chown, chgrp

– touch (command), utime (sys call)

– ln (command ), link(syscall)

– rm (command), unlink(syscall)

Attributes query by:– system calls stat or fstat


File Permissions

all UNIX files have associated a set of owner permissions bits, that are used by the OS to determine the access

classes of users: owner, group, other permissions: owner group other

rwx rwx rwx

111 111 111 octal 777

111 101 101 octal 755

- default permissions = creation_mask XOR umask_value

- creation mask (set at system initialization) is usually 0777


File Permissions

Command shell:

umask new_mask_value (display/change umask value)

System call:

mode_t umask(mode_t mask);

- to modify umask for all your processes edit your startup file , .login (.profile) and insert the line umask nnn (where nnn is the new umask value)

- change permissions

- command shell:

chmod guorwx file


File Permissions

System calls: <sys/types.h>

<sys/stat.h>

int chmod(const char *path, mode_t mode);

int fchmod(int fid, mode_t mode);

Example: Use system call stat and determine the inode information

Name of the file is passed as program’s argument.


File Permissions

#include <sys/types.h>#include <sys/stat.h>

void main(int argc, char * argv[])struct stat statbuf;

if (stat(argv[1], &statbuf)==-1) {/* print error message */exit(1); }

else {if (statbuf.st_mode & S_IFDIR) {

printf(“ %s is a directory \n”, argv[1]);printf(“user ID is %s \n”, stat_buf.st_uid);

}

}


File Owner Information

real user ID (UID) real group ID (GID)

- information is obtained from the password file entry (/etc/passwd)

group ID (GID) numbers should map to group names from /etc/group effective user ID (EUID) effective group ID (EGID)

- usually UID identical with EUID



Example: command passwd (change password) need to have access to a

system file (as a root) a non-privileged user is able to run a command temporarily as

root if suid permission is set: rws r-x r-x root /etc/passwd



Command shell id is used to display owner information (UID, GUID, EUID, EGID)

System Calls:

<sys/types.h>

<unistd.h>

uid_t getuid(void);

uid_t geteuid(void);

gid_t getgid(void);

gid_t getegid(void);


File Representation

UNIX File System Organization: BOOT BLOCKSUPERBLOCKINODE TABLEDATA BLOCK

Boot Block– contains a program that loads UNIX kernel file from the file system– transfer control to the start address of the kernel code– if the system is already running and the disk is accessed by mount, boot block is

ignored

SUPERBLOCK- contains all information that defines the current state of the file system- size of i-list Iinode list table)- size in blocks of the files system- number of free blocks available in the file system- the address of the free inodes list- number of total free blocks in the system- files system name and file system disk name- superblock is read when mounting the device


File Representation

INODE - identifies a UNIX file Inode Structure

– file attributes

– physical disk address where file date is located

Inode Table (UNIX System V)– an inode entry for each file in the file system

Directory File - maps file name to inodes numbers


File Representation

Inode Table inode structure

File Information- size (bytes)- owner information (UID, GID, EUID, EGID)- relevant times- number of links- number of blocks- permissions

Data Blocks- direct pointers to beginning of file blocks- indirect pointers- double indirect pointers- triple indirect pointers

Data Block …………….

Data Block


Directory Structure

/root

/bin /etc unix /dev /usr /tmp

Absolute path name: /usr/project/bin/a.out

Relative path name: ./a.out

../../a.out

File name: NAME_MAX in <limits.h>

Path name: PATH_MAX -’’-

Hard Link - the path name of a file

ln /usr/myproj/a.out /usr/project/bin/a.out


Directory Representation

Directory Representation– UNIX directory - a file containing a correspondence between filenames

and inodes

– the inode does not contain file name

– data blocks of the directory file are directory entries

– a directory entry contains: inode number + file name

– command as rename/move (mv) affect the name filed from directory entry,

– only one physical copy of the file exists on disk (file may have several names on different directories or the same name)

– for a newly created directory (ex. mkdir dir_name) OS makes a new directory entry for it and assign an inode to it


Accessing Files

Example of Directory file:

Inode Number File Name

78 .

109 ..

115 sample

240 test.c

300 a.out

115 sample_data_l


Directory Files

Record-oriented files– file name

– inode number

Record data type is the structure dirent in SV/POSIX

<dirent.h>

direct in BSD

<sys/dir.h>


Directory Functions

Opens a directory file, return a handle of type DIR*– opendir

Reads next record from directory file– readdir

Closes a directory file– closedir

Sets file pointer at the beginning – rewinddir

Random directory access– telldir, seekdir


Hard vs. Symbolic Links

Hard Links: ln fileA fileB– pathnames of a file (directory entries)– NO new inode is created– cannot link files across file systems– cannot link directories (as reg.user)– increase the hardlink counter per inode– ln /dirA/name1 /dirB/name2

Effect: creates a hard link, both names point to the same inode structure)

Symbolic Links: ln -s fileA fileB– Symbolic link is a file containing the name of another file(directory)– a new inode is created– can link files across file systems– can link directories (as reg.user)– ln -s /dirB/name2 /dirA.name1– Effect: if there is an entry 12345 – name1 in /dirA

creates new inode 56789new entry 56789 – name2 in /dirBthe block of date from inode 56789 contains “/dirA/name1”


UNIX File System Calls (APIs)

Header file: <sys/types.h>

<fcntl.h> open - opens a file for data access

Header file: <sys/types.h>

<unistd.h>creat - creates regular file

read - reads data from a file

write - writes data to a file

close - terminate the access to a file


UNIX File System Calls (APIs)

Header files: <sys/types.h>

<unistd.h>

lseek - random access to a file

link - creates a hard link to a file

unlink - removes a hard link of a file

chown - changes UID or GID of a file

chmod - changes access perm. of a file

Header files: <sys/types.h>

<sys/stat.h>

stat, fstat - queries attributes of s file


File Handles

C stream pointers (FILE *) Conversions, header file <stdio.h>

int fileno(FILE *stream_pointer)

FILE *fdopen(inf fd, char *open_mode);

C Library Func. UNIX System Callfopen open

fread, fget, fgets, fscanf read

fwrite,fput,fputs,fprintf write

fseek, ftell, frewind lseek

fclose close


FILE handle

#include <stdio.h>

FILE *fp;

if (( fp = fopen(“test.dat”, “w”)== NULL)

fprintf(stderr, “cannot open file\n”);

else

fprintf(fp, “This is a text”);

“This is a text”);

Process

File Descriptor Table (FDT)

0 - stdin

1 - stdout

2 - stderr

3

4

5

6

System

File Descriptor Table (SDT)

0 - stdin

1 - stdout

2 - stderr

3

4

5

6

User area Kernel area

fp

3


Buffering

Minimize # of write/read calls Types of Buffering

– Fully / Line / No Buffering

Change Buffering Type#include <stdio.h>void setbuf(FILE *fp, char *buf)void setbuf(FILE *fp, char *buf, int mode, size_t size)mode: _IOFBF, _IOLBF, _IONBFsize: default is BUFSIZ

Flush a bufferint fflush(FILE *fp);


Parent Process opens file test.dat before fork

Parent FDT

0

1

2

3 FD test.dat

4

5

6

SFTChild FDT

0

1

2

3 FD test.dat

4

5

6


Parent Process opens file test.dat after fork

Parent FDT

0

1

2

3 FD test.dat

4

5

6

SFTChild FDT

0

1

2

3 FD test.dat

4

5

6


Pipe

Think at a pipe as being a special file that can store a limited amount of data in a FIFO manner

Size: 512 K on most Unix systems (<limits.h> or <sys/param.h>)

Interaction rule: one process writes to the pipe (as it were a file) and other process reads from the pipe

Data are written at one end of the pipe and are read at the other end

System keeps track of the current location of last read/write


Process synchronization using pipes

If a writing process attempts to write to a full pipe the system blocks it until pipe is able to receive data

If a process attempts to read from an empty pipe the system blocks it until data is available

A process will block if it tries to read from a pipe that has been opened for reading but no process has opened it for writing

If a write is made to a pipe not open for reading SIGPIPE is generated and errno is set to ``broken pipe''


write() System Call

Data are written to a pipe using unbuffered I/O write

Include files: <unistd.h> Calling pattern:

– size t write (int fileds, void *buf, size t nbytes);

– fileds is a file descriptor

Return values: – On success: #bytes written

– On failure: 1

– Set errno? Yes


Writing to a pipe

Each write request is appended to the end of the pipe Write requests of P IPE BUF size or less are guaranteed

not to be interleaved with other write requests to the same pipe

Write request may cause the process to block; the constants

O NONBLOCK, O NDELAY in !sys/fcntl.h? can be set by fcntl() to non blocking strategy

When O NONBLOCK, O NDELAY are set and the request to write P IPE BUF size bytes or less fails the value returned by write is shown in the next slide Table ==>


Setting the constants that control piping strategy

The table shows what write returns in function of constants Blocking constants for write()

If a write is made to a pipe not open for reading SIGPIPE is generated and errno is set to ``broken pipe''

O NONBLOCK O NDELAY Value returned

set clear -1

clear set 0

clear clear process blocks until space available

set set process does not block


read() system call

Data are read from a pipe using unbuffered I/O read Include files: <sys/types.h>,

<sys/uio.h>,

<unistd.h> Calling pattern:

– size t read (int fileds, void *buf, size t nbyte);

– fileds is a file descriptor;

Return values: – On success: #bytes read

– On failure: 1

– Set errno? Yes


Reading from a pipe

Read actions are initiated from the current position If pipe is open for writing by another process but it is empty read

returns as shown bellow Blocking constants for read

If the pipe is not opened for writing by another process read returns 0 (as in case of O NDELAY set)

O NONBLOCK O NDELAY Value returned

set clear -1

clear set 0

clear clear process blocks until data available

set set process does not block


Types of pipes

unnamed pipes, used with related processes (example: parent/child or child/child)

named pipes, used with unrelated process


Unnamed pipes ; pipe() system call

An unnamed pipe is constructed with pipe() system call, Include files <unistd.h> Calling pattern int pipe ( int filedes[2] ); Return values:

– On success: 0 – On failure: 1 – Set errno? Yes

Functionality: – pipe() operates on a full duplex or half duplex basis

Writing/reading from a pipe

filedes[1] ) filedes[0]

filedes[0] ) filedes[1] half duplexfull duplex


Actions performed by pipe()

If successful, it returns two file descriptors, filedes[0], filedes[1] that reference two data streams

Historically pipes were non duplex (i.e., unidirectional); if two way communication was needed two pipes were opened, one for reading and the other for writing

Currently file descriptors returned by pipe() are full duplex (i.e., bi directional), both opened for reading and writing.

In a full duplex if the process writes to filedes[0] then filedes[1] is used for reading, and vice versa

In a half duplex, filedes[0] is used for reading and filedes[1] for writing

an attempt to use filedes[1] for reading or filedes[0] for writing produces an error


Example: a parent process sends to a child the first argument from its command line and child acknowledges the

receipt /* Using a pipe to send data from parent to child */ #include <stdio.h> #include <unistd.h> #include <stdlib.h>#include <string.h> #include <errno.h>

int main (int argc, char *argv[]) {int fdes[2]; static char message[BUFSIZ]; if (argc != 2) {fprintf(stderr,''Usage: %s message``n'', *argv); exit(1); }if (pipe(fdes) == 1) { perror(''Pipe''); exit(2); }switch (fork()) {case 1: { perror(''Fork''); exit(3); }



receipt /* Child process: closes fdes[1] and read fdes[0] to obtain the message */

case 0:

{

close(fdes[1]);

if (read(fdes[0], message, BUFSIZ) != 1)

{

printf(''Message received by child is: [%s]``n'', message);

fflush(stdout);

}

else { perror(''Read''); exit(4); }

break;

}



receipt /* Parent closes fdes[0] and write message to fdes[1] */

default:

{

close(fdes[0]);

if (write(fdes[1], argv[1], strlen(argv[1])) != 1)

{

printf(''Message sent by parent is: [%s]``n'', argv[1]);

fflush(stdout);

}

else { perror(''Write''); exit(5); }

break;

}

}

exit(0);

}


Redirecting stdin and stdout

Examples:– cat > test.dat

cat“0”

“2”

“1” cat“0”

“2”

“1”

test.dat0 stdin

1 stdout

2 stderr

0 stdin

1 test.dat

2 stderr


Redirecting stdin and stdout; dup() system call

Create a pipe Associate stdin and stdout with the created pipe using

dup() system call

Include files <unistd.h> Calling pattern

– int dup ( int fdes );

Return values: – On success: next available file descriptor

– On failure: 1

– Set errno? Yes or No


dup() functionality

dup() duplicates an original open file descriptor; new descriptor references SFT (System File Table) entry

for the next available non negative file descriptor new descriptor shares the same file pointer (offset), has

the same access mode as original and remain open across exec.

since actions necessary for redirection are as in the code: – int fdes[2]; pipe(fdes); close(fileno(stdout));dup(fdes[1]);

– there is a chance that descriptor returned by dup() is not the one closed;


dup2() system call

Using dup2() system call for file redirection Include files <unistd.h> Calling pattern int dup2 ( int fdes1, int fdes2 ); Return values:

– On success: next available file descriptor >=0

– On failure: 1

– Set errno? Yes

dup2() functionality: – dup2() closes the file and duplicates the file descriptor atomically

– therefore dup2() avoid the limitation of dup()

both dup() and dup2() can be implemented with fcntl()


Redirecting stdin and stdout

Example: int fd;

mode_t fd_mode = S_IRUSR | S_IWUSR | S_IRGRP | S_IROTH;

fd = open(“test.dat”, O_WRONLY | O_CREAT, fd_mode);

dup2(fd, STDOUT_FILENO);

…..

close(fd);

0 stdin

1 stdout

2 stderr

3 Write to

test.dat

0 stdin

1 Write to

test.dat

2 stderr

Process FDT after open

0 stdin

1 Write to

test.dat

2 stderr

3 Write to

test.dat

Process FDT after dup2

Process FDT after close


Example: implementing ''last | sort '' command

/* A home grown last | sort command */ #include <stdio.h>#include <unistd.h> #include <stdlib.h>#include <errno.h> void main (void) {int fdes[2]; if (pipe(fdes) == 1) { perror(''Pipe''); exit(1); } switch (fork()) {case 1: { perror(''Fork''); exit(2); } case 0: { dup2(fdes[1], fileno(stdout));

close(fdes[0]); close(fdes[1]); execlp(''last'', ''last'', 0); exit(2);

}

default:

{

dup2(fdes[0],fileno(stdin));

close(fdes[0]);

close(fdes[1]);

execlp(''sort'', ''sort'', 0);

exit(4);

}

}

}


Per Process FDT and SFT before dup2()

Parent FDT

0 - stdin

1 - stdout

2 - stderr

3 fdes[0]

4 fdes[1]

5

6

SFTChild FDT

0 - stdin

1 - stdout

2 - stderr

3 fdes[0]

4 fdes[1]

5

6


Per Process FDT and SFT after dup2()

Parent FDT

0 – fdes[0]

1 – stdout

2 - stderr

3 fdes[0]

4 fdes[1]

5

6

SFTChild FDT

0 - stdin

1 – fdes[1]

2 - stderr

3 fdes[0]

4 fdes[1]

5

6


Per Process FDT and SFT after close()

Parent FDT

0 – fdes[0]

1 – stdout

2 - stderr

3

4

5

6

SFTChild FDT

0 - stdin

1 – fdes[1]

2 - stderr

3

4

5

6


Summary for communication via unnamed pipes

Create the pipe(s) needed Generate the child process(es) Close/duplicate file descriptors to properly associate the

ends of the pipe Close the unneeded ends of the pipe Perform the communication activities Close any remaining open file descriptors If appropriate, wait for child processes to terminate


popen() library function

popen() encapsulates the sequence of activities, Generate a pipe;

– Fork a child process – Duplicate file descriptors created by pipe; – Passes command execution information from one process to

another Include files <stdio.h> Calling pattern

– FILE *popen (char *command, char *type ); Return values:

– On success: ptr to a FILE – On failure: NULL pointer – Set errno? no


popen() functionality

create a child process; child process exec's a Bourne shell which will execute the

shell command passed to popen() Input to and output from the child process is

accomplished via pipe: – if mode type in popen is ``w'' parent process can write to the

standard input of the shell command run by the child and child process can read the data at its standard input;

– if mode type in popen is ``r'' parent process can read from the standard output of the shell command run by the child and child process can write the data at its standard output;


pclose() library function

pclose(): closes a data stream opened by popen(), Include files <stdio.h> Calling pattern int pclose (FILE *stream ); Return values:

– On success: Exit status of command

– On failure: 1

– Set errno? no


Example: using popen and pclose

/* Using popen and pclose */ #include <stdio.h>#include <unistd.h>#include <stdlib.h>#include <limits.h>int main (int argc, char *argv[]) {FILE *fin, *fout; char buffer[PIPE_BUF]; int n; if (argc ! 3) { fprintf(stderr, ''Usage: %s cmd1 cmd2\n'', argv[0]); exit(1); }fin = popen(argv[1], ''r''); fout = popen(argv[2], ''w''); while ((n = read(fileno(fin), buffer, PIPE_BUF)) > 0) write (fileno(fout), buffer, n); pclose (fin); pclose (fout); exit(0); }


Example: using popen and pclose

Observations: This program requires two command line arguments, i.e., two shell

commands whose standard output/input will be redirected via pipes generated by popen()

The first popen() call with option ``r'' directs the system to fork a child that will execute first shell command whose output will be redirected so it can be read by the parent using fin.

The second popen() call with option ``w'' directs the system to fork a new child that will execute the second shell command whose input will be the data written by parent using fout.

The while() loop is used to copy data from the output end of one pipe to the input end of the other pipe.

Documents

TDC368 UNIX and Network Programming