File Management

Slide 13-1

Copyright © 2004 Pearson Education, Inc. Operating Systems: A Modern Perspective, Chapter 13

Slide 13-2


13FileManagement

Slide 13-3


Fig 13-2: The External View of the File Manager

Hardware

ApplicationProgram

ApplicationProgram

Fil

e M

gr

Dev

ice

Mgr

Mem

ory

Mgr

Pro

cess

Mgr

UNIXF

ile

Mgr

Dev

ice

Mgr

Mem

ory

Mgr

Pro

cess

Mgr

Windows

open()read()

close()

write()

lseek()

CreateFile()ReadFile()CloseHandle()

SetFilePointer()

WriteFile()mount()

Slide 13-4


• Persistent storage• Shared device

Why Programmers Need Files

HTMLEditor

HTMLEditor

<head>…</head><body>…</body>

WebBrowser

WebBrowser

• Structured information• Can be read by any applic

• Accessibility• Protocol



foo.html

FileManager

FileManager

FileManager

FileManager

Slide 13-5


File Management• File is a named, ordered collection of

information• The file manager administers the collection

by:– Storing the information on a device– Mapping the block storage to a logical view– Allocating/deallocating storage– Providing file directories

• What abstraction should be presented to programmer?

Slide 13-6


Information Structure

Records

Applications

Structured Record Files

Record-Stream Translation

Stream-Block Translation

Byte Stream Files

Storage device

Slide 13-7


Byte Stream File Interface

fileID = open(fileName)close(fileID)read(fileID, buffer, length)write(fileID, buffer, length)seek(fileID, filePosition)

Slide 13-8


Low Level Files

Stream-Block Translation

b0 b1 b2 bi......

fid = open(“fileName”,…);…read(fid, buf, buflen);…close(fid);

int open(…) {…}int close(…) {…}int read(…) {…}int write(…) {…}int seek(…) {…}

Storage device response to commands

Slide 13-9


Structured Files

Records

Record-Block Translation

Slide 13-10


Record-Oriented Sequential Files

Logical Record

fileID = open(fileName)close(fileID)getRecord(fileID, record)putRecord(fileID, record)seek(fileID, position)

Slide 13-11



...H byte header k byte logical record

Logical Record

Slide 13-12



...H byte header k byte logical record

...

FragmentPhysical Storage Blocks

Logical Record

Slide 13-13


Electronic Mail Examplestruct message {/* The mail message */ address to; address from; line subject; address cc; string body;};struct message *getRecord(void) { struct message *msg; msg = allocate(sizeof(message)); msg->to = getAddress(...); msg->from = getAddress(...); msg->cc = getAddress(...); msg->subject = getLine(); msg->body = getString(); return(msg);}

putRecord(struct message *msg) { putAddress(msg->to); putAddress(msg->from); putAddress(msg->cc); putLine(msg->subject); putString(msg->body);}

Slide 13-14


Indexed Sequential File

• Suppose we want to directly access records

• Add an index to the file

fileID = open(fileName)close(fileID)getRecord(fileID, index)index = putRecord(fileID, record)deleteRecord(fileID, index)

Slide 13-15


Indexed Sequential File (cont)

Account #012345123456294376...529366...965987

Index

ik

j

index = i

index = k

index = j

Application structure

Slide 13-16


More Abstract Files

• Inverted files– System index for each datum in the file

• Databases– More elaborate indexing mechanism– DDL & DML

• Multimedia storage– Records contain radically different types– Access methods must be general

Slide 13-17


Implementing Low Level Files

• Secondary storage device contains:– Volume directory (sometimes a root directory

for a file system)– External file descriptor for each file– The file contents

• Manages blocks– Assigns blocks to files (descriptor keeps track)– Keeps track of available blocks

• Maps to/from byte stream

Slide 13-18


Disk Organization

Blk0Blk0 Blk1

Blk1 Blkk-1Blkk-1

BlkkBlkk Blkk+1

Blkk+1 Blk2k-1Blk2k-1

Track 0, Cylinder 0

Track 0, Cylinder 1

BlkBlk BlkBlk BlkBlk Track 1, Cylinder 0

BlkBlk BlkBlk BlkBlk Track N-1, Cylinder 0

BlkBlk BlkBlk BlkBlk Track N-1, Cylinder M-1

…

…

…

…

…

…

…

…

Boot Sector Volume Directory

Slide 13-19


Low-level File System Architecture

b0 b1 b2 b3 bn-1 … …

Block 0

...

Sequential Device Randomly Accessed Device

Slide 13-20


File Descriptors•External name•Current state•Sharable•Owner•User•Locks•Protection settings•Length•Time of creation•Time of last modification•Time of last access•Reference count•Storage device details

Slide 13-21


An open() Operation

• Locate the on-device (external) file descriptor

• Extract info needed to read/write file• Authenticate that process can access the file • Create an internal file descriptor in primary

memory• Create an entry in a “per process” open file

status table• Allocate resources, e.g., buffers, to support

file usage

Slide 13-22


File Manager Data Structures

External File Descriptor

Open FileDescriptor

Copy info from external to the open file descriptor

1

Process-FileSession

Keep the state of the process-file session

2

Return a reference to the data structure

3

Slide 13-23


Opening a UNIX File

fid = open(“fileA”, flags);…read(fid, buffer, len);

0 stdin1 stdout2 stderr3 ...

Open File Table

File structure

inode

Internal File Descriptor

On-Device File Descriptor

Slide 13-24


Block Management

• The job of selecting & assigning storage blocks to the file

• For a fixed sized file of k blocks– File of length m requires N = m/k blocks

– Byte bi is stored in block i/k

• Three basic strategies:– Contiguous allocation– Linked lists– Indexed allocation

Slide 13-25


Contiguous Allocation

• Maps the N blocks into N contiguous blocks on the secondary storage device

• Difficult to support dynamic file sizes

Head position 237…First block 785Number of blocks 25

File descriptor

Slide 13-26


Linked Lists• Each block contains a header with

– Number of bytes in the block– Pointer to next block

• Blocks need not be contiguous

• Files can expand and contract

• Seeks can be slowFirst block…

Head: 417...

Length

Byte 0

Byte 4095...

Length

Byte 0

Byte 4095...

Length

Byte 0

Byte 4095...

Block 0 Block 1 Block N-1

Slide 13-27


Indexed Allocation

• Extract headers and put them in an index

• Simplify seeks

• May link indices together (for large files)

Index block…

Head: 417...

Byte 0

Byte 4095...

Byte 0

Byte 4095...

Byte 0

Byte 4095...

Block 0

Block 1

Block N-1

Length

Length

Length

Slide 13-28


DOS FAT Files

DiskBlock

File Descriptor

DiskBlock

DiskBlock

…43

107254

File Access Table (FAT)

DiskBlock

DiskBlock

DiskBlock

…43

107

10743

254

254

File Descriptor

Slide 13-29


UNIX FilesDatamode

owner…Direct block 0Direct block 1…Direct block 11Single indirectDouble indirectTriple indirect

inode

Data

Data

Index

Data

DataIndexIndex

IndexData

Data

Index

Index

IndexIndex

Index

Data

Data

Slide 13-30


Unallocated Blocks

• How should unallocated blocks be managed?

• Need a data structure to keep track of them– Linked list

• Very large

• Hard to manage spatial locality

– Block status map (“disk map”)• Bit per block

• Easy to identify nearby free blocks

• Useful for disk recovery

Slide 13-31


Marshalling the Byte Stream

• Must read at least one buffer ahead on input

• Must write at least one buffer behind on output

• Seek flushing the current buffer and finding the correct one to load into memory

• Inserting/deleting bytes in the interior of the stream

Slide 13-32


Full Block Buffering

• Storage devices use block I/O• Files place an explicit order on the bytes• Therefore, it is possible to predict what is likely to

be read after bytei

• When file is opened, manager reads as many blocks ahead as feasible

• After a block is logically written, it is queued for writing behind, whenever the disk is available

• Buffer pool – usually variably sized, depending on virtual memory needs– Interaction with the device manager and memory

manager

Slide 13-33


Directories

• A set of logically associated files and sub directories

• File manager provides set of controls:– enumerate– copy– rename– delete– traverse– etc.

Slide 13-34


Directory Structures

• How should files be organized within directory?– Flat name space

• All files appear in a single directory

– Hierarchical name space• Directory contains files and subdirectories

• Each file/directory appears as an entry in exactly one other directory -- a tree

• Popular variant: All directories form a tree, but a file can have multiple parents.

Slide 13-35


Directory Implementation

• Device Directory– A device can contain a collection of files– Easier to manage if there is a root for every file

on the device -- the device root directory

• File Directory– Typical implementations have directories

implemented as a file with a special format– Entries in a file directory are handles for other

files (which can be files or subdirectories)

Slide 13-36


UNIX mount Command

/

bin usr etc foo

bill nutt

abc

/

blah

cde xyz

FS

/

bin usr etc foo

bill nutt

abc

/

blah

cde xyz

mount FS at foo

FS

Slide 13-37


VFS-based File Manager

File System IndependentPart of File Manager

File System IndependentPart of File Manager

Exports OS-specific API

Virtual File System SwitchVirtual File System Switch

MS-DOS Part ofFile Manager

MS-DOS Part ofFile Manager

ISO 9660 Part ofFile Manager

ISO 9660 Part ofFile Manager

ext2 Part ofFile Manager

ext2 Part ofFile Manager

…

Rensselaer Polytechnic InstituteCSCI-4210 – Operating SystemsDavid Goldschmidt, Ph.D.

very small

very large

very fast

very slow

volatile

non-volatile

managed aspart of thefilesystem

A File Management System is a set of OS services that supports files and directories for user applications, OS programs, etc. Data should be organized and

available in a convenient andefficient manner

Files are the basicbuilding blocks

File attributes include: Human-readable symbolic name Type (e.g. executable, directory, text, PDF, etc.) Logical location (i.e. containing directory or

path) Physical location on disk Size (in bytes) Protection or security (i.e. permissions) Timestamps (created, last modified, last

accessed)

The pathname (or just path) of a file specifies the sequence of directoriesone must traverse to locatethe file An absolute path starts

at the root node A relative path starts

anywhere

A link provides a shortcut to a file and may circumvent the given directory hierarchy A hard link in Unix is

indistinguishable fromthe original file

A symbolic link in Unixis merely a shortcut

A Windows shortcut is just a symbolic link

File creation requires space allocation Opening a file returns a handle or file

descriptor Read and write operations

use the handle and anoffset (or file pointer)

The close operation simplydeletes the handle anddeallocates any memory

Deleting a file deallocates all disk spacemarked as in use by the file But likely does not erase file contents Deleted files are recoverable until the disk

space is used for (and overwritten by) other file(s)

The delete operation also removes the corresponding entry in the containing directory

In a contiguous disk space allocation scheme, files are allocated to contiguous blocks of disk space

A A A B B C C C D

A A A C C C D B B B B

A A A C C C B B B B D D

A A A C C C B B B B D D

In a clustered disk space allocation scheme, files are allocated to clusters of disk space on an as needed basis

A A A B B C C C D

A A A B B C C C D B B

A A A B B C C C D B B D

File accesses may cross cluster boundaries,causing slower response times

A file allocation table (FAT) maps logical filesto their physical addresses on a disk A pure FAT would have an entry

for each sector (e.g. 512 bytes) To improve performance, clusters

of contiguous sectors are used Clusters range from 4 to 64 sectors

(e.g. 2,048 to 32,768 bytes)

Microsoft FAT-32 filesystem maps to clusters May require linked clusters to house

entire FAT

Linux uses indexed clusters

In Unix, file information is stored in an index node (I-node), which contains: Mode and type of file Number of links to the file File owner’s userid and groupid Size of file (in bytes) Last access and last modified times Number of blocks allocated to the file Pointers to the first twelve blocks Pointers to three additional blocks of

pointers

inode12

1024

10242

10243

clustercluster

cluster

…

cluster

cluster

cluster

inode1024

single indirect

…

inode1024

double indirect

inode1024

inode1024

… …

…

inode10242

triple indirect

…

A virtual filesystem provides transparent access to different filesystem types on multiple device types and disk partitions

A virtual machine is a layered approach that logically combines the kernel operating system and hardware Creates the illusion

of multiple processes,each executing on itsown virtual processorwith its own virtualmemory

http://www.computerworld.com/action/article.do?command=viewArticleBasic&articleId=9002552

Non-virtual machine Virtual machine

Java programs execute on a native Java Virtual Machine (JVM)

Virtual machines provide complete protection of system resources Each virtual machine is isolated from

all other virtual machines▪ which prohibits direct sharing

of system resources Virtual machines can be difficult to

implementdue to the effort required to providean exact duplicate of each underlying machine

Documents

File Management