File System of Mac OS X

Akila sivapathasekaran

Poly ID: 0479990

CS 6233, Spring 2012

Polytechnic Institute of Newyork University

ContentsPart – I.....................................................................................................................................................4

1. Abstract......................................................................................................................................4

2. Introduction and history.............................................................................................................4

3. File System Concepts..................................................................................................................4

4. Apple Partitioning Scheme.........................................................................................................4

5. Disk Arbitration..........................................................................................................................5

6. File Descriptors..........................................................................................................................5

7. Disk Images................................................................................................................................5

8. Virtual File System Layer...........................................................................................................5

9. File System Types......................................................................................................................6

9.1. MS-DOS......................................................................................................................................6

9.2. New Technology File System.....................................................................................................6

9.3. Universal Disk Format...............................................................................................................6

9.4. Apple filing Protocol.................................................................................................................6

9.5. Hierarchical File System Plus....................................................................................................6

Part – II....................................................................................................................................................7

10. Hierarchical File System Plus Filing System..............................................................................7

10.1. Features...................................................................................................................................7

10.2. Structure of HFS+ file system...................................................................................................7

10.3. Reserved Areas........................................................................................................................8

10.4. Volume Header........................................................................................................................8

10.5. HFS Wrapper............................................................................................................................8

10.6. Special files..............................................................................................................................8

10.6.1. Allocation File..........................................................................................................................8

10.6.2. Catalog File..............................................................................................................................8

10.6.3. Extent Overflow File.................................................................................................................9

10.6.4. Attribute File............................................................................................................................9

10.6.5. Startup File...............................................................................................................................9

10.7. Examining HFS+ features........................................................................................................9

10.7.1. Case Sensitivity........................................................................................................................9

10.7.2. File Encoding.........................................................................................................................10

10.7.3. Repairing Permission.............................................................................................................10

10.7.4. Quotas....................................................................................................................................10

10.7.5. Hardlinks...............................................................................................................................10

10.7.6. Aliases and Symbolic Links....................................................................................................10

10.7.7. Resource Forks:.....................................................................................................................10

10.8. Optimization..........................................................................................................................11

10.8.1. On-the Fly Defragmentation.................................................................................................11

10.8.2. Hot File Clustering.................................................................................................................11

11. Comparison of HFS+, NTFS and AFP..................................................................................11

12. Conclusion.............................................................................................................................11

References.............................................................................................................................................12

Part – I

1. Abstract File System of an operating system provides a conceptual view of data on storage devices used by computer systems. Like any other computer system, Apple computers store data in a hard disk drive which are divided into sectors and addressed using Logical Block Addressing (LBA). Disks are further divided into partitions. Partitions contain volume, which is an instance of file system. pdisk command is used in Apple partitioning system to view the partition tables of a disk. Mac OS X is an extensively used modern Operating System. This paper deals with the file system of the apple computers with the spotlight mainly on the HFS Plus File System, a favorite file system type of the Mac OS X. A comparison is made at the end of the paper between the various file systems implemented by the Mac OS X based on the task achieved by them.

2. Introduction and brief historyApple has a very big fascinating technical history and has been present since 1976. The Mac OS X operating system of Apple has successful ideologies and technologies that has evolved over all these years to make it a big hit. Starting off with Star Trek, Apple’s journey went a long way crossing Raptor, NuKernel, Copland, Accent and Mach which was the successor of accent. It was in 1996 that the first Apple operating system was released and gradually grabbed a huge audience for its efficient features. This was followed by various other successors with more and more enhanced characteristics.

3. File System Concepts The system designed for arranging data on a storage medium is called as file system. Any file system instance can be called a volume which can be split as slices or partitions. A particular HFS+ volume can be extended across multiple partitions. Blocks contain space which is a manifold of the sector size. Common sector sizes are 512 bytes and 2KB.But Mac OS has the allocation block of size 4KB and a HFS+ volume is divided into a number of allocation blocks. Extents are series of continuous allocation blocks. A number of separate parts of the file are called forks. HFS+ supports multiple byte streams in a file. The distinct files supported by HFS+ are data and resource forks. Most data are stored in B-trees which are balanced binary search trees that can have large branching factor. All data in B-Tree exist in in the leaf nodes.

4. Apple Partitioning Scheme Apple computer systems use a specific partition layout called Universal HD. This is just one of the partitioning mechanism used by the Apple system (others include PC Style Partitioning and GUID Based Partitioning). It contains several legacy partitions. For example, apple partitioning scheme consist of 64 blocks each of 512 bytes. First physical block has first 2 bytes set to ‘ER’. ER is the apple partitioning scheme mark. Block size is stored in bytes three and four. The next four bytes include the number of blocks on disk. The 63 blocks stores ‘PM’ as their first two bytes alongside with partitions storing offset, size and type. Any amount of partitions can be formed that would fit in the partition map.

5. Disk Arbitration Mac OS X Disk Arbitration maintains disks. It is done by diskarbitrationd() and DiskArbitration.FrameWork. System call diskarbitrationd() is the central authority for disk management. Disk Arbitration framework identifies a disk, participates in some mounting decisions and interacts with IO Kit regarding notifications. This framework presents API which accomplish following functions of controlling disk objects.

DADiskMount()- mount the volume at the given disk DADiskUnmount()- unmount the volume at the given disk DADiskEject()- ejects the given disk DADiskRename()- volume is renamed at the given disk DADiskGetOptions()- to get disk options DADiskSetOptions()- to set disk options DADiskClaim()- disk is claimed to use exclusively DADiskUnclaim()- disk is unclaimed

6. File Descriptors File descriptors correspond to the open files. They take action as an index into process file descriptor table in the kernel. These descriptors are used in the system calls. Following are their Mac OS X’s file descriptors along with their type.

DTYPE_VNODE - struct node DTYPE_SOCKET - struct socket DTYPE_PSXSHM - struct pshmnode DTYPE_PSXDEM - struct psemnode DTYPE_KQUEUE - struct kqueue DTYPE_PIPE - struct pipe DTYPE_FSEVENTS - struct fsevent_handle

7. Disk Images There is risk in every activity. Similarly, file system has their jeopardy of losing entire data. For this, virtual disks or disk images are used. These disk images act very similar to their physical counterparts.

Mac OS X comprises a framework DiskImages.framework. This provides wide-ranging support for disk images.

8. Virtual File System Layer Mac OS X provides virtual file system interface called the VFS Layer. This allows multiple file system to coexist in orderly manner. Vfsops structure of this interface acts as function-pointer for the following functions.

Vfs_mount()- implement the mount() system call. Vfs_start()- any operation that the file system wishes is started after the mount() is

successfully performed. Vfs_unmount()- implements the unmount() system call. Vfs_root()- retrieves the root node of the file system. Vfs_getattr()- vfs_attributes are populated with file system attributes Vfs_statfs()- file system statistics are obtained.

9. File System Types Mac OS X supports number of file systems. Since, Mac OS X takes out from many sources, it has many file system API’s like carbon file manager, NSFile Manager and Family and BSD System calls. Following are file systems supported by Mac OS X:

Local File System: These use local storage. Eg: HFS+, HFS, MS-DOS, NTFS, and UDF. Network File System: Computers that are connected over a network access file from each

other using these file systems. Apple Filing Protocol (AFP), FTP File System, NFS. Pseudo File System: These file systems provide file-like view for non-file information.

9.1. MS-DOS Mac OS X support FAT alternative of MS-DOS file system. This file system is not loaded into kernel but is present as a loadable kernel extension.

9.2. New Technology File System Mac OS X includes read-only support for New Technology File system (NTFS). NTFS file system driver is hearted on free BSD NTFS driver. This is also not compiled into the kernel but loaded by mount_ntfs() when required.

9.3. Universal Disk Format Universal Disk Format (UDF) is the file system used by DVD-ROM disks and CD’s. It is implemented as a kernel extension and loaded by mount_udf () when required.

9.4. Apple filing Protocol Apple Filing Protocl (AFP) is a session based protocol used by networked computers to share files. User Authentication Method (UAM) is used by systems to validate other systems on the network.

9.5. Hierarchical File System PlusHierarchical File System Plus (HFS+) is the much preferred file system of the Mac OS X. It is also called as Extended File System. Its predecessor is HFS. It is very rich in features and uses a B-Tree based Catalog file to store and maintain the file system’s hierarchical structure.

Part – II

10. Hierarchical File System plus Filing System

10.1. Features Provide storage for file of size up to the range of 2^63 bytes in size. Name encoding is made based on Unicode character set (range – 255 16 bit characters which

are stored in decomposed form). Indexing is based on the B+ tree structure which is hierarchical. The extents that outnumber the specified values are stored in the B+ tree. The size of the extended attribute is limited to 4 KB. B+ tree stores the arbitrary metadata. A File object can employ means like aliases, hard link and symbolic link to refer to another

file. Hot File Clustering method is adopted to adaptively look up frequently accused files. To maintain continuity, dynamic relocation of small fragmented files is performed from time

to time. File permission are applied in UNIX. File flags are BSD based which designate file as append-only, immutable and undeletable. Supports volume-level user and group quotas. Metadata stores finder information of a file to make certain per-file properties like file-

extension hiding and color-coded labels. Method call searchfs() performs search for file object based on given search criteria. Numerous time-stamps are stored for a single file system object. Case sensitivity is achieved. Dedicated startup files which can be used by non-Mac OS X to boot from HFS+ volume.

10.2. Structure of HFS+ file system Reserved Areas are present at the beginning and end of volume. Volume Header stores information about volume. Alternate Volume Header is a copy of the volume header. Catalog B Tree** stores the metadata about files and directories Extents Overflow B Trees** store the extents that overflow.

Attribute B Tree** maintains the extended attributes of files and directories. Allocation File** is a bitmap structure to denote filled allocation blocks. Private Metadata Folder stores data about deleted files and hardlinks. Hot Files B Tree store information about frequently accessed files. Start-up File** is used to boot system using some files present in HFS+. Journal Files holds files about journals. Quota Files maintains group and user quota information.

**These special files need not be in contiguous locations and are not user-visible and hence they do not contribute to the total count of files on volume header.

10.3. Reserved Areas The volume’s first two and last logical portions are reserved. The first sectors are used as boot blocks while the last one is reserved for storing manufacturing information. They contained the following system files that are used for booting.

System : system file Finder : shell file Macsbug : debugger Disassembler : another debugger Startupscreen : file containing start up Clipboard : system scrap file

10.4. Volume Header After the first reserved area present is the volume header. It is 512 bytes in size. This contains information about other data structures. While other structure changes location; volume header has a fixed one. It is at a fixed offset from the start. Alternate volume header stores disk and file system repair utilities. Volume header contains extent of HFS+ B Trees and other files. Kernel block called Volume Control Block (VCB) holds volume header info once as HFS+ volume is mounted. It also stores dynamic information about the volume.

10.5. HFS Wrapper Some computers does not support HFS+ in ROM. Apple had to ensure that these computers were able to boot Mac OS from HFS+ volume. Wrapping mechanism was hence originated. By this HFS+ volume was wrapped so that it appears as HFS to ROM. The System file of HFS would refer to the embedded HFS+ code, mount it and perform boot by using System file of HFS+ volume. Thus, HFS wrapper volume contains boot blocks, Master Directory Block (MDB), HFS Data, Volume Header, HFS+ data and reserved space.

10.6. Special files

10.6.1. Allocation File It is basically to track if an allocation block is being used. It is a bit map structure i.e., when a block is holding user data or assigned to file system data structure, the respective bit is set in allocation blocks. Since allocation file is also a file, it itself occupies some blocks. This file has the property of being grown or shrunk and so space can be flexibly manipulated based on the use. hfsdebug is used to view contents of the file.

10.6.2. Catalog FileThis file contains the vital information about all files and folders and also acts as their catalog. The file and folder names are stored as HFSUnistr 255 structure which consists of a length and 255-element double byte Unicode character array. The files and folders are identified by a unique Catalog Node ID (CNID). Catalog files are organized as B-Tree which gives way for quick and efficient searching. Each file has 2 leaf records in catalog file. File record, File thread record and each folder has Folder record, Folder thread records. File Record (HFS+CatalogFile) has standard file metadata like files CNID, time stamps, Unix-style permission, finder information and initial extents of files data. Folder Record: (HFS+CatalogFolder) has standard folder metadata but has no data. Every folder has a valence value representing the number of files and immediate subdirectories. HFSPlusCatalogThread represent file and folder thread records. A record contains name and parent CNID of the catalog node it represents. Thread records represent the hierarchical structure of the file systems by threading together the qualified organization of files and folders.

10.6.3. Extent Overflow FileAllocation of storage to a file can be non-contiguous. A file record in catalog has space

for holding each extent record for the corresponding file data resource forks. If a fork has more than 8 fragments, its remaining extents are stored in leaf nodes of Extent Overflow File. The data structure associated with Extent Overflow File is HFSplusExtentRecord. Its keys are fixed and structure consists of forktype, fileID and startBlock attributes.

10.6.4. Attribute FileAttribute file sits in the implementation of named forks. Named forks are associated with

files or folders and are used to implement extended attributes for files and folder. ACL in Mac OS X 10.4 uses extended attributes store ACL data attached to files and folders. All extended attribute contain Name (Unicode string in decomposed and canonical order) and random data. Mac OS X 10.4 supports only values that will fit in B-Tree. This file uses HFSPlusAttrKey structure which included fileIO, startBlock and AttrName. System calls Setattr(), getxattr(), listattr(), removeattr() are used to operate on file names.

10.6.5. Startup FileHFS+ supports an optional startup file that contains secondary bootloader while booting

the system. The location of startup file is at well-known position a fixed offset from volume header. So there is no necessity to have built in HFS+ support mechanism like the one used for HFS wrapper.

10.7. Examining HFS+ features

10.7.1. Case SensitivityHFS+ is case preserving, case insensitive file system. This is often an undesirable feature. For this HFS+ was introduced as extension of HFS+. HFS+ support case sensitivity by passing –s option to newfs- hfs. KeyCompareType field of catalog B-Tree ensures case sensitivity by having its value oxcf instead oxbc.

10.7.2.File EncodingUnicode encoding done for files, folders and extended attributes. This consists of 16-bit

length with up to 255 double-byte Unicode characters. These characters are fully decomposed with the composing characters in canonical order. When strings between HFS+ and user space are exchanged, they are encoded by kernel as ASCII compatible UTI 8 bytes. Older Mac OS encoded using Apple only text encoding likes MacGreek, MacRoman etc.

10.7.3.Repairing PermissionSometimes, there may happen an unintended or misbehaving application that corrupts

permission. This problem is handled by Mac OS X with the help of bill of materials (bom) that is installed along with each package. This bom contains all files along with its metadata (File Unix Permission). Bom files are used by tools that repair permission to get the original permission.

10.7.4.QuotasHFS+ supports volume level quotas based on user and group IDs. The catalog node

structure contains an array of disk quota usage records. The array contains 2 elements in MAC OS X10.4 one for user quotas and one for group quotas. Quota filenames are .quota.user and .quota.group. Each file contains a header followed by hash table of structures specifying various quota limits and usage values for user or group IDs.

10.7.5. HardlinksThey are special features represented by link count value of which will indicate the type

of relationship between files. Values of for link count will denote that two files have different pathnames but refer to same physical file on disk. Value of 2 denotes that two files are equivalent. Mac OS X implementation of hard links uses a special file called hard-link file for each directory entry.

10.7.6. Aliases and Symbolic LinksBoth provide reference to files and folders but aliases being light weight references.

While moving a target on the volume does not break alias, it necessarily causes harm to symbolic links. CNID works as unique identifier that does not change when target is moved. When accessing alias, one of two references to files lost deleted can be recovered by using the update new alias value. This features helps to rename files appears even without breaking their shortcuts.

10.7.7. Resource Forks:Resource forks of HFS and HFS+ are used to hold resources. These forks usually store

icons, menus, dialog box, executable code, license information for an application and font details, preview pictures, windows location for a document. Forks has got limit. Data fork accessed by Unix API on Mac OS X. this multifork structure of HFS+ is incompatible with other file system and so care must be taken to transfer files to other system.

10.8. OptimizationThe Mac OS X HFS+ implements adaptive optimization in order to improve performance

and reduce fragmentation. HFS+ employs following mechanisms to achieve optimization.

1. On-the Fly Defragmentation2. Hot File Clustering

10.8.1. On-the Fly Defragmentation The file is moved to a location by calling hfs_relocate() which attempts to find contiguous blocks for the file. To perform this relocation following conditions are met.

File is a regular file File is not read-only File is not already open File fork is not 0 and size less than or equal to 20 MB Fork fragmented into 8 or more extents Bootstrapping has finishing completely.

10.8.2. Hot File Clustering Hot File Clustering (HFC) is adaptive, multistage clustering scheme based on the premise that frequently accessed files are small in number and size these files are called hot files. The temperature of these files is calculated which is the ratio of the number of bytes is read from file during recording time to file size. Thus, more frequently accessed files have more temperature value. HFS+ uses the dumpsize field of HFSPlusForkData structure to record the data read from a fork.

11. Comparison of HFS+, NTFS and AFP HFS+ and AFP support a huge file size of 2TB while NTFS does not .Only HFS+ is case sensitive of all three. Hard links that contain reference to other files are supported all file systems except AFP. Symbolic links which are light weight references to other files are offered by all of the file system types. Time stamps for the root directory are not stored in any of the file system types. Also none of the file system dynamically repositions the blocks that are allocated but never written. These are just few features. But most of them are supported by HFS+ which is what makes it the much preferred file system type by the Mac OS X.

12. Conclusion Hence, we have gained an insight about the functioning of the Mac OS X file system and its default file system type. Analyzing all the features, it is quite obvious that files in Mac OS X are handled in a very different and efficient way which has paved way for its popularity. Huge file size with a strong file system mechanism is what is expected from a computer system. Having achieved the latter, it is time to device a huge storage at the lowest cost which forms the hardware part of the system.

References 1. www.developer.apple.com 2. Amit Singh, 2006. Mac OS X Internals : A systems Approach 3. www.Mac-fourm.com