46
Mac OS X File Systems

Mac OS X File Systems - osxbook.com OS X File Systems. ... fuse-dbfs TagsFS Cromfs mysqlfs yacufs ZFS ferrisfuse FUR ntfs-3g ... MacFUSE VFS kernel extension Mac OS X patch for libfuse

Embed Size (px)

Citation preview

Page 1: Mac OS X File Systems - osxbook.com OS X File Systems. ... fuse-dbfs TagsFS Cromfs mysqlfs yacufs ZFS ferrisfuse FUR ntfs-3g ... MacFUSE VFS kernel extension Mac OS X patch for libfuse

Mac OS XFile Systems

Page 2: Mac OS X File Systems - osxbook.com OS X File Systems. ... fuse-dbfs TagsFS Cromfs mysqlfs yacufs ZFS ferrisfuse FUR ntfs-3g ... MacFUSE VFS kernel extension Mac OS X patch for libfuse

Had to give it a shot!

Page 3: Mac OS X File Systems - osxbook.com OS X File Systems. ... fuse-dbfs TagsFS Cromfs mysqlfs yacufs ZFS ferrisfuse FUR ntfs-3g ... MacFUSE VFS kernel extension Mac OS X patch for libfuse

Most ideas come from previous ideas.

Alan Kay

Page 4: Mac OS X File Systems - osxbook.com OS X File Systems. ... fuse-dbfs TagsFS Cromfs mysqlfs yacufs ZFS ferrisfuse FUR ntfs-3g ... MacFUSE VFS kernel extension Mac OS X patch for libfuse

Looking Back

Page 5: Mac OS X File Systems - osxbook.com OS X File Systems. ... fuse-dbfs TagsFS Cromfs mysqlfs yacufs ZFS ferrisfuse FUR ntfs-3g ... MacFUSE VFS kernel extension Mac OS X patch for libfuse
Page 6: Mac OS X File Systems - osxbook.com OS X File Systems. ... fuse-dbfs TagsFS Cromfs mysqlfs yacufs ZFS ferrisfuse FUR ntfs-3g ... MacFUSE VFS kernel extension Mac OS X patch for libfuse
Page 7: Mac OS X File Systems - osxbook.com OS X File Systems. ... fuse-dbfs TagsFS Cromfs mysqlfs yacufs ZFS ferrisfuse FUR ntfs-3g ... MacFUSE VFS kernel extension Mac OS X patch for libfuse
Page 8: Mac OS X File Systems - osxbook.com OS X File Systems. ... fuse-dbfs TagsFS Cromfs mysqlfs yacufs ZFS ferrisfuse FUR ntfs-3g ... MacFUSE VFS kernel extension Mac OS X patch for libfuse

This means that the return which isexecuted immediately after the callto aretu actually returns from the lastroutine which did the savu.

They are notexpected tounderstand this.

You are not expectedto understand this.

Page 9: Mac OS X File Systems - osxbook.com OS X File Systems. ... fuse-dbfs TagsFS Cromfs mysqlfs yacufs ZFS ferrisfuse FUR ntfs-3g ... MacFUSE VFS kernel extension Mac OS X patch for libfuse

The most importantrole of UNIX is to

provide a file system.

UNIX

Page 10: Mac OS X File Systems - osxbook.com OS X File Systems. ... fuse-dbfs TagsFS Cromfs mysqlfs yacufs ZFS ferrisfuse FUR ntfs-3g ... MacFUSE VFS kernel extension Mac OS X patch for libfuse

This means that the return which isexecuted immediately after the callto aretu actually returns from the lastroutine which did the savu.

They are notexpected tounderstand this.

You are not expectedto understand this.

Page 11: Mac OS X File Systems - osxbook.com OS X File Systems. ... fuse-dbfs TagsFS Cromfs mysqlfs yacufs ZFS ferrisfuse FUR ntfs-3g ... MacFUSE VFS kernel extension Mac OS X patch for libfuse

The most importantrole of UNIX is to

provide a file system.

UNIX

Everything is a file.

Page 12: Mac OS X File Systems - osxbook.com OS X File Systems. ... fuse-dbfs TagsFS Cromfs mysqlfs yacufs ZFS ferrisfuse FUR ntfs-3g ... MacFUSE VFS kernel extension Mac OS X patch for libfuse

System Call Layer

User Program

System Calls

User

Kernel

The File System

Page 13: Mac OS X File Systems - osxbook.com OS X File Systems. ... fuse-dbfs TagsFS Cromfs mysqlfs yacufs ZFS ferrisfuse FUR ntfs-3g ... MacFUSE VFS kernel extension Mac OS X patch for libfuse

User

Kernel

File System 1

System Call Layer

User Program

File System 2

Which?

System Calls

Page 14: Mac OS X File Systems - osxbook.com OS X File Systems. ... fuse-dbfs TagsFS Cromfs mysqlfs yacufs ZFS ferrisfuse FUR ntfs-3g ... MacFUSE VFS kernel extension Mac OS X patch for libfuse

User

Kernel

FS1

System Call Layer

User Program

FS2 FS3 FSn...

Which?

System Calls

Page 15: Mac OS X File Systems - osxbook.com OS X File Systems. ... fuse-dbfs TagsFS Cromfs mysqlfs yacufs ZFS ferrisfuse FUR ntfs-3g ... MacFUSE VFS kernel extension Mac OS X patch for libfuse

Some File System

User

Kernel

Create()Remove()Read()Write()...

node1

noden

...

Page 16: Mac OS X File Systems - osxbook.com OS X File Systems. ... fuse-dbfs TagsFS Cromfs mysqlfs yacufs ZFS ferrisfuse FUR ntfs-3g ... MacFUSE VFS kernel extension Mac OS X patch for libfuse

... ...v1

vn

vn+1

vn+m

Vnode/VFS

HFS+

hfs_create()

hfs_remove()

hfs_read()

hfs_write()...

...h1

hn

...

UFS

u1

um

ufs_create()

ufs_remove()

ufs_read()

ufs_write()...

Kernel

User

System Call Layer

User Program

File Systems

VNOP_CREATE()

VNOP_REMOVE()

VNOP_READ()

VNOP_WRITE()...

Page 17: Mac OS X File Systems - osxbook.com OS X File Systems. ... fuse-dbfs TagsFS Cromfs mysqlfs yacufs ZFS ferrisfuse FUR ntfs-3g ... MacFUSE VFS kernel extension Mac OS X patch for libfuse

HFS+

HFS

UFS

UDF

ISO9660

NTFS

MS-DOS

AFP

SMB/CIFS

NFS

WebDAV

FTP

cddafs

devfs

fdesc

deadfs fifofs specfs

synthfs union volfs

ZFS

Many File Systems

Page 18: Mac OS X File Systems - osxbook.com OS X File Systems. ... fuse-dbfs TagsFS Cromfs mysqlfs yacufs ZFS ferrisfuse FUR ntfs-3g ... MacFUSE VFS kernel extension Mac OS X patch for libfuse

Malloc/Free Abstraction

No partitions (just storage pool)

Automatically grows/shrinks

Pool’s storage shared across file systems

Transactional (no journal, no fsck)

Copy-on-write, checksummed

ZFS

Page 19: Mac OS X File Systems - osxbook.com OS X File Systems. ... fuse-dbfs TagsFS Cromfs mysqlfs yacufs ZFS ferrisfuse FUR ntfs-3g ... MacFUSE VFS kernel extension Mac OS X patch for libfuse

Inside HFS+

Page 20: Mac OS X File Systems - osxbook.com OS X File Systems. ... fuse-dbfs TagsFS Cromfs mysqlfs yacufs ZFS ferrisfuse FUR ntfs-3g ... MacFUSE VFS kernel extension Mac OS X patch for libfuse

...

# Top 10 Files with the Most Extents

1 632 183.10 2163846 Macintosh HD:/.Spotlight-V100/.store.db

2 618 187.24 2163845 Macintosh HD:/.Spotlight-V100/store.db

...

Out of 2252467 non-zero data forks total, 2252128 (99.98 %) have no fragmentation.

sudo hfsdebug -f -t 10$

Page 21: Mac OS X File Systems - osxbook.com OS X File Systems. ... fuse-dbfs TagsFS Cromfs mysqlfs yacufs ZFS ferrisfuse FUR ntfs-3g ... MacFUSE VFS kernel extension Mac OS X patch for libfuse

# Free Contig Starting@ Ending@ Space

...

101813 0x2e1097b 0x2e2972f 397.71MB

127220 0x48f0c 0x67fff 496.95MB

238304 0x2dca9cd 0x2e04cac 930.88MB

...

sudo hfsdebug -0 | sort -n +1$

Page 22: Mac OS X File Systems - osxbook.com OS X File Systems. ... fuse-dbfs TagsFS Cromfs mysqlfs yacufs ZFS ferrisfuse FUR ntfs-3g ... MacFUSE VFS kernel extension Mac OS X patch for libfuse

...

next allocation search at block = 0x2c0f7d6

next unused catalog node ID = 4042316

file system write count = 138418087

...

sudo hfsdebug -m$

Page 23: Mac OS X File Systems - osxbook.com OS X File Systems. ... fuse-dbfs TagsFS Cromfs mysqlfs yacufs ZFS ferrisfuse FUR ntfs-3g ... MacFUSE VFS kernel extension Mac OS X patch for libfuse

Demos

Page 25: Mac OS X File Systems - osxbook.com OS X File Systems. ... fuse-dbfs TagsFS Cromfs mysqlfs yacufs ZFS ferrisfuse FUR ntfs-3g ... MacFUSE VFS kernel extension Mac OS X patch for libfuse

File System Activity

Page 26: Mac OS X File Systems - osxbook.com OS X File Systems. ... fuse-dbfs TagsFS Cromfs mysqlfs yacufs ZFS ferrisfuse FUR ntfs-3g ... MacFUSE VFS kernel extension Mac OS X patch for libfuse

KernelFirmware

// Obtain handle for simple read file accessStatus = OpenSimpleReadFile( FALSE, // loading a boot selection? NULL, // optional source buffer 0, // source buffer size &FilePath, &DeviceHandle, // OUT &SimpleReadHandle // OUT );...Status = ReadSimpleReadFile( SimpleReadHandle, // file handle (UINTN)NBytes, // offset &ReadSize, // number of bytes (VOID *)Buffer); // buffer

EFIKernel Core File

Quota Files

Kernel Symbols File

File System Journals

HFS+ B-Tree (Hotfiles)

...

Kernel

GeneratingFile System Activity

int error;

vnode_t vp;

error = vnode_open(“file”, O_CREAT|FWRITE,

..., &vp, ...);

...

error = vn_rdwr(UIO_WRITE, vp, ...);

...

error = vnode_close(vp, FWRITE, ...);

User

Kernel

HFS+ UFS NFS...

Vnode/VFS Layer

BSD Interfaces (libSystem)

CarbonCarbon File Manager

CocoaNSFileManager

Carbon App Cocoa App X11 App

“Terminal” Commands

Page 27: Mac OS X File Systems - osxbook.com OS X File Systems. ... fuse-dbfs TagsFS Cromfs mysqlfs yacufs ZFS ferrisfuse FUR ntfs-3g ... MacFUSE VFS kernel extension Mac OS X patch for libfuse

MonitoringFile System Activity

fsevents

Kauth

kdebug

DTrace (Leopard onwards)

Page 28: Mac OS X File Systems - osxbook.com OS X File Systems. ... fuse-dbfs TagsFS Cromfs mysqlfs yacufs ZFS ferrisfuse FUR ntfs-3g ... MacFUSE VFS kernel extension Mac OS X patch for libfuse

User

Kernel

User Program

System Call Layer

System CallHandler Functions

/dev/fsevents

...

FS Events

Page 29: Mac OS X File Systems - osxbook.com OS X File Systems. ... fuse-dbfs TagsFS Cromfs mysqlfs yacufs ZFS ferrisfuse FUR ntfs-3g ... MacFUSE VFS kernel extension Mac OS X patch for libfuse

Kernel

vnode_authorize()

VFS Layer

File Systems File Execution

Kauth Layer

Result

No ListenersHave Listeners

Call Each Listener

Page 30: Mac OS X File Systems - osxbook.com OS X File Systems. ... fuse-dbfs TagsFS Cromfs mysqlfs yacufs ZFS ferrisfuse FUR ntfs-3g ... MacFUSE VFS kernel extension Mac OS X patch for libfuse

User

Kernel

fs_usage sc_usagelatency

...

Kdebug Trace Buffer

KERNEL_DEBUG*()

Kernel Code

Page 31: Mac OS X File Systems - osxbook.com OS X File Systems. ... fuse-dbfs TagsFS Cromfs mysqlfs yacufs ZFS ferrisfuse FUR ntfs-3g ... MacFUSE VFS kernel extension Mac OS X patch for libfuse

ModifyingFile System Activity

Permissions and ACLs

Kauth

Storage Filter Scheme

Page 32: Mac OS X File Systems - osxbook.com OS X File Systems. ... fuse-dbfs TagsFS Cromfs mysqlfs yacufs ZFS ferrisfuse FUR ntfs-3g ... MacFUSE VFS kernel extension Mac OS X patch for libfuse

User

Kernel

User Program

Device Services

Transport Services

Physical Interconnect

Device

...

YourKernel

Extension

Filter

Write Read

Page 33: Mac OS X File Systems - osxbook.com OS X File Systems. ... fuse-dbfs TagsFS Cromfs mysqlfs yacufs ZFS ferrisfuse FUR ntfs-3g ... MacFUSE VFS kernel extension Mac OS X patch for libfuse

Demos

Page 34: Mac OS X File Systems - osxbook.com OS X File Systems. ... fuse-dbfs TagsFS Cromfs mysqlfs yacufs ZFS ferrisfuse FUR ntfs-3g ... MacFUSE VFS kernel extension Mac OS X patch for libfuse

HostingFile System Activity

Page 35: Mac OS X File Systems - osxbook.com OS X File Systems. ... fuse-dbfs TagsFS Cromfs mysqlfs yacufs ZFS ferrisfuse FUR ntfs-3g ... MacFUSE VFS kernel extension Mac OS X patch for libfuse

NFSClient

NFSServer

FTP FS

User Program

NFS ProtocolUser

Kernel

Page 36: Mac OS X File Systems - osxbook.com OS X File Systems. ... fuse-dbfs TagsFS Cromfs mysqlfs yacufs ZFS ferrisfuse FUR ntfs-3g ... MacFUSE VFS kernel extension Mac OS X patch for libfuse

User

Kernel

WebDAVVFS

KernelExtension

mount_webdavUserProgram

Custom Protocol(AF_UNIX Socket)

...

Local Cache

Page 37: Mac OS X File Systems - osxbook.com OS X File Systems. ... fuse-dbfs TagsFS Cromfs mysqlfs yacufs ZFS ferrisfuse FUR ntfs-3g ... MacFUSE VFS kernel extension Mac OS X patch for libfuse

HostingFile System Activity

Difficult.

Page 38: Mac OS X File Systems - osxbook.com OS X File Systems. ... fuse-dbfs TagsFS Cromfs mysqlfs yacufs ZFS ferrisfuse FUR ntfs-3g ... MacFUSE VFS kernel extension Mac OS X patch for libfuse

Until Today.

Page 39: Mac OS X File Systems - osxbook.com OS X File Systems. ... fuse-dbfs TagsFS Cromfs mysqlfs yacufs ZFS ferrisfuse FUR ntfs-3g ... MacFUSE VFS kernel extension Mac OS X patch for libfuse

MacFUSE

Page 40: Mac OS X File Systems - osxbook.com OS X File Systems. ... fuse-dbfs TagsFS Cromfs mysqlfs yacufs ZFS ferrisfuse FUR ntfs-3g ... MacFUSE VFS kernel extension Mac OS X patch for libfuse

MacFUSEVFS

libfuse

YourFile

System User Program

/dev/fuseUser

Kernel

Sourceof

Data

Page 41: Mac OS X File Systems - osxbook.com OS X File Systems. ... fuse-dbfs TagsFS Cromfs mysqlfs yacufs ZFS ferrisfuse FUR ntfs-3g ... MacFUSE VFS kernel extension Mac OS X patch for libfuse

sshfs

owfs

encfs

Run-Time-Access

PhoneBook

KIO Fuse Gateway

LUFS Bridge

Bluetooth

mcachefs

RelFS

gphoto2-fuse-fs

CvsFS

Siefs User-Level Versioning

Offline Media Content DB

ntfsmount

cddfs

BitTorrent

djmount

fusecram

fuseiso Logic

FUSE&DPAPcompFUSED

DBToy

FUSECompres

CopyFS

GnomeVFS2FUSE

archivemount

FUSE::DBI

LoggedFS

LZOlayer

fusepak

FunionFS

MythTVfs

Captive NTFS

unionfs-fuse

Lkarmafs

Apretujado

CryptoFSAfuse

FUSEPod

Mountlo

fuse_adfsrarfs

localfs

dereferencefs

fsfipi

LAFSunpackfs

hierfs

fuse-dbfsTagsFS

Cromfs

mysqlfs

yacufs

ZFS

ferrisfuse

FURntfs-3g

playlistfs

gcfuse wadfs

mp3fs

DVDfs

Hyppocampus

ltspfsrofs

beaglefs gitfs

fusedaap

spgmfs

fdmsfs

mediamax-fuse FunFS

SMB for FUSE

Fusedav

GmailFSSMBNetFS

GfarmFS

httpfs

Clustered RAID

HTTP-FUSE

WikipediaFS

wdfsFuseFTP

FlickrFS

FSFS NOOFSGrifi

BlogFS

OpenomyFS

ELFI

CurlFtpFS

wikifuse

9pfuse

PUFSGlusterFS

usmb

TrackerFS

obexfs

afpfs-ng

GSpaceFS

Page 42: Mac OS X File Systems - osxbook.com OS X File Systems. ... fuse-dbfs TagsFS Cromfs mysqlfs yacufs ZFS ferrisfuse FUR ntfs-3g ... MacFUSE VFS kernel extension Mac OS X patch for libfuse

Demos

Page 43: Mac OS X File Systems - osxbook.com OS X File Systems. ... fuse-dbfs TagsFS Cromfs mysqlfs yacufs ZFS ferrisfuse FUR ntfs-3g ... MacFUSE VFS kernel extension Mac OS X patch for libfuse

What’s Being Released

MacFUSE VFS kernel extension

Mac OS X patch for libfuse

Mac OS X patch for sshfs

A Universal Binary tarball

Page 45: Mac OS X File Systems - osxbook.com OS X File Systems. ... fuse-dbfs TagsFS Cromfs mysqlfs yacufs ZFS ferrisfuse FUR ntfs-3g ... MacFUSE VFS kernel extension Mac OS X patch for libfuse

Thanks

FUSE Developers on Linux and FreeBSD

Google

My Team

You

Page 46: Mac OS X File Systems - osxbook.com OS X File Systems. ... fuse-dbfs TagsFS Cromfs mysqlfs yacufs ZFS ferrisfuse FUR ntfs-3g ... MacFUSE VFS kernel extension Mac OS X patch for libfuse

Q/A