A Label-Based File System

1

A Label-Based File System

Shang Rong Tsai

Department of Information Management

Chang-Jung University

2

Outlines

• Background and Introduction

• Network Storage/SAN/NAS

• Object-Based Storage

• A Label-Based File System

• Epilogue

3

The Epoch of Data Explosion

• Electronic data are continuously growing• Data and storage managements are becoming more and more

important– Continuous service and data availability– Data backup and restore– Storage and data sharing– Efficiency in data and storage management

• Expecting using less man power• Flexibility in system configuration• Efficiently expanding storage capacity

4

Personal Data

• Capacity of disks is growing

• Huge number of files on a single disk– Documents got from Internet– Photos by Digital Cameras

• Sometimes we have problems to find a file

• More ways to locate files are needed

• Giving labels to files for locating

5

Network Storage Systems

• Storages are moving to networks for sharing and efficient management

• Such demands push the emerge of SAN (Storage Area Network)• NAS (Network Attached Storage) has been used for longer time• Object-Based Storage System

– Both SAN and NAS have their own merits and drawbacks.– SAN supports data access at block level, not good for applications in data

sharing– NAS imposes bottleneck on servers and has bad scalability– Object-Based approach potentially eliminates their drawbacks

• Network storage technologies offer a new platform for networking and storage people to play a new game.

6

Conceptual Model of SAN

clientclient client

client

DiskDisk Disk

tape

Linux Windows SunOS

SAN to connect hosts and storages

Network to connect clients and hosts

Hosts

Clients

Storages

7

What is a SAN

• SAN is a high speed network (traditionally Fiber Channel) to connect storages to servers (hosts). The network is basically used as a replacement of storage bus in traditional shared bus storage systems, thus enhances the possible sharing scale and possible fail over, etc.

• Access to SAN storage is at block level

8

Why SANs (1/2)

• From the communication aspect, the SAN can bypass the possible communication bottleneck. It enables communication between– Server-to-server– Server-to-storage (typical model)– Storage-to-storage (e.g. for backup without

servers’ intervention)

9

Why SANs (2/2)

• Improvements to application availability: Storage is accessible through multiple data paths for better reliability, availability, and serviceability.

• Higher application performance: Storage processing is off-loaded from servers

• Virtualized storage: Storages on SAN can be flexibly configured as a logical volume of any size, thus easily sharing storage space at configuration

• Data backup to remote sites: enabled for disaster protection

• Simplified centralized management: Single image of storage systems simplifies management.

10

Storage Virtualization (1/2)

• SNIA defines storage virtualization as: “The act of integrating one or more (back end)

services or functions with additional (front end) functionality for the purpose of providing useful abstractions. Typically, virtualization hides some of the back-end complexity, or adds or integrates new functionality with existing back end services. Examples of virtualization are the aggregation of multiple instances of a service into one virtualized service, or to add security to an otherwise insecure service. Virtualization can be nested or applied to multiple layers of a system.”

11

Storage Virtualization (2/2)

• To be more practical, storage virtualization is the aggregation of physical storage from multiple network storage devices into a single logical storage device that is managed and used by a central host.

• Logical Volume Manager (LVM) is basically the concept of storage virtualization which has been used for many years.

• By storage virtualization we can easily and flexibly configure the size of a logical disk.

• For some applications or sites, the amount of storage required grows at unprecedented rates. Try to think about if a disk partition is becoming full.

12

SAN connectivity

• Traditionally, SAN used Fibre Channel technology to implement the storage networks

• Fibre Channel SANs support high bandwidth storage traffic at 200 MB/s and enhancements to 10 Gb/s in the near future. This will be mostly used for inter switch links (ISL) between switches.

• iSCSI SAN (SCSI over TCP/IP) is a relatively new approach for storage networks.

• Fibre Channel over IP (FCIP) • FCIP can be used as WAN bridging transport for b

oth FC and iSCSI based campus SANs

13

iSCSI

• iSCSI stands for Internet Small Computer System Interconnect

• iSCSI is a protocol for encapsulating SCSI commands on a TCP/IP network

• The iSCSI protocol enables universal access to storage devices and Storage Area Networks over TCP/IP network

14

SCSI Protocol Layers

• SCSI command layer– Generic commands (for all devices)– Device specific commands

• Transport Layer

• Physical Layer (Connectivity Layer)

15

SCSI application (e.g. File systems)

SCSI block commands SCSI stream commandsSCSI commands for other types of devices

SCSI Generic (Primary) commands

SCSIcommandslayer

SCSI transportlayer

Physical layer

iSCSI over TCP/IP

TCP

IP

Layer 2 (Ethernet)

Parallel SCSI SCSI over Fiber Channel

Fiber ChannelParallel SCSI Bus

SCSI Protocol Layer

16

iSCSI PDU (from SNIA)

17

Overview of iSCSI

• iSCSI provides initiators and targets with unique names and a discovery method.

• The iSCSI protocol establishes communication sessions between initiators and targets, and provides methods for them to authenticate one another.

• An iSCSI session may contain one or more TCP connections and provides recovery in case of connection failures.

• SCSI CDBs (Command Descriptor Block) are passed from the SCSI command layer to the iSCSI transport layer. The iSCSI transport layer encapsulates the SCSI CDB into an iSCSI Protocol Data Unit (PDU) and forwards it to the Transmission Control Protocol (TCP) layer.

• iSCSI provides the SCSI command layer with a reliable transport.

18

What is NAS

• Network Attached Storage (NAS) is basically a LAN attached file server that provides shared file access using a file sharing protocol such as Network File System (NFS) or Common Internet File System (CIFS)

• Access to NAS is at file level.

19

Why NAS

• NAS technology has been used for decades to share files, thus to save storage space and to keep data consistency. (in contrast to file copy, like FTP)

• Data sharing in NAS is at file level, which matches the semantic for applications. In contrast, sharing in SAN is at block level which is very difficult (if not impossible) for applications to R/W data sharing. (Applications recognize files, not blocks)

20

What is an Object-Based Storage

• Storage devices that operate at object-level• Traditional storage devices (Such as DAS,

SAN) operate at block-level• Objects are typically files which match the

semantic level that applications manipulate data.

• In traditional systems, files are mapped to blocks before they are stored in storages.

21

Why Object-Based Storage (1/2)

• Drawbacks for NAS– Most of the processing on file access are on the

file server => Poor scalability– Difficult to distribute users’ files to multiple

servers while preserve a single global file namespace.

• We may delegate the management of the subset of the whole file namespace to a file server, however, the load distributed to each server may be very uneven.

22

Why Object-Based Storage (2/2)

• Drawbacks of SAN– SANs operate at block-level, so sharing data bet

ween applications may still need upper layer software.

• file read/write sharing• Record/file locking

• Object-Based Storage can potentially eliminate both the drawbacks of NAS and SAN

23

The Value of Objects (from SNIA)

• Better security via capabilities– Each object can have its own security domain– All I/O is authorized by the device

• Easier to share data– Files and records can be stored as objects– Low-level metadata managed by device

• Opportunities for intelligence– Attribute-based learning for resource allocation

• Better caching, pre-fetching and staging of data– Self-configuring storage w/ continuous reorganization

• Layout objects to best serve client requests

24

Two Basic Components in File Service

• Directory Service– Providing the global file name space visible to users

and applications

– Mapping file pathname to unique file id

– May need the flat file system to access directories

• Flat File System– Given the file id, returning the file contents

– Storing file attributes

– Managing file allocation on disks

25

Typical Handling in File Access

• File pathname lookup : open /a/b/c– Symbolic file name => file id (inode in Unix)– For each file path component, get the file id, get

the file contents until getting file id of the target

• Get the attributes (including the location of the file on disks) of the file

• Ready for W/R

26

How to Restructure the file system to fit the Object-Based Storage Architecture ?

• We may (as Lustre did) partition the functions into– MDS (Meta Data Server)– OSD (Object-Storage Device)

• What is the total system architecture?• What are the data and management functions delegated t

o MDS?• What are the data and management functions delegated t

o OSDs? • Important observation: Typically, among the file process

ing in systems, 90% processing is by OSDs, 10% by MDS => To balance the loads

27

Traditional structure vs. Object-Based structure

28

Lustre• A system developed at CMU, a very early working syste

m demonstrate the concept of object-based storage systems.

• Modifying the Linux kernel• A few number of MDSs• A large number of OSDs• Majorly targeting at the applications in large scale comp

uting.– Large number of users– File service for high performance computing

• Good Scalability

29

OST 1

OST 2

OST 7

OST 3

OST 6

OST 5

OST 4

GigE

QSW Net

Lustre Clients(1,000 Lustre Lite)

Up to 10,000’s

MDS 1(active)

MDS 2(failover)

Lustre Object Storage Targets (OST)

Linux OST

Servers with disk arrays

3rd party OST Appliances

SAN

Lustre Architecture (from CFS Inc.)

30

Lustre Components and Functions (1/2)

• Basic Components– Client Filesystems

• Interfacing to local file system (VFS) on clients

• As MDS client

• As OST client

– MetaData Servers

• All the meta-data operations - creating new directories, files, symbolic links, or acquiring and updating inodes, are handled by the MDS.

– Object Storage Target

• All the file I/O related operations are directed to the OST’s.

31

Lustre Components and Functions (2/2)

• The role of the client filesystem is to provide a directory tree, subdivided into filesets, which provides cluster-wide Unix file sharing semantics.

• The client filesystem interacts with the meta-data servers for meta-data handling, i.e. for the acquisition and updates of inodes and directory information.

• File I/O, including the allocation of blocks, striping, and security enforcement, is contained in the protocol between the client filesystem and the object storage targets.

• A third protocol exists between the OST and the MDS, largely for pre-allocation and recovery purposes.

32

SNIA T10 Specification

• A specification developed for Object-based Disk

• The SCSI command set defined to provide efficient peer-to-peer operation of input/output logical units that manage the allocation, placement, and accessing of variable-size data-storage containers, called objects.

33

Some Operations defined in T10

• Format OSD -defining OSD structure on device• Create Object Group -defining a set in which to create

objects• Create object -creating an object, returning the object

ID fid• Open fid• Read (fid,starting byte,length)• Write (fid,starting byte,length)• Close fid• Get attributes of an object

34

Possible Intelligent functions

• OSDs serve and manage data at object (most often representing files) level. This enables OSDs to smartly enforce management on a per-object basis– Automatic replication/backup

– QoS

– Caching/Prefetching

– Optimal layout

– Preprocessing/Postpressing (such as data compression/decompression, data filtering)

35

How to support the intelligent functions ?

• Extensible file attributes would be a good way• Users can define their own file attributes. A special ‘co

de’ attribute can be assigned to a particular file. Setting the code attribute to a file is in effect installing the code to the OSD to execute the intelligent functions within the OSD.

• How to set the code attribute in practice ?– Consideration for platform (OSD) dependence – Sharing the code on OSD

• How to command the OSDs to execute the intelligent functions automatically

36

Label-Based File System(LBFS)

• A project cooperated with Industrial Technology Research Institute

• A file system using object-based structure• Features

– Capability to give labels to files for easy locating

– Flexibility in restructuring the namespace– object-based structure

37

Why LBFS ? • Number of files are growing on a single disk• Traditional directory-based file systems can not

satisfy the requirements in locating files– Sometimes we forget in which directories the files reside– Content-based file search takes times– Typical file attributes are not enough

• The namespace of the whole file system can not be reorganized flexibly– Can each user have his own file namespace given a

single set of files ?

38

Overview of LBFS (1/2)

OSD

MDSClient

Network

CategoryA CategoryB

Namespace and meta data service

Object access

service

CategoryA

39

Overview of LBFS (2/2)

• MDS providing meta data and namespace services• MDS providing a browsing structure called Category• OSD providing object storage• System abstraction

– General Objects (GO)

– Attributes of file objects

– Labels

– Collections

– Categories

40

General Object and Label (1/2)

• GO is the basic unit to store a file object• We plan a higher level object called (XO)

XML objects for encapsulating an XML information unit

• Label is basically a keyword attached to a GO by users

• Searching a file by labels is much faster than by content matching

41

GO attributes

data

General attributes: Object Name、 Object Size、 Time Stamp …

Indexing attributes: Label1、 Label2 …

01.jpg

File attributes + Labels

File contents

General Object and Label (2/2)

42

GO Metadataid Unique ID of GO

name Name of GO

size Size of GO

uid Owner ID

gid Group ID

mode Access mode of GO

atime Last access time

ctime creation time

mtime Last modified time

contentType content-type of GO

referenceCount Reference count of GO

objects GO data objects list

labels list of labels attached to GO

typical file metadata

GO extended metadata

43

Collection (1/3)

• Collection is logically an objects container• A collection logically contains a set of symbolic

links, each of which links to a file object • Each collection has an expression defined for it to

specify the set of objects contained in the collection

Collection

i-expression : Picture Picture

Picture

Doc

Picture

Picture

Doc

Music

44

Collection (2/3)

• When users create an object and give it a label, the object will be “included” in the corresponding collection automatically

Collection A

i-expression : Picture

Collection B

i-expression : Music

User

Music

Picture

Picture

Music

Creating objects and labeling

45

Collection (3/3)

• In traditional file systems, a directory and 和 files under it are bound in a fixed way

• In an LBFS, a collection and file objects in it are bound in a flexible way, by matching the expression assigned to the collection and the labels attached to the objects

• The expression , called i-expression– AND label-name– OR label-name– NOT label-name

46

Category (1/3)

• An Category is logically a namespace of file objects, it is organized as a tree structure, where leaf nodes are objects and non-leaf nodes are collections

Category GOs

47

Category ???(2/3)

Music Category

GOs

音樂

古典流行精選

中國西洋

i-expression :Null

i-expression :Null

i-expression: 音樂 +中國 +古典

i-expression : 音樂 +流行

i-expression : 音樂 +精選i-expression: 音樂 +西洋 +古典

音樂

中國古典精選

音樂

西洋古典精選

音樂

中國古典

音樂

西洋

音樂

西洋

音樂

流行

音樂

流行精選

照片

風景

照片

人物

林志玲

照片

風景

阿里山

文件

論文OSD

文件

投影片

OSD

48

Category (3/3)

• Multiple Categories can be organized to bind a single set of objects

George

GeorgeGeorge

Photo

Photo

Photo

PhotoPhoto

Photo

Photo Photo

Photo

Mary

Mary

Family

Family Family

Family

Family

Mary PhotoMary

George

Family

Mary

Category

CategoryCategory

49

Overview of the prototype

MDS

VirtualPartitionManager

OSDManager

GOMetadataService

Metadata Storage Pool

Networking

SecurityManager

• Networking ：與 Client 使用 Metadata Service Protocol ，與 OSD 使用 iSCSI Protocol

• MDS ：提供 MDS 對外的服務• VP Manager ：負責管理各個 VP ，包含

根據 Storage Policy 決定物件的配置• OSD Manager ：負責管理各 OSD• Security Manager ：負責 T10 OSD

Security Model 中所提供的功能，包含credential 的發放

• GO Metadata Service ：負責管理 GO Metadata • Metadata Storage Pool ：真正儲存 metadata 的地方

50


• 測試用的雛型系統使用資料庫儲存 Category 、Collection 、 Label 等相關資訊。

• Metadata Storage Pool 介面明確 (metadata storage API) ，方便未來更換不同的儲存媒體。

MDS

VirtualPartitionManager

OSDManager

GOMetadataService


Networking

SecurityManager

51

Metadata Service Protocol

• Metadata Service Protocol is designed for exchanging messages between Clients and MDS

• Messages are XML-based

Element Name Description Attribute Name Description

<Function> MDS 執行的 function name Function 名稱

<ArgList> Function 參數

<Return> MDS 回傳內容 name Function 名稱

code 錯誤代碼

52

Message Example• Client request

<?xml version="1.0" encoding="utf-8" ?>

<Function name="collection_list">

<ArgList>

<catId>Category_4</catId>

</ArgList>

</Function>

• MDS response<?xml version="1.0" encoding="utf-8" ?>

<Return function="collection_list" code="">

<![CDATA[Category_4 Collection_35 Video {} Category_4 Collection_36 Music {}]]>

</Return>

53

OSD Implementation

• 使用去年學研計劃研發之 OSD– 以 Intel iSCSI/OSD 為藍本修改而來– 使用 T10 命令操作物件– 運轉在 Linux 系統上

54

Client Implement

• Networking ：使用 Metadata Service Protocol ，與 iSCSI Protocol 和 MDS 及 OSD 通信。

• OSD Client : 與 OSD 的客戶端軟體。• Cache : 處理 GO 、 Category 、 Collection 、 Label 的快取管理。• MDS Client : MDS 的客戶端軟體。• Security : 使用者帳號、登入等安全控管。• LBOFS : 系統提供給開發者銜接 LBFS

的函數庫。• Label Based Object FS Manager : 提供給使用者的 LBFS 操作介面。 Cache

OSDclient MDS

client

Networking

LBOFS

Security

Label Based ObjectFS Manager

LBOFS API

55

LBOFS API – Account & Label

Function Name Description

lbofs_login (userName,passwd) 登入 Virtual Partition

lbofs_logout 登出

Lbofs_label_add (vpId,lblName) 新增 Labellbofs_label_del (vpId,lblId) 刪除 Labellbofs_label_list (vpId) 列出使用者在 VP 中的 Label

56

LBOFS API – Category

lbofs_category_add (vpId,catName) 建立 Category

lbofs_category_del (vpId,catId) 刪除 Category

lbofs_category_list (vpid) 列出 VP 中所有 Category

lbofs_category_get (vpId,catId,buf) 取得 Category metadata

lbofs_category_rename (vpId,catId,catName) 更改 Category 名稱lbofs_category_sharable (vpId,catId,share) 設定 Category 共享lbofs_category_query (vpId,contition) 查詢 VP 中的 Category

57

LBOFS API – Collection

lbofs_collection_add(vpId,catId,colParentId,coName) 建立 collection

lbofs_collection_del (vpId,catId,colId) 刪除 collection

lbofs_collection_list (vpId,catId)列出 Category 中所有Collection

lbofs_collection_list_children (vpId,catId,parentId)列出 Collection 下一層的 Collection

與 GO

lbofs_collection_get (vpId,catId,colId) 取得 Collection metadata

lbofs_collection_rename (vpId,catId,colId,colName) 更改 Collection 名稱

lbofs_collection_set_parent (vpId,catId,colId,parentId) 設定 Collection 之 Parent Id

lbofs_collection_set_expr (vpId,catId,colId,expr) 設定 Collection 之條件式lbofs_collection_query (vpId,catId,condition) 查詢 Category 中的 Collection

58

LBOFS API – GOlbofs_go_add (vpId,goName) 建立 GO

lbofs_go_del (vpId,goId) 刪除 GO

lbofs_go_get (vpId,goId,buf) 取得 GO metadata

lbofs_go_rename (vpId,goId,newGoName) 更改 GO 名稱lbofs_go_label_add (vpId,goId,lblId,lblName) 增加 GO 上的 Label

lbofs_go_label_del (vpId,goId,lblId) 刪除 GO 上的 Label

lbofs_go_chmod (vpId,goId,mode) 更改 GO 上的存取權限lbofs_go_chgrp (vpId,goId,grpId) 更改 GO 上的使用者群組lbofs_go_set_time (vpId,goId,atime,ctime,mtime) 設定 GO 上的時間戳記lbofs_go_query (vpId,condition) 列出符合條件的 GO

lbofs_go_read_data (vpId,goId) 讀取 GO 資料lbofs_go_write_data (vpId,goId) 寫入 GO 資料lbofs_go_set_attribute (vpId,goId,page,attribute,value) 設定 GO attribute

lbofs_go_flush (value) 強制將 OSD 上的資料寫入lbofs_go_lock (vpId,goId) 對 GO Lock

lbofs_go_unlock (vpId,goId) 解除 GO Lock

59

Label Based Object FS Manager (1/4)

• Exporting Windows files to LBOFS

• Labeling the exported files

Local FS LBOFS

Export to LBOFSB

B

60


61


62


63

Epilogue (1/2)

• Network Storage Technologies get more attention by IT industry for data sharing and effective data management for huge and dynamically growing amount of data.

• SAN and NAS have their roles to play, currently the major approaches for network storages.

• SAN suffers the difficulty in direct data sharing.• NAS suffers the difficulty in system scalability• Object-Based Storage emerges as a new approach to solve

both problems.• Object-Based Storage offers new opportunities for intelligent

storage devices which potentially derive many research topics

64

Epilogue (2/2)

• LBFS uses labels to assist locating files quickly• LBFS uses collections to flexibly bind file objects to file

containers• LBFS provides category abstraction for various browsing

structures to fit individual need • We plan to modify the current prototype to serve as a

system for storing long-term useful documents with the following capabilities:– Automatic file backup– Label-based file search