PETAL:DISTRIBUTED VIRTUAL DISKS

E. K. LeeC. A. ThekkathDEC SRC

• Paper presents a distributed storage management system:– Petal consists of a collection of network-

connected servers that cooperatively manage a pool of physical disks

– Client see Petal as a highly available block-level storage partitioned into virtual disks

Highlights

Introduction

• Petal is a distributed storage system that– Tolerates single component failures– Can be geographically distributed to tolerate site

failures– Transparently reconfigures to expand in

performance or capacity– Uniformly balances load and capacity– Provides fast efficient support for backup and

recovery

Petal User Interface

• Petal appears to its clients as a collection of virtual disks:– Block-level interface– Lower-level service than a DFS – Makes system easier to model, design,

implement and tune– Can support heterogeneous clients and

applications

Client view

Scalable Network

BSD FFSBSD FFS NTFS EXT2 FS NTFS

Virtualdisks

Petal

Physical view

Storage Server Storage Server Storage Server

Scalable Network

BSD FFSBSD FFS NTFS EXT2 FS NTFS

Petal Server Modules

Global StateModule

RecoveryModule

Data AccessModule

LivelinessModule

Virtual toPhysical

Overall design (I)

• All state information is maintained on servers– Clients maintain only hints

• Liveness module ensures that all servers will agree on the system operational status– Uses majority consensus and periodic

exchanges of “I’m alive”/”You’re alive?” messages

Overall design (II)

• Information describing– current members of storage system and– currently supported virtual disksis replicated across all servers

• Global state module keeps this information consistent– Uses Lamport’s Paxos algorithm– Assumes fail-silent failures of servers

Overall design (III)

• Data access and recovery modules– Control how client data are distributed and stored– Support

• Simple data striping w/o redundancy• Chained declustering

– It distributes mirrored data in a way that balances load in the event of a failure

Address translation (I)

• Must translate virtual addresses <virtual-disk ID, offset>

into physical addresses<server ID, disk ID, offset>

• Mechanism should be fast and fault-tolerant

Address translation (II)

• Uses three replicated data structures– Virtual disk directory:

translates virtual disk ID into a global map ID– Global map:

locates the server responsible for translating the given offset (block number)

– Physical map:Locates physical disk and computers physical offset within that disk

Virtual to physical mapping

VDir

GMap

PMap0

Server 0

VDir

GMap

PMap2

Server 2

VDir

GMap

PMap1

Server 1vdiskID

offset

VDir

GMap

PMap2

Server 2

diskID and diskOffseton this server

Address translation (III)

• Three step process:1. VDir translates virtual disk ID given by client

into a GMap ID2. Specified GMap finds server that can translate

given offset3. PMap of server translates GMap ID and offset

to a physical disk and a disk offset• Last two steps are almost always performed by

same server

Address translation (IV)

• There is one GMap per virtual disk• That GMap specifies

– Tuple of servers spanned by the virtual disk– Redundancy scheme used to protect data– GMaps are immutable

• Cannot be modified• Must create a new GMap

Address translation (V)

• PMaps are similar to page tables– Each PMap entry maps 64 KB of physical

disk space – Server that performs the translation will

usually perform the disk I/O• Keeping GMaps and PMaps separate minimizes

amount of global information that must be replicated

Support for backups

• Petal supports snapshots of virtual disks• Snapshots are immutable copies of virtual disks

– Created using copy-on-write• VDir maps <virtual-disk ID, epoch(?)> into

<GMap ID, epoch>– Epoch identifies current version of virtual

disks and snapshots of past versions

Incremental reconfiguration (I)

• Used to add/remove new servers and new disks• Three simple steps

1. Create new GMap2. Update VDir entries 3. Redistribute the data

• Challenge is to perform the reconfiguration concurrently with normal client requests

Incremental reconfiguration (II)

• To solve the problem– Read requests will

• Try first new GMap • Switch to old GMap if new GMap has

no appropriate translation– Write requests will always use new GMap

Incremental reconfiguration (III)

• Observe that new GMap must be created before any data are moved

– Too many read requests will have to consult both GMaps

• Seriously degrades system performance• Do instead incremental changes over a fenced

region of a virtual disk

Chained declustering (I)

Virtual Disk

Server 2

D2

D1

D6

D5

Server 1

D1

D0

D5

D4

Server 0

D0

D3

D4

D7

Server 3

D3

D2

D7

D6

Chained declustering (II)

• If one server fails, its workload will be almost equally distributed among remaining servers

• Petal uses a primary/secondary scheme for managing copies– Read requests can go to either primary or

secondary copy– Write requests must go first to

primary copy

Petal prototype

• Four servers– Each has fourteen 4.3 GB disks

• Four clients• Links are 155 Mb/s ATM links• Petal RPC interface has 24 calls

Latency of a virtual disk

Throughput of a virtual disk

Throughput is mostly limited by CPU overhead(233 MHZ CPUs!)

File system performance

(Modified Andrew Benchmark)

Conclusion

• Block-level interface s simpler and more flexible than a FS interface

• Use of distributed software solutions allows geographic distribution

• Petal performance is acceptable but for write requests– Must wait for primary and secondary copies to

be successfully updated

Paxos: the main idea

• Proposers propose decision values from an arbitrary input set and try to collect acceptances from a majority of the accepters

• Learners observe this ratification process and attempt to detect that ratification has occurred

• Agreement is enforced because only one proposal can get the votes of a majority of accepters

Paxos: the assumptions

• Algorithm for consensus in a message-passing system

• Assumes the existence of Failure Detectors that let processes give up on stalled processes after some amount of time

• Processes can act as proposers, accepters, and learners– A process may combine all three roles

Paxos: the tricky part

• The tricky part is to avoid deadlocks when– There are more than two proposals – Some of the processes fail

• Paxos lets– Proposers make new proposals– Accepters release their earlier votes for losing

proposals