Leases and-caching final

Soumya KoduriPoornima G

Lease Locks and Caching

Gluster design SummitMay. 2015

Agenda

Quick intro to Lease locks

Why implement in Gluster

Design

Network Partitions

Challenges / limitations

Future enhancements

Client side caching with leases

Locking mechanism, which guarentees the client, certain semantics with respect to the sharing of the file/directory with other clients.

Granted on regular files (file/directory).

Protocol for cache consistency and aggressive caching.

Enhances performance.



Client 1Server

Request for lease on file1file1file1

Granted,cache file1Client 2

Request for access to file1 Lease Break file1Flush all the cached data

Flush file1,Unlock lease

6.Granted accessto file1126435

Why support it in Gluster

NFS-Ganesha/Samba applications

Multi-protocol access on Gluster

A new client side caching xlator based on leasesFile data

Directory entries (+ readdirplus)

Metadata

Lease locks and its manifestations

Linux lease locks:- Read lease- Write lease- Process based

NFS V4 delegations:- Read delegations- Write delegations- Directory delegations- Client based

Lease locks and its manifestations

SMB- Oplocks (opurtunistic locking)

Level 1 (exclusive lock) Level 2 (Read lock) Batch (handle caching + exclusive lock) Fd based

Leases introduced in SMB 2.1 Read Read/Handle Read/Write Read/Write/handles Client based

Oplocks was introduced in SMB 2?Leases were introduced in SMB 2.1 considered as oplocks done right. Major differences being:- Introduction of new typrs of leases- Introduction of lease Key

On Gluster

Currently supported- Read Lease- Write Lease- fd based

Plan is to support a super set of all standards:- Handle Lease, Read handle, Read Write Handle Lease- Lease Key support i.e. client based.- Lease upgrade/downgrade.- Directory Lease

Doesn't adhere to the Posix standard of one unlock per process.

Design

Gluster Server

Locks xlatorLease xlator

libgfapiProtocol clntProtocol server

glfs_adv_lock()

Lk Request

Lk Response

Recall LeaseUses Upcall Infra- Stores the lease info

- Check conflict,grant/ Recall lease

- Timer based recall

Command to enable Lease lock support for a volume:Gluster volume set leaeses on

Recall Lease implemented via the file system notification feature in Gluster

The lease lock requests are not sent to the Posix locks xlator, and hence it should be placed above the Posix locks, for it to function right.

Design

New API introduced in libgfapi to request lease locks.int glfs_*advanced*_lock ();

Lease xlator on server side above Posix locks:- Maintains the lease state of a file- Checks for conflicts, Grants, Recall, lease locks

Recall lease is implemented using Gluster file system notifications.

Timer based recall




Network Parition

Gluster Server disconnects:

Replica brick will serve the lease state.

Non-replica - the lease state is lost forever.

Gluster Client disconnects:

All the locks held by this client will be unlocked.

Gluster Server/Client Reconnects:

The client will replay all the lease locks it has held.




Challenges / Limitations

Rebalance/Tiering

Self-Heal

Client lock replay not enabled.

Inconsistency when used along with performance xlators

Delegation throttling

Tracking non fd based fops and the actual client associated

Future Enhancements

Super set of all leases

Upgrade and downgrade leases

Heuristics based lease grantConflict frequency

Limit on leases granted

Quota limits

Migrate/heal lease state

Locks recovery during network disconnects

Client side caching

Common challenges of caching

Challenges with caching

Mostly in memory hence less aggressive

Not distributed cache coherent

Performance hit with cache-validation requests

Not tolerant with file-system notification failures

architectures; this work relies on reliable, synchronousbroadcast communication as provided by the system bus.A distributed system, however, can experience partial fail-ures: a host may crash or messages may be lost. Existingapproaches to consistency for file caches fall into two cat-egories: those that assume reliable broadcast, and so donot tolerate communication failures, and those that requirea consistency check for every read, and so fail to delivergood performance.

Benifits of using leases

Cache Coherency

Aggressive persistent data caching

Reduces application-level response time

Suitable for distributed systems of larger scale with their larger ratio of processor speed to network delay

Performance improvement in the workloads with little write-sharing

architectures; this work relies on reliable, synchronousbroadcast communication as provided by the system bus.A distributed system, however, can experience partial fail-ures: a host may crash or messages may be lost. Existingapproaches to consistency for file caches fall into two cat-egories: those that assume reliable broadcast, and so donot tolerate communication failures, and those that requirea consistency check for every read, and so fail to delivergood performance.

Benifits of using leases

Directory leases will help in caching readdir and readdirplus coherently

Well suited for home directory use cases

Decreases file creation latency

Client cache based on leases

Gluster Server

Locks xlatorLease xlator

Libgfapi/FUSEProtocol clntProtocol server

Open file1

Open + Lease req

Lease Granted rsp

Lease Cache

Open

Open+lease

Cache file1 if lease grantedGluster Client




Thank You!

GlusterFS Meetup

GlusterFS Meetup

Red Hat Proprietary

Click to edit the title text formatClick to edit Master title style

Click to edit the outline text formatSecond Outline LevelThird Outline LevelFourth Outline LevelFifth Outline LevelSixth Outline LevelSeventh Outline LevelClick to edit Master text stylesSecond levelThird levelFourth level

Fifth level

Technology

Leases and-caching final