Nache: Design and Implementation of a Caching Proxy for NFSv4

Almaden

Rice University

Nache: Design and Implementation of a Caching Proxy for NFSv4

Ajay Gulati, Rice University Manoj Naik, IBM Almaden Renu Tewari, IBM Almaden

Almaden

© 2003 IBM Corporat2006n

Rice

Talk Outline

Federated File System

NFSv4: Overview and New Features

Nache Architecture

Nache Implementation

Experimental Results

Conclusions and Discussion

Almaden


Rice

Federated File System

File system federation with unified namespace

Flexible data management

– Replication, migration and read/write caching across distributed file servers

What are the options?

– AFS, DCE/DFS provide distributed FS over WAN, but…

• difficult to deploy, not widely used

– GPFS, Lustre are cluster filesystems, but…

• good for controlled environments, no federation

– NFS is a standard, but…

• v2/v3 have been designed mainly for LAN, are chatty and not suited for WAN, firewall issues

– NFSv4?

Almaden


Rice

Why NFSv4?

Open industry standard

Optimizations for WAN

– Compound operations• A single RPC request contains multiple ops, reduces round-trips

– Read-Write delegations • Efficient cache consistency, no need to contact server once a

delegation is obtained• Delegations can be recalled on conflicting access

Federation support

– Client redirection• Server returns NFS4ERR_MOVED to redirect clients• Client requests fs_locations attribute• Client connects to new location

Almaden


Rice

AlmadenAlmaden HoustonHouston New YorkNew York

Admin

ClientsClients

Filesystem Namespace Unification

Separate namespace across servers

Unified namespace for clients

Almaden


Rice

ClientsClients(San Jose)(San Jose)

Server (San Server (San Jose)Jose)

Server Server

(New York)(New York)

/project

1. OPEN /project/p12. NFS4ERR_MOVED3. GETATTR4. fs_locations

Data Access using NFSv4 Redirection

Limitations High network latency when clients access remote servers Delegation are unsuited for data sharing, fewer awarded and frequently broken No unified caching, every client has to access remote server

6. /project/p1

7. DELEGRETURN

5.Client connects to New York server

8. Server responds

Almaden


Rice

What is a Proxy Cache (Nache)?

An NFSv4 server acts as a proxy for another remote

NFS server

– caches data for reads and writes using delegations

– becomes the temporary “owner” of the data servicing

opens, lock requests and reads and writes

Multiple clients can share a proxy cache using a single

server-to-proxy delegation

Extends beyond web-proxy caching

Almaden


Rice

Data Access using a Proxy Cache

ClientsClients(San Jose)(San Jose)

Proxy Cache Proxy Cache

(San Jose)(San Jose)

ServerServer

(New York)(New York)

/project

3./project/p1

1.OPEN /project/p1

Benefits with NFSv4 Proxy Reduced network traffic and delay

– Clients access files through local proxy cache– Clients can share delegations – fewer conflicts/recalls

Improved performance for cached files – Clients can collectively take advantage of aggressive readahead, write back and local locking at the proxy– Both data and delegations are shared

2. Delegate /project/p1 to proxy

Almaden


Rice

Nache Architecture

Nache Server Nache Client

VFS Layer

Buffer Cache cacheFS

Local Client(export to)

Remote Server

(mount from)

(Linux nfsd module) (Linux nfs module)

Nache

Almaden


Rice

Nache Implementation

Cascaded mounts

NFS operation forwarding

Sub-operation RPC call handling

Almaden


Rice

Cascaded Mounts

Proxy exports a NFS mounted file system

Added export functionality for NFS

– export_ops added, not available in vanilla NFS

Nache Server

/watson/almaden

mount server:/ /almaden

ls /net at client shows contents of /watson at server

Client

mount nache:/ /net

/net /export

/export

Almaden


Rice

NFS Operation Forwarding

NFS ops are redirected from nfsd to nfs using vfs layer

Issues: certain operations are stateful

– OPEN, CREATE, LOCK, ULOCK, CLOSE

ClientClient NacheNache ServerServer

VFS

Nache Client NFSD

VFSVFS

NFS Client Nache Server

Application

Almaden


Rice

Sub-Operation RPC calls

Each FS operation at Nache server translates to a separate RPC call from Nache client

Solution:– Remove such redundant checks from code path

Nache Server Nache Client

VFS Layer

Local Client Remote Server

(Linux nfsd module) (Linux nfs module)1.OPEN

2.LOOKUP3.ACCESS4.OPEN

Almaden


Rice

Experimental Methodology

Setup

– 2-6 local NFS clients (San Jose)

– One local proxy with modified kernel modules (San Jose)

– One local NFS server (San Jose)

– One remote NFS server (New York)

Benchmarks

– Filebench (developed by Sun)

– Compilation of various packages – Linux kernel, emacs, gdb

– Our micro-benchmarks

Almaden


Rice

Experiment Categories

Delegation Benefits

Nache Benefits

Nache Overhead

Almaden


Rice

Read Delegation Benefits

Client repeats open-read-close on 100 files

Server ops reduced by 50% due to delegations

0

2

4

6

8

10

12

14

16

1 2 4 8 16 32

Number of Iterations

NF

S O

ps a

t ser

ver

(in'0

00) No Delegations

With Delegations

Almaden


Rice

Write Delegation Benefits

A client repeats open-read/write-close on 100 files

Server ops reduced by 4 times

0

10

20

30

40

50

60

1 2 4 8 16 32

Number of Iterations

NFS

Ops

at s

erve

r (in

'000

)

No Delegations

With Delegations

Almaden


Rice

Delegation Benefits (Server load)

Server ops reduced due to delegations

kernel emacs gdb webserver varmail

NF

S O

ps a

t ser

ver

(in'0

00)

With Delegations

Without Delegations

49

848

1.62.8

5.4

20.8

1529

2225

Almaden


Rice

Delegation Benefits (Client performance)

Time taken is lower and ops/sec is higher due to delegations

kernel emacs gdb webserver varmail

Thr

ough

put (

Ops

/sec

) or

Tim

e T

aken

(m

ins)

With Delegation

Without Delegation

14

29

2 6.5

565

390 9592

Time (minutes)

Ops/second

Almaden


Rice


Delegation Benefits

Nache Benefits

Nache Overhead

Almaden


Rice

Nache Benefits - Filebench (webserver)

Ops at server reduces by 38% with 4 clients

0

8

16

24

32

40

1 Client 2 Clients 3 Clients 4 Clients

NF

S O

ps a

t ser

ver

per

100

benc

hmar

k op

s

Without Nache

With Nache

Almaden


Rice

Nache Benefits – Kernel compile

Ops at server unaffected by increase in number of clients

0

30

60

90

120

150

180

210

240

1 Client 2 Clients 3 Clients 4 Clients

NF

S O

ps a

t ser

ver

(in '0

00)

Without Nache

With Nache

Almaden


Rice

Nache Benefits – Compile Time

Compile time unaffected by increase in number of clients

1

10

100

1000

Kernel Gdb Emacs

Tim

e ta

ken

over

WA

N (

min

utes

)

No Nache

With Nache (1 Client)

With Nache (2 Clients)

With Nache (4 Clients)

Almaden


Rice


Delegation Benefits

Nache Benefits

Nache Overhead

Almaden


Rice

Nache Overhead over WAN (Throughput)

Micro benchmark

Ops/s (Nache)

Ops/s (no Nache)

Overhead(%)

Create-Files 9 16 43

Random-Reads 77 48 -60

Random-Writes 8.6 10 14

Random-Append 2.2 2.4 8

Nache induces an overhead of 8-43% Code can be further optimized to reduce overheads

Almaden


Rice

Nache Overhead over WAN (Latency)

Micro benchmarkLatency (Nache)

Latency (no Nache)

Overhead (%)

Create-Files 5217 3009 42

Random-Reads 12.9 20.7 -37

Random-Writes 116.3 98 18

Random-Append 883.4 821 8

Proxy induces an overhead of 8-42% Code can be further optimized to reduce overheads

Almaden


Rice

Result Summary

Delegations provide better throughput, lower latency and reduced load at server

Nache improves performance in presence of sharing among clients

Overhead of Nache in absence of sharing is not too high except for few cases and can be further reduced

Almaden


Rice

Conclusions

NFSv4 is a good alternative for building federated file

systems

Delegations help in reducing the number of server

operations and provide better caching

Nache can be integrated with a federated file system

architecture and improve performance in presence of

sharing

Documents

Nache: Design and Implementation of a Caching Proxy for NFSv4