Let’s Make Parallel File System More Parallelqingzhen/talk/batchfs_talk_ci15.pdfLet’s Make Parallel File System More Parallel ... type=[file] time=2015-07-27 … id ... convert

Let’s Make Parallel File System More Parallel[LA-UR-15-25811]

Qing Zheng1, Kai Ren1, Garth Gibson1, Bradley W. Settlemyer21Carnegie MellonUniversity

2Los AlamosNationalLaboratory

HPC defined by …

LANL/Summer_SchoolParallel_Data_Lab - http://www.pdl.cmu.edu/ 2

Parallel scientific appslow-latency network for msg passing

tired cluster deployments

PFS for highly scalable storage I/O

compute nodes(10,000+)

storage nodes(100+)

App2

App1App3

Parallel File System[Lustre]

Failure Handling …



storage nodes(100+)

App2

App1App3

Parallel File System[Lustre]Nodes/network will fail

apps use checkpoints to avoid

complete re-execution

each proc dumps its memory to a file

Failure Handling …



storage nodes(100+)

App2

App1App3

Parallel File System[Lustre]When failure happens

an app is simply re-scheduled

and resumes

execution from a latest checkpoint

Checkpointing …



storage nodes(100+)

App2

App1App3

Parallel File System[Lustre]

640K open()/close()N * 640K write()

• Assuming 20,000 nodes and 32 CPUs per node

1 if (proc_id == 0) {2 mkdir(“/proj/a/chk/001”);3 }4 sync();5 int fd = open(“/proj/a/chk/001/<proc_id>”,6 O_CREAT | O_EXCL | O_WRONLY);7 write(fd, “<…..>”);8 write(fd, “<…..>”);9 close(fd);

YES? NO?[ DATA ] [ METADATA]


Will existing PFS deliver sufficient perf?

NO?


[ METADATA]open(), close(), unlink(),

mkdir(), rmdir(),rename(), getattr(), chmod(), readdir(), …

Metadata

Namespace Tree1

Data Location3

File Attributes2

1• hierarchical directory structure

• file name, file size, last modification time, …

• where to find file/directory data ?

Decoupled PFSParallel File System

metadata service data service[a single (or a few) machines] [a large collection of machines]

Allow data to scale without scaling metadata

e.g. Lustre MDS e.g. Lustre OSS


NOFS only stores

large files

Isn’t Metadata a Problem?


NOmetadata is small in size

NO90% of ops are

I/O

NOFS only stores

large files


Median file size in actually tiny/small

• < 64KB in cloud computing data centers

• < 64MB in super computing environments


64MB is the default block size for Google File System



NOmetadata is small in size

NO90% of ops are

I/O

bigger & bigger cluster• # app processes

• metadata size

• # of metadata op

HPC is growing Fast

Tomorrow we will have EXASCALE computing facilities

more intensive METADATA WORKLOADS


Metadataeventually a huge problem !!

NO!![ METADATA]


Will existing PFS deliver sufficient perf?

PARALLEL DATA/METADATA


GOAL


Middleware Design


Parallel Scientific Applications

Underlying Storage Infrastructure[Object Storage/Parallel File System]

metadata ops

data storage metadata storage

Middleware Design



metadata storagedata/metadata storage

Parallel Scientific Application

Primary Server

metadata operations

fast interconnect

Client Proc Private Server

Middleware Design


metadata storagedata/metadata storage

Parallel Scientific Application

Primary Server

metadata operations

fast interconnect

Client Proc Private Server

Enables metadata to be potentially servedfrom compute nodes


Client-funded File System Metadata Architecture

Agenda


BulkInsertion

Metadata Representation

1 2


Metadata Representation1

Block-based Metadata


data block mapinode map

superblock

inode blocks data blocks

id=161

size=64

type=[file]

time=2015-07-27

…

id=157

size=4096

type=[directory]

time=2015-07-27

…

inode [..] -> 132

[.] -> 157

zhengq-> 158

kair -> 159

garth -> 160

bws -> 161

directory entry list

UNIX Model

Block-based Metadata

data block mapinode map

superblock

inode blocks data blocks

id=161

size=64

type=[file]

time=2015-07-27

…

id=157

size=4096

type=[directory]

time=2015-07-27

…

[..] -> 132

[.] -> 157

zhengq-> 158

kair -> 159

garth -> 160

bws -> 161

inode directory entry list

file creates -> disk seeks, liner directory entry search costzero per-directory concurrency


Table-based Metadata

KEY = parent_id + hash(fname), VALUE = an embedded inode + fname

proj (id=1)

batchfs (id=5)

src (id=2)

fs.h fs.c

ROOT (id=0)

orde

red

KV p

airs


0,h(src) id=2, type=dir, fname=src

0,h(proj) id=1, type=dir, fname=proj

1,h(batchfs) id=5, type=dir, fname=batchfs

2,h(fs.h) id=3, type=file, fname=fs.h

2,h(fs.c) id=4, type=file, fname=fs.c

key valuereaddir “[ROOT]“

readdir “/src “

Table-based Metadata

KEY = parent_id + hash(fname), VALUE = an embedded inode + fname

0,h(src) id=2, type=dir, fname=src

0,h(proj) id=1, type=dir, fname=proj

proj (id=1)

batchfs (id=5)

src (id=2)

fs.h fs.c

ROOT (id=0)

1,h(batchfs) id=5, type=dir, fname=batchfs

2,h(fs.h) id=3, type=file, fname=fs.h

2,h(fs.c) id=4, type=file, fname=fs.c

orde

red

KV p

airs

readdir “[ROOT]“

readdir “/src “

key value

A large distributed sorted directory entry tablewith embedded inodes


Table RepresentationLog-structured Merge Trees [LSM]

create file/directory

In-mem B-Tree

A collection of B-trees at different levels

k/v

k/v k/v

k/v

k/v k/v

k/v k/v k/v

k/v

k/v k/v

k/v k/v k/v

k/v k/v k/v k/v

merge merge

Level-0 Level-1

Level-2


level-0 always sits in memory



In-mem B-Tree


k/v

k/v k/v

k/v

k/v k/v

k/v k/v k/v

k/v

k/v k/v

k/v k/v k/v

k/v k/v k/v k/v

merge merge

Level-0 Level-1

Level-2


merge level-0 into level-1

FULL



In-mem B-Tree


k/v

k/v k/v

k/v

k/v k/v

k/v k/v k/v

k/v

k/v k/v

k/v k/v k/v

k/v k/v k/v k/v

merge merge

Level-0 Level-1

Level-2


merge partial level-1 into level-2

FULL



In-mem B-Tree


k/v

k/v k/v

k/v

k/v k/v

k/v k/v k/v

k/v

k/v k/v

k/v k/v k/v

k/v k/v k/v k/v

merge merge

Level-0 Level-1

Level-2


convert random disk I/O into sequential I/Oavoids disk seeks

(optimized for K/V insertion)

LSM - Updates

Convert K/V updates to K/V insertion operations

chmod(“/proj/batchfs”, …)


proj (id=1)

batchfs (id=5)

src (id=2)

fs.h fs.c

ROOT (id=0)

1,h(batchfs) perm=xxx, fname=batchfs, seq=245

1,h(batchfs) perm=yyy, fname=batchfs, seq=361

seq 361>245no write in-place

LSM - Deletions

Convert K/V deletions to K/V insertion operations

rmdir(“/proj/batchfs”, …)


proj (id=1)

batchfs (id=5)

src (id=2)

fs.h fs.c

ROOT (id=0)

1,h(batchfs) live=true, fname=batchfs, seq=245

1,h(batchfs) live=false, fname=batchfs, seq=361

seq 361>245no explicit deletion

LSM - Deletions

Convert K/V deletions to an K/V insertion operations

rmdir(“/proj/batchfs”, …)

proj (id=1)

batchfs (id=5)

src (id=2)

fs.h fs.c

ROOT (id=0)

1,h(batchfs) live=true, fname=batchfs, seq=245

1,h(batchfs) live=false, fname=batchfs, seq=361

seq 361>245no explicit deletion1. immutable data structure

2. snapshotting a file system image is trivialLANL/Summer_SchoolParallel_Data_Lab - http://www.pdl.cmu.edu/ 31


LSM - Storage


k/v

k/v k/vk/v

k/v k/v

k/v k/v k/v

k/v

k/v k/v

k/v k/v k/v

k/v k/v k/v k/v

represented formatted

namespace

LSM-Tree

T1 T2

T3 T4

32MB each

LSM - Storagek/v

k/v k/vk/v

k/v k/v

k/v k/v k/v

k/v

k/v k/v

k/v k/v k/v

k/v k/v k/v k/v

represented formatted

namespace

LSM-Tree

T1 T2

T3 T4

e.g. 32MB each

Pack metadata into large filesReuse data path to deliver scalable metadata


Experiments


ExperimentsEach client process creates 1 private directoryand inserts a set of empty files into that directory

(CHECKPOINT WORKLOAD)


Hadoop File System (HDFS) ClusterDataNode

DataNode

DataNode

DataNode

DataNode

DataNode

DataNode

DataNode

NameNode

Each node has two CPUs, 8GM RAM, one HDD SATA disk, and one 1Gb Ethernet port

[metadata node]

ExperimentsEach client process creates 1 private directoryand inserts a set of empty files into that directory

(CHECKPOINT WORKLOAD)

Hadoop File System (HDFS) ClusterDataNode

DataNode

DataNode

DataNode

DataNode

DataNode

DataNode

DataNode

NameNode

Each node has two CPUs, 8GM RAM, one HDD SATA disk, and one 1Gb Ethernet port

[metadata node]The original Hadoop file system gives 600 op/s


Experiment Settings


HDFS Data Node

HDFSName Node 1 BatchFS Server

1-8 BatchFS clients

HDFS Data Node

1-8 BatchFS clients

HDFS Data Node

1-8 BatchFS clients

DISK DISK DISK

…

1 million files inserted without bulk insertion

HDFS Baseline v.s. BatchFS


0.6 0.6 0.6 0.6

11

13 1312

0

2

4

6

8

10

12

14

Thro

ughp

ut (K

op/

s)

20X 20X 20X 20X

8 client processes 16 client processes 32 client processes 64 client processesEfficient Metadata Representation


Bulk Insertion2

Traditional Model

ParallelScientific Application

DedicatedMetadata Server

mkdir(), create()

T1 T2 T3 T4

on-disk namespace storage

write tree files

Shared Underlying Storage Infrastructure


Traditional Model



mkdir(), create()

T1 T2 T3 T4


write tree files

Shared Underlying Storage Infrastructure


Sync. Interface

Strong Consistent

Traditional Model



mkdir(), create()

T1 T2 T3 T4


write tree files

Shared Underlying Storage Infrastructure1. Dedicated service doesn’t work in exascale


320K client processes

Sync. Interface

Strong Consistent

bottleneck

Traditional Model



mkdir(), create()

T1 T2 T3 T4


write tree files

Shared Underlying Storage Infrastructure1. Dedicated service doesn’t work in exascale

2. Traditional model overkill for scientific applications LANL/Summer_SchoolParallel_Data_Lab - http://www.pdl.cmu.edu/ 43

320K client processes

Sync. Interface

Strong Consistent

bottleneck

Bulk Insertion




write tree files

T5 T6

client’s metadata mutations

(1) write tree files


T1 T2 T3 T4

mkdir()create()

via private servers

Bulk Insertion




write tree files

T5 T6

(2) bulk submit



T1 T2 T3 T4

finishes execution by as easily as picking upall submitted tree files

Bulk Insertion




write tree files

T5 T6

(2) bulk submit


T1 T2 T3 T4


Similar to database pre-loadingData inserted via a low-level protocol instead of SQL


Bulk Insertion




write tree files

T5 T6

(2) bulk submit


T1 T2 T3 T4


Similar to database pre-loadingData inserted via a low-level protocol instead of SQL


1. More efficient h/w utilization2. less calls to dedicated servers: more scalable metadata

Concurrency Control


Total ordering of mutations from different clients

client1

1 chmod(“/proj”, …)

client2

1 rmdir(“/proj”, …)

client1


client2


client1

1 mkdir(“/proj”, …)

client2

1 mkdir(“/proj”, …)

client1

1 rename(“/proj”, “/a”)

client2

1 rename(“/proj”, “/b”)

Optimistic Locking


proj

batchfs

src

fs.h fs.c

ROOT

proj

batchfs

src

fs.h fs.c

ROOT

checkpoint

ck1checkpoint

ck1

batchfs

SNAPSHOT

BatchFSClient

BOOTSTRAP

CHECK/MERGE

SUBMIT

Optimistic Locking

proj

batchfs

src

fs.h fs.c

ROOT

SNAPSHOT CHECK/MERGE proj

batchfs

src

fs.h fs.c

ROOT

checkpoint

ck1checkpoint

ck1

batchfs

BatchFSClient

BOOTSTRAP SUBMIT

Similar to source code control (github/svn)Except there is no data copying (we do copy-by-ref)


Optimistic Locking

proj

batchfs

src

fs.h fs.c

ROOT

SNAPSHOT CHECK/MERGE proj

batchfs

src

fs.h fs.c

ROOT

checkpoint

ck1checkpoint

ck1

batchfs

BatchFSClient

BOOTSTRAP SUBMIT

Similar to source code control (github/svn)Except there is no data copying (we do copy-by-ref)


Fundamental AssumptionScientific applications rarely produce conflicts

Phase 1: BranchingClient instantiates a private namespace

from a global snapshot

global branch

bulk_insert(…)

Client snapshot(…)

mkdir(…)chmod(…)

T

T1 T2 T3

global namespace

client’s private branch

a global snapshot


KV pairs

T1 T2 T3 T4 T5

Phase 2: MergingServer picks up and schedules a check

on client’s metadata mutations

global branch

bulk_insert(…)

Client snapshot(…)

mkdir(…)chmod(…)

T1 T2 T3

global namespace

client’s private branch

a global snapshot


KV pairs

T1 T2 T3 T4 T5

open(…) Client2T

tentative accepted,subject to future rejection

Phase 3: Verification


T5

T6

T7


Log

metadata operation log view

SST Interpreter

global namespace

soft re-execution

T1 T2 T3 T4

concurrent updatesthat mostly don’t produce conflicts

T1 T2 T3 T4

COMMIT

T5 T6 T7 T8

conflict resolution

Experiments


Previous Setting


HDFS Data Node


1-8 BatchFS clients

HDFS Data Node

1-8 BatchFS clients

HDFS Data Node

1-8 BatchFS clients

DISK DISK DISK

…

1 million files inserted without bulk insertion

New Setting


HDFS Data Node


1-8 BatchFS clients

HDFS Data Node

1-8 BatchFS clients

HDFS Data Node

1-8 BatchFS clients

DISK DISK DISK

…

8 million files inserted with bulk insertion

No v.s. w/ Bulk Insertion


0.6 0.6 0.6 0.611 13 13 12

139

188203

216

0

50

100

150

200

250

Thro

ughp

ut (K

op/

s)

8 client processes 16 client processes 32 client processes 64 client processes

8X15X 15X 18X

Bulk Insertion - 20X * 18X = 360X faster then HDFS

Client-funded File System Metadata Architecture

Agenda


BulkInsertion

Metadata Representation

1 2

Why FS is slow?

Inefficient metadata representation

At least one RPC per operation

Synchronous metadata interface

Pessimistic concurrency control

Dedicated authorization service


Client-funded HPC


PrimaryMetadata Server

compute nodes

pre-executesmetadata ops privately

Underlying Storage

not in critical path

per-batchsynchronizationExascale PFS architecture

• Move metadata computation from servers to apps

• Better h/w utilization

• FS scales w/ # of clients

App1App3

App2

Client-funded HPCPrimary

Metadata Server

compute nodes

pre-executesmetadata ops privately

Underlying Storage

not in critical path

per-batchsynchronizationExascale PFS architecture

• Move metadata computation from servers to apps

• Better h/w utilization

• FS scales w/ # of clients

App1App3

App2

Apps have long had rich h/w resourcesNow they can buy themselves scalable metadata


Future Work


ImplementationLANL/Summer_SchoolParallel_Data_Lab - http://www.pdl.cmu.edu/ 64

Metadata TracesLANL/Summer_SchoolParallel_Data_Lab - http://www.pdl.cmu.edu/ 65

Reference

Scaling the File System Control Plane with Client-Funded Metadata Servers (PDSW14)

Scaling File System Metadata Performance with Stateless Caching and Bulk Insertion (SC14)


QUESTIONS

Documents

Let’s Make Parallel File System More Parallelqingzhen/talk/batchfs_talk_ci15.pdfLet’s Make Parallel File System More Parallel ... type=[file] time=2015-07-27 … id ... convert