!""

BetrFS: A right-optimized, write-optimized file system

#$%%$&'"(&))*)+"(,)"-,&)+"-&)."/0&)+"1'2.0"1340$)5&%&+"(20)"64'*5+"-$70*)."

($&2+"1)3,8"9$:&%+";8&40&)5";&)<*=+";0&)**)<8&">*<<=+"?*$@"#&%40+"9$A0&*%"

B*)<*8+"9&8C)"D&8&A0EF2%52)+">2G"(20)42)+"B8&<%*="FH"I,47'&,%+"&)<"J2)&%<"6H"

;285*8"

"K52)="B8223"L)$M*84$5=+"N23,5*3"O)AH+">,5.*84"L)$M*84$5=+"9&44&A0,4*:4"O)4C5,5*"

2@"N*A0)2%2.=""

"

P""

•  J$43"G&)<Q$<50"4R*AS"!PT"9BU4"

•  #283%2&<S"!V$B"4*W,*)C&%"

Q8$5*"

•  ext4"G&)<Q$<50S"!  !XY"9BU4"

"

ext4 is good at sequential I/O

0

40

80

120

*higher is better

MB/

s ext4raw disk

Sequential I/O

Z""

ext4 struggles with random writes

•  J$43"G&)<Q$<50"4R*AS"!PT"9BU4"

•  #283%2&<S"K'&%%+"8&)<2'"

Q8$5*4"2@"A&A0*<"<&5&"

•  ext4"Q8$5*"G&)<Q$<50S"!  !HT"9BU4"

"

0

40

80

120

*higher is better

MB/

s ext4raw disk

Random Overwrites

Y""

•  >&)<2'"Q8$5*"R*8@28'&)A*"<2'$)&5*<"G="

4**34"

•  B&A3E2@E50*E*)M*%2R*S"!  1M*8&.*"<$43"4**3"C'*"$4"!!'4"

!  K**3"@28"*M*8="YIB"Q8$5*"!  O'R%$*4"'&[$','"XHY9BU4"G&)<Q$<50"

•  ;8*M$2,4"G*)A0'&83"G*)*\54"@82'"%2A&%$5=+".22<"OU]"

4A0*<,%$)."

What is going on here?

T""

•  ;824S"! Q8$C)."<&5&"$4"^,45"&)"&RR*)<"52"50*"%2."

•  F2)4S"!  \%*"G%2A34"A&)"G*A2'*"4A&:*8*<"2)"<$43"

!  8*&<$)."<&5&"G*A2'*4"4%2Q"

•  ?2..$)."4C%%"R8*4*)54"&"58&<*2_"G*5Q**)"8&)<2'EQ8$5*"&)<"4*W,*)C&%EOU]"R*8@28'&)A*"

Avoiding seeks: log-structured file systems

`""

•  L4*"!"#$%&'()*#+%,-#.,%/%0-a#]O4b"

!  2)E<$43"<&5&"458,A5,8*4"50&5"8&R$<%="$).*45")*Q"<&5&"Q0$%*"'&$)5&$)$)."%2.$A&%"%2A&%$5="

•  F8*&5*"&"4A0*'&"50&5"'&R4"\%*"2R*8&C2)4"52"

*cA$*)5"#]O"2R*8&C2)4"

•  O'R%*'*)5*<"$)"50*"?$),["3*8)*%"

!  *[R24*<")*Q"R*8@28'&)A*"2RR285,)$C*4"

BetrFS

d""

•  ;8$28"Q283S"#]O4"A&)"&AA*%*8&5*"DK"2R*8&C2)4"

!  N23,DK"e64'*5+"B*)<*8+"D&8&A0EF2%52)+"I,47'&,%"f!Pg+"IhDK"eK0*:=+"KR$%%&)*+"9&%R&)$+"

1)<8*Q4+"K*=45*8+"&)<"/&<23"f!Zg+"N&G%*DK"e>*)"&)<"V$G42)"f!Zg+""

!  ;8$28"#]DK4"$)",4*8"4R&A*"

•  B*58DK".2&%S"*[R%28*"&%%"50*"Q&=4"Q8$5*E2RC'$7&C2)"

A&)"G*",4*<"$)"&"\%*"4=45*'"

!  *[R%28*"50*"$'R&A5"2@"Q8$5*E2RC'$7&C2)"2)"50*"

$)5*8&AC2)"Q$50"50*"8*45"2@"50*"4=45*'"

Advancing write-optimized FSes

i""

•  BjE58**4S"&)"&4='R52CA&%%="2RC'&%"3*=EM&%,*"4528*"

•  BjE58**4"&4='R52CA&%%="<2'$)&5*"%2.E458,A5,8*<"

'*8.*E58**4"

•  #*",4*"D8&A5&%"N8**4+"&)"2R*)E42,8A*"BjE58**"

$'R%*'*)5&C2)"@82'"N23,5*3"

BetrFS uses Bε-Trees

k""

•  O'R%*'*)5"&"<$AC2)&8="2)"3*=EM&%,*"R&$84"

!  insert(k,v)!!  v = search(k)!!  delete(k)!!  k’ = successor(k)!!  k’ = predecessor(k)!

•  l*Q"2R*8&C2)S"!  upsert(k, ƒ)!!

Bε-Tree Operations

!X""

•  m,*8$*4"aR2$)5"&)<"8&).*b"A2'R&8&G%*"52"BE58**4"

! Q$50"A&A0$).+"n!"4**3"o"<$43"G&)<Q$<50"!  0,)<8*<4"2@"8&)<2'"W,*8$*4"R*8"4*A2)<"

•  6[58*'*%="@&45"$)4*854"

!  5*)4"2@"502,4&)<4"R*8"4*A2)<"

Bε-trees search/insert asymmetry

!!""

upsert(k,ƒ)-

•  1)"upsert"4R*A$\*4"&"*1$2)'."52"&"M&%,*"!  *H.H"$)A8*'*)5"&"8*@*8*)A*"A2,)5"

!  *H.H"'2<$@="50*"T50"G=5*"2@"&"458$)."

•  upserts"&8*"*)A2<*<"&4"'*44&.*4"&)<"$)4*85*<"

$)52"50*"58**"

!  <*@*8"&)<"G&5A0"*[R*)4$M*"W,*8$*4"!  Q*"A&)"R*8@28'"5*)4"2@"502,4&)<4"2@"upserts"R*8"4*A2)<"

upsert = update + insert

!P""

•  9&$)5&$)"5Q2"4*R&8&5*"BjE58**"$)<*[*4S"

*%$2,2$2-#.,%/3-------------path -> struct stat!,2$2-#.,%/3----(path,blk#) -> data[4096]!

•  O'R%$A&C2)4S"

!  @&45"<$8*A528="4A&)4"!  <&5&"G%2A34"&8*"%&$<"2,5"4*W,*)C&%%="

File System ! Bε Tree

!Z""

Operation Roundup

8*&<"

Q8$5*"

'*5&<&5&",R<&5*"

readdir!mkdir/rmdir!unlink!rename!

8&).*"W,*8=" upsert! upsert" 8&).*"W,*8=""""upsert"*<*%*5*"*&A0"G%2A3"*<*%*5*"50*)"""8*$)4*85"*&A0"G%2A3""

]R*8&C2)"""""""""""""""""O'R%*'*)5&C2)"

!Y""

•  #8$5*EG&A3"A&A0$)."A&)"A2)M*85"4$).%*EG=5*"52"

@,%%ER&.*"Q8$5*4"

•  upserts"*)&G%*"B*58DK"52"&M2$<"50$4"Q8$5*"&'R%$\A&C2)"

Integrating BetrFS with the page cache

!T""

Page cache integration #1: blind write

;&.*"A&A0*"

U02'*UG$%%U@22H5[5"

upsert(/home/bill/foo.txt, )!

write(/home/bill/foo.txt, )!

upsert(/home/bill/foo.txt, )!

!`""

;&.*"A&A0*"

U02'*UG$%%U@22H5[5"

upsert(/home/bill/foo.txt, )!

write(/home/bill/foo.txt, )!

upsert(/home/bill/foo.txt, )!

N&8.*5"R&.*"

$4"A&A0*<H"

Page cache integration #2: write-after-read

!d""

;&.*"A&A0*"

U02'*UG$%%U@22H5[5"

write(/home/bill/foo.txt, )!

42"5%$-(25%-#0-6267%,8-

Page cache integration #3: write to mmap’ed file

!i""

•  B="8*50$)3$)."50*"$)5*8&AC2)"G*5Q**)"50*"R&.*"A&A0*"&)<"50*"\%*"4=45*'+"Q*"G*)*\5"

'28*"50&)"4$'R%="4R**<$).",R"$)<$M$<,&%"

2R*8&C2)4"

!  ,4*"upserts"52"&M2$<",))*A*44&8="8*&<4"!  ,4*"upserts"52"&M2$<"Q8$5*"&'R%$\A&C2)"

Page-cache takeaways

!k""

System Architecture

hDK"

*[5Y"

;&.*"F&A0*"

J$43"

,)'2<$\*<p"

)*Q"A2<*"

PX""

•  J2"Q*"'**5"2,8"R*8@28'&)A*".2&%4"@28"4'&%%+"

8&)<2'+",)&%$.)*<"Q8$5*4q"

•  O4"B*58DK"A2'R*CCM*"@28"4*W,*)C&%"OU]q"

•  J2"&)="8*&%EQ28%<"&RR%$A&C2)4"G*)*\5q"

Performance Questions

P!""

•  J*%%"2RCR%*["<*4352RS"!  YEA28*"ZHY"Vr7"$d+"Y"VB">19"

!  dPXX>;9"PTXVB"K*&.&5*"B&88&A,<&"

•  F2'R&8*"Q$50"G58@4+"*[5Y+"[@4+"7@4"

!  <*@&,%5"4*s).4"@28"&%%"

•  1%%"5*454"&8*"A2%<"A&A0*"

Experimental Setup

PP""

0.1

1

10

100

*lower is better

Tim

e (s

) BetrFSbtrfsext4xfszfs

1000 Random 4−byte writes

Small, random, unaligned writes are an order-of-magnitude faster

•  !"V$B"\%*+"8&)<2'"<&5&"

•  !+XXX"8&)<2'"YEG=5*"Q8$5*4"

•  fsync()"&5"*)<"

PZ""

! !!

!! !

! !! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! !

100

1000

10000

100000

0 1M 2M 3MFiles Created

*higher is better

File

s/se

cond

! BetrFSbtrfsext4xfszfs

Small File Creation

Small file creates are an order-of-magnitude faster

•  A8*&5*"Z"'$%%$2)"\%*4"&)<"

""""""""Q8$5*"PXXEG=5*4"52"*&A0"

•  G&%&)A*<"<$8*A528="58**""

""""""""Q$50"@&)2,5"!Pi"

•  R*8@28'&)A*"2M*8"C'*"

PY""

Sequential I/O

0

25

50

75

100

read writeOperation

*higher is better

MiB

/s

BetrFSbtrfsext4xfszfs

1GiB Sequential I/O •  #8$5*"8&)<2'"<&5&"52"\%*+"

""""""""!X"YIEG%2A34"&5"&"C'*"

•  K*W,*)C&%%="8*&<"<&5&"G&A3"

PT""

BetrFS forgoes indirection for locality: delete, rename O(n)

!

!

!

!

!

0

100

200

300

256M

iB

512M

iB1G

iB2G

iB4G

iB

File Size

Tim

e (s

)

! BetrFS

BetrFS Delete Scaling •  Q8$5*"8&)<2'"<&5&"52"\%*+"

""""""""fsync()"$5"•  <*%*5*"\%*"

P`""

0

20

40

60

80

Tim

e (s

)

BetrFSbtrfsext4xfszfs

grep −r

0

5

10

15

20

Tim

e (s

)

GNU Find

BetrFS forgoes indirection for locality: fast directory scans

•  8*A,84$M*"4A&)4"@82'"8225"2@"

""""""""""?$),["ZH!!H!X"42,8A*""

•  VlL"find"4A&)4"\%*"""""""""""'*5&<&5&"

•  grep –r"4A&)4"\%*""""""""""""A2)5*)54"

Pd""

0

200

400

600

*lower is better

Tim

e (s

) BetrFSbtrfsext4xfszfs

IMAP(50% read, 50% mark or move)

•  J2M*A25"PHPH!Z"'&$%"4*8M*8"

"""""""",4$)."*2#9,#"-•  P`+XXX"sync()"2R*8&C2)4"

BetrFS Benefits Mailserver Workloads

Pi""

BetrFS Benefits rsync!

0

10

20

30

*higher is better

MB

/ s

BetrFSbtrfsext4xfszfs

In−place rsync ofLinux 3.11.10

•  rsync"?$),["42,8A*"58**"52"""""""""52")*Q"<$8*A528="2)"!"#$"DK"•  A2R=$)."52"&)"*'R5="<$8*A528="

Pk""

•  J2"Q*"'**5"2,8"R*8@28'&)A*".2&%4"@28"4'&%%+"

8&)<2'"Q8$5*4q"

•  O4"B*58DK"A2'R*CCM*"@28"4*W,*)C&%"OU]q"

! 928*"Q283"52"<2"0*8*"

•  J2"&)="8*&%EQ28%<"&RR%$A&C2)4"G*)*\5q"! 928*"*[R*8$'*)54"$)"R&R*8"

Performance Questions

ZX""

•  F&3*"tt"6&5S"])*"\%*"4=45*'"A&)"0&M*".22<"

4*W,*)C&%"&)<"8&)<2'"OU]"R*8@28'&)A*""

•  #]O"R*8@28'&)A*"8*W,$8*4"8*M$4$C)."'&)="

<*4$.)"<*A$4$2)4"

!  $)2<*4"! Q8$5*E5082,.0"M4H"Q8$5*EG&A3"A&A0$)."!  R*8@28'"G%$)<"Q8$5*4"Q0*)*M*8"R244$G%*"

betrfs.org"u".$50,GHA2'U24A&8%&GUG*58@4"

BetrFS