An Experiment on Bare-Metal BigData Provisioning · Bigdata Configuration Bigdata Installation OS Reboot Firmware Initialization Post Setup Software Installation Package Installation

An Experiment on Bare-Metal BigData Provisioning

Ata Turk, Ravi S. Gudimetla, Emine Ugur Kaynar, Jason Hennessey, Sahil Tikale, Peter Desnoyers, Orran Krieger

1

BigData Analytics on the Cloud

• BigData deployments are moving to the cloud • On-demand usage (Cost), Elasticity,

Agility, Simplicity, … • Virtualized IaaS solutions: Amazon

EMR, Azure HDInsight, …

• Virtualization drawbacks • Overhead, unpredictability, security

concerns, device functionality, … • Bare-metal cloud solutions: IBM,

Rackspace, and Internap, …

2

Bare-Metal BigData Cloud Solutions

• Bare-Metal cloud provisioning • Automated provisioning: Ironic,

MaaS, … • Image copy to local disk => long

waits => loss of agility & elasticity

• OS streaming*, Lazy copy & de-virtualization**

• What about network booting? • incur an ongoing unacceptable

overhead during runtime

** Y. Omote, T. Shinagawa, and K. Kato, “Improving Agility and Elasticity in Bare-metal Clouds,” in ASPLOS’15, pp. 145–159, 2015.

3

* David Clerc, “OS Streaming Deployment”, in IPCCC’10, pp. 169–179, 2010.

• Large parts of the HPC community has been doing it for the last 20 years.

• Virtualized IaaS is doing it all the time.

• Why not bare-metal cloud?

4

Network-Mounted BigData System

• Clients access kernel and init ramdisk via PXE

• Mount OS & BigData apps from a remote iSCSI volume

• Use local disk for ephemeral storage (HDFS, /swap, /tmp,…)

5

Bare-Metal BigData Provisioning Prototype

6


Hardware Isolation Layer: A service to allocate bare-metal nodes out of a shared pool and isolate network

7


CEPH: Central image store hosting user images with BigData applications

8


Provisioning VM: Gateway between isolated servers and image store

9


10


11


12


13

Provisioning Time

Local Disk iSCSI0

200

400

600

800

1000

1200

1400

Elap

sed

Tim

e (S

ecs)

Bigdata ConfigurationBigdata InstallationOS RebootFirmware InitializationPost Setup Software InstallationPackage InstallationOS Boot(inc. Kernel+Initrd Download)DHCP requestFirmware InitializationHaas Power CycleCeph CloningHaas Initilization

14

Firmware initialization

OS Boot (inc. kernel+initrd download)

OS Installation

OS Reboot

BigData App Installation

Post Config

DHCP Request

Local diskinstallation iSCSI boot

Provisioning Time

Local Disk iSCSI0

200

400

600

800

1000

1200

1400

Elap

sed

Tim

e (S

ecs)


15

Firmware initialization

OS Boot (inc. kernel+initrd download)

OS Installation

OS Reboot

BigData App Installation

Post Config

DHCP Request

Local diskinstallation iSCSI boot

Provisioning Time Scaling

2 Node 4 Node 8 Node0

50

100

150

200

250

300

Elap

sed

Tim

e (S

ecs)

Bigdata Post ScriptBootingCeph CloningHaas Initilization

16

Read Traffic over Boot Drive

Initi

alPr

ovis

ioni

ngD

ata

Gen

erat

ion

1

Sort

1

Dat

aG

ener

atio

n 2

Sort

2

Dat

aG

ener

atio

n 3

Sort

3

Dat

aG

ener

atio

n 4

Sort

4

Dat

aG

ener

atio

n 5

Sort

50

100

200

300

Cum

ulat

ive

iSC

SI re

ads

per n

ode

(MB)

iSCSI Reads: Runs with 256GB DataiSCSI Reads: Runs with 128GB Data

17


Initi

alPr

ovis

ioni

ngD

ata

Gen

erat

ion

1

Sort

1

Dat

aG

ener

atio

n 2

Sort

2

Dat

aG

ener

atio

n 3

Sort

3

Dat

aG

ener

atio

n 4

Sort

4

Dat

aG

ener

atio

n 5

Sort

50

100

200

300

Cum

ulat

ive

iSC

SI re

ads

per n

ode

(MB)


~170MB / 8GB Boot Image => 2%18


Initi

alPr

ovis

ioni

ngD

ata

Gen

erat

ion

1

Sort

1

Dat

aG

ener

atio

n 2

Sort

2

Dat

aG

ener

atio

n 3

Sort

3

Dat

aG

ener

atio

n 4

Sort

4

Dat

aG

ener

atio

n 5

Sort

50

100

200

300

Cum

ulat

ive

iSC

SI re

ads

per n

ode

(MB)


3KB/sreadafter initial boot

19

Write Traffic over Boot Drive

Initi

alPr

ovis

ioni

ngD

ata

Gen

erat

ion

1

Sort

1

Dat

aG

ener

atio

n 2

Sort

2

Dat

aG

ener

atio

n 3

Sort

3

Dat

aG

ener

atio

n 4

Sort

4

Dat

aG

ener

atio

n 5

Sort

50

100

200

300

400

500

600

700

Cum

ulat

ive

iSC

SI w

rites

per

nod

e (M

B)

iSCSI Writes - Runs with 256GB DataiSCSI Writes - Runs with 128GB Data

20

Write Traffic over Boot Drive

Initi

alPr

ovis

ioni

ngD

ata

Gen

erat

ion

1

Sort

1

Dat

aG

ener

atio

n 2

Sort

2

Dat

aG

ener

atio

n 3

Sort

3

Dat

aG

ener

atio

n 4

Sort

4

Dat

aG

ener

atio

n 5

Sort

50

100

200

300

400

500

600

700

Cum

ulat

ive

iSC

SI w

rites

per

nod

e (M

B)

iSCSI Writes - Runs with 256GB DataiSCSI Writes - Runs with 128GB Data 14KB/s

write

21

Runtime Performance of Network-Mounted Boot Drive

171 31

9

616

1187

2314

171 31

8

617

1176

2281

64 115 30

0

542

1073

69 120 23

8

555

1361

60 75 76 118 19

9

52 63 86 125 20

1

Data Size (GB)0

400

800

1200

1600

2000

2400

2800

Elap

sed

Tim

e (s

ecs)

WordCount - Local DiskWordCount - iSCSI MountedSort - Local DiskSort - iSCSI MountedGrep - Local DiskGrep - iSCSI Mounted

8GB 16GB 32GB 64GB 128GB

22


171 31

9

616

1187

2314

171 31

8

617

1176

2281

64 115 30

0

542

1073

69 120 23

8

555

1361

60 75 76 118 19

9

52 63 86 125 20

1

Data Size (GB)0

400

800

1200

1600

2000

2400

2800

Elap

sed

Tim

e (s

ecs)



23


171 31

9

616

1187

2314

171 31

8

617

1176

2281

64 115 30

0

542

1073

69 120 23

8

555

1361

60 75 76 118 19

9

52 63 86 125 20

1

Data Size (GB)0

400

800

1200

1600

2000

2400

2800

Elap

sed

Tim

e (s

ecs)



24

Take-aways• Network booting the OS for bare-metal BigData

• uses only a fraction of boot disk during start-up • improves provisioning time with no runtime degradation

• provisioning time < 5 mins, boot disk reads: ~3KB/s, writes: ~14KB/s

• Enormous effort on bare-metal provisioning on local disks may be unnecessary, especially for BigData deployments

• We are building a new Bare Metal Imaging Service using remote network boot mechanisms • enable capabilities available on virtualized platforms (e.g.

snapshotting, cloning, …) to bare metal cloud solutions

25

Questions

26

Provisioning Time

Local Disk iSCSI0

200

400

600

800

1000

1200

1400

Elap

sed

Tim

e (S

ecs)


27

Local diskinstallation iSCSI boot Emulab* Ironic*

* A. Chandrasekar and G. Gibson, “A comparative study of baremetal provisioning frameworks,” Parallel Data Laboratory, Carnegie Mellon University, Tech. Rep. CMU-PDL-14-109, 2014.

Provisioning Time

Local Disk iSCSI0

200

400

600

800

1000

1200

1400

Elap

sed

Tim

e (S

ecs)


28

Local diskinstallation iSCSI boot Emulab* Ironic*

* A. Chandrasekar and G. Gibson, “A comparative study of baremetal provisioning frameworks,” Parallel Data Laboratory, Carnegie Mellon University, Tech. Rep. CMU-PDL-14-109, 2014.

copy (rather than install) an image to local disk

Documents

An Experiment on Bare-Metal BigData Provisioning · Bigdata Configuration Bigdata Installation OS Reboot Firmware Initialization Post Setup Software Installation Package Installation