Cellular Disco:Resource management using virtual clusters on shared-memory multiprocessors

Kinshuk Govil, Dan Teodosiu*, Yongqiang Huang, and Mendel Rosenblum

Computer Systems Laboratory, Stanford University* Xift, Inc., Palo Alto, CA

www-flash.stanford.edu

2

Motivation• Why buy a large shared-memory machine?

– Performance, flexibility, manageability, show-off• These machines are not being used at their

full potential– Operating system scalability bottlenecks– No fault containment support– Lack of scalable resource management

• Operating systems are too large to adapt

3

Previous approaches• Operating system: Hive, SGI IRIX 6.4, 6.5

+Knowledge of application resource needs– Huge implementation cost (a few million lines)

• Hardware: static and dynamic partitioning+Cluster-like (fault containment)– Inefficient, granularity, OS changes, large apps

• Virtual machine monitor: Disco+Low implementation cost (13K lines of code)– Cost of virtualization

4

Questions• Can virtualization overhead be kept low?

– Usually within 10%

• Can fault containment overhead be kept low?– In the noise

• Can a virtual machine monitor manage resources as well as an operating system?– Yes

5

Virtual Machine Monitor

Hardware

Overview of virtual machines• IBM 1960s• Trap privileged

instructions• Physical to machine

address mapping• No/minor OS

modifications

Virtual Machine

OS

AppVirtual Machine

OS

App

6

Cellular Disco

CPU CPU CPU CPU CPU CPU CPU

Interconnect

Avoiding OS scalability bottlenecks

Virtual MachineVMVMApplication App App App

OSOS Operating System

32-processor SGI Origin 2000

. .

.

7

Experimental setup

• Workloads– Informix TPC-D (Decision support database)– Kernel build (parallel compilation of IRIX5.3)– Raytrace (from Stanford Splash suite)– SpecWEB (Apache web server)

32P Origin 2000

IRIX 6.4Cellular Disco

32P Origin 2000

IRIX 6.2

vs.

8

MP virtualization overheads

32-processor overheads

020406080

100120

InformixTPC-D

Kernelbuild

Raytrace SpecWEBNorm

aliz

ed e

xecu

tion

time

IRIXCellular Disco

• Worst case uniprocessor overhead only 9%

+10% +20%+1% +4%

9

VM

Cellular Disco

CPU CPU CPU CPU CPU CPU CPU CPU

Interconnect

Fault containment

VM VM

• Requires hardware support as designed in FLASH multiprocessor

10

Fault containment overhead 0%

• 1000 fault injection experiments (SimOS): 100% success

0

20

40

60

80

100

120

InformixTPC-D

Kernelbuild

Raytrace SpecWEBNorm

aliz

ed e

xecu

tion

time 1 cell

8 cells+1% -2% +1% +1%

11

Resource management challenges

• Conflicting constraints– Fault containment– Resource load balancing

• Scalability• Decentralized control• Migrate VMs without OS support

12

VM

Cellular Disco

CPU CPU CPU CPU CPU CPU CPU CPU

Interconnect

CPU load balancing

VM VM

VM VM VM

13

Idle balancer (local view)

• Check neighboring run queues (intra-cell only)

• VCPU migration cost: 37µs to 1.5ms– Cache and node memory affinity: > 8 ms

• Backoff• Fast, local

A0B1

CPU 0 CPU 1 CPU 2 CPU 3A1

B0 B1 VCPUs

14

Periodic balancer (global view)

• Check for disparity in load tree

• Cost– Affinity loss– Fault

dependencies

1 3

4

A0

1CPU 0

0CPU 1

2CPU 2

1CPU 3A1

B0 B1B1

fault containment boundary

15

CPU management results

0

50

100

150

200

250

One VM Four VMs Eight VMs

Nor

mal

ized

exe

cutio

n tim

e

Ideal

Cellular Disco

• IRIX overhead (13%) is higher

+0.3%

+9%

16

VM

Cellular Disco

RAM RAM RAM RAM RAM RAM RAM RAM

Interconnect

Memory load balancing

VM VMVM

17

Memory load balancing policy• Borrow memory before running out• Allocation preferences for each VM• Borrow based on:

– Combined allocation preferences of VMs– Memory availability on other cells– Memory usage

• Loan when enough memory available

18

Memory management results

• Ideally: same time if perfect memory balancing

Cellular Disco DB

4Interconnect

4 4 4 4 4 4 4 Cellular Disco

DB

32 CPUs, 3.5GBInterconnect

Only +1% overhead

19

• Operating system (IRIX6.4)• Hardware partitioning

– Simulated by disabling inter-cell resource balancing

Comparison to related work

Cellular Disco TPC-D

8 CPUs 8 CPUs 8 CPUs 8 CPUsInterconnect

16 processRaytrace

20

• CPU utilization: 31% (HW) vs. 58% (VC)

216 231221 229

434

325

0

100

200

300

400

500

Raytrace Database

Tim

e (s

econ

ds)

OperatingsystemVirtual clusters

Hardwarepartitioning

Results of comparison

21

Conclusions• Virtual machine approach adds flexibility

to system at a low development cost• Virtual clusters address the needs of large

shared-memory multiprocessors– Avoid operating system scalability bottlenecks– Support fault containment– Provide scalable resource management– Small overheads and low implementation cost