Virtualization: Towards More Flexible and Efficient Grids

Virtualization: Towards More Flexible and Efficient Grids

Kate [email protected]

Argonne National Laboratory

13/09/05 Kate Keahey, PPAM 2005

The Grid Metaphor

How do we store energy?

How do we charge for energy?

How do we reliably deliver energy?

What happens if a power station fails?

How do we ensure quality of service?

What elements make for a safe and efficient power Grid?

How do we make sure that supply meets demand?


Computational Grids

How do we store computing?

How do we charge for computing?

How do we reliably deliver cycles?

What happens if a power station fails?

How do we ensure quality of service?

What elements make for a safe and efficient power Grid?

How do we make sure that supply meets demand?

What is the “unit” of resource usage?

How can we manage different computing environments?

How can we ensure that disk, CPUs, network are all available?

How can we negotiate for computation?

NCSA

ANLCaltech

SDSC

Tera Grid

Grid Middleware

We need a “computon” that will combine environment and enforcement aspects

as well as a way of managing the multi-dimensional nature of the Grid


Grids Today Grid Middleware Tools

Security, Data Management, Resource Management & Scheduling, Monitoring

Standards: GGF, OASIS Implementations: Globus Toolkit, Condor and others Many new services are being developed

Significant deployments and use of Grid infrastructure TeraGrid, Open Science Grid (OSG), Grid 3, many

European deployments Multiple projects making production use of Grid

infrastructure. Still issues: heterogeneity, lack of satisfactory control

and accounting, no on-demand computing


The Virtualization Layer

Virtual Grids: virtualize computers, networks, disks, memory Overlay networks, virtual storage…

Use middleware to map the virtualized constructs onto physical hardware Trust middleware to map and remap the

virtual environment as needed Trust market forces to ensure that physical

resources are plentiful when you need them


Virtual Workspace

For now: focus on “virtual workspace” Unit of enforcement, a “computon” for the Grid Representation of a desired environment

Later: put all elements of the system together into a virtual Grid

We need progress in the following areas: Protocols to dynamically negotiate and describe a

workspace Ongoing work at GGF: WS-Agreement, JSDL spec

A unit of enforcement A “critical mass” implementation Recent revival in virtual machine technologies provides

potential for such an implementation


Virtual Machine Basics

Hardware

Virtual Machine Monitor (VMM) / Hypervisor

Guest OS(Linux)

Guest OS(NetBSD)

Guest OS(Windows)

A VM can serialize all of its state (including RAM) A VM image is simply a collection of files

Disk partitions, RAM, configuration file Such image can be easily moved (migrated) between hypervisors

of the same type Such image can also be saved and used for rollbacks

VM VM VM

AppApp AppAppApp


Different Hypervisor Implementations Depending on the layer you virtualize you will

end up with a different VM API: language VMs (JVM) ISA: system VMs (VMware)

Different types of system virtual machines Full virtualization (VMware)

Run multiple unmodified guest OSs Para-virtualization (Xen, UML, Denali)

Run multiple guest OSs ported to a special architecture Single OS image (Vserver)

What is the cost of using VMs? Paper: “From Sandbox to Playground: Dynamic

Virtual Environments in the Grid”, Grid 2004


The Need for Speed

L X V U

SPEC INT2000 (score)

L X V U

Linux build time (s)

L X V U

OSDB-OLTP (tup/s)

L X V U

SPEC WEB99 (score)

0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1.0

1.1

Benchmark suite running on Linux (L), Xen (X), VMware Workstation (V), and UML (U)

Paper: “Xen and the Art of Virtualization”, SOSP 2003


Licensing and Distribution

License Open source (Xen, UML)

Visible effects of open source community at work Commercial (VMware)

Also, XenSource Distribution/Installation

Para-virtualization requires kernel modifications Yes, but … everything else stays the same Xen is (or soon to be) part of multiple distributions: Fedora Core

4, Debian, inofficial: Gentoo, Mandrake and SUSE distributions Work on making Xen part of the Linux kernel

Privilege Xen (root, patch kernel, domain 0 privileges setup) VMware Workstation (root, installation only) UML: user-level


What Makes VMs Great Summary of VM properties:

Good isolation properties Generally enhanced security, audit forensics

Excellent enforcement potential Details depend on implementation

Customizable software configuration Library signature, OS, maybe even 64/32-bit architectures

Serialization property VM images (include RAM), can be copied

The ability to pause and resume computations Allow migration

How do we make VMs available over the network and manage them so as to leverage this potential? Challenges: security, enforcement, protocols


Grid Services

Web Service Resource Framework An extension of Web Services Provides standard mechanisms for

Creation Lifetime Management State management, inspection (notification)

Globus Toolkit 4 Implementation of the WSRF framework

Available since April 2005 Provides secure authentication, authorization as well as

tools for fast transfer, replica management, monitoring, and others.


What are Virtual Workspaces?

Virtual Workspaces: environments that can be made available dynamically the Grid well-defined properties in terms of environment definition

and resource usage enforcement Examples:

A physical cluster booted to a desired configuration (e.g. Cluster on Demand)

A Grid3 node dynamically configured using Pacman A cluster partition configured with a hypervisor A VM representing an OSG configuration enforcing memory

and CPU usage Workspaces can be implemented using a variety of

technologies VMs are the most promising


Virtual Workspace Environment Aspect (workspace meta-data)

Information/state that outlives its deployment Generic information (name, time to live) Attested software partition information: OS, “OSG configuration”,

“application installation”, etc. Services: ssh, GRAM, pre-configured job

Resource allocation request (deployment time) Flexibly negotiated within desired constraints

See GGF WS-Agreement standard Memory, disk, networking, etc.

See GGF JSDL standard On deployment the actual resource allocation information

becomes available for inspection Atomic workspaces and virtual clusters

Clusters are simply aggregate workspaces


Deploying Workspaces in the Grid

Define workspace environment

Manage workspace

Negotiate workspace deployment characteristic

WorkspaceWizard

(VW Factory)

Workspace Management

Service(VW Repository)

Workspace Service

(VW Manager)

request a workspace

workspace meta-data

manage workspace environment

workspace metadata

Workspace

terminate workspace deployment

negotiate workspace deployment

manage/monitor/renegotiate workspace deployment

manage activities within the workspace


Current Implementation Current prototype using Globus Toolkit 4

Leveraging standard Grid Service features Workspace Wizard

Returns workspace meta-data Very rudimentary implementation

Workspace Service Create: takes workspace meta-data and a deployment

descriptor Manage:

renegotiate resource allocation Also traditional Grid Service management: TTL, etc.

Destroy Different options: pause, shutdown or destroy

First tech preview release expected later this month


How dynamic is the deployment?

Automatic Protocol-based Moving towards better articulation of migration Renegotiation of resource allocation

How fast is this deployment? Deployment of workspace for EMBOSS suite:

Manual: ~45 minutes Based on pre-configured Vmware VMs: ~6 minutes Based on pre-configured Xen VM: < 1 second

How much overhead does workspace deployment add over what we have today?


How much deployment overhead are we adding?

Using a paused VM allows us to “save” on initiation time

8

8

8

0.7

0.7 1.7

0.8

0.8

0 2 4 6 8 10 12

a)

b)

c)

job startup scenario

time (in seconds)

VM setup

VM boot

job setup

GRAM job

a) GRAM job executionb) GRAM job execution in a paused Xen VMc) job execution in a booted Xen VM (pre-configured job)


Workspace Service: Virtual Clusters


Workspace Deployment Across Technologies

Basic node configuration (+/-boot from image) Cluster on Demand, PXE, bcfg On the order of many minutes (~30 minutes)

Refining configuration, creating access Dynamic account with workspace service: < 1s

(mostly GT4 request processing time) Refining Installation: ~2 hours to configure an ATLAS

node using Pacman Virtual machines

Deploying images Xen: ~100 ms VMware Workstation: ~ several seconds


Nested Workspaces

Physical machineprocure hardware

program program program

…VM

Hypervisor/OSdeploy hypervisor/OS workspace

VM VMdeploy VM workspace (with hypervisor/OS)

It is easier to maintain a few hypervisor configurations than thousands of user configurations.

Those can be deployed in virtual machines.


Virtual Playgrounds

Application

Virtual Grid

http://www.csc.uni-frankfurt.de/pics/cluster.gif

http://www.csc.uni-frankfurt.de/pics/cluster.gif


Ongoing Work on Workspaces Dynamic resource management with VMs

Virtual clusters, fine-grained resource mangement, migration, moving towards economic management

X. Zhang, T. Freeman

IP overlay network for virtual machines Management infrastructure for VM IP addresses

T. Freeman & L. Chen

Secure management of VM images Image attestation and verification Handling image distribution Managing workspace identity

W. Lu, T. Freeman, F. Siebenlist

Deployment Edge Services for OSG: with F. Wuertherwein & A. Rana


Related Work In-Vigo

VM-based infrastructure for the Grids VM deployment, virtual storage, virtual networks Renato Figueiredo, Jose Fortes

Virtuoso VNET: virtualizing networks Peter Dinda & lab

VIOLIN Isolated, virtual networks for VMs Dongyan Xu & lab

Cluster on Demand Clusters of VMs on demand, also networking, resource

management Jeff Chase and lab


The Challenges that Lie Ahead Deployment

How do I prepare a cluster for VM execution? Reserve and publish

Site-specific versus Grid-specific What security trade-offs are acceptable? How will VM usage change site configuration? And many, many others

Environment configuration management How to configure and manage a VM?

GGF CDDLM working group Packaging infrastructures

Security Huge potential: how are we going to leverage it?

Economics, Grid markets, and many others


Conclusions Virtual is the new real!

Virtualization is emerging as an important abstraction layer in the Grids

Virtual workspaces are cornerstone of this new abstraction layer

Rapidly developing VM technology has the potential to implement a “computon” for the Grids Fast, accessible VMs “critical mass” implementation for virtual workspaces

Two sides to providing “computation on tap” Abstractions and enforcement mechanisms Protocols

There is much ongoing work in VMs but even more challenges still like ahead

If you like a challenge, give us a call

[email protected]

Documents

Virtualization: Towards More Flexible and Efficient Grids