Considerations and Strategies for Deploying Virtual Clusters

Giovanni MarchettiSr. Technical EvangelistMicrosoft Corp.WSV202

Session Objectives

Brief intro to HPC ClustersWhy virtualize HPC clusters?Where do they make sense?How do we deploy and manage them?What configuration issues arise?How do we schedule jobs on them?Are they so exotic?

What Is High Performance Computing?

A branch of computer science that studies systems designs and programming techniques to extract the best computational performance out of microprocessors. It is usually associated with parallel computing either on specialized machines or on clusters of commodity machines.

Industries Using HPC

Education & ResearchGeological ServicesManufacturingFinancial ServicesMoviesMedicineArchaeology???!!!

Real-world Application Example

Courtesy ofUniversity of Bologna, Italy

CAT scan of a 3000 year-old catData and computation – intensiveLimited comms. among nodesTypical parametric sweep

MPINetwork

PrivateNetwork

PublicNetwork

Corporate IT Infrastructure

Compute NodeHead Node

AD

DNS

DHCP

Clients Monitoring SystemsManagement

Compute Cluster

Possible Cluster Topologies

Admin / User Cons

WDS

Job Scheduler

MPI

Management

NAT

Node Manager

MPI

Management

Compute Node

Node Manager

MPI

Management

The Opportunity

Low usage on physical servers (e.g. at night)Make computers compute for a living

Virtualized infrastructureTreat computing as just another functionIntegrate with “real” HPC clusters

Maintain isolation for security and ease of management reasons

Cycle scavenging does not

The Applications

Suitable for parametric sweeps (e.g., Montecarlo simulations) or web services

Many independent processes, possibly multi-threadedLarge set of inputsNot interactive

NOT suitable for MPI applicationsVirtualization comms. overhead (reduced with R2)Unpredictable latenciesLimited topologies

The Options

We have 3 alternatives:Install the HPC Pack on all the x64 serversDual bootUse Hyper-V and virtual machines with HPC 2008

Coming in R2 / v3 Boot from networkBoot from vhd

HPC Pack on all servers

ProsLittle overhead when not in useEasy to deploy and manageAlways ready

Cons:No isolation at all

Whoever can launch a job, can run anything on your computer / x64 workstation

Only works on the domain your computer is inHead node may not necessarily be in that domain

No control on resource consumption

Dual Boot

Pros:“Clean” solution, as long as you have a 2nd partition100% of resources available in designated timeNo perf. Overhead

Cons:Licenses: 2 per machine!!!100% of resources available ONLY in designated timeRequires careful management of OS transitions

Diskpart is powerful and dangerous.Limited isolation: must manage access to partitions carefully

Hyper-V

ProsComplete isolationControl resource allocation Rapid provisioning / de-provisioningDomains and VLANs for VM can differ from hostVirtualization rights included or free hyper-v server

Cons:Performance and management overheadMust have virtualization management solutionMust work on your own scheduler integration

Hyper-V: Quick Tips

Works only on x64 servers with DEP and VT (or AMD-V) enabled

Most x64 workstations can run it;Install “desktop experience”

Once installed, everything is a virtual machine, including the management partitionPatch early, patch often and be sure you know what to patch

Hyper-V Management

Requires Virtual Machine ManagerVMM will install agent, hyper-v role on targeted servers in trusted domainsCan manage servers in different domains

Deploy these patches on hyper-v hosts:KB956589 – perf. counters for synthetic storageKB956774 – BITS client handling volume IDs

Virtual Machine Manager Server

Connector

Self Service Web Portal

Administrator’s Console

Management Interfaces

SAN Storage

Operator’s Console

Web Console

Windows PowerShell

Operations Manager Server

Virtual Server Host

VM

VM

VMM Library Server

VM Template

ISO ScriptVHD

VMware VI3

Virtual Center Server

ESX Host

VM VM VM VM

VM VM VM VM

VM VM VM VM

Windows® PowerShell

VMM 2008 Architecture

VM Deployment

Build “prototype node” VMCreate hardware and o/s profile based on itExtract a template from it with SCVMM

Will run sysprep and make the node unavailableSelect target servers to run hyper-v

Note: RDMA adapters (e.g. Infiniband) will not be virtualized; they will be used as any other IPoIB interface

Deploy node template on target servers (GUI or powershell script)

VM Placement

CPU, Network,and Disk Load

Physical Disk and Memory Requirements

Configuration

VM

Result of Disk Capacity and Memory Check

Disk Capacity and Memory Check

Capacity PlanningTechnology

Normalized Host + VM Load

Existing Load Configuration

Host Performance and Configuration

RatingFunction

VM Placement

Memory is the main constraintVM can be moved and RAM allocation altered, but not dynamically

HPC VMs will most often either run at 100% CPU or 0% within the VM

Does not necessarily map to real cpu utilizationHPC job scheduler does not understand “50% of CPU”

VM Placement Considerations

Specify number of CPUs in VM and assume 100%Total Number of active logical CPUs ≤ number of physical CPUs on hostLogical will be mapped to physical by hyper-v scheduler

Hyper-V Architecture Provided by:

OS

MS / XenSource / Novell

ISV/IHV/OEM

Hyper-V

Windows hypervisor

“Designed for Windows” Server Hardware

Non hypervisor aware OS

Windows Server 2003, 2008

Applications

WindowsKernel VSC

WindowsKernel

Windows Server 2008

VSP

VMBus Emulation

Parent Partition

Kernel Mode

User Mode

Xen-enabledLinux Kernel Linux

VSCs

VMBus

Hypercall Adapter

Applications

Child Partitions

VMBus

Virtualization Stack

VMService

WMI Provider VM WorkerProcess

Applications

VM Placement Considerations

SCVMM hardware profiles ≠ hardware constraintsIndication of expected hw performance

Hyper-V always reports up to 1 quad-core cpu of underlying kind

Co-located tasks in VM may not actually beIdentical hardware profiles on same kind of host simplify life for the scheduler and the admin

Not always practicalPlace for performance, not density (in R2)

On the HPC side

Create host groupsRule of thumb: 1 per VM template deployed

Use job templates to enforce constraints for jobs on VM host group (e.g. max runtime)Schedule jobs at appropriate time

At <time> job submit <job parameters>Or specify host group, then activate the nodes in question

At <time> node resume <node list>

Rapid Provisioning / De-provisioning

Create several templatesDifferent kind of nodes for different kind of jobs> 1 template deployed to same physical host, 1 active

Test scalability by deploying N, 2N, etc… VMs. on same hostsNo need to delete VMs

Take nodes offline and shut them downWhen required, turn them back on

Provisioning on demand

1 more layer of complexityUse SCOM to track n. of tasks in the queueCause VM provisioning or activation when high threshold is reached (run psh script)De-provision or shut down when low threshold is reached

SCOM / SCVMM / Pro tipsMay cause VMs to be movedMay be irrelevant to particular HPC workload

VMs always run 100% or 0 %

Dynamically Grow / Shrink

Dynamically grow or shrink pool of resources for jobs in HPC Server 2008Ties in with dynamic provisioning / de-provisioning

Provision serversGrow pool available to running jobOffline servers -> shrink poolDe-provision or pause VMs

Job keeps running

SCOM rules for on-demand VMsdemo

Availability advantages

High availability of head nodes is quite complexSCVMM takes care of HA for VMs on clusters

Head node can be a VM tooIn this case, you’re protected against node failure only

What if time on a machine has run out, but you still need to compute?

Shrink pool, move VM to available node, bring backWith R2, live migration means you need not interrupt the computation either!

Moving Computing Nodesdemo

Call to Action

Just try it!www.microsoft.com/hpcwww.microsoft.com/hyper-vwww.microsoft.com/scvmm

questions & answers

www.microsoft.com/teched

Sessions On-Demand & Community

http://microsoft.com/technet

Resources for IT Professionals

http://microsoft.com/msdn

Resources for Developers

www.microsoft.com/learningMicrosoft Certification and Training Resources

www.microsoft.com/learning

Microsoft Certification & Training Resources

Resources

Related ContentVIR208 Virtual Machine Manager 2008: Technical Overview and R2 PreviewVIR209 How to Build an Efficient Application Infrastructure through VirtualizationVIR302 Hyper-V & Virtual Machine Manager 2008: Tips, Tricks, and Best PracticesVIR303 Hyper-V & System Center: Practical Orchestration & Process Automation

VIR03-INT Virtual Computing Clusters

VIR04-HOL Introduction to Hyper-V VIR05-HOL Introduction to Microsoft System Center Virtual Machine Manager 2008

Windows Server ResourcesMake sure you pick up your copy of Windows Server 2008 R2 RC from the Materials Distribution Counter

Learn More about Windows Server 2008 R2: www.microsoft.com/WindowsServer2008R2

Technical Learning Center (Orange Section): Highlighting Windows Server 2008 and R2 technologies•Over 15 booths and experts from Microsoft and our partners

Complete an evaluation on CommNet and enter to win!

© 2009 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries.The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS,

IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.