Peter Lauterbach, Akorri - Storage Networking Industry ......We will discuss two major IT virtualization initiatives. Storage Virtualization Server Virtualization (and the combination

Best Practices in Managing Virtualized Environments

Peter Lauterbach, Akorri

Best Practices in Managing Virtualized Environments © 2009 Storage Networking Industry Association. All Rights Reserved. 2

SNIA Legal Notice

The material contained in this tutorial is copyrighted by the SNIA. Member companies and individual members may use this material in presentations and literature under the following conditions:

Any slide or slides used must be reproduced in their entirety without modificationThe SNIA must be acknowledged as the source of any material used in the body of any document containing material from these presentations.

This presentation is a project of the SNIA Education Committee.Neither the author nor the presenter is an attorney and nothing in this presentation is intended to be, or should be construed as legal advice or an opinion of counsel. If you need legal advice or a legal opinion please contact your attorney.The information presented herein represents the author's personal opinion and current understanding of the relevant issues involved. The author, the presenter, and the SNIA do not assume any responsibility or liability for damages arising out of any reliance on or use of this information.

NO WARRANTIES, EXPRESS OR IMPLIED. USE AT YOUR OWN RISK.


Abstract

Best Practices in Managing Virtualized EnvironmentsToday, data center environments are increasingly complex with virtualization at all layers of the IT stack, including network, server, SAN and storage. IT professionals are often challenged in diagnosing application performance issues, optimizing infrastructure resource utilization, and planning for future changes. The best practices for managing complex data center environments include cross domain management orientation, watching the infrastructure response time for cross-domain performance, looking for application contention and contention-based latency in the storage layer, best fit analysis of workloads to storage resources, and working toward infrastructure performance SLAs. Key requirements for this new breed of management software include agent-less discovery and SMI-S support.


Virtualization is Everywhere

SAN SAN

NETWORK

App Servers Web Servers Security

Array Virtualization

Storage Network

Server Virtualization

Client Network

Tremendous BenefitsPooling of resourcesRapidly deploy new applicationsIncrease resource utilizationOver-subscribe resourcesLower acquisition cost and TCO

Traditional system management practices may no longer work


What’s “Real” about Virtualization?

Like the Emperor’s new (virtualized) clothes –A logical interface presenting a normalized “resource” that isn’t “all there” Built over physical and other virtual layers that do not look at all like the presented logical resource

We will discuss two major IT virtualization initiativesStorage VirtualizationServer Virtualization(and the combination of the two!)

Check out SNIA Tutorial:

Virtualization 1- What, Why, Where, and How


Virtualization Pools Resources

SAN SAN

CLIENT NETWORK

Server Pool

STORAGE NETWORLK

Storage Pool Tier 1Tier 2

Archive

CLIENT NETWORK

Physical Infrastructure Model Virtual Infrastructure Model


Managing Virtualized Environments

Managing through Virtualization is ChallengingDiagnosing Performance ProblemsOptimizing Resource UtilizationPlanning for Future Changes

Virtualization Feature “New” Admin Challenge

Clients Reserve and ShareResource Capacity

Resource Performance still Degrades Non-linearly with Load

Dynamic Infrastructure Finding Transitional bottlenecks

Increased Resource Utilization Optimal Resource Deployment

Easy to provision new VMs Predicting if the next VM fits


The Bottom Line…

Applications share resourcesPoor performance is caused by:

Hard-to-find I/O bottlenecks and resource contentionMis-alignment between layers of virtualizationUnder-provisioning shared resources

Over-provisioning of shared resources as insurance negates ROI

Inhibitors to successVirtualized data center complexityLack of cross-domain managementLack of cross-domain communication



Solving Old Problems in a New EnvironmentRecommended Best Practices -

1. Cross Domain Analysis and Shared Resource Contention2. Adopt an Application View of Performance3. Use Automation Wisely4. “Effective Capacity” Management5. Model-based Optimization and Planning


1. Cross Domain Analysis

Virtualization Management is “Cross-Domain” -Create a Cross-Domain Baseline (discover and collect)

Mapping from multiple layers (app, server, storage, physical & virtual) Aim for agent-less and “on-line”Standards like SMI-S are essential for heterogeneous environments

Check Configuration First Don’t optimize or “plan a baseline” from a poorly configured systemChecklist vendor configuration best practices

Newer technologies (Thin-wide arrays, 10 GbE networks, SSDs) move performance bottlenecks elsewhere.

Check out SNIA Tutorial:

Solving Business-Oriented Goals with SMI-S


I/O Paths Through VirtualizationApplications and Servers

Virtual Server Hosts

Virtual StorageStorage Arrays


Find Shared Resource Contention

Stepping Through a Virtual Looking Glass -Need to Map through Virtualization Layers

Map relationships at every levelExponential problem of server virtualization over storage virtualizationSum up the loads from every client that shares each resource

Quantify Application Contention due to SharingCalculate performance impact back to each application

Root cause is mostly figuring out What’s Changed when Capacity runs out

If Load changed, was it aberrant behavior or growth?If Configuration changed, does it violate policy or show thrashing? If Contention arose, who is new to the pool?


Application Contention

Cross Domain visibility is naturally “foggy”

Domain specific management has limited viewVirtualization makes it harder

Management requires end-to-end picture

Best Practices in Managing Virtualized Environments © 2009 Storage Networking Industry Association. All Rights Reserved.

A common map helps different domain admins communicate

Need a map throughall the indirection

Sharing can be dynamic – maps must be too

Long data path from application to array…

14

Cross-Domain: Navigating the Virtualized Environment


2. Adopt Application View of Performance

The Customer is Always Right –Application Infrastructure Performance

How long do it take an I/O to complete from the application point of view (Response Time)Some applications ($$$) are more loved than others

Manage to this “Service” PerformanceElement utilizations are interesting,but service performance is the goal

Look for Abnormal “Service” BehaviorNot just default rule-of-thumb thresholds on utilizations


Service Layer Metrics

Customer Resource

Throughput @ Response Time


Look for Abnormal Behavior

Check for Abnormal Behavior

Calculate baselineA statistical analysis of variance of performanceover time

Compare data to baseline

Shared Resources tend to average out peaks that will show in dedicated resources

Helps Justify Virtualization

Acceptable Variance


4. “Effective Capacity” Management

Capacity Management Isn’t Just “Enough GBs”Storage has both space and time constraints(server folk have it easy!)Manage to the total “Effective Capacity”

Maximum utilization that gives good performanceNot to total actual utilization (aka “saturation”)

Build in Automation for ScalabilityVirtualized environments tend to sprawlAnd they can change dynamically Check out SNIA Tutorial:

Storage Virtualization II –Effective Use of Virtualization


Effective Capacity = Optimal Usage


4. Use Automation Wisely

Build in Automation for ScalabilityVirtualized environments tend to sprawlAnd they can change dynamically

Almost everything can be automatedEvent Monitoring Performance collection and reportingAnalysis of Performance and Configuration

correlation of events with performance, first and second order analysisProvisioning, Reconfiguration and Migration

Don’t forget to leave an audit trail

Feedback loopWhat where the effects of the change? Check out SNIA Tutorial:

Storage Virtualization II –Effective Use of Virtualization


5. Model based Optimization and Planning

Moving Towards a Real-Time Datacenter -Constantly Increase Operational Efficiency

Most working infrastructure is sub-optimizedDedicated resources “If it ain’t broke, don’t fix it” attitudes (or capabilities)

However, when everything is shared, everyone goes down together…Real-er Time Capacity Planning

Utilizations are related to Response Time through Queuing TheoryNeed to predict performance degradation under future application load changesNeed to predict performance improvements from possible architectural/technology changes

Planning and tuning will go from large cyclical events to smaller, more dynamic perturbations


Queuing Theory to The Rescue…

Queuing Models create Response Time curvesBased on established mathematics (Buzen, et.al – see www.cmg.org )

Useful analytically (historically) as well as predictivelyFor a simple example think of a check-out line at the grocery store

Complex Queuing Network Models can representnested and virtualized IT domains

Advanced cross-domain solutions model IT virtualization

http://www.cmg.org/



In Summary -1. Cross Domain Analysis and Shared Resource Contention

Virtualization is about sharing across IT domains,and that’s often the problem

2. Adopt an Application View of PerformanceManage to customer service levels

3. Use Automation WiselyDoing more with less time and fewer errors

4. “Effective Capacity” ManagementShared resources still obey the laws of physics

5. Model-based Optimization and PlanningLeverage Prediction to Improve your Future


Q&A / Feedback

Please send any questions or comments on this presentation to SNIA: [email protected]

Many thanks to the following individuals for their contributions to this tutorial.

- SNIA Education Committee

Rich Corley Rob StrechayKen Hu Brian SchoferTom Joyce Lisa CreweKevin Faulkner

mailto:[email protected]

Documents

Peter Lauterbach, Akorri - Storage Networking Industry ......We will discuss two major IT virtualization initiatives. Storage Virtualization Server Virtualization (and the combination