VMWare + EMC storage.pdf

8/10/2019 VMWare + EMC storage.pdf

1/102

EMC Corporation

Corporate Headquarters:

Hopkinton, MA 01748 -91031-508 -435 -1000

www.emc.com

EMC Host Connectivity Guide

for VMware ESX Server v2.x

P/N 300-002-304REV A02


2/102

EMC Host Connectivity Guide for VMware ESX Server v2.xii

Copyright 2001-2005 EMC Corporation. All rights reserved.

Updated March 2005

EMC believes the information in this publication is accurate as of its publication date. Theinformation is subject to change without notice.

THE INFORMATION IN THIS PUBLICATION IS PROVIDED "AS IS." EMC CORPORATIONMAKES NO REPRESENTATIONS OR WARRANTIES OF ANY KIND WITH RESPECT TO THEINFORMATION IN THIS PUBLICATION, AND SPECIFICALLY DISCLAIMS IMPLIEDWARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE.

Use, copying, and distribution of any EMC software described in this publication requires anapplicable software license.

Trademark Information


3/102

EMC Host Connectivity Guide for VMware ESX Server v2.x iii

Preface ............................................................................................................................ ix

Chapter 1 Introduction

Purpose of this Document.............................................................. 1-2Related Documentation .................................................................. 1-2VMware ESX Server v2.x Features................................................ 1-3

VMotion ..................................................................................... 1-3Raw Device Mappings ............................................................. 1-3

Chapter 2 Installation Information for VMware ESX Server v2.xwith Symmetrix and CLARiiON Arrays

Overview........................................................................................... 2-2VMware ESX Server Options.................................................. 2-4VMkernel Parameters .............................................................. 2-8

Driver Configuration in VMware ESX Server v2.x................... 2-23Rescanning the SCSI Bus .............................................................. 2-26Persistent Binding.......................................................................... 2-30LUN Spanning................................................................................ 2-34

Setting Up and Installing Virtual Machines............................... 2-37Additional Notes............................................................................ 2-45

Chapter 3 VMware ESX Server v2.x and Symmetrix Storage ArrayFibre Channel Environment

VMware/Symmetrix Environment............................................... 3-2Host and Hardware Connectivity.......................................... 3-2

Symmetrix Connectivity.......................................................... 3-2Booting the VMware ESX Server v2.5 from the Symmetrix ...... 3-6

Contents


4/102

EMC Host Connectivity Guide for VMware ESX Server v2.xiv

Contents

Fabric Zoning ................................................................................... 3-8

Required Storage System Configuration...................................... 3-8Useful VMware ESX Server Utilities and Functions .................. 3-9Addressing Symmetrix Devices .................................................. 3-10

Fabric Addressing .................................................................. 3-10SCSI-3 FCP Addressing ......................................................... 3-11

Chapter 4 VMware ESX Server v2.x and CLARiiON Storage ArrayFibre Channel Environment

VMware/CLARiiON Environment .............................................. 4-2Host and Hardware Connectivity.......................................... 4-2CLARiiON Connectivity ......................................................... 4-2Booting the VMware ESX Server v2.5 from CLARiiONStorage Arrays .......................................................................... 4-5Fabric Zoning............................................................................ 4-7Required Storage System Configuration .............................. 4-7

Useful VMware ESX Server Utilities and Functions ........... 4-7Failover Functionality in VMware ESX Server v2.x for Attachto CLARiiON Storage Arrays ........................................................ 4-9Manual Registration of VMware ESX Server v2.x for Attachto CLARiiON Storage Arrays ...................................................... 4-13Adding the VMware ESX Server Host to a Storage Group..... 4-20Manual Trespass on CLARiiON Arrays to Recover theOriginal Path .................................................................................. 4-21

Index................................................................................................................................ i-1


5/102

EMC Host Connectivity Guide for VMware ESX Server v2.x v

2-1 VMware ESX Server Options ...................................................................... 2-42-2 Startup Profile ............................................................................................... 2-52-3 Security Settings ........................................................................................... 2-72-4 VMkernel Parameter Listing ....................................................................... 2-9

2-5 Storage Management > Disks and LUNs Tab ........................................ 2-372-6 Storage Management > Disks and LUNs > Core Dump Partition ...... 2-382-7 Access Mode ................................................................................................ 2-392-8 Add Virtual Machine Dialog .................................................................... 2-402-9 Status Monitor > Virtual Machines .......................................................... 2-402-10 VMware Remote Console > Connect to Virtual Machines ................... 2-412-11 Configuration Editor, Hardware, Connection ....................................... 2-433-1 One Host, Two Switches, and One Symmetrix Array ............................ 3-4

3-2 One Host, Two Switches, and Four Symmetrix Arrays .......................... 3-54-1 One Host, Two Switches, and One CLARiiON Array ............................ 4-34-2 One Host, Two Switches, and Four CLARiiON Arrays ......................... 4-44-3 Connectivity Status dialog ........................................................................ 4-15

Figures


6/102

vi EMC Host Connectivity Guide for VMware ESX Server v2.x

Figures


7/102

EMC Host Connectivity Guide for VMware ESX Server v2.x vii

2-1 VMkernel Parameters, Descriptions, and Default Values .................... 2-123-1 Symmetrix SCSI-3 Addressing Modes .................................................... 3-124-1 Useful VMware ESX Server Utilities and Functions ............................... 4-7

Tables


8/102

EMC Host Connectivity Guide for VMware ESX Server v2.x

Tables

viii


9/102

EMC Host Connectivity Guide for VMware ESX Server v2.x ix

Preface

As part of an effort to improve and enhance the performance and capabilitiesof its product line, EMC from time to time releases revisions of its hardwareand software. Therefore, some functions described in this document may not

be supported by all revisions of the software or hardware currently in use.For the most up-to-date information on product features, refer to your

product release notes.

If a product does not function properly or does not function as described inthis document, please contact your EMC representative.

This guide describes the features and setup procedures for VMware

ESX Server host interfaces to EMC Symmetrix and CLARiiONStorage Array systems.

Audience This guide is intended for use by storage administrators, systemprogrammers, or operators who are involved in acquiring, managing,or operating Symmetrix, CLARiiON, and host devices.

Readers of this guide are expected to be familiar with the followingtopics:

Symmetrix or CLARiiON system operation

VMware ESX Server operating environment


10/102

x EMC Host Connectivity Guide for VMware ESX Server v2.x

Preface

Organization Here is an overview of where information is located in this guide.

Chapter 1, Introduction This document is meant to assist in theinstallation and configuration of VMware ESX Server v2.x attached toEMC CLARiiON and Symmetrix storage systems.

Chapter 2, Installation Information for VMware ESX Server v2.x withSymmetrix and CLARiiON Arrays This chapter covers installationinformation for the VMware ESX Server v2.x with Symmetrix andCLARiiON arrays.

Chapter 3, VMware ESX Server v2.x and Symmetrix Storage Array FibreChannel Environment This chapter lists Symmetrix Fibre Channelsupport information specific to the VMware ESX Server environment.

Chapter 4, VMware ESX Server v2.x and CLARiiON Storage Array FibreChannel Environment This chapter lists CLARiiON Fibre Channelsupport information specific to the VMware ESX Server environment.


11/102

EMC Host Connectivity Guide for VMware ESX Server v2.x xi

Preface

Conventions Used inThis Guide

EMC uses the following conventions for notes and cautions.

A note presents information that is important, but not hazard-related.

CAUTION

A caution contains information essential to avoid damage to thesystem or equipment. The caution may apply to hardware orsoftware.

Typographical ConventionsEMC uses the following type style conventions in this guide:

AVANTGARDE Keystrokes

Palatino,bold

Dialog box, button, icon, and menu items in text Selections you can make from the user interface,

including buttons, icons, options, and fieldnames

Palatino,italic

New terms or unique word usage in text Command line arguments when used in text Book titles

Courier,italic

Arguments used in examples of command linesyntax.

Courier System prompts and displays and specificfilenames or complete paths. For example:

working root directory [/user/emc]:

c:\Program Files\EMC\Symapi\db

Courier,

bold

User entry. For example:

symmpoll -p

Options in command line syntax


12/102

xii EMC Host Connectivity Guide for VMware ESX Server v2.x

Preface

Where to Get Help For questions about technical support, call your local sales office orservice provider.

If you have a valid EMC service contract, contact EMC CustomerService at:

Follow the voice menu prompts to open a service call and select theapplicable product support.

If you are located outside the North America, call the nearest EMCoffice for technical assistance.

Sales and Customer

Service Contacts

For the list of EMC sales locations, please access the EMC home page

at:http://www.emc.com/contact/

For additional information on the EMC products and servicesavailable to customers and partners, refer to the EMC Powerlink Website at:

http://powerlink.emc.com

Your Comments Your suggestions will help us continue to improve the accuracy,organization, and overall quality of the user publications. Please senda message to [email protected] your opinions ofthis guide.

United States: (800) 782-4362 (SVC-4EMC)

Canada: (800) 543-4782 (543-4SVC)

Worldwide: (508) 497-7901


13/102

Introduction 1-1

1Invisible Body Tag

This document is meant to assist in the installation and configuration of

VMware ESX Server v2.x attached to EMC CLARiiON and Symmetrix

storage systems.

Purpose of this Document ................................................................1-2 Related Documentation.....................................................................1-2 VMware ESX Server v2.x Features ..................................................1-3

Introduction


14/102

1

1-2 EMC Host Connectivity Guide for VMware ESX Server v2.x

Introduction

Purpose of this Document

This document is meant to assist in the installation and configuration of

VMware ESX Server v2.x attached to EMCCLARiiONand Symmetrix

storage systems.

Related Documentation

For the EMC Support Matrix (ESM)for supported server and HBAcombinations, go to:

http://powerlink.EMC.com/

For Navisphereinformation, refer to the Navisphere ManagerAdministrators Guide.

For VMware specific documentation, such as the VMware ESX Server

Release Notes, ESX Server Administration Guide, and the ESX ServerInstallation Guide, go to:

http://www.vmware.com/support/


15/102

1

VMware ESX Server v2.x Features 1-3

Introduction

VMware ESX Server v2.x Features

VMware ESX Server is similar to Linux, but is its own operatingsystem. VMware GSX Server is an application that runs on top of astandard Linux or Windows operating system.

VMware ESX Server consists of virtualization software that providesserver consolidation and allows several instances of operatingsystems running as virtual machines on one physical machine.

Currently, VMware ESX Server installs and operates as a 32-bitoperating system on Intel-based and Opteron-based servers. Theoperating systems qualified to run as virtual machines include 32-bitversions of Linux, Windows, and NetWare.

VMotion VMotion is part of VMware Virtual Center, a systems managementand provisioning tool used by system administrators. VMotiontechnology allows the migration of virtual machines from onephysical server to another physical server. For instance, VMotioncould be used to migrate virtual machines from one server to anotherso that maintenance may be performed on the original server. Thisallows the operating systems running in the virtual machinepartitions to continue running their workloads.

Raw Device

Mappings

VMware ESX Server v2.5 introduced a new technology called raw

device mapping. Essentially, raw device mapping is a SCSIpass-through technology that allows Virtual Machines to pass SCSIcommands directly to the physical hardware.

Note that using raw disk mappings in Shared mode with Fibre ChannelHBAs is not supported. The HBAs must be allocated solely to the VirtualMachines in order to use raw disks.


16/102

1


Introduction


17/102

Installation Information for VMware ESX Server v2.x with Symmetrix and CLARiiON Arrays 2-1

2Invisible Bo

This chapter covers installation information for the VMware ESXServer v2.x with Symmetrix and CLARiiON arrays.

Overview.............................................................................................2-2 Driver Configuration in VMware ESX Server v2.x.....................2-23 Rescanning the SCSI Bus.................................................................2-26 Persistent Binding ............................................................................2-30 LUN Spanning..................................................................................2-34 Setting Up and Installing Virtual Machines.................................2-37 Additional Notes..............................................................................2-45

Installation Information for

VMware ESX Server v2.x withSymmetrix and CLARiiON Arrays


18/102

2


Installation Information for VMware ESX Server v2.x with Symmetrix and CLARiiON Arrays

Overview

The fibre channel HBA driver functions as a device driver layerbelow the standard VMware SCSI adapter driver. The Fibre Channelinterface is therefore transparent to the VMware disk administrationsystem.

Note:VMware ESX Server is NOT Linux and, therefore, requires its ownEMC-qualified drivers, not the standard EMC-qualified Linux drivers.

Install VMware ESX Server from the CD. The VMware ESX Serverinstallation is based on a modified Red Hat v7.2 v2.4.9 kernel.However, VMware ESX Server is NOT Linux.

VMware recommends partitioning the internal hard drive into fourpartitions. For example:

Note that a swap partition is required by both the service console and theVMkernel. The reference to the swap partition in this example is for theservice console. The swap partition and core dump partitions for theVMkernel can be created using the GUI after the initial installation of theservice console OS.

As stated in the Preface, EMC does not recommend fabric boot of theVMware ESX Server itself over the fabric, but we do support fabric boot with

VMware ESX Server v2.5. The Virtual Machines are expected to boot over thefabric using the QLogic or Emulex HBAs.

Refer to the ESM for specifics on which operating systems aresupported as Virtual Machines.

You may customize the installation according to your server and theamount of memory and hard disk space you have available.

/boot 50 MB primary

swap 256 MB to 1 GB primary

/ 2.5 GB primary

/vmimages 4 GB


19/102

2

Overview 2-3


The following are a few recommendations for the installation:

Use static IP addresses. Set the hardware clock when prompted.

Create at least one user account other than root.

Note that emacs, samba, and NFS are not enabled by default in theConsole OS.

Reboot the system after completing the installation.

For VMware ESX Server v2.1.x installations, when the systemreboots, you will be prompted with two options in the LILO bootprompt:

linux-up,linux

Assuming the system on which you are installing has multiple CPUs,the default boot image will be linux. This is expected andacceptable to use.

For VMware ESX Server v2.5 installations, when the system reboots,you are prompted with three options in the LILO boot prompt:

esx, linux-up,linux

The default boot image for VMware ESX Server v2.5 is esx.

You can now log in to the VMware server using IE or Netscape with

the host name or IP address. This allows you access to the VMwarehost using the Management User Interface (MUI).

After you log into the host, the first dialog is the VMware ESX ServerStatus Monitordialog.

The Status Monitordialog provides a summary reporting of thesystem CPU and processor usage by the Virtual Machines as well as avisual status report of the Virtual Machines themselves.


20/102

2



VMware ESX Server Options

Select the Options tab. The Optionsdialog provides you with a listof actions that can be performed to configure or modify the VMwareESX Server, as shown in Figure 2-1.

Figure 2-1 VMware ESX Server Options

From the Optionstab, select the Startup Profilemenu. The StartupProfile creates the VMware ESX Server boot configuration and allows

you to specify whether the server resources should be allocated to theService Console, the Virtual Machines, or shared between the two.

Fibre Channel and SCSI HBAs can be allocated for use with theVirtual Machines only or can be shared between the Virtual Machinesand Service Console. In the case of booting from the fabric, the FCHBAs must be shared so that the Service Console and the VirtualMachines can access the boot device.

Networking cards cannot be shared between the Service Console andthe Virtual Machines so they must be allocated either to the Service


21/102

2

Overview 2-5


Console or to the Virtual Machines. Therefore, a minimum of twoNICs must be installed in the system.

For an example of the Startup Profiledialog, see Figure 2-2.

Figure 2-2 Startup Profile

After the changes appropriate to your system are made, save theconfiguration by clicking OK.

After saving the configuration changes, reboot the host so that thechanges can take effect.


22/102

2



After the VMware host reboots, proceed with the VMkernelconfiguration.

The Network Connectionsmust be set up according to thenetwork interface cards (NICs) being used and the customerenvironment. Note that the server must have at least two NICs.Because NICs may not be shared, one must be allocated to theConsole and at least one must be allocated to the VirtualMachines.

It is not necessary to reboot the VMware ESX Server into the esxkernelin order for network configuration changes to take effect.

The Security Settingsselected are entirely dependent upon thecustomers desires. VMware recommends that customers use theHigh Security setting so all connections into ESX will beencrypted. If FTP and Telnet are required, the Medium Securitysetting will need to be selected. The Security Settingdialog is

shown in Figure 2-3 on page 2-7


23/102

2

Overview 2-7


Figure 2-3 Security Settings

Swap Configuration creates, modifies, and activates swap filesthat allow your virtual machines to use more memory than isphysically available. Whether or not you want to enable a swapfile depends on the amount of memory in the server. It isrecommended that a swap file be created so that the virtualmachines will be able to use more memory than is actuallyavailable in the server.

When booting from an internal hard drive, it is recommended that theswap file resides on the internal device, rather than a SAN-attacheddevice.


24/102

2



If required, the SNMP agent is enabled in the SNMPConfigurationsection.

The Advanced Settingssection allows you to modify theVMkernel parameters. Currently, EMC tests with and supportsthe majority of the default VMkernel parameters. The tablebelow shows the parameters, their descriptions, and the VMwareand EMC default settings. It is highly recommended that theEMC default parameters are used when attached to EMC StorageArrays.

VMkernel Parameters

A number of parameters define the behavior of the VMkernel.Currently, EMC recommends that the majority of the parametervalues remain the same as the VMware defaults. The exception atthis time is the Disk.MaxLunparameter.

These values can be viewed and modified using the MUI and in thesubdirectories within the /proc/vmware/configdirectory.

To view the listing of directories in the /proc/vmware/configdirectory, change to the directory and perform a listing:

cd /proc/vmware/config

ls la

dr-xr-xr-x 2 root root 0 Jan 18 06:07 Cpudr-xr-xr-x 2 root root 0 Jan 18 06:07 Disk

dr-xr-xr-x 2 root root 0 Jan 18 06:07 FileSystemdr-xr-xr-x 2 root root 0 Jan 18 06:07 Irqdr-xr-xr-x 2 root root 0 Jan 18 06:07 Memdr-xr-xr-x 2 root root 0 Jan 18 06:07 Migratedr-xr-xr-x 2 root root 0 Jan 18 06:07 Miscdr-xr-xr-x 2 root root 0 Jan 18 06:07 Netdr-xr-xr-x 2 root root 0 Jan 18 06:07 Numadr-xr-xr-x 2 root root 0 Jan 18 06:07 Scsi

These directories provide the types of parameters that are listed ineach. For instance, the /proc/vmware/config/Diskdirectorycontains the parameters pertaining to the Disk subsystem. Anychanges made to the parameters in this directory will be specific toDisks. For example, Disk.MaxLUNis part of the/proc/vmware/configfile located in the Disk directory and theparameter itself is theMaxLUNsetting.

If desired, changes to the values of these parameters made byperformed using the MUI. Log into the server via the MUI and selectthe Optionstab. In the line for Advanced Settings | Service Console


25/102

2

Overview 2-9


Settings, select Advanced Settings. You will then be presented with awindow that contains the parameters and their values and

descriptions of the parameters.

The parameters are listed in alphabetical order. An example of theMUI dialog listing the beginning of the parameter list is shown inFigure 2-4.

Figure 2-4 VMkernel Parameter Listing

The values of the VMkernel parameters also can be set at thecommand line. This is done by echoing the /proc filesystem. Asmentioned previously, the parameters are ultimately listed in the/proc/vmware/configdirectory. The following exampledemonstrates the steps to view and change the value of a particularsetting.

I t ll ti I f ti f VM ESX S 2 ith S t i d CLARiiON A


26/102

2



In this particular example, the LogMultiPathparameter is beingenabled.

[root@l82bi199 /]# cd /proc/vmware/config/Scsi[root@l82bi199 Scsi]# lsConflictRetries LogAborts LogMultiPath PassthroughLocking[root@l82bi199 Scsi]# cat LogMultiPathLogMultiPath (Log path state changes) [default = 0]: 0[root@l82bi199 Scsi]# echo 1 >> LogMultiPath[root@l82bi199 Scsi]# cat LogMultiPathLogMultiPath (Log path state changes) [default = 0]: 1

If the parameter is changed by echoing the/procat thecommand line, this change will not persist through a reboot.

Although it is not recommended, if you prefer to use thecommand line, you may modify the /etc/rc.localfile so thatthe changes you have via echoing /procat the command line willbe run upon a reboot.

The /etc/rc.localfile is that script that contains your

initialization parameters and is run after all of the standardsystem init scripts. It is recommended that you append yourmodifications to the end of the file.

For example, see the end of the /etc/rc.localfile for theaddition of the /proc modifications:

[root@l82bi199 root]# more /etc/rc.local

#!/bin/sh## This script will be executed *after* all the other init scripts.# You can put your own initialization stuff in here if you don't# want to do the full Sys V style init stuff.

# BEGINNING_OF_VMWARE_RC_DOT_LOCALif ( uname -a | grep -q vmnix ); thenR="VMware ESX Server 2.5.0"else

R="Linux"fiarch=$(uname -m)a="a"case "_$arch" in _a*) a="an";; _i*) a="an";;esac

NUMPROC=`egrep -c "^cpu[0-9]+" /proc/stat`if [ "$NUMPROC" -gt "1" ]; then

Installation Information for VMware ESX Server v2 x with Symmetrix and CLARiiON Arrays


27/102

2

Overview 2-11


SMP="$NUMPROC-processor " if [ "$NUMPROC" = "8" -o "$NUMPROC" = "11" ]; then

a="an" else a="a" fifi

# This will overwrite /etc/issue at every boot. So, make any changesyou

# want to make to /etc/issue here or you will lose them when you reboot.echo "" > /etc/issue

echo "$R" >> /etc/issueecho "Kernel $(uname -r) on $a $SMP$(uname -m)" >> /etc/issue

cp -f /etc/issue /etc/issue.netecho >> /etc/issue

#Echo the /proc filesystem to enable the#multipath logging. This will ensure that#the change persists through reboots.

echo 1 >> /proc/vmware/config/Scsi/LogMultiPath

# END_OF_VMWARE_RC_DOT_LOCAL

By default, the maximum number of LUNs scanned per disk array byESX v2.x is 8. Depending upon the EMC storage array configurationand the devices that are being allocated to the VMware host, it isrecommended that the following parameter be changed in the

VMkernel configuration to accommodate the LUNs allocated to thehost.

For example:

Disk.MaxLun == 128(The default is 8.)

If the Symmetrix or CLARiiON Storage Group has LUNs allocated tothe VMkernel addressable as greater than eight, then this parameter

must be increased. The DiskMaxLunparameter can be as large as 255.However, the maximum number of LUNs that can be managed by asingle VMware ESX Server is 128. The VMware ESX Server willsimply stop scanning the SCSI bus once it has seen 128 actual LUNs.

Please note that increasing the Disk.MaxLunparameter may slow down theSCSI scanning time.

In VMware ESX Server v2.1.x, the VMkernel is configured withsupport for sparse LUNs by default. The Disk.SupportSparseLun

Installation Information for VMware ESX Server v2 x with Symmetrix and CLARiiON Arrays


28/102

2



parameter allows the VMkernel to scan past non-existent LUNs, thatis, allows LUN skipping. When attaching to EMC storage arrays, it is

strongly recommended that the Disk.SupportSparseLunparameterbe set at the default of 1.

The following table lists the VMkernel parameters with theircorresponding descriptions and the VMware default and EMCrecommended values.

Table 2-1 VMkernel Parameters, Descriptions, and Default Values

VMkernel Parameter and DescriptionVMwareDefaultValue

EMCRecommended

Value

Cpu.BoundLagQuanta

Number of global quanta before bound lag [1-100]

8 8

Cpu.CellMigratePeriodMilliseconds between opportunities to migrate across cells 1000 1000

Cpu.ConsoleMinCpu

Min percentage of CPU 0 to dedicate to console [0-100]

8 8

Cpu.ConsoleOSWarpPeriod

Period in milliseconds [0-100]

20 20

Cpu.CreditAgePeriod

Period in milliseconds [500-10000]

3000 3000

Cpu.IdlePackageRebalancePeriod

Usec between chances to rebalance idle packages (0 to disable, 100000 max)

541 541

Cpu.MachineClearThreshold

Machine clears per million cycles to trigger quarantine

100 100

Cpu.MigratePenalty

Penalty in milliseconds [0-2000]

100 100

Cpu.MigratePeriod

Milliseconds between opportunities to migrate across CPUs

20 20

Cpu.PreemptPenalty

Penalty in milliseconds [0-2000]

10 10

Cpu.Quantum

Quantum in milliseconds [1-1000]

50 50



29/102

2

Overview 2-13


Cpu.RunnerMovePeriod

Milliseconds between opportunities to move currently-running vcpu

200 200

Cpu.SharesPerVcpuHigh

Shares per vcpu for high CPU priority [100-10000]

2000 2000

Cpu.SharesPerVcpuLow

Shares per vcpu for low CPU priority [100-10000]

500 500

Cpu.SharesPerVcpuNormal

Shares per vcpu for normal/default CPU priority [100-10000]

1000 1000

Cpu.SkewSampleThreshold

Number of skew samples allowed before co-deschedule (0 to disable skew checking)

3 3

Cpu.SkewSampleUsecInterval between vsmp skew tests

500 500

Disk.DelayOnBusy

Delay in milliseconds for completion of commands with a BUSY status

400 400

Disk.MaskLUNs

LUNs to mask from kernel. Format: ::

[empty] [empty]

Disk.MaxLUNMaximum number of LUNs per target that we scan for

8 128Default: 8

Disk.MaxResetLatency

Delay in milliseconds before logging warnings and spawning new reset worlds if a reset isoverdue or taking too long

2000 2000

Disk.MaxVCNotReadyTime

The number of seconds a VC path can report NOT READY status before being declared dead

60 60

Disk.PathEvalTime

The number of seconds between FC path evaluations

300 300

Disk.ResetLatency

Delay in milliseconds between reset thread wake-ups

1000 1000

Disk.ResetMaxRetries

Max number of bus reset retries (0=infinite)

0 0

Table 2-1 VMkernel Parameters, Descriptions, and Default Values (continued)

VMkernel Parameter and Description VMwareDefaultValue

EMCRecommendedValue



30/102

2


y y

Disk.ResetOnFailover

Issue a SCSI reset when failing over to an alternate HBA

0 0

Disk.ResetOverdueLogPeriod

Delay in seconds between logs of overdue reset

60 60

Disk.ResetPeriod

Delay in seconds between bus resets retries

30 30

Disk.ResetThreadExpires

Life in seconds of an inactive reset handle thread

1800 1800

Disk.ResetThreadMax

Max number of reset handler threads

16 16

Disk.ResetThreadMinMin number of reset handler threads

1 1

Disk.RetryUnitAttention

Retry all SCSI commands that return a unit attention error

1 1

Disk.SANDevicesWithAPFailover

SAN devices with Active/Passive Path Failover. Format: ::...

[empty] [empty]

Disk.SchedNumReqOutstanding

Number of outstanding commands to a target with competing worlds

16 16

Disk.SchedQControlSeqReqs

Number of consecutive requests from a VM required to raise the outstanding commands tomax

128 128

Disk.SchedQControlVMSwitches

Number of switches between commands issued by different VMs required to reduceoutstanding commands to CONFIG_DISK_CIF

6 6

Disk.SchedQuantum

Number of consecutive requests from one World

8 8

Disk.SectorMaxDiff

Distance in sectors at which disk BW sched affinity stops

2000 2000

Disk.SharesHigh

Shares for high disk priority [100-10000]

2000 2000



EMCRecommendedValue

2Installation Information for VMware ESX Server v2.x with Symmetrix and CLARiiON Arrays


31/102

2

Overview 2-15

Disk.SharesLow

Shares for low disk priority [100-10000]

500 500

Disk.SharesNormal

Shares for normal/default disk priority [100-10000]

1000 1000

Disk.SupportSparseLUNSupport for sparse LUNs if set to one

1 1

Disk.UseDeviceReset

Use device reset (instead of bus reset) to reset a SCSI device

1 1

Disk.UseLunReset

Use LUN reset (instead of device/bus reset) to reset a SCSI device

1 1

Disk.UseReportLUNUse the REPORT LUN command to speed up scanning for devices

1 1

FileSystem.LockRetries

Maximum number of retries when encountering file system lock

15 15

FileSystem.RemoteFSEnabled

Enable remote VMFS support

0 0

Irq.BestVcpuRouting 1 1

Irq.IRQMaxLoadPct

Maximum percentage of a CPU's resources that should be devoted to interrupts

70 70

Irq.IRQRebalancePeriod

Time in ms between attempts to rebalance interrupts

500 500

Irq.RoutingPolicy

Policy for interrupt routing, 1 for idle-routing, 2 for random, 0 to disable moving IRQs

1 1

Mem.AdmitHeapMin

Free heap space required to power on VM, in KB [256-10240]

1024 1024

Mem.AllocHighThreshold

Threshold (in MB) at which we start allocating memory above 4GB

768 768

Mem.BalancePeriod

Period in seconds [1-120], 0 to disable

15 15



EMCRecommendedValue


32/102



33/102

2

Overview 2-17

Mem.SwapFileOnSAN

Enables swap files on the SAN [0-1]

1 1

Mem.SwapIORetry

Retry timeout for swap I/O in seconds [1-2678000], 0 to disable timeout

0 0

Mem.SwapSharedSet the max ref count of a swappable shared page [0-100]

2 2

Migrate.Enabled

Enable hot migration support

0 0

Migrate.MemChksum

Checksum VM's memory while migrating

0 0

Migrate.MinReservationReserve a percentage of a CPU for use by migration helper worlds[0, 100]

30 30

Migrate.NetTimeout

Timeout for migration network operations

10 10

Migrate.PageInProgress

Time in seconds after which a pagein will be killed if there is no progress[5-1800]

15 15

Migrate.PageInTimeoutTime in seconds to wait for pagein to finish [10-1800]

180 180

Migrate.PreCopyLeftMB

Maximum modified memory left over after pre-copy is done

16 16

Migrate.PreCopyMinProgressMB

Minimum reduction in modified memory after a pre-copy iteration

16 16

Migrate.TSMasterPseudo-synchronize clocks for migration to/from this machine[0, 1]

0 0

Misc.BlueScreenTimeout

Timeout in seconds, 0 is no timeout

0 0

Misc.ConsoleShareXfer

Transfer shares to console on RPC calls

1 1

Misc.CosCorefile

VMFS to write the Service Console core file to [must be symbolic filesystem name]

[empty] [empty]



EMCRecommendedValue



34/102

2


Misc.DebugMemEnable

Enable memory debugging, 0 to disable

0 0

Misc.EnableHighDMA

Enable DMA above 4GB

0 0

Misc.KVMapGuardUncachedUse guard pages around uncached kvmap mappings, [0-1]

1 1

Misc.LogToFile 1 1

Misc.LogToSerial 1 1

Misc.LogWldPrefix 0 0

Misc.MemAdmitMapEntriesMin

Free KVMap entries required to power on VM, [0-100]

30 30

Misc.MemMapEntriesLow

Point at which to start conserving KVMap entries, [0-1024]

200 200

Misc.MinimalPanic 0 0

Misc.ProcVerbose

Option unused

[empty] [empty]

Misc.PsodOnCosPanic

PSOD VMkernel on Console OS panic / oops [0-1]

0 0

Misc.SerialBaudRate 115200 115200

Misc.SerialPort 1 1

Misc.TimerHardPeriod

Hard timer interrupt period in microseconds

1000 1000

Misc.TimerMinGuestPeriod

Minimum period for guest timer callbacks in microseconds

100 100

Misc.VmkperfPerWorld

Should performance counters be maintained per-world [0-1]

0 0

Net.ClusterHaltCheck

1 to check for clustered tx/rx packets on halt

1 1



EMCRecommendedValue



35/102

2

Overview 2-19

Net.CopiesBeforeRemap

Copies before remapping, 0 to disable

10 10

Net.DisableWatchdog

Disable network watchdog timeout handler

0 0

Net.MaxMallocPacketsMaximum number of packets that can be allocated via the memory allocator

256 256

Net.MaxPrivateSKBs 256 256

Net.MaxRecvPackets 64 64

Net.NotifySwitch

Broadcasts an arp request on net handle enable [0 = disabled, 1 = enabled]

1 1

Net.PktsDeferXmitIntIfKeeping

Defer xmit completion interrupts if this many packets still pending

6 6

Net.PktsPerXmitInterrupt

Packets to transmit before raising completion interrupt

5 5

Net.ProcVerbose

Verbose procfs output for networking, 0 to disable

0 0

Net.RXClusterCPUSamplePeriod

CPU %idle sample period (msec) [1-10000]

1000 1000

Net.RXClusterDelayOff

Weighting factor for deactivating clustering

10 10

Net.RXClusterDelayOn

Weighting factor for activating clustering

2 2

Net.RXClusterDelayTDec

Decrement timer freq this often

1000 1000

Net.RXClusterDelayTInc

Increment timer freq this often

0 0

Net.RXClusterTMaxFreq

Max timer freq as a power of two [1-13]

10 10



EMCRecommendedValue



36/102

2


Net.RXClusterTMigrateDelay

Timer CPU migration intervals [1-100]

5 5

Net.RXClusterTMigrateThresh

Timer CPU migration threshold (delta %idle)

20 20

Net.RXClusterTMinFreqMin time freq as a power of two [1-13]

6 6

Net.RXClusterThreshOff

Interrupts/sec to deactivate clustering

2000 2000

Net.RXClusterThreshOn

Interrupts/sec to activate clustering

4000 4000

Net.RXClusterTimerCPUWhich cpu to run the timer on, 0 for migratory timer

0 0

Net.RecvClusterDynAdjRate 50 50

Net.RecvClusterDynQueueMax 20 20

Net.RecvClusterOffCount 30 30

Net.RecvClusterOnCount 40 40

NetRecvClusterQueueMax 10 10

Net.RecvClusterTimeoutCount 1 1

Net.RecvClusterTimeoutMS 10 10

Net.RecvCopyLength 200 200

Net.SwitchFailoverBeaconEtherType

NIC Teaming switch failover beacon Ether Type [0x6000 - 0x9000]

36864 36864

Net.SwitchFailoverBeaconInterval

NIC Teaming switch failover beacon interval [1-60]

1 1

Net.SwitchFailoverThreshold

NIC Teaming switch failover threshold, a positive number to enable switch failover policy

2 2

Net.TCPIPLog

TCP/IP logging, 0 to disable

0 0



EMCRecommended

Value



37/102

Overview 2-21

Net.TcpSegmentationOffload

Allow TSO, 0 to disable

1 1

Net.VlanTrunking

1 to enable 802.1Q VLAN Tagging by VMkernel

1 1

Net.XmitClusterOffCount 30 30

Net.XmitClusterOnCount 40 40

Net.XmitClusterQueueMax 10 10

Net.XmitClusterTimeoutCount 1 1

Net.XmitClusterTimeoutMS 10 10

Net.XmitInterruptIfStoppedRaise an interrupt if a xmit packet is returned and the guest has stopped transmitting

0 0

Net.XmitInterruptTimeout

Timeout in milliseconds to check for completed transmits

10 10

Net.XmitMinWinZeroCopyLen

For windows guests the minimum size of packets to try to avoid the copy

256 256

Net.ZeroSpeedLinkDownZero speed means link down, 1 to enable it

0 0

Numa.AutoMemAffinity

1 to set mem affinity automatically based on cpu affinity, 0 to disable

1 1

Numa.Debug

Level of NUMA scheduler debugging information to print [0-4]

0 0

Numa.MigImbalanceThresholdMinimum deviation in owed ms between nodes, per second, to trigger migration

75 75

Numa.PageMig

1 to permit NUMASched to manipulate page migration, 0 to disallow it

1 1

Numa.Rebalance

1 to use NUMASched rebalancer, 0 to disallow it

1 1

Numa.RebalancePeriodFrequency of NUMA node rebalancing, in milliseconds 2000 2000



EMCRecommended

Value



38/102


It is not necessary to reboot the VMware ESX Server in order for anychanges to the parameter values to take effect.

Scsi.ConflictRetries

Maximum number of retries when encountering reservation conflict

4 4

Scsi.LogAborts

Log timing of aborted or reset I/Os

0 0

Scsi.LogMultiPathLog path state changes

0 0

Scsi.PassthroughLocking 1 1



EMCRecommended

Value


39/102



40/102


The following VMnix kernel images are defined on your system:

Boot image configuration: vmnix Image file: /boot/vmlinuz-2.4.9-vmnix2 Memory: 384M Service Console devices: Ethernet controller: BROADCOM Corporation NetXtreme BCM5703 Gigabit Ethernet

(rev 02) RAID storage controller: Compaq Computer Corporation Smart Array 5i/532 (rev

01) (shared) VM devices: Ethernet controller: BROADCOM Corporation NetXtreme BCM5703 Gigabit Ethernet

(rev 02) Fiber storage controller: QLogic Corp QLA231x/2340 (rev 02) Ethernet controller: Intel Corporation 82557 [Ethernet Pro 100] (rev 08) RAID storage controller: Compaq Computer Corporation Smart Array 5i/532 (rev

01) (shared)

Type in the name of the boot image configuration you wish to configureor type "new" to create a new image [vmnix]:

Selected configuration was 'vmnix'.

Configuring your setup. Enter in the options that you want. Defaults have been provided.

Name of this configuration [vmnix]:

Now allocating memory to the Service Console. The memory that is not allocatedto the Service Console will be allocated to the VMs. Be sure to indicate anumber smaller than you actually have on your system or you may not be able to

run any VMs.

Your system is reporting approximately 2,559 MB of memory.Note: you cannot allocate more than 800 MB to the Service Console.

If you plan to run 4 virtual machines or fewer, 128 MB should suffice.Set this to 192 MB for 8 virtual machines, 272 MB for 16 virtual

machines, 384 MB for 32 virtual machines, or 512 MB for more than32 virtual machines.

Amount of memory (in megs) to allocate to the Service Console [384]:Enable hyperthreading? (Only works with 8 or less physical processors) [y]:

Now to divide the SCSI controllers and network adapters. In this process,you will select whether to give the previous devices to the Service Console orthe virtual machines that you will be running. In order to ensure optimalperformance for your virtual machines, we suggest you allocate as many devicesas possible to the virtual machines, leaving only the ones necessary toconfigure and maintain the Service Console.

Type 'c' to allocate to the Service Console or 'v' to allocate to the virtual



41/102

Driver Configuration in VMware ESX Server v2.x 2-25

machines, or 's' to share between both.

RAID storage controller Compaq Computer Corporation Smart Array 5i/532 (rev 01)running on bus 1 device

3 function 0 (cciss.o) [vmhba4] [s]:Ethernet controller BROADCOM Corporation NetXtreme BCM5703 Gigabit Ethernet (rev

02) running on bus 2device 1 function 0 (bcm5700.o) [] [c]:Ethernet controller BROADCOM Corporation NetXtreme BCM5703 Gigabit Ethernet (rev

02) running on bus 2device 2 function 0 (bcm5700.o) [vmnic0] [v]:Fiber storage controller QLogic Corp QLA231x/2340 (rev 02) running on bus 6 device

1 (qla2300_604.o) [vmhba2] [v]:Ethernet controller Intel Corporation 82557 [Ethernet Pro 100] (rev 08) running

on bus 6 device 2 function0 (e100.o) [vmnic1] [v]:

Configuration complete.Commit changes (y/n)? [y]:



42/102


Rescanning the SCSI Bus

For use in the VMkernel itself, rather than using the standard Linux/procfile system to obtain information on the HBAs, refer to the/proc/vmware/scsidirectory. This directory contains entries foreach HBA in the system allocated to the Virtual Machines. Each HBAis denoted byvmhbaN(where Nindicates the file for each adapterowned by the Virtual Machines).

You may confirm that the QLogic or Emulex driver is loaded in the

VMkernel with the vmkload_mod command. For example, thefollowing command may be issued on this host to list the QLogicdriver:

root@l82bi199 /]# vmkload_mod -lName R/O Addr Length R/W Addr Length ID Loadedvmklinux 0x4dd000 0xf000 0x12414a0 0x53000 1 Yesnfshaper 0x4ec000 0x1000 0x1297f28 0x1000 2 Yesqla2300_604 0x4ed000 0x19000 0x1298f30 0x22000 3 Yes

bond 0x506000 0x2000 0x150e1d8 0x2000 4 Yes [Assuming that the devices have been allocated to the VMware hostand that the VMkernel has been configured appropriately, theStorage Managementdialog in the MUI reports the devices that areallocated to the host.

You may also view the device listing at the command line. This maybe done with the following commands:

1.more /proc/scsi/qla2300/N

where Nis the sequential value of each QLogic HBA installed inthe system, beginning with the number after the last host adapternumber entry in the file. (The first host adapter number entrybegins with zero.) This is the same as it would be for standardLinux.

2.ls la /proc/vmware/scsi/vmhbaN

where Nindicates the file for each adapter owned by theVMkernel and, subsequently, the Virtual Machines

After the host is added to the Storage Group, the SCSI bus needs to berescanned.


43/102



44/102


For example:

vmkfstools -s vmhba0

If you are using the MUI, then go to the Storage Managementdialog and click on the Rescan SANoption to rescan the SCSIbus.

5. For either QLogic or Emulex HBAs, you may also remove andreinsert the driver from the VMkernel to initiate a rescan of theSCSI bus.

For QLogic:vmkload_mod lvmkload_mod u qla2300_604vmkload_mod /usr/lib/vmware/vmkmod/qla2300_604.o vmhba

For Emulex:

vmkload_mod lvmkload_mod u lpfcddvmkload_mod /usr/lib/vmware/vmkmod/lpfcdd.o vmhba

It is recommended in the case of using either the QLogic or Emulex HBAsthat the Rescan SANoption in the MUI be used to rescan the SCSI bus.

For VMware ESX Server v2.5:

1. It is recommended that the MUI be used to rescan the SCSI bus.Simply go to the Storage Managementdialog, and click on theRescan SANoption to rescan the SCSI bus. This process calls thecos-rescan.sh.

2. If the command line is preferred, you may use thecos-rescan.shscript to initiate the rescan.

For example:

[root@l82bi199 /]# cos-rescan.sh vmhba2

Rescanning vmhba2...done.On scsi1, removing:On scsi1, adding:

3. You may reboot the host, but since this is disruptive, it is notrecommended.



45/102

Rescanning the SCSI Bus 2-29

4. If booting from an internal hard drive rather than from the SAN,you may also remove and reinsert the driver from the VMkernel

to initiate a rescan of the SCSI bus. This is pertinent for bothQLogic and Emulex HBAs.

For QLogic:

vmkload_mod lvmkload_mod u qla2300_604vmkload_mod /usr/lib/vmware/vmkmod/qla2300_604.o vmhba

For Emulex:

vmkload_mod lvmkload_mod u lpfcddvmkload_mod /usr/lib/vmware/vmkmod/lpfcdd.o vmhba

In VMware ESX Server v2.5, the vmkfstoolscommand should not beused to rescan the SCSI bus; otherwise, the Service Console will notsee or report the changes made.



46/102


Persistent Binding

EMC highly recommends that persistent binding be enabled inVMware ESX Server v2.x. When persistent binding is enabled, thetarget IDs are assigned to specific storage array devices. Thisassignment persists through reboots so the association remains thesame. This is particularly helpful when using raw devices.

Persistent binding may be enabled or disabled using the MUI or thecommand line.

If using the MUI, select the Optionstab, and then select the StorageManagementtab. Under Storage Management, there are threeoptions: Disks and LUNs, Failover Paths, and Adapter Bindings.To view the current information for your system, select the AdapterBindingsoption.

An example of the output from the Adapter Bindingsdialog isbelow:


T i d h i bi di f h d li


47/102

Persistent Binding 2-31

To view and reset the persistent binding from the command line,follow the steps below.

1. Log in as root.

2. To view the current persistent binding entries for the HBAs in thesystem, look at the pbindingsfile.

An example of what the pbindingsfile may look like is asfollows:

[root@l82bi199 /]# more /etc/vmware/pbindings6.1.1.qla2300_607 = "scsi-qla1-tgt-0-di-0-port=50060160082012bb;scsi-qla1-tgt-1-di-0-port=50060168082012bb;scsi-qla1-tgt-2-di-0-port=5006016010601270;scsi-qla1-tgt-3-di-0-port=5006016810601270;"6.1.0.qla2300_607 = "scsi-qla0-tgt-0-di-0-port=50060168082012bb;scsi-qla0-tgt-1-di-0-port=50060160082012bb;scsi-qla0-tgt-2-di-0-port=5006016010601270;scsi-qla0-tgt-3-di-0-port=5006016810601270;"

3. To remove the existing persistent binding settings, either thepbindingsfile may be deleted or the pbind.plscript may be run.

Thepbind.pl

is located in the/usr/sbin

directory and may berun as root from any directory location.

To delete the file, use the following command:

rm f /etc/vmware/pbindings

To remove the settings using the pbind.plscript, use the Dswitch. This switch deletes the persistent binding information forall of the adapters within the system.

pbind.pl -D

For example:

[root@l82bi199 /]# pbind.pl -DDeleting ALL bindings

4. Reboot the host.

5. After the system reboots successfully, log in and again run thepbind.plscript. This time use the A switch which will add thepersistent binding information for all of the adapters within thesystem.

pbind.pl A


F l


48/102


For example:

[root@l82bi199 /]# pbind.pl -A

+-----------------------------------------+| Setting up the following bindings... |+--------+---------+----------------------+| Host | Target | World Wide Name || 01 | 000 | 50060160082012bb || 01 | 001 | 50060168082012bb || 01 | 002 | 5006016010601270 || 01 | 003 | 5006016810601270 || 00 | 000 | 50060168082012bb || 00 | 001 | 50060160082012bb || 00 | 002 | 5006016010601270 || 00 | 003 | 5006016810601270 |+-----------------------------------------+

6. Edit the vmwareinitialization script. This script is run when thesystem is booting up and starts and manages the services neededby the VMkernel.

vi /etc/init.d/vmware

7. Search for the line containing the textsave_san_persistent_bindings. The code should be locatedapproximately around line 896 in the file.

896 # Save the SAN persistent bindings for the next boot897 save_san_persistent_bindings

8. Comment out the line save_san_persistent_bindings.

This is only a temporary change. When this line is commentedout, the adapter to target bindings will not be saved andtherefore, will not persist through reboots.

896 # Save the SAN persistent bindings for the next boot897 #save_san_persistent_bindings

9. Save the file and reboot the host again.

10. Once the system is rebooted, the persistent binding line will needto be uncommented.

Change to the /etc/init.d/vmwaredirectory again:

vi /etc/init.d/vmware

Uncomment the line for save_san_persistent_bindingsandsave the file.


11 Now run wwpn pl to associate the targets HBA WWPN and


49/102

Persistent Binding 2-33

11. Now runwwpn.plto associate the targets, HBA WWPN, andvmhba.

[root@l82bi199 /]# wwpn.pl -vWWPN 1.02 Copyright VMware 2003Display WW port names and VMHBA information for fibre channel adaptersFor each vmhba here are the corresponding Qlogic and Emulex WW Port NamesAdapter WWPN PCI (decimal)vmhba1: 210000e08b0910a7 (Qlogic) 6:1:0 /proc/scsi/qla2300/0vmhba1:0: 50060168082012bb scsi-qla0-port-0=0000000000000000:50060168082012bb;vmhba1:1: 50060160082012bb scsi-qla0-port-1=0000000000000000:50060160082012bb;vmhba1:2: 5006016010601270 scsi-qla0-port-2=0000000000000000:5006016010601270;

vmhba1:3: 5006016810601270 scsi-qla0-port-3=0000000000000000:5006016810601270;vmhba2: 210100e08b2910a7 (Qlogic) 6:1:1 /proc/scsi/qla2300/1vmhba2:0: 50060160082012bb scsi-qla1-port-0=0000000000000000:50060160082012bb;vmhba2:1: 50060168082012bb scsi-qla1-port-1=0000000000000000:50060168082012bb;vmhba2:2: 5006016010601270 scsi-qla1-port-2=0000000000000000:5006016010601270;vmhba2:3: 5006016810601270 scsi-qla1-port-3=0000000000000000:5006016810601270;

12. The persistent binding is now reset to the most recentinformation.



50/102


LUN Spanning

When a volume is spanned, the volume is extended to includemultiple VMFS-2 disks or partitions. A physical extent is defined bythe VMFS-2 disks or partitions that compose a logical spannedVMFS-2 volume.

There are some restrictions to using spanned volumes. They are:

Spanning volumes is supported only with VMFS-2 volumes, notwith VMFS-1 volumes.

The maximum number of files may not be changed.

A spanned volume may not be removed.

Data on the physical extents are lost when they are added to theVMFS-2 spanned volume. Therefore, it is strongly recommendedthat only newly created partitions or newly added devices areused for spanning volumes.

A VMFS-2 volume may span up to a maximum of 32 LUNs.

To span a volume using the MUI, select the Optionstab then selectStorage Management. Select the Disks and LUNscategory. In theDisks and LUNscategory, you are provided with a list of the devicesthat are assigned to the host. Select the one that you want to expand.The configuration window for the device you chose will be shown onyour screen and will provide you with the specifics on the selected

device plus the devices that may be added to the selected device tocreate a spanned VMFS-2 volume.

See the figure that follows for an example using the devicevmhba1:2:13.



51/102

LUN Spanning 2-35

After you have spanned a VMFS-2 volume, you may view thechanges immediately because the /vmfsdirectory is updateddynamically. The command vmkfstoolsshould be used to list theinformation on the particular device and to provide the spanning andphysical extent information.

For example:

[root@l82bi199 /]# cd /vmfs[root@l82bi199 /]# vmhba1:0:0:1 vmhba1:2:11:1 vmhba1:2:13:1 vmhba1:2:3:1[root@l82bi199 /]# vmkfstools --list vmhba1:2:13:1

Name: 1270data13 (public)Capacity 2826683392 (2679 file blocks * 1048576), 1701838848 (1623 blocks) avail


Permission Uid Gid Attr Bytes (Blocks) Last Modified Filename


52/102


rw------- 0 0 disk 1107296256 ( 1056) Aug 26 06:28 w2k3_data4.dsk

[root@l82bi199 /]# vmkfstools -P vmhba1:2:13:1vmhba1:2:13:1 is a VMFS-2.11 volume spanning 4 physical extents.

Volume label (if any): 1270data13UUID (if any): 409bb858-be8577b8-ccf1-0002b3248a69Physical Extents: vmhba1:2:13:1 vmhba1:2:5:1 vmhba1:2:6:1 vmhba1:2:7:1

To view the same volume information from the MUI, go to theOptionstab and select Storage Management. Select the Disks andLUNscategory and scroll to view the VMFS-2 volume that wasspanned. Please see the reference to Disk vmhba1:2:13in the figurethat follows for an example.


S tti U d I t lli Vi t l M hi


53/102

Setting Up and Installing Virtual Machines 2-37

Setting Up and Installing Virtual Machines

The devices to be used for the Virtual Machines with VMFS areformatted in the Storage Management > Disks and LUNs dialog.

When the Create Volumesoption is selected, you are prompted toselect the method with which the devices will be created.

The Typicaloption formats the entire device.

The Customoption allows you to manually create one or more

partitions according to the size required.Select the Typicalformatting option, as shown in Figure 2-5. If yoursystem has not been configured with a Core Dump Partition, thenyou will be prompted to create one.

Figure 2-5 Storage Management > Disks and LUNs Tab

When booting from an internal hard drive, it is recommended thatthe core dump partition be created on the hard disk internal to thesystem, not a SAN attached device. If booting from a SAN device,

then the core dump partition must be created on a SAN device aswell.

An example of the dialog providing this prompt is in Figure 2-6 onpage 2-38.



54/102


Figure 2-6 Storage Management > Disks and LUNs > Core Dump Partition

Select Yes,and then select Finish.

After the partition has been created, it should be formatted withVMFS. You also can provide it with a Volume Label, or alias, that islisted as the VMFS Name.

For VMFS file systems, you can specify whether the file system isPublic(accessible by multiple physical servers) or Shared(accessibleby multiple physical servers, and can be used simultaneously byvirtual machines on these servers).

The accessibility should be set to the default which is Public in almostall cases. The accessibility should be set to Shared if the VMFS willhold disks to be used for MSCS clustering.

An example of the accessibility setting window is shown inFigure 2-7.



55/102


Figure 2-7 Access Mode

After the devices are formatted with the VMFS file system, the VirtualMachines can be configured and installed.

In the main screen of the VMware ESX Server, go to the StatusMonitortab and select Add Virtual Machine. The Wizard will guideyou through the Virtual Machine configuration.

An example of the Virtual Machine Configurationdialog for aWindows 2000 virtual machine is in Figure 2-8 on page 2-40.



56/102


Figure 2-8 Add Virtual Machine Dialog

In this dialog, you can allocate a name for the virtual machine and alocation for the configuration file.

The name allocated in the Virtual Machinesdialog will be displayedin the Status Monitordialog as shown in Figure 2-9as well as listedin the Remote Console listing:

Figure 2-9 Status Monitor > Virtual Machines


After the Virtual Machines are configured, the OS must be installedon that Virtual Machine To access a virtual machine the VMware


57/102


on that Virtual Machine. To access a virtual machine, the VMwareRemote Console software needs to be installed on either a Windowsor Linux machine.

The Remote Console software is available to download from theStatus Monitordialog, as shown in Figure 2-9 on page 2-40, whichprovides a listing of the Virtual Machines configured on the VMwareServer. The Remote Console software is available as executable forWindows NT/Windows 2000 and as an RPM and a tar archive forLinux. Select and download the appropriate application for your

operating system.

After you have installed the VMware Remote Console software onyour operating system, start the application and log onto theVMware ESX Server.

Figure 2-10is an example of what the dialog will look like.

Figure 2-10 VMware Remote Console > Connect to Virtual Machines

Select a Virtual Machine, and select the Power Onbutton. This willstart up the virtual machine.


58/102


The figure below is an example of what the console will look likewhen using the Editoption.


59/102


g p

Select the option to Use physical drive,and connect the CD-ROMdevice if an actual CD is to be used for the virtual machine OSinstallation.

If an ISO image is to be used for the OS installation on the virtualmachine, select Use ISO image, and browse to the location of the ISOimage. In Figure 2-11, an ISO image is selected for installation.

Figure 2-11 Configuration Editor, Hardware, Connection

After the installation has completed, the Virtual Machine must berebooted to be ready for use.


The following is an example of the Virtual Machine Overviewdialogwith multiple Virtual Machines installed:


60/102


p


Additional Notes


61/102

Additional Notes 2-45

Currently, EMC software will function on neither the ServiceConsole nor the Virtual Machines as the currently-releasedversions of the CLARiiON-based software applications do notinclude support for VMware ESX Server.

Currently, VMware ESX Server v2.x does not support accessingSnapView, MirrorView, or SAN Copy LUNs.

Again, please note that PowerPathis not available for the

VMkernel. The failover supported with VMware ESX Server v2.xand EMC Storage Arrays is native to the VMware ESX Serverv2.x itself. The native failover functionality performs failoveronly, not load balancing.

The VMware ESX Server host needs to be registered manually onthe CLARiiON array as the Naviagent is not supported forVMware ESX Server.

Neither the Naviagent nor the NaviCLI are supported forVMware ESX Server at this time.

It is recommended that Virtual Machines running on VMwareESX Server v2.x boot from the array.

For the Virtual Machines, it is recommended, but not required,that the boot LUNs be in a CLARiiON RAID group that isseparate from that RAID group for the data volumes.

Prior to VMware ESX Server v2.5, the VMware ESX Server itselfwas not supported booting from the array. VMware ESX v2.5adds support for booting from the SAN for the ESX Server itself.

When booting VMware ESX Server v2.5.0 from an EMC storagearray, it is recommended that the boot LUN be in a RAID groupseparate from the data volumes.

When booting the VMware ESX Server from an internal drive, the

VMware Core Dump and Swap Files should be created oninternal drives rather than SAN attached devices.

It is highly recommended that the vmware-tools be installed oneach Virtual Machine. The use of the vmware-tools will save onresources and provide better memory management. Thevmwaretools will also provide a VGA device driver and aheartbeat mechanism to communicate with the VMkernel.


62/102

3

VMware ESX Server v2.x andSymmetrix Storage Array


63/102

VMware ESX Server v2.x and Symmetrix Storage Array Fibre Channel Environment 3-1

This chapter lists Symmetrix Fibre Channel support informationspecific to the VMware ESX Server environment.

VMware/Symmetrix Environment.................................................3-2

Booting the VMware ESX Server v2.5 from the Symmetrix ........3-6 Fabric Zoning......................................................................................3-8 Required Storage System Configuration ........................................3-8 Useful VMware ESX Server Utilities and Functions.....................3-9 Addressing Symmetrix Devices.....................................................3-10

Symmetrix Storage ArrayFibre Channel Environment

3VMware ESX Server v2.x and Symmetrix Storage Array Fibre Channel Environment

VMware/Symmetrix Environment


64/102


This section lists some Symmetrix Fibre Channel support informationspecific to the VMware ESX Server environment.

Host and Hardware Connectivity

Refer to the EMC Support Matrix or contact your EMC representativefor the latest information on qualified hosts, host bus adapters, and

connectivity equipment.

EMC does not support mixing HBAs from different vendors in the same host.

Symmetrix Connectivity

The Symmetrix system is configured by an EMC Customer Engineer

via the Symmetrix service processor.

The EMC Customer Engineer (CE) should contact the EMCConfiguration Specialist for updated online information. Thisinformation is necessary to configure the Symmetrix system tosupport the customers host environment.

When attaching a VMware ESX Server host to a Symmetrix, the Linuxdefault FA settings should be used. They are as follows:

SCP3 SCSI-3

PTOP: Point to Point

VCM: Volume Logix

C: Common Serial Number

EAN: Enable Auto Negotiation

Volume Logix must be enabled on the Symmetrix.After the EMC CE has assigned target IDs and LUNs and brought theSymmetrix channel and disk directors online, reboot the networkoperating systems, and go into the configuration program.

All qualified HBAs are listed in the EMC Support Matrix.


Note that the VMware ESX Server installer will recognize LUNs25MB or less as management LUNs. This includes any gatekeepersassigned to the VMware host via Solutions Enabler


65/102

VMware/Symmetrix Environment 3-3

assigned to the VMware host via Solutions Enabler.

It is recommended that each VMware ESX Server host have amaximum of 8 paths to Symmetrix Storage Arrays. A path is definedby one HBA port that is zoned to one SP port.

With a recommended maximum of 8 paths from a VMware ESXServer to Symmetrix Storage Arrays, it is recommended that eachVMware ESX Server host be zoned to a maximum of 4 storage arrays.

A couple of possible configuration scenarios are described in Example1on page 3-4and Example 2on page 3-5.


Example 1 In this example as shown in Figure 3-1, one host with two HBAs isattached to one Symmetrix array using two separate switches. TwoFA ports on each of two FAs within the array are being used. HBA1 is


66/102


po s o e c o o s e y e e g se szoned to 15D0 and to 16C1. HBA2 is zoned to 15D1 and to 16C0. Thezones should be composed of a single initiator and a single target sothey would be created with one HBA and on FA port. Two HBAswith two paths each to the array totals four paths and if using singleinitiator/single target zoning, there are 4 zones.

In this particular example, two switches are used. Using only oneswitch is supported, but such a configuration would lack

redundancy. Preferably, a minimum of two switches should be usedto add another level of redundancy. Alternatively, for additionalredundancy, two separate fabrics could be utilized.

Figure 3-1 One Host, Two Switches, and One Symmetrix Array

MirroredFA 15D Symm 1

d0 d1 c0 c1

Server

H

B

A

0

H

B

A

1

MirroredFA 16C Symm 1

d0 d1 c0 c1


Example 2 In this example as shown in Figure 3-2, one host with two HBAs isattached using a two-switch fabric to four Symmetrx arrays FA ports.In this configuration, the zones are created with one HBA and one FA


67/102

VMware/Symmetrix Environment 3-5

g

port. That is,

HBA1 is zoned to one 15D0 port on each of the four arrays.

HBA2 is zoned to one 16C1 port on each of the four arrays.

Figure 3-2 One Host, Two Switches, and Four Symmetrix Arrays

All qualified HBAs are listed in the EMC Support Matrix.

When assigning Symmetrix LUNs to a VMware ESX Server host, theLUNs should be assigned to the host across both FAs since theSymmetrix is an active/active array.

M i rr o r edF A 1 5 D S y m m 4

d 0 d 1 c 0 c 1

S e r v e r

H

B

A

0

H

B

A

1

M i r ro r e dF A 1 5 D S y m m 3

d 0 d 1 c 0 c 1

M i r ro r e dF A 1 5 D S y m m 1

d 0 d 1 c 0 c 1M i r ro r e d

F A 1 5 D S y m m 2

d 0 d 1 c 0 c 1

M i r ro r e dF A 1 6 C S y m m 2

d 0 d 1 c 0 c 1


d 0 d 1 c 0 c 1


d 0 d 1 c 0 c 1


d 0 d 1 c 0 c 1


Booting the VMware ESX Server v2.5 from the Symmetrix


68/102


VMware ESX Server v2.5 hosts have been qualified for booting fromSymmetrix devices interfaced through Fibre Channel as specified inthe EMC Support Matrix.

EMC does not recommend booting VMware ESX Server from theSymmetrix system unless the host is using VMwares native failoverfunctionality.

Note that if VMwares native failover functionality is not used and it isnecessary to use a Symmetrix device as a boot disk, you should shut downthe Virtual Machines and the ESX Server during any maintenance proceduresthat might cause the boot disk to become unavailable to the host.

The Symmetrix device that is to contain the Master Boot Record(MBR) for the host must have a lower logical unit number (LUN)than any other device visible to the host. This device must be

mapped as /dev/sdaby the Linux operating system for the boot tosucceed from the device.

Note that this includes the case of the Symmetrix Volume Logixdatabase device. The Volume Logix device is write-protected so theinstaller will fail to write the MBR to this device.

To force the installer to avoid an attempt to write to this device,EMC recommends masking the LUN. The administrative host

must first initialize the Volume Logix database. The LUN maythen be masked by modifying the active configuration file toenable the fba flag2to restrict access to the Volume Logixdatabase.

An alternative to masking the VCM DB, is to map the VolumeLogix database device so that it is the highest LUN presented tothe host However, please be aware that the LUN number shouldnot be higher than 254 (FE) if it is to be used by Solutions Enabler

or Control Center running on a Microsoft host.

The EMC-recommended method is to use LUN masking.

When attaching the VMware host to the Symmetrix storage array, usethe adapter in the lowest-numbered PCI slot in the server.

To keep the configuration and installation simple, it is recommended

that only that HBA be cabled to the array. Ensure that the boot BIOS


or firmware has been applied to your HBA and that the boot BIOShas been enabled on the HBA to be used for boot.


69/102

Booting the VMware ESX Server v2.5 from the Symmetrix 3-7

Prior to the installation on a Symmetrix LUN, the Linux host HBAmust have successfully logged into the array. Using SolutionsEnabler from another host, at least one LUN must be assigned to thehost.

During the installation procedure, it is recommended, but notrequired, that only one LUN be allocated to the host for ease of use.After the installation has completed, additional LUNs can beassigned to the host.

When performing the installation, you will need to manually typebootfromsanor bootfromsan-textat the installation prompt.

By selecting this option, the HBAs will automatically be set to beshared between the Service Console and the Virtual Machines.Remember that this parameter should not be changed when creatingyour VMware ESX Server configuration.

Cautions

CAUTION

If VMware loses all paths to the array for a long enough period, thedisks disappear from the system. A hard reboot is required tobring the system back to a usable state.

Any of these events could crash a system booting from a Symmetrixstorage array:

Lost connection to the Symmetrix system (pulled or damagedcable connection).

Symmetrix service and upgrade procedures, such as on-lineSymmetrix microcode upgrades and/or configuration changes.

Symmetrix director failures, including failed lasers.

Symmetrix system power failure.

Storage area network service/upgrade procedures such asfirmware upgrades or hardware replacements.


Restrictions Emulex HBAs are not supported when booting from the fabric;only QLogic HBAs are supported.

R di k i i d h b i f h


70/102


Raw disk mapping is not supported when booting from thefabric.

Only FC-SW environments are supported when booting from theSAN.

Fabric Zoning

When using VMware ESX Server hosts in a fabric environment, thezoning must be set up as single initiator and single target zoning. Asingle initiator/single target zone is composed of one HBA and oneSymmetrix storage array port.

Multi-initiator zones are not supported in a VMware ESX Server fabricenvironment.

Required Storage System Configuration

Symmetrix system configuration is performed by an EMC CustomerEngineer through the Symmetrix service processor.

Using Volume Logix to mask LUNs is highly recommended.

To verify that the VMware ESX Server host can see all of theSymmetrix target devices, configure the host as described in theremainder of the chapter.


Useful VMware ESX Server Utilities and Functions


71/102

Useful VMware ESX Server Utilities and Functions 3-9

Utility/Function Description

fdisk Command used to create and manipulate partition tables.

grep Command used to search through a file or files to find specific text.

vmkpcidivy Command used to create a boot configuration for the VMkernel.

vmkfstools Command used to create and manipulate files on LUNs owned by the VMware ESX Server host.

vmkload_mod Command used to view, load, remove driver modules in the VMkernel.

vmkmultipath Command used to display and set the configuration on a particular device and the paths attachedto that device.

vm-support Command used to gather information about the VMware ESX Server itself and virtual machines toassist in debugging issues or to obtain performance information for the virtual machines.

vmkdump Command used to manage the VMkernels dump partition.


Addressing Symmetrix Devices


72/102


Fabric Addressing

Each port on a device attached to a fabric is assigned a unique 64-bitidentifier called a World Wide Port Name (WWPN). These names arefactory-set on the HBAs in the hosts, and are generated on the FibreChannel directors in the Symmetrix system.

For comparison to Ethernet terminology, an HBA is analogous to a NIC card,and a WWPN to a MAC address.

The ANSI standard also defines a World Wide Node Name (WWNN), butthis name has not been consistently defined by the industry.

When an N_Port (host server or storage device) connects to the fabric,

a login process occurs between the N_Port and the F_Port on thefabric switch. During this process, the devices agree on suchoperating parameters as class of service, flow control rules, and fabricaddressing. The N_Ports fabric address is assigned by the switchand sent to the N_Port. This value becomes the source ID (SID) onthe N_Port's outbound frames and the destination ID (DID) on theN_Port's inbound frames.

The physical address is a pair of numbers that identify the switch andport, in the format s,p, where s is a domain ID and p is a valueassociated to a physical port in the domain. The physical address ofthe N_Port can change when a link is moved from one switch port toanother switch port. The WWPN of the N_Port, however, does notchange. A Name Server in the switch maintains a table of alllogged-in devices, so an N_Port may adjust automatically to changesin the fabric address by keying off the WWPN.

The highest level of login that occurs is the process login. This is usedto establish connectivity between the upper-level protocols on thenodes. An example is the login process that occurs at the SCSI FCPlevel between the HBA and the Symmetrix system.


SCSI-3 FCP Addressing

The Symmetrix director extracts the SCSI Command DescriptorBlocks (CDB) from the frames received through the Fibre Channel


73/102

Addressing Symmetrix Devices 3-11

Blocks (CDB) from the frames received through the Fibre Channellink. Standard SCSI-3 protocol is used to determine the addressingmode and to address specific devices.

The Symmetrix supports three addressing methods based on asingle-layer hierarchy as defined by the SCSI-3 Controller Commands(SCC):

Peripheral Device Addressing Logical Unit Addressing

Volume Set Addressing

All three methods use the first two bytes (0 and 1) of the 8-byte LUNaddressing structure. The remaining six bytes are set to 0s.

For Logical Unit and Volume Set addressing, the Symmetrix portidentifies itself as an Array Controller in response to a hosts Inquirycommand sent to LUN 00. This identification is done by returningthe byte 0x0C in the Peripheral Device Type field of the returneddata for Inquiry. If the Symmetrix system returns the byte 0x00 in thefirst byte of the returned data for Inquiry, the Symmetrix system isidentified as a direct access device.

Upon identifying the Symmetrix system as an array controller device,the host should issue a SCSI-3 Report LUNS command (0xA0), in

order to discover the LUNs.


The three addressing modes, contrasted in Table 3-1, SymmetrixSCSI-3 Addressing Modes, differ in the addressing schema (Target ID,LUN, and Virtual Bus) and number of addressabledevices.


74/102


The addressing modes are provided to allow flexibility in interfacing withvarious hosts. In all three cases the received address is converted to theinternal Symmetrix addressing structure. Volume Set addressing is thedefault for Symmetrix systems. Select the addressing mode that isappropriate to your host.

Table 3-1 Symmetrix SCSI-3 Addressing Modes

Addressing Mode CodeaABit

VBit

Response toInquiry

LUNDiscovery

Method

PossibleAddresses

MaximumLogical

Documents

VMWare + EMC storage.pdf