Tips and Tricks for CreatingTips and Tricks for CreatingGood VMs:Good VMs:ItIt’’s Not as Easy as Yous Not as Easy as You’’d Think!d Think!
Greg ShieldsGreg ShieldsPartner and Principal TechnologistConcentrated Technologywww.ConcentratedTech.com
This slide deck was used in one of our many conference presentations. We hope you enjoy it, and invite you to use it
within your own organization however you like.
For more information on our company, including information on private classes and upcoming conference appearances, please
visit our Web site, www.ConcentratedTech.com.
For links to newly-posted decks, follow us on Twitter:@concentrateddon or @concentratdgreg
This work is copyright ©Concentrated Technology, LLC
3030 Awesome Tiplets for Getting the Most Awesome Tiplets for Getting the Most
Out of Your Virtual MachinesOut of Your Virtual Machines
Greg ShieldsGreg ShieldsPartner and Principal TechnologistConcentrated Technologywww.ConcentratedTech.com
#1: Purchase Compatible Hardware#1: Purchase Compatible Hardware
…and not just “compatible with ESX”. Purchase hardware compatible with each
other.– Particularly considering vMotion needs.
#2: Buy Nehalem/Opteron#2: Buy Nehalem/Opteron
Intel Nehalem & AMD Opteron include support for Intel EPT / AMD RVI processor extensions.– Together, referred to as Second Level Address
Translations, or SLAT– Includes hardware-assisted memory management unit
(MMU) virtualization.– Significantly faster for certain workloads, such as
those with large context switches.– Finally, full support for Remote Desktop Services /
XenApp
Note that these support Large Memory Pages, which will disable ESX’s page table sharing.
#3: Mind NIC Oversubscription#3: Mind NIC Oversubscription
One of the greatest benefits of iSCSI is its linear scalability.– Need more throughput, just add another NIC!
However, VLANs and link aggregation introduce the notion of NIC oversubscription.– Ceteris Paribus, Storage traffic >>> Regular traffic.
Even with VLANs, always segregate storage NICs from production networking NICs.– If possible/affordable use segregated network paths.
– Monitor! This will kill your performance faster than anything!
#4: Consider Further#4: Consider FurtherSegregating Heavy WorkloadsSegregating Heavy Workloads
Some VMs run workloads that make heavy use of their attached disks.– Consider segregating these workloads onto their
own independent NICs and paths.– Keep an eye on your IOPS.
#5: vSphere 4.0 VMs Don#5: vSphere 4.0 VMs Don’’t Backup t Backup Applications Correctly!Applications Correctly!
vSphere 4.1 added full support for Microsoft VSS on Server 2008 guests.– This support is only automatic if the guest was
initially created on a vSphere 4.1 host.– Hosts upgraded from vSphere 4.0 aren’t properly
backing up their applications.
Fix this by setting disk.EnableUUID to True.– Power off machine.– Edit Settings | Options | General | Configuration
Parameters | Add Row– Power on machine.
#6: HBA Max Queue Depth#6: HBA Max Queue Depth
One solution for poor fibre storage performance can be adjusting your HBA maximum queue depth.– More queues can mean more performance, but less
cross-device and cross-VM optimizations.– 32 by default.– This is not a task taken lightly.– Kind of like adjusting air/fuel mix on a carburetor.
Multi-step process. See http://kb.vmware.com/kb/1267 for
details.
#7: Consider Hardware iSCSI#7: Consider Hardware iSCSI
…but, perhaps, don’t buy them…
ESX’s software iSCSI initiator works well.– However, using it incurs a small processing
overhead.– Hardware iSCSI NICs offload this overhead to the
card.– NFS/NAS storage also experience this behavior.
#7: Consider Hardware iSCSI#7: Consider Hardware iSCSI
…but, perhaps, don’t buy them…
ESX’s software iSCSI initiator works well.– However, using it incurs a small processing overhead.– Hardware iSCSI NICs offload this overhead to the card.– NFS/NAS storage also experience this behavior.
Newer NICs reduce this effect, those with…– Checksum offload– TCP segmentation offload (TSO)– 64-bit DMA addressing– Multiple Scatter Gather elements per Tx frame– Jumbo frames
#8: Set NICs to Autonegotiate#8: Set NICs to Autonegotiate
VMware’s recommendation is to set all NICs to autonegotiate, full duplex.– This is sort of “duh” these days.– But its worth mentioning, because…
– Some old school network admins still prefer to manually set speed/duplex due to a crazy old race condition bug that happened a long, long time ago.
#8: Set NICs to Autonegotiate#8: Set NICs to Autonegotiate
VMware’s recommendation is to set all NICs to autonegotiate, full duplex.– This is sort of “duh” these days.– But its worth mentioning, because…
– Some old school network admins still prefer to manually set speed/duplex due to a crazy old race condition bug that happened a long, long time ago.
– Just smack around those old coots.
#8: Set NICs to Autonegotiate#8: Set NICs to Autonegotiate
VMware’s recommendation is to set all NICs to autonegotiate, full duplex.– This is sort of “duh” these days.– But its worth mentioning, because…
– Some old school network admins still prefer to manually set speed/duplex due to a crazy old race condition bug that happened a long, long time ago.
– Just smack around those old coots. – “You can take your Token Ring and your IPX
and go home now!”
#9: Do Not Team Storage NICs#9: Do Not Team Storage NICs
What? Don’t team them?– Well, I guess I mean “team” as in the classic sense
of network teaming.
#9: Do Not Team Storage NICs#9: Do Not Team Storage NICs
What? Don’t team them?– Well, I guess I mean “team” as in the classic sense
of network teaming.
Remember that storage NICs leverage MPIO for link aggregation.– MPIO is a superior technology over link aggregation
anyway for storage.– ‘tis also easier to use, and better for routing!
vCenter’s GUI wizards make this hard not to do, but be aware that extra steps are required…
#10: Enable Hyperthreading#10: Enable Hyperthreading
Early in ESX’s days we debated whether hyperthreading improved or decreased overall performance.– That debate is over. The winner is “increase”.
Today, hyperthreading adds a non-linear additional quantity of processing capacity.– Like 20-30% (???), not a full extra proc.
But you know this.– Enable it in your servers’ BIOS.– Just turn it on, OK?
#11: Allocate Only the CPUs #11: Allocate Only the CPUs You NeedYou Need
Allocate only as many vCPUs as a VM requires.– Start with only one as your baseline. Rarely
deviate. Circle this bullet point. No, really.– Don’t use dual vCPUs if single-threaded application.– Don’t assign more vRAM than necessary.
#11: Allocate Only the CPUs #11: Allocate Only the CPUs You NeedYou Need
Allocate only as many vCPUs as a VM requires.– Start with only one as your baseline. Rarely deviate.
Circle this bullet point. No, really.– Don’t use dual vCPUs if single-threaded application.– Don’t assign more vRAM than necessary.
More vCPUs equals more problems.– More vCPUs equals more interrupts– Extra overhead in maintaining consistent memory view
between vCPUs. This is tough, especially with today’s descheduled processing.
– Some OSs migrate single-threaded workloads between multiple CPUs, adding a performance tax.
More CPUs good for CPU spike handling.
#12: Disconnect Unused#12: Disconnect UnusedPhysical Hardware DevicesPhysical Hardware Devices
COM, LPT, USB, Floppy, CD/DVD, NICs, etc.all consume interrupt resources.– High priority resources.– It is a big deal to insert a CD/DVD/USB.
Connected Windows guests will poll CD/DVD drives very frequently, significantly affecting performance.– Disconnect these in VM properties when not in use.– There’s a reason why the “Connected” checkbox
exists!– Note! Connected devices can prevent a vMotion!
#13: Upgrade to VM Version 7#13: Upgrade to VM Version 7
Virtual hardware version 7 offers some very significant performance improvements.– VMXNET3 paravirtualized NIC driver– PVSCSI paravirtualized SCSI driver– Upgrade VMware tools. Reboot.– (More on these in a minute)
Note that VMv7 hardware cannot be vMotioned to ESX servers prior to 4.0.– Be careful of this.
#14: Don#14: Don’’t Fear Scaling Outt Fear Scaling Out
Creating VMs is easy, so we create them.– You’ll eventually run out of CPU resources.– You’ll probably run out of RAM first.– Don’t run more VMs than processing/memory capacity.
When running very close to capacity, use CPU reservations to guarantee 100% CPU availability for the console.– Host | Configuration | System Resource Allocation |
Edit– Particularly important if you have software installed
there.– This is unnecessary in ESXi.
#15: 80% is Nice#15: 80% is Nice
VMware recommends maintaining an administrative ceiling on utilization at 80%.– This reserves enough capacity for failure, service
console.– VMware suggests that 90% should be a warning for
overconsumption.
Less workload dynamics can shift this up.– …but, seriously, who can really state that?
#16: With Older OSs,#16: With Older OSs,Use UP HAL When PossibleUse UP HAL When Possible
Newer OSs (Vista, W7, 2008) use the same HAL for all UP/SMP conditions.
Older OSs leverage two HALs– A Uniprocessor HAL– A Multiprocessor HAL
An SMP HAL that is only given a single vCPU will run slightly slower.– Slightly more synchronization code.
Note that this will impact hot add.
#17: Mind Scheduling Affinity#17: Mind Scheduling Affinity
It is possible to tag a VM to a particular pProc.– Good for ensuring that VM has processing
resources during contention.– Setting Code Sharing to None prevents any other
vProc from using a pProc on the same core. Like disabling HT.
– Setting Code Sharing toInternal prevents vProcson other VMs from usingpProc on same core.Only same VM.
– Just set this to Any.
#18: Don#18: Don’’t Touch this Setting.t Touch this Setting.
Exceptionally rare are the cases when this setting shouldbe adjusted.– So, no touchy.
#18: Don#18: Don’’t Touch this Setting.t Touch this Setting.
Exceptionally rare are the cases when this setting shouldbe adjusted.– So, no touchy.
– I will tell youwhen.
– I havevery reasonableconsulting rates.
#19: Don#19: Don’’t Just Keep Up thet Just Keep Up theOld (and Dumb) Habit of Assigning 4G Old (and Dumb) Habit of Assigning 4G of RAM to Every Stinking Virtual of RAM to Every Stinking Virtual Machine, No Matter What Workload it Machine, No Matter What Workload it Runs. Really.Runs. Really.
Consciously consider the amount of RAM that a VM needs, and assign it that RAM.– Yes, VMware has memory ballooning.– But overallocating unnecessarily increases VM
overhead.– Ballooning isn’t automatic. Ballooning is slow.
Ballooning is reactive.
#19: Don#19: Don’’t Just Keep Up thet Just Keep Up theOld (and Dumb) Habit of Assigning 4G Old (and Dumb) Habit of Assigning 4G of RAM to Every Stinking Virtual of RAM to Every Stinking Virtual Machine, No Matter What Workload it Machine, No Matter What Workload it Runs. Really.Runs. Really.
Consciously consider the amount of RAM that a VM needs, and assign it that RAM.– Yes, VMware has memory ballooning.– But overallocating unnecessarily increases VM overhead.– Ballooning isn’t automatic. Ballooning is slow.
Ballooning is reactive.
Note to Self: Talk about Hyper-V’s Dynamic Memory here. Very cool.
#20: Stop with the Snapshots#20: Stop with the Snapshots
Snapshots are (were) a significant selling point in the early days of virtualization.– About to do something risky? Snapshot!
Its like a career protection device!
#20: Stop with the Snapshots#20: Stop with the Snapshots
Snapshots are (were) a significant selling point in the early days of virtualization.– About to do something risky? Snapshot!
Its like a career protection device!
However, snapshots aren’t (and never were) meant for long-term storage.– And I mean “no more than just a few minutes” long.– They’re not meant for backups.– Reverting to an aged snapshot can break computer
trust relationships to Windows domain.– Managing snapshots, particularly linked ones,
significantly reduces overall VM performance.
#21: Perform vSphere Tasks#21: Perform vSphere Tasksin the Off Hoursin the Off Hours
Some vSphere tasks are actually quite impactful on VM operations.– Provisioning virtual disks– Cloning virtual machines– svMotion– Manipulating file permissions– Backups– Anti-virus (bleh)
Do these tasks during off hours, or you may impact performance for other running VMs.
#22: Mind Affinities#22: Mind Affinities
Some VMs need to regularly communicate with each other with high throughput.– “Keep Virtual Machines Together”– Make sure these machines share the same
vSwitch.– Collocation forces inter-VM traffic through the
system bus rather than pNICs, significantly increasing speed.
The loss of other VMs could be bad, if both are collocated on the same host.– “Separate Virtual Machines”
#23: Disable Screen Savers#23: Disable Screen Savers
And Window animations.
Screen savers represent a machine interrupt, particularly those with heavy graphics.– “Pipes”, I’m looking right at you!– This interrupt is particularly impactful on collocated
VMs.
– …and, plus, screen savers on servers is sooooo 2002.
#24: Use NTP, not VMware#24: Use NTP, not VMwareTools for Time SyncTools for Time Sync
…and here’s one out of the odd files…
VMware suggests configuring VMs to sync time from an external NTP server.– They prefer this even over their own internal
timekeeper.– Their timekeeper uses a much lower resolution
than NTP.– NTP = milliseconds– NT5DS = 1 second– VMware Tools = ?
#25: Never Use PerfMon #25: Never Use PerfMon Inside the VM, Except…Inside the VM, Except…
Not that you’d ever actually use PerfMon, but…– Measuring performance from within a virtual
machine fails to take into account for unscheduled time.
– Essentially, when the ESX server isn’t servicing the VM, no time passes within that VM.
– Also, in-VM PerfMon doesn’t recognize virtualization overhead.
– Most important, in-VM PerfMon can’t see down into layers below the VM: Storage, processing, etc.
#25: Never Use PerfMon #25: Never Use PerfMon Inside the VM, Except…Inside the VM, Except…
Not that you’d ever actually use PerfMon, but…– Measuring performance from within a virtual machine
fails to take into account for unscheduled time.– Essentially, when the ESX server isn’t servicing the VM,
no time passes within that VM.– Also, in-VM PerfMon doesn’t recognize virtualization
overhead.– Most important, in-VM PerfMon can’t see down into
layers below the VM: Storage, processing, etc.
VMware Tools adds PerfMon counters to VMs.– These are OK to use, as they’re synched from ESX.
#26: Paravirtualization is #26: Paravirtualization is Your FriendYour Friend
VM Hardware Version 7 adds two new paravirtualized drivers.– VMXNET3 replaces E1000– PVSCSI replaces BusLogic/LSILogic
Paravirtualized drivers are superior to emulation– They are “aware” they’ve been virtualized. Can
work directly with host without needing emulation.– Mexican menus versus French menus.
VMXNET3 supports TSO & Jumbo Frames,in the VM!– Even if the physical hardware doesn’t support TSO!
#27: Turn on Jumbo Frames,#27: Turn on Jumbo Frames,but Do it Everywherebut Do it Everywhere
If you plan to use Jumbo Frames…– MTU size is usually set to 9000– Make sure you enable it everywhere.– This brings particular assist with large file transfers
(think WDS, virtual disk provisioning, etc.) and storage connections.
Not all network equipment supports Jumbo Frames.– Test, test, test.
#28: DRS Will Prioritize#28: DRS Will PrioritizeFaster Hosts over Slower OnesFaster Hosts over Slower Ones
A neat fact (that I didn’t know):– When potential hosts for a DRS relocation have
compatible CPUs but different CPU frequencies and/or memory capacity…
– …DRS will prioritize relocating VMs to the system with the highest CPU frequency and more memory.
– This won’t be the case if that CPU is already at capacity.
#29: Disable FT, Unless#29: Disable FT, UnlessYouYou’’re Using Itre Using It
…and most of you aren’t.
You can “turn on” but not “enable” FT.– Problem: Turning on Fault Tolerance automatically
disables some features that enhance VM performance.
– Hardware virtual MMU is one.
Or, just don’t use that horrible feature. Har!– (Is there anyone from VMware in the audience…?)
#30: Match Configured OS#30: Match Configured OSwith Actual OSwith Actual OS
Big oops here, usually during OS migrations. This setting also
sets a few important low-level kerneloptimizations.
Make sureyours arecorrect!
BONUS TIP #31: Follow the NumbersBONUS TIP #31: Follow the Numbers
Private Clouds are all about quantifying performance in terms of supply and demand.
vSphere gives you those numbers. Just sum ‘em up.
Final ThoughtsFinal Thoughts
See! Creating good VMs isn’t all that easy.– Our jobs aren’t going away any time soon!– These little optimizations add up
Be smart with your virtual environment and always remember…
Final ThoughtsFinal Thoughts
See! Creating good VMs isn’t all that easy.– Our jobs aren’t going away any time soon!– These little optimizations add up
Be smart with your virtual environment and always remember…
…you cannot change the laws of Physics!
Tips and Tricks for CreatingTips and Tricks for CreatingGood VMs:Good VMs:ItIt’’s Not as Easy as Yous Not as Easy as You’’d Think!d Think!
Greg ShieldsGreg ShieldsPartner and Principal TechnologistConcentrated Technologywww.ConcentratedTech.com
Please fill out evaluations,or no free beer for you!
!!!
This slide deck was used in one of our many conference presentations. We hope you enjoy it, and invite you to use it
within your own organization however you like.
For more information on our company, including information on private classes and upcoming conference appearances, please
visit our Web site, www.ConcentratedTech.com.
For links to newly-posted decks, follow us on Twitter:@concentrateddon or @concentratdgreg
This work is copyright ©Concentrated Technology, LLC