I A N G ’ S J O U R N E Y F R O M L E G A C Y B L A D E S E R V E R S T O C I S C O U N I F I E D C O M P U T I N G S Y S T E M , W I T H N O ( N O T I C E A B L E ) V M W A R E

D O W N T I M E

UCS Nirvana

Lee BurlesonNetwork and Systems EngineerIowa National Guard

Cisco ARNG Training EventMarch 2011

Disclaimer

This presentation contains information on Iowa National

Guard migration from a legacy blade server environment

to a Cisco UCS system. The content was created for the

audience attending the 5th Annual Training Event and

not for distribution beyond the attendees without the

express permission of the Iowa National Guard .

Introduction

whoami

My credentials

Outline

Can’t do a UCS AO class in 45m

Legacy Environment

2 Dell 1855 chassis

18 ea 1955 Blade servers

Great for IANG when purchased

Legacy Environment, cont.

The aura wears off…

Less-than-ideal chassis management interface

Cabling: power, FC, Ethernet, management, KVM

Design limitations, especially when virtualizing

Module management

Eth, FC

Points of management

Photos

Photos, cont.

Overview and Background

Project goals

Modernize

Improve management

Boot from SAN

Reduce complexity

Increase performance

Blade Chassis Research

Read both vendor-produced AND independent literature

Drank Cisco Kool-Aid @ Minneapolis

Wrote specs (anyone need those?)

Received 2 ea chassis & 8 ea B200M2 blades

UCS Basics

Fabric Interconnects are heart of system (2)

40c x 8b = 320b*

Not just for virtualization, but does it very well

Single and well-done mgmt UI

All config kept in FI – hardware independence

Built-in, transparent fabric redundancy

UCS PhotoFront view

UCS PhotoRear view

Hardware Installation

Chassis are heavy

Very clean cabling

Power cable challenges with our PDUs• Ordered C19-C14 cables

Per chassis:

2 ea 1M cords

2 ea 2M cords

Hardware Installation, cont.

Ordered reduced-length IOM interconnects (SFP-H10GB-CU1M)

Remember: only 1,2,or 4 IOM connections are allowed!

Management

Baby steps

Connected only mgmt ports of both Fabric Interconnects

Must reserve IP space in same subnet for KVM access to blades

Used management interface (Java) to explore and design system without external impact

Management, cont.

Able to create entire system without upstream connectivity or even the blades themselves

Resource pools Server pools

MAC address pools

WWNN pools

KVM IP address pools

vNIC templates

Etc…

Service profile templates

Service profiles (7b+1ub)

Storage

Configure switches based on UCS config

WWNN Aliases + Zoning = Port independence

Enable NPIV (globally + UCS ports)

Connected FIs to FC fabric; verified config

May want to fully alias and zone in both fabrics; depends on your design

No production VMFS presentation yet!

Built initial test ESXi server as SAN-boot POC

Network

Single 1GbE link from FI pair connected upstream -

helped prove failover and identify issues

Spent a while in this state

4 x 10Gb transition was later seamless

Ease of adding VLANs

VMware Integration

Wanted to improve VM I/O, even past basic host

improvements gained with upgrade

Hypervisor bypass/PCI pass-through/VMDirectPath

I thought this was what I wanted, until I tried it

Various painful consequences

Recommendation: don’t do it! (until….)

VMware Integration, cont.

Our host design: 4 network interfaces (vNICs) + 2 FC vHBAs

vNIC0 - vswitch0 – “mgmt” - fabric A/B

vNIC1 - vswitch1 – “vmotion” - fabric B/A

vNIC2 - FI dvs – “vm_Uplink01” - fabric A only

vNIC3 - FI dvs – “vm_Uplink02” - fabric B only

No trunking!

Pass-through switching - VN-Link

vMotion works great

Reduces host CPU, reduces latency, improves I/O

Creates dynamic vNIC## – vm int xx - FI dvs – balanced fabric connections!

VMware Integration, cont.VMware host interfaces, UCS screenshot

VMware Integration, cont.VMware host interfaces, VMware screenshot

VMware Integration, cont.

Created addl cluster for isolation

Joining the datacenter

DVS connection

Different DVSs available: VMware, Nexus 1000V,

Nexus 1010, 6100 FI

VM Migration

The culmination of all our work

Recommend temporarily setting DRS to Partial so you don't fight it

Ran into first problem

VM Migration, cont.Changing the VM Network Connection

Second problem: destination host didn’t have a Port Group for the VM to connect to

VM Migration, cont.Changing the VM Network Connection

VM Migration, cont.Moving vCenter itself

The last challenge, so I thought

Catch 22?

Procedure used: [Bad memory block] Did something bad; I lost my vCenter

connection

Direct-connect to vCenter-owning host

Power down vCenter VM

Remove from Inventory

Direct-connect to UCS crossover host

Import VM, verify network, and power it on

Connect to vCenter and change adapter over to dvs (as in standard procedure)

VM Migration, cont.Moving vCenter itself (also cont.)

Procedure that I should have used:

Migrate vCenter VM to crossover host

Connect to vCenter and change adapter over to dvs (as in

standard procedure)

Still, be careful

VMware Integration, cont.Balanced fabric screenshot

Goals Met?

Definitely modernized

Vastly improved management

100% boot from SAN (even the MSSQL blade)

Reduced complexity

Improved performance:

vMotion SharePoint VM with 16GB RAM

Before: ~3 minutes, After: 40 seconds

Can take down a host running a dozen VMs within 5 minutes

Future

Nexus 5548

Topology considerations - are the 5548s the new de

facto datacenter core with L3?

HP iSCSI gateway for EVA – 10Gb iSCSI for VMs

The Nexus 1000V is no longer needed with

VMWare on UCS (or is it?)

Acknowledgements/Resources

www.bradhedlund.com

blog.scottlowe.com

www.cisco.com/go/ucs

bladesmadesimple.com