HyperFlex Configuration Best Practices and Troubleshootingclnv.s3.amazonaws.com/2017/eur/pdf/BRKVIR-2700.pdf · HyperFlex Configuration Best Practices and Troubleshooting ... Customer

HyperFlex Configuration Best Practices and Troubleshooting Scenarios

Clifford Aldan, Customer Support Engineer, Cisco Global TACJonathan Gorlin, Solutions Architect, HyperFlex

BRKVIR-2700

© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public

Abstract

HyperFlex is Cisco's HyperConverged Infrastructure solution designed to meet today's Virtual Desktop Infrastructure and Virtual Server Infrastructure workloads. This session enables professionals administering the datacenter to understand best practices when configuring HyperFlex and troubleshoot common scenarios.

BRKVIR-2700 3


The main goals of this session

• Configurations in key areas of the HyperFlex solution – Northbound Switch, UCS, vSphere, and the HX Data Platform (HXDP)

• Common troubleshooting scenarios experienced on HyperFlex

BRKVIR-2700 4


This session does not include

• HX performance deep dive

• Replication, backup, and DR scenarios

• VDI configuration and deployment

• Distributed switch configuration (Nexus 1000v)

• HX 2.0

• Roadmap

BRKVIR-2700 5


Related Cisco Live Sessions

Session-ID Date and Time Session Name

BRKCOM-2004 Thursday, 11:30 AM Cisco HyperFlex Deep Dive and Performance

BRKINI-4003 Tuesday, 11:15 AM What's New with Cisco HyperFlex Systems 2

PSODCN-2375 Wednesday, 1:15 PM Panel: What’s New with HyperFlex All Flash Release and Learn How ForePaaS Built a Hybrid

Cloud Service on HyperFlex

LTRDCN-1700 Tuesday, 9:30 AM Deployment and operation of Cisco Hyperflex HX Data Platform

BRKGS-2004 Wednesday, 12:35 PM Learn How to enable your Always-On Enterprise with Cisco HyperFlex and Veeam!

BRKGS-1344 Wednesday, 2:10 PM Increase productivity, consolidate data management, and ensure business continuity with

Commvault and Cisco UCS

BRKPLT-2000 Tuesday, 4:15 PM Enabling Secure Delivery of Apps and Data with Superior Performance and Agility via Citrix

and Cisco

BRKVIR-2700 6













and Cisco

BRKVIR-2700 7


Cisco SparkAsk Questions, Get Answers, Continue the Experience

Use Cisco Spark to communicate with the Speaker and fellow participants after the session

Download the Cisco Spark app from iTunes or Google Play

1. Go to the Cisco Events mobile app

2. Find this session

3. Click the Spark button under Speakers in the session description

4. Enter the room, room name = BRKVIR-2700

5. Join the conversation!

The Spark Room will be open for 2 weeks after Cisco Live

BRKVIR-2700 8

• Prerequisites

• Northbound Switch

• UCS

• vSphere

• HX Data Platform

• Troubleshooting Scenarios

Agenda

Prerequisites


Have a planning call with all parties that will be involved in the HX deployment

Server Team

Network Team

Storage Team

Security Team

Cisco or Partner

UCS, VMware, HX

Switches

Routers

FC/FCoE Switches

External SAN/NAS

Account Team

TAC Support

Firewalls

Security Gateways

HX HCI

Initiative

BRKVIR-2700 11


Fill out the latest version of the HX pre-install checklist

BRKVIR-2700 12


Fill out the latest version of the HX pre-install checklist

• UCS Information

• VLANs

• IP Addresses (ESXi, SCVM)

• vSphere Information (vCenter, ESXi, Datacenter name, Cluster name)

• HXDP Information

• DNS, NTP, Time Zone

• Auto Support

BRKVIR-2700 13


Review the interoperability requirements to ensure compatibility between hardware and software

!

HX240cM4

Co

nso

le

1 6 12 18 24

HyperFlex

!

HX240cM4

Co

nso

le

1 6 12 18 24

HyperFlex

!

HX240cM4

Co

nso

le

1 6 12 18 24

HyperFlex

ESXi 5.5 U3

Enterprise Plus

6.0 U1b, U2

Enterprise Plus, Standard,

Essentials Plus, ROBO

UCS

2.2(6f) – 1.7.1

2.2(7c) – 1.7.1-14835

HyperFlex

1.7.1, 1.7.1-14835

ESXi 5.5 U3

Enterprise Plus

6.0 U1b, U2



UCS

2.2(7c)

HyperFlex

1.7.3

ESXi 5.5 U3 (HX240 only)

Enterprise Plus

6.0 U1b, U2, U2 P3



UCS

3.1(2b) – 1.8.1a, 1.8.1b,

1.8.1c, 1.8.1e

3.1(2c) – 1.8.1c, 1.8.1e

HyperFlex

1.8.1a, 1.8.1b, 1.8.1c, 1.8.1e

BRKVIR-2700 14


Review the interoperability requirements to ensure compatibility between hardware and software

!

HX240cM4

Co

nso

le

1 6 12 18 24

HyperFlex

!

HX240cM4

Co

nso

le

1 6 12 18 24

HyperFlex

!

HX240cM4

Co

nso

le

1 6 12 18 24

HyperFlex

ESXi 5.5 U3

Enterprise Plus

6.0 U1b, U2



UCS

2.2(6f) – 1.7.1

2.2(7c) – 1.7.1-14835

HyperFlex

1.7.1, 1.7.1-14835

ESXi 5.5 U3

Enterprise Plus

6.0 U1b, U2



UCS

2.2(7c)

HyperFlex

1.7.3

ESXi 5.5 U3 (HX240 only)

Enterprise Plus

6.0 U1b, U2, U2 P3



UCS

3.1(2b) – 1.8.1a, 1.8.1b,

1.8.1c, 1.8.1e

3.1(2c) – 1.8.1c, 1.8.1e

HyperFlex

1.8.1a, 1.8.1b, 1.8.1c, 1.8.1e

BRKVIR-2700 15


Validate Network Services are stable, reachable, and continuous (for the lifetime of the cluster)

!

HX240cM4

Co

nso

le

1 6 12 18 24

HyperFlex

!

HX240cM4

Co

nso

le

1 6 12 18 24

HyperFlex

!

HX240cM4

Co

nso

le

1 6 12 18 24

HyperFlex

HyperFlex Cluster

BRKVIR-2700 16


Allow required ports through the firewall

BRKVIR-2700 17



HTTP HTTPS KVM SNMP

80 443 2068 161

TCP TCP TCPSNMP

POLL

INBOUND

BRKVIR-2700 18



HTTP HTTPS KVM SNMP

80 443 2068 161

TCP TCP TCPSNMP

POLL

SMTP DNS NTP SNMP

25 53 123 162

TCPTCP

UDPTCP

SNMP

TRAP

INBOUND OUTBOUND

BRKVIR-2700 19



HTTP HTTPS KVM SNMP

80 443 2068 161

TCP TCP TCPSNMP

POLL

SMTP DNS NTP SNMP

25 53 123 162

TCPTCP

UDPTCP

SNMP

TRAP

SSH POP3 IMAP4VCENTER

ESXi

VCENTER

HTTPS

HX

INSTALLER

22110

(995)

143

(993)902 9443 8089

TCP TCP TCP TCP/UDP TCP TCP

INBOUND OUTBOUND

INBOUND/OUTBOUND

BRKVIR-2700 20

• Prerequisites


• UCS

• vSphere



Agenda

Northbound Switch


Assess your northbound switches for features, licenses, port count, port config, cable requirements

What features need to be enabled/licensed?

Do I have enough ports/port licenses?

What’s the switchport configuration?

BRKVIR-2700 23






BRKVIR-2700 24






BRKVIR-2700 25






BRKVIR-2700 26


Create VLANs for the following network traffic types

•Used for all VM traffic

•L3 routable

• VM live migrations

•L2 or L3

•ESXi/SCVM storage traffic

•HXDP replication

•L2 Non-routed

•vCenter, ESXi, SCVM communication

•L3 Routable

Management Storage Data

VM NetworkvMotion

OPTIONAL

External

Storage

iSCSI, NFS,

FCoE

BRKVIR-2700 27


Use VLANs other than VLAN 1

VLA

N 1

Default VLAN in which all switch control plane traffic use – telnet, SNMP, syslog

Protocol communication between switches (i.e., CDP, STP, VTP, etc)

All switchports are part of VLAN 1

Will cause networking issues, especially for disjoint Layer 2 (i.e., DMZ)

CONTROL PLANE

BRKVIR-2700 28


Set switchports to UCS FIs as trunk ports to allow multiple VLANs to traverse each uplink

CISCO UCS 6248UP 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32

STAT

ID

CISCO UCS 6248UP 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32

STAT

ID

switch# configure terminal

switch(config)# interface port-channel 201

switch(config-if)# switchport mode trunk

switch(config-if)# switchport trunk allowed vlan

100, 200, 300, 400

FI A

FI B

BRKVIR-2700 29


Set switchports connecting to UCS FIs as edge ports to support End Host Mode

FIs seen as a host with a large number of NICs and MAC addressesEth1/1

CISCO UCS 6248UP 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32

STAT

ID

Northbound Switch

STP

BRKVIR-2700 30


Set switchports connecting to UCS FIs as edge ports to support End Host Mode

For Cisco NX-OS switchports to UCS FIs switch# configure terminal


switch(config-if)# spanning-tree port type edge trunk

For Cisco IOS switchports to UCS FIs switch# configure terminal


switch(config-if)# spanning-tree portfast edge trunk (IOS release 12.2(33)SXI and later)

switch(config-if)# spanning-tree portfast trunk

(IOS releases prior to 12.2(33)SXI)

BRKVIR-2700 31


Enable jumbo frames for interfaces connecting to the FIs

• For Cisco NX-OS, use a policy-map and system qospolicy-map type network-qos FCoE-with-Jumbo

class type network-qos class-fcoe

pause no-drop

mtu 2158

class type network-qos class-default

mtu 9216

multicast-optimize

system qos

service-policy type network-qos FCoE-with-Jumbo

• For Cisco IOS, configure at interface level6500(config)# interface gigabitEthernet 1/1

6500(config-if)# mtu ?

<1500-9216> MTU size in bytes

6500(config-if)# mtu 9216

Jumbo Frame (9216 MTU)

BRKVIR-2700 32


Enable jumbo frames end-to-end to support Storage Data and vMotion traffic

QoS

Policy Map

Switchport

QoSSystem Class

vNIC vSwitchVMkernelAdapter

CISCO UCS 6248UP 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32

STAT

ID

CISCO UCS 6248UP 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32

STAT

ID

UCSM!

HX240cM4

Co

nso

le

1 6 12 18 24

HyperFlex

ESXi Hypervisor

BRKVIR-2700 33


Enable jumbo frames end-to-end to support Storage Data and vMotion traffic

QoS

Policy Map

Switchport

QoSSystem Class


CISCO UCS 6248UP 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32

STAT

ID

CISCO UCS 6248UP 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32

STAT

ID

UCSM!

HX240cM4

Co

nso

le

1 6 12 18 24

HyperFlex

ESXi Hypervisor

BRKVIR-2700 34


Map the CoS values to the CoS settings of the UCS QoS System Class

• For Cisco NX-OSswitch# configure terminal

switch(config)# class-map type qos class_platinum

switch(config-cmap-qos)# match cos 5

• For Cisco IOSswitch# configure terminal

switch(config)# class-map class_bronze

switch(config-cmap)# match cos 1

Platinum

5

Silver

2

Bronze

1

Gold

4

Platinum

5

Silver

2

Bronze

1

Gold

4

CISCO UCS 6248UP 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32

STAT

ID

CISCO UCS 6248UP 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32

STAT

ID

BRKVIR-2700 35


Configure port-channel for redundancy, wider bandwidth, and load balancing

BRKVIR-2700 36



CISCO UCS 6248UP 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32

STAT

ID

CISCO UCS 6248UP 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32

STAT

ID

vPC

BRKVIR-2700 37



CISCO UCS 6248UP 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32

STAT

ID

CISCO UCS 6248UP 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32

STAT

ID

CISCO UCS 6248UP 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32

STAT

ID

CISCO UCS 6248UP 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32

STAT

ID

vPC Non-vPC

BRKVIR-2700 38


Sample configuration: vPC with 4 links from each Fabric Interconnect Switch 1 Configuration

interface Ethernet1/25-26

description “Link to HX-UCS-FI-A Eth1/25-26”

channel-group 201 mode active

interface port-channel201

description "vPC to HX-UCS-FI-A"

switchport mode trunk

switchport trunk allowed vlan 100,200,300,400

spanning-tree port type edge trunk

speed 10000

vpc 201


description “Link to HX-UCS-FI-B Eth1/27-28”



description "vPC to HX-UCS-FI-B"




speed 10000

vpc 202

CISCO UCS 6248UP 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32

STAT

ID

CISCO UCS 6248UP 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32

STAT

ID

Switch 1 Switch 2

vPC

BRKVIR-2700 39


Sample configuration: vPC with 4 links from each Fabric Interconnect Switch 2 Configuration





description "vPC to HX-UCS-FI-B"




speed 10000

vpc 202





description "vPC to HX-UCS-FI-A"




speed 10000

vpc 201

CISCO UCS 6248UP 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32

STAT

ID

CISCO UCS 6248UP 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32

STAT

ID

Switch 2Switch 1

vPC

BRKVIR-2700 40


Sample configuration: port-channel with 4 links from each Fabric Interconnect

Switch 1 Configuration interface Ethernet1/25-26




description "PC to HX-UCS-FI-A Eth1/25-26"




speed 10000





description "PC to HX-UCS-FI-B Eth1/27-28"




speed 10000

CISCO UCS 6248UP 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32

STAT

ID

CISCO UCS 6248UP 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32

STAT

ID

Switch 1 Switch 2

Non-vPC

BRKVIR-2700 41


Sample configuration: port-channel with 4 links from each Fabric Interconnect

Switch 2 Configuration interface Ethernet1/25-26




description "PC to HX-UCS-FI-B Eth1/25-26"




speed 10000





description "PC to HX-UCS-FI-A Eth1/27-28"




speed 10000

CISCO UCS 6248UP 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32

STAT

ID

CISCO UCS 6248UP 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32

STAT

ID

Switch 2Switch 1

Non-vPC

BRKVIR-2700 42


OPTIONAL – Set up FC/FCoE ports to connect HyperFlex to an external FC/FCoE SAN

CISCO UCS 6248UP 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32

STAT

ID

4

1

5

2

6

3

7 8HX220c

M4

Console!

HyperFlex

4

1

5

2

6

3

7 8HX220c

M4

Console!

HyperFlex

4

1

5

2

6

3

7 8HX220c

M4

Console!

HyperFlex

CISCO UCS 6248UP 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32

STAT

ID

FC SAN

FC Only Switches

HyperFlex

CISCO UCS 6248UP 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32

STAT

ID

4

1

5

2

6

3

7 8HX220c

M4

Console!

HyperFlex

4

1

5

2

6

3

7 8HX220c

M4

Console!

HyperFlex

4

1

5

2

6

3

7 8HX220c

M4

Console!

HyperFlex

CISCO UCS 6248UP 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32

STAT

ID

FC/FCoESANSwitches

with Unified Ports –EthernetFC/FCoE

HyperFlex

BRKVIR-2700 43

• Prerequisites


• UCS

• vSphere



Agenda

UCS


Set up Call Home to get email alerts from UCS Manager (UCSM)

Refer to the Cisco UCS Manager

System Monitoring Guide,

Release 3.1

(OPTIONAL) Fill out this section if

using Smart Call Home (SCH)

• Contact your Cisco Account

team to obtain information

BRKVIR-2700 46


IF NEEDED – Use Auto-Install to upgrade/downgrade the UCS firmware (infra/server)

BRKVIR-2700 47


Make sure all UCS components are at the correct version and the Activate Status is in “Ready” state

BRKVIR-2700 48


Configure unified ports – network uplink, server, or FC uplink ports

BRKVIR-2700 49


CISCO UCS 6248UP 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32

STAT

ID

CISCO UCS 6248UP 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32

STAT

ID


Network Uplink ports

BRKVIR-2700 50


CISCO UCS 6248UP 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32

STAT

ID

CISCO UCS 6248UP 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32

STAT

ID FI BFI A


PSU

2

770W AC

PP

770W AC

PP

M 12

SS

PC

Ie 2

PC

Ie 1

ACT LINK ACT LINK

PORT

1PORT

2

1

Server ports

BRKVIR-2700 51


CISCO UCS 6248UP 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32

STAT

ID

CISCO UCS 6248UP 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32

STAT

ID FI B

1400W AC

1400W ACM

SS

PC

Ie 4

ACT LINK ACT LINK

PORT

1PORT

2

5

PC

Ie 5

PC

Ie 6

21

PC

Ie 1

PC

Ie 2

PC

Ie 3

1

2

PSUMLOM


Server ports

FI A

BRKVIR-2700 52


CISCO UCS 6248UP 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32

STAT

ID

CISCO UCS 6248UP 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32

STAT

ID

FAN STATUS

FAN STATUS

FAN STATUS

FAN STATUS

FAN STATUS

FAN STATUS

FAN STATUS

FAN STATUS

CHS A56

FAN 1 FAN 5 FAN 2 FAN 6 FAN 3 FAN 7 FAN 4 FAN 8

!

UCS 2204XP

4

3

2

1

UCS 2204XP

4

3

2

1


Server ports

FI BFI A

BRKVIR-2700 53


CISCO UCS 6248UP 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32

STAT

ID

CISCO UCS 6248UP 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32

STAT

ID


FC Uplink ports

BRKVIR-2700 54


When converting to FC uplinks, use the Configure Unified Ports wizard (FI reboot required)

BRKVIR-2700 55


When converting to FC uplinks, use the Configure Unified Ports wizard (FI reboot required)

BRKVIR-2700 56


When considering VLANs to create, note that the HX installer will create certain VLANs in UCSM

Can be a comma separated list of guest

VLANs

BRKVIR-2700 57


VLANs

2000, 4000, 67,

5, 4, 1000, 500,

6, 3000

Create the remaining VLANs to match those configured on the northbound switches

CISCO UCS 6248UP 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32

STAT

ID

BRKVIR-2700 58


When deploying disjoint Layer 2, prune the VLANs to the corresponding network uplinks

CISCO UCS 6248UP 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32

STAT

ID

CISCO UCS 6248UP 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32

STAT

ID

DMZ

4000

Production

10, 20

Dev

2000

FI A FI BVLAN 1 allowed on all ports

VLAN 1 cannot be pruned off the FIs

BRKVIR-2700 59



VLAN 1

not

listed!

BRKVIR-2700 60



• Do this on both FIs

BRKVIR-2700 61



• Do this on both FIs

• Assign all VLANs to their respective uplinks

BRKVIR-2700 62


HX installer configures QoS and sets Jumbo Frames for Storage and vMotion networks

• Refer to this table when configuring CoS value (1, 2, 4, 5) on the northbound switches

BRKVIR-2700 63


HX installer configures QoS and sets Jumbo Frames for Storage and vMotion networks

• Refer to this table when configuring CoS value (1, 2, 4, 5) on the northbound switches

• While HyperFlex initial configuration does not use Priority Best Effort, it can be used for other traffic types

BRKVIR-2700 64


Eight updating vNIC templates are created and assigned a QoS policy, MTU size, and a VLAN*

Silver

hv-mgmt-a

hv-mgmt-b

MTU 1500

VLAN hx-inband-

mgmt

Gold

vm-network-a

vm-network-b

MTU 1500

VLAN vm-network

Bronze

hv-vmotion-a

hv-vmotion-b

MTU 9000

VLAN hx-vmotion

Platinum

storage-data-a

storage-data-b

MTU 9000

VLAN hx-storage-

data

!

HX240cM4

Co

nso

le

1 6 12 18 24

HyperFlex

!

HX240cM4

Co

nso

le

1 6 12 18 24

HyperFlex

!

HX240cM4

Co

nso

le

1 6 12 18 24

HyperFlex

BRKVIR-2700 65


QoS

Policy Map

Uplink Interface

QoSSystem Class


Enable jumbo frames end-to-end to support HX storage and vMotion traffic

CISCO UCS 6248UP 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32

STAT

ID

CISCO UCS 6248UP 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32

STAT

ID

UCSM!

HX240cM4

Co

nso

le

1 6 12 18 24

HyperFlex

ESXi Hypervisor

BRKVIR-2700 66


OPTIONAL: Create additional vNIC templates and MAC pools for other traffic types

!

HX240cM4

Co

nso

le

1 6 12 18 24

HyperFlex

!

HX240cM4

Co

nso

le

1 6 12 18 24

HyperFlex

!

HX240cM4

Co

nso

le

1 6 12 18 24

HyperFlex

BRKVIR-2700 67


OPTIONAL: Create vHBA templates and WWNN/WWPN pools for FC/FCoE SAN connection

!

HX240cM4

Co

nso

le

1 6 12 18 24

HyperFlex

!

HX240cM4

Co

nso

le

1 6 12 18 24

HyperFlex

!

HX240cM4

Co

nso

le

1 6 12 18 24

HyperFlex

FC/FCoE SAN

FC/FCoE Switches

BRKVIR-2700 68


For compute nodes, configure local disk config and boot policies if ESXi is not installed on SD cards

BRKVIR-2700 69

• Prerequisites


• UCS

• vSphere



Agenda

vSphere


Deploy vCenter outside of the HX cluster if one is not present

!

HX240cM4

Co

nso

le

1 6 12 18 24

HyperFlex

!

HX240cM4

Co

nso

le

1 6 12 18 24

HyperFlex

!

HX240cM4

Co

nso

le

1 6 12 18 24

HyperFlex

HyperFlex VDI/VSI Cluster

vCenter

Virtual Server Farm

LAN

DNS vCenter

BRKVIR-2700 72


If a vCenter server is already present, double check that it meets the HX interoperability

DNS

vCenter

• vCenter version must be equal to or higher than the ESXi version on the HX node

• vCenter meets the requirements to manage all the Hosts and VMs in the environment – Tiny, Small, Medium, Large

• TCP ports used by HX are NOT used by other 3rd party software or plugins (i.e., port 8089, port 80 for vCenter on Windows)

BRKVIR-2700 73


Use the local SSO account during the HX installation

[email protected]

• Local SSO account has root-level privileges to vCenter

• It’s a member of the SystemConfigurationAdministators group by default

BRKVIR-2700 74


Use the local SSO account during the HX installation

[email protected]

• Local SSO account has root-level privileges to vCenter

• It’s a member of the SystemConfigurationAdministators group by default

BRKVIR-2700 75


Considerations when purchasing HX with vSphere licenses

!

HX240cM4

Co

nso

le

1 6 12 18 24

HyperFlex

ESXi 6.0 U1b , ESXi 6.0 U2 P3

BRKVIR-2700 76



!

HX240cM4

Co

nso

le

1 6 12 18 24

HyperFlex

OEM version of ESXi 6.0 Enterprise Plus factory-installed on HX nodes


No physical PAK or license keys provided

BRKVIR-2700 77



!

HX240cM4

Co

nso

le

1 6 12 18 24

HyperFlex



vSphere OEM license keys are obfuscated

i.e. 0N085-XXXXX-XXXXX-XXXXX-10LHH

All factory-installed HX nodes share the SAME OEM license key


BRKVIR-2700 78



!

HX240cM4

Co

nso

le

1 6 12 18 24

HyperFlex


No pre-installation of ESXi Standard, Essentials Plus, or ROBO editions


vSphere OEM license keys are obfuscated

i.e. 0N085-XXXXX-XXXXX-XXXXX-10LHH

All factory-installed HX nodes share the SAME OEM license key


BRKVIR-2700 79


Considerations when purchasing HX without vSphere licenses

!

HX240cM4

Co

nso

le

1 6 12 18 24

HyperFlex


BRKVIR-2700 80



!

HX240cM4

Co

nso

le

1 6 12 18 24

HyperFlex


ESXi license keys must be supplied by end-customer

BRKVIR-2700 81



!

HX240cM4

Co

nso

le

1 6 12 18 24

HyperFlex

OEM version of ESXi 6.0 Foundation factory-installed on HX nodes



BRKVIR-2700 82



!

HX240cM4

Co

nso

le

1 6 12 18 24

HyperFlex

OEM version of ESXi 6.0 Foundation factory-installed on HX nodes

After initial setup, license can be changed to any supported edition



BRKVIR-2700 83


Set ESXi to use an external syslog server

Confirm on your syslog server that logs are being received

BRKVIR-2700 84


Configuring an external syslog server can also be done using the vSphere client

BRKVIR-2700 85


Reset the ESXi Host root password from the default on all HX nodes

BRKVIR-2700 86


Alternatively, reset the ESXi Host root password from the DCUI

BRKVIR-2700 87


Alternatively, reset the ESXi Host root password from the DCUI

BRKVIR-2700 88


If the ESXi version needs to be upgraded, use the bundle.zip file available at Cisco.com

BRKVIR-2700 89


Set reserve vSphere HA failover CPU and Memory capacity

• n/cluster-size

• Ex. For a 4 node cluster with N+1, reserve ¼ or 25%

• Ex. For an 8 node cluster with N+1, reserve 1/8 or 13%

• Try to avoid using “host failures” admission control as it becomes inflexible in many cases with large reservations and wastes resources with unbalanced clusters

BRKVIR-2700 90


Accept the default setting Datastore for Heartbeating (automatic)

BRKVIR-2700 91


Leave default DRS Settings as is – Fully automated

BRKVIR-2700 92


Enable EVC BEFORE adding any workloads to the HX cluster

• Prepares the cluster for

• Adding HX nodes with newer CPUs (v4 CPU, Broadwell)

• Upgrading CPUs on existing HX nodes

• Use lowest CPU model in cluster (currently v3 CPU, Haswell)

BRKVIR-2700 93


HX installer will create vSwitches, port groups and assign VLANs and vmnics accordingly

BRKVIR-2700 94



CISCO UCS 6248UP 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32

STAT

ID FI-ACISCO UCS 6248UP 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32

STAT

ID FI-B

!

HX240cM4

Co

nso

le

1 6 12 18 24

HyperFlex

Active Stby Active Active Stby Active Active Stby

vmnic0 vmnic1vmnic2 vmnic3vmnic4 vmnic6 vmnic5 vmnic7

hv-mgmt-a

storage-data-a

hv-mgmt-b

storage-data-b

vm-network-b

vm-network-a

hv-vmotion-a

hv-vmotion-b

Port 1 Port 2

VIC 1227

BRKVIR-2700 95



CISCO UCS 6248UP 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32

STAT


STAT

ID FI-B

!

HX240cM4

Co

nso

le

1 6 12 18 24

HyperFlex



hv-mgmt-a

storage-data-a

hv-mgmt-b

storage-data-b

vm-network-b

vm-network-a

hv-vmotion-a

hv-vmotion-b

Port 1 Port 2

VIC 1227

BRKVIR-2700 96



CISCO UCS 6248UP 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32

STAT


STAT

ID FI-B

!

HX240cM4

Co

nso

le

1 6 12 18 24

HyperFlex



hv-mgmt-a

storage-data-a

hv-mgmt-b

storage-data-b

vm-network-b

vm-network-a

hv-vmotion-a

hv-vmotion-b

Port 1 Port 2

VIC 1227

BRKVIR-2700 97



CISCO UCS 6248UP 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32

STAT


STAT

ID FI-B

!

HX240cM4

Co

nso

le

1 6 12 18 24

HyperFlex



hv-mgmt-a

storage-data-a

hv-mgmt-b

storage-data-b

vm-network-b

vm-network-a

hv-vmotion-a

hv-vmotion-b

Port 1 Port 2

VIC 1227

BRKVIR-2700 98



CISCO UCS 6248UP 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32

STAT


STAT

ID FI-B

!

HX240cM4

Co

nso

le

1 6 12 18 24

HyperFlex



hv-mgmt-a

storage-data-a

hv-mgmt-b

storage-data-b

vm-network-b

vm-network-a

hv-vmotion-a

hv-vmotion-b

Port 1 Port 2

VIC 1227

BRKVIR-2700 99


QoS

Policy Map

Uplink Interface

QoSSystem Class


Enable jumbo frames end-to-end to support HX storage and vMotion traffic

CISCO UCS 6248UP 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32

STAT

ID

CISCO UCS 6248UP 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32

STAT

ID

UCSM!

HX240cM4

Co

nso

le

1 6 12 18 24

HyperFlex

ESXi Hypervisor

BRKVIR-2700 100


vMotion and VM networks are supported to migrate to a distributed switch (vDS or Nexus 1000v)

Mirror the configuration from the standard vSwitch settings

BRKVIR-2700 101


For vMotion, set the MTU size to 9000 at the dvSwitch and VMkernel port group levels

BRKVIR-2700 102

• Prerequisites


• UCS

• vSphere



Agenda

HX Data Platform


After HX deployment, SSH into the HX installer and execute the post_install script

root@Cisco-HX-Data-Platform-

Installer:~# post_install

BRKVIR-2700 105


After HX deployment, SSH into the HX installer and execute the post_install script

root@Cisco-HX-Data-Platform-

Installer:~# post_install

• Helps streamline some of the post-

install tasks

• Can be re-run to add additional

VLANs on existing HX nodes

• Additional converged or compute

nodes would require manual

configuration

BRKVIR-2700 106


Storage

Cluster

Management IP

Configure ASUP to get information about failures and cluster health

SCVM

SCVM

SCVMYes

asup.hyperflex.io

[email protected]

BRKVIR-2700 107


For ASUP, enter the SMTP information, enable ASUP services, add email recipients, and review

saipan-hx-cluster

asup.hyperflex.io

[email protected]

[email protected]

BRKVIR-2700 108



root@SpringpathController:~# stcli services smtp set --smtp

asup.hyperflex.io --fromaddress [email protected]

saipan-hx-cluster

asup.hyperflex.io

[email protected]

[email protected]

BRKVIR-2700 109

mailto:[email protected]



root@SpringpathController:~# stcli services smtp set --smtp

asup.hyperflex.io --fromaddress [email protected]

root@SpringpathController:~# stcli services smtp show

smtpServer: asup.hyperflex.io

fromAddress: [email protected]

saipan-hx-cluster

asup.hyperflex.io

[email protected]

[email protected]

BRKVIR-2700 110





root@SpringpathController:~# stcli services asup enable

saipan-hx-cluster

asup.hyperflex.io

[email protected]

[email protected]

BRKVIR-2700 111




root@SpringpathController:~# stcli services asup recipients set –-recipients

[email protected]

root@SpringpathController:~# stcli services asup recipients add –-recipients

[email protected]

saipan-hx-cluster

asup.hyperflex.io

[email protected]

[email protected]

BRKVIR-2700 112






root@SpringpathController:~# stcli services asup recipients set –-recipients

[email protected]

root@SpringpathController:~# stcli services asup recipients add –-recipients

[email protected]

root@SpringpathController:~# stcli services asup show

recipientList:

----------------------------------------

[email protected]

----------------------------------------

[email protected]

----------------------------------------

enabled: True

saipan-hx-cluster

asup.hyperflex.io

[email protected]

[email protected]

BRKVIR-2700 113






Send a test email to verify that SCVM can send emails to the test recipient

root@SpringpathController:~# echo "i am testing email

capability of this SCVM" | mailx -s "this is a test"

[email protected]

BRKVIR-2700 114





[email protected]

• Check the target mailbox

BRKVIR-2700 115





[email protected]

• Check the target mailbox

root@SpringpathController:~# sendasup –t

• You can search ASUP in a text editor using CTRL+F

BRKVIR-2700 116


Change the SCVM password from the default during post deployment

Running this command will apply the password to all SCVMs

BRKVIR-2700 117


Manage your HX datastores through the HX plugin or HX Dashboard – i.e, creation, resizing, renaming

BRKVIR-2700 118


Manage your HX datastores through the HX plugin or HX Dashboard – i.e, creation, resizing, renaming

BRKVIR-2700 119


Monitor free space on the HX cluster and put capacity planning measures in place

BRKVIR-2700 120


Use HX Data Platform to create VM snapshots

BRKVIR-2700 121



BRKVIR-2700 122



BRKVIR-2700 123


When cloning VMs, use HX ReadyClones

BRKVIR-2700 124


When cloning VMs, use HX ReadyClones

BRKVIR-2700 125


When cloning VMs, use HX ReadyClones or normal VMware Clone Wizard

BRKVIR-2700 126

• Prerequisites


• UCS

• vSphere



Agenda

Troubleshooting Scenarios


Confirm network connectivity if problems are encountered during install

• SSH to HX installer

• Ping the UCSM cluster IP

• Ping the vCenter server IP

• Ping all ESXi hosts management IP (vmk0)

• Verify DNS resolution functional:

• vCenter

• SSO URI

• UCSM

• ESXi hosts

BRKVIR-2700 129


Synchronize time fails if not using an internal time source

• Use a reliable NTP time source (internal server or domain controller)

• Avoid using public NTP servers

BRKVIR-2700 130


VM migration fails due to missing VMkernel adapter

• vMotion VMkernel adapter is not created automatically unless post_install script is run after deployment or if it is manually created

BRKVIR-2700 131


Create the VMkernel adapter either in CLI or through the vSphere client with MTU 9000

[root@esxi-01:~] esxcfg-vmknic -a -i 192.168.4.158 -n

255.255.255.0 –m 9000 -p vMotion

BRKVIR-2700 132


Confirm that the vMotion VLAN is on the port-group, vNICs, uplink interfaces, and switchports

BRKVIR-2700 133


Confirm that the vMotion VLAN is on the port-group, vNICs, uplink interfaces, and switchports

BRKVIR-2700 134


Confirm that the vMotion VLAN is on the port-group, vNICs, network uplink ports, and switchports

BRKVIR-2700 135



BRKVIR-2700 136



BRKVIR-2700 137


Test vMotion connectivity with vmkping between HX nodes

A regular vmkping will confirm initial connectivity between HX nodes

BRKVIR-2700 138


Test vMotion connectivity with vmkping between HX nodes

A jumbo vmkping will confirm jumbo frames between HX nodes is functional

BRKVIR-2700 139


If it fails with jumbo MTU, check vSwitch, vNICs, QoS, northbound switch MTU and CoS for misconfig

BRKVIR-2700 140



BRKVIR-2700 141



BRKVIR-2700 142



BRKVIR-2700 143



vMotion traffic is localized to FI-A unless a network failure event causes a failover to FI-B

BRKVIR-2700 144


Other issues that would cause migration to fail

VM Level Issues

• Incompatible CPUs – mixed CPU families with HX or compute-only

• VM port group is not created on destination host

• Additional VMDKs (external storage) not available to all hosts

• VM connected to physical hardware (PCI passthrough, USB, etc)

• Not enough resources to accommodate VM

BRKVIR-2700 145


If the HX node is NOT re-imaged with the HX custom ESXi ISO, the ESXi configuration step fails

BRKVIR-2700 146



• Check the latest Cisco documentation

BRKVIR-2700 147




• Use the HX custom ISO file as it contains an integrated install script that follows the factory imaging procedures

BRKVIR-2700 148





• Mount this ISO via KVM and re-image all HX nodes (non-interactive)

BRKVIR-2700 149





• Mount this ISO via KVM and re-image all HX nodes (non-interactive)

• WARNING: Re-imaging ESXi removes the OEM license key

• Open a TAC case to recover the license

BRKVIR-2700 150


During HX upgrade, java.net.ConnectionException: Connection refused error message appears

BRKVIR-2700 151



• Caused by SSH on ESXi being disabled

BRKVIR-2700 152



• Caused by SSH on ESXi being disabled

• Enable SSH on all HX nodes to continue with the HX upgrade

BRKVIR-2700 153


HX Upgrades: Multiple Clusters Registered to One vCenter

• Each cluster must be independently bootstrapped

• First bootstrap will update the plugin for all clusters, but you must ensure each cluster has the latest packages installed by running cluster-bootstrap.sh

BRKVIR-2700 154


HX Upgrades: UCS Not Running Expected Version

This will occur if you fail to upgrade the UCS infrastructure before beginning the upgrade. Follow the earlier steps to upgrade the UCS infrastructure first.

BRKVIR-2700 155


HX Upgrades: Node Maintenance Mode Failed

Will occur after a 1 hour timeout if you fail to vacate the running VMs from that host.

Re-try the upgrade again and be sure to check on the status of VM migration referencing vCenter and the event log for the current node

• If DRS is enabled, a VM might get stuck due to incompatibility to vMotion (network problem, ISO mounted, CPU features missing, etc.)

• If DRS is disabled or enabled but not in fully automatic mode, you will have to manually migrate your VMs on each node

BRKVIR-2700 156


HX Upgrades: Error Occurred While Configuring UCS

Occurs after a 2 hour timeout if the server fails to complete firmware upgrades & full boot into ESXi

1. Check UCSM for the progress of the firmware update

2. Remediate the issue

3. Wait for ESXi to boot

4. Exit maintenance mode

5. Re-try upgrade again

BRKVIR-2700 157


HX Upgrades: Validation Failed – UCS Version

• Ensure you have uploaded the packages to UCSM

• Check the exact syntax of your input into target version textbox

• E.g. 3.1(2b)

BRKVIR-2700 158


HX Upgrades: No Connection Error

1. If a converged node is in maintenance mode with controller VM shutdown

2. Didn’t upgrade HXDP first before FW in split upgrade

3. Any stCtlVM not running stNodeMgr• Can run “status stNodeMgr” to see if it is

running

• If it isn’t running, it is preferred to restart the CVM

• Do not attempt to do this without TAC assistance!

• You might bring down an online cluster if you fail to check failure tolerances

BRKVIR-2700 159


HX Upgrades: Monitoring Progress

• Best to watch progress in vCenter• If your session is disconnected, you can reconnect to the plugin and resume monitoring the

upgrade

• You can also monitor via the CLI• CLI session will disconnect when the controller VM you are connected to is rebooted

• stcli cluster upgrade --status

• Based on current step, you can find more details within that management console (UCSM, ESX event log, etc.)

• Don’t assume the upgrade is frozen – many times we are healing behind the scenes• Use “stcli cluster storage-summary --detail” to check healing status

BRKVIR-2700 160


• HyperFlex Field Install Guide http://cisco.box.com/v/hx-install

• HyperFlex Field Upgrade Guidehttp://cisco.box.com/v/hx-upgrade

• HyperFlex Customer Cleanup/Reinstall Guidehttp://cisco.box.com/v/hx-cleanup

HyperFlex Field Guides

http://cisco.box.com/v/hx-install

http://cisco.box.com/v/hx-upgrade

http://cisco.box.com/v/hx-cleanup

• Prerequisites


• UCS

• vSphere


Agenda


Complete Your Online Session Evaluation

Don’t forget: Cisco Live sessions will be available for viewing on-demand after the event at CiscoLive.com/Online

• Please complete your Online Session Evaluations after each session

• Complete 4 Session Evaluations & the Overall Conference Evaluation (available from Thursday) to receive your Cisco Live T-shirt

• All surveys can be completed via the Cisco Live Mobile App or the Communication Stations

BRKVIR-2700 163

CiscoLive.com/Online


Continue Your Education

• Demos in the Cisco campus

• Visit the Hub (2.2) for:

• DevNet Zone related sessions and demos

• Walk-in Self-Paced Labs

• Technical Solutions Clinic

• Lunch & Learn

• Meet the Engineer 1:1 meetings

• Related sessions

• Join the session’s Spark room

BRKVIR-2700 164













and Cisco

BRKVIR-2700 165


Cisco SparkAsk Questions, Get Answers, Continue the Experience

Use Cisco Spark to communicate with the Speaker and fellow participants after the session

Download the Cisco Spark app from iTunes or Google Play

1. Go to the Cisco Events mobile app

2. Find this session

3. Click the Spark button under Speakers in the session description

4. Enter the room, room name = BRKVIR-2700

5. Join the conversation!

The Spark Room will be open for 2 weeks after Cisco Live

BRKVIR-2700 166

Thank You

Appendix


Links• Documentation Roadmap:

http://www.cisco.com/c/en/us/td/docs/hyperconverged_systems/HyperFlex_HX_DataPlatformSoftware/HX_Documentation_Roadmap/HX_Series_Doc_Roadmap.html

• Pre-install Checklist: http://www.cisco.com/c/en/us/td/docs/hyperconverged_systems/HyperFlex_HX_DataPlatformSoftware/HyperFlex_preinstall_checklist/b_HyperFlexSystems_pre_install_checklist_1_8.html

• Getting Started Guide: http://www.cisco.com/c/en/us/td/docs/hyperconverged_systems/HyperFlex_HX_DataPlatformSoftware/GettingStartedGuide/1-8/b_HyperFlexSystems_GettingStartedGuide_1_8_c.html

• Configuration Guides: http://www.cisco.com/c/en/us/support/hyperconverged-systems/hyperflex-hx-data-platform-software/products-installation-and-configuration-guides-list.html

• End-User Guides: http://www.cisco.com/c/en/us/support/hyperconverged-systems/hyperflex-hx-data-platform-software/products-user-guide-list.html

• Cisco Validated Design for HCI: http://www.cisco.com/c/en/us/solutions/enterprise/data-center-designs-cloud-computing/hyperconverged-infrastructure-design.html

• HX220/HX240 Node Installation Guides: http://www.cisco.com/c/en/us/support/hyperconverged-systems/hyperflex-hx-series/products-installation-guides-list.html

BRKVIR-2700 170

http://www.cisco.com/c/en/us/td/docs/hyperconverged_systems/HyperFlex_HX_DataPlatformSoftware/HX_Documentation_Roadmap/HX_Series_Doc_Roadmap.html

http://www.cisco.com/c/en/us/td/docs/hyperconverged_systems/HyperFlex_HX_DataPlatformSoftware/HyperFlex_preinstall_checklist/b_HyperFlexSystems_pre_install_checklist_1_8.html

http://www.cisco.com/c/en/us/td/docs/hyperconverged_systems/HyperFlex_HX_DataPlatformSoftware/GettingStartedGuide/1-8/b_HyperFlexSystems_GettingStartedGuide_1_8_c.html

http://www.cisco.com/c/en/us/support/hyperconverged-systems/hyperflex-hx-data-platform-software/products-installation-and-configuration-guides-list.html

http://www.cisco.com/c/en/us/support/hyperconverged-systems/hyperflex-hx-data-platform-software/products-user-guide-list.html

http://www.cisco.com/c/en/us/solutions/enterprise/data-center-designs-cloud-computing/hyperconverged-infrastructure-design.html

http://www.cisco.com/c/en/us/support/hyperconverged-systems/hyperflex-hx-series/products-installation-guides-list.html


Links• Release Notes: http://www.cisco.com/c/en/us/support/hyperconverged-systems/hyperflex-hx-data-platform-software/products-

release-notes-list.html

• Upgrade Guides: http://www.cisco.com/c/en/us/support/hyperconverged-systems/hyperflex-hx-data-platform-software/products-installation-guides-list.html

• Disjoint Layer 2: http://www.cisco.com/c/en/us/solutions/collateral/data-center-virtualization/unified-computing/white_paper_c11-692008.html

• Cisco HyperFlex HX-Series Hardware and Software Interoperabiliy Matrix in UCSM Release 3.1(2): http://www.cisco.com/c/dam/en/us/td/docs/unified_computing/ucs/interoperability/matrix/b_hcl_HX-rel-3-1-2.pdf

• Video: Cisco HyperFlex Day Zero Configuration: http://www.cisco.com/c/en/us/td/docs/hyperconverged_systems/HyperFlex_HX_DataPlatformSoftware/videos/cisco_hyperflex_day_zero_configuration.html

• Video: Cisco HyperFlex Compute Only Node Addition: http://www.cisco.com/c/en/us/td/docs/hyperconverged_systems/HyperFlex_HX_DataPlatformSoftware/videos/cisco_hyperflex_compute_only_node_addition.html

BRKVIR-2700 171

http://www.cisco.com/c/en/us/support/hyperconverged-systems/hyperflex-hx-data-platform-software/products-release-notes-list.html

http://www.cisco.com/c/en/us/support/hyperconverged-systems/hyperflex-hx-data-platform-software/products-installation-guides-list.html

http://www.cisco.com/c/en/us/solutions/collateral/data-center-virtualization/unified-computing/white_paper_c11-692008.html

http://www.cisco.com/c/dam/en/us/td/docs/unified_computing/ucs/interoperability/matrix/b_hcl_HX-rel-3-1-2.pdf

http://www.cisco.com/c/en/us/td/docs/hyperconverged_systems/HyperFlex_HX_DataPlatformSoftware/videos/cisco_hyperflex_day_zero_configuration.html

http://www.cisco.com/c/en/us/td/docs/hyperconverged_systems/HyperFlex_HX_DataPlatformSoftware/videos/cisco_hyperflex_compute_only_node_addition.html

Appendix: Prerequisites


HyperFlex Pre-Install Checklist: UCS Fabric Interconnect Information

Uplink Switch Model

Fabric Interconnect Cluster IP Address

UCS FI-A Address

UCS FI-B Address

Pool for KVM IPs (One per HX node is required)

Subnet Mask

Default Gateway

MAC Pool Prefix

User ID

Password

BRKVIR-2700 173


HyperFlex Pre-Install Checklist: VLAN Information

Network VLAN ID Description

hx-inband-mgmtUsed for management traffic between ESXi, HX, and

vCenter. Must be routable

hx-storage-data Used for storage traffic and requires L2

hx-vmotion Used for live migration of VMs

vm-network Used by VM/Application network

additional network

additional network

additional network

BRKVIR-2700 174


HyperFlex Pre-Install Checklist: IP Address Information

Management Network Storage Data Network

ESXi vmk0 Storage Controller Eth0 ESXi vmk1 Storage Controller Eth1

Node 1 IP Address

Node 2 IP Address

Node 3 IP Address

Node 4 IP Address

Subnet Mask Subnet Mask

Default Gateway Default Gateway

BRKVIR-2700 175


HyperFlex Pre-Install Checklist: IP Address Information

• 3 VLANs recommended – 2 required

• Number of IP addresses• VLAN X (management): 2 x number of nodes + 2

• E.g. 4 node cluster needs 10 IPs. 8 node cluster needs 18 IPs

• VLAN Y (storage data): 2 x number of nodes + 1 [can all be non-routable/private]• E.g. 4 node cluster needs 9 IPs. 8 node cluster needs 17 Ips

• VLAN Z (vMotion): 1 x number of nodes [can be non-routable/private]

• UCSM VLAN (typically X): 1 x number of nodes + 3 [for new deployments only]

• In total:• E.g. 4 node cluster needs 30 IPs, 8 node cluster needs 54 IPs

• Be very careful IPs fall in proper subnet and check for duplicate IPs!

BRKVIR-2700 176


HyperFlex Pre-Install Checklist: vMotion Network IP Address (configured by post_install script)

IP Address HX Node

IP Address Server 1

IP Address Server 2

IP Address Server 3

IP Address Server 4

Subnet Mask: Default Gateway:

If vMotion network is on its own dedicated TCP/IP stack, include the subnet mask and default gateway here

BRKVIR-2700 177


HyperFlex Pre-Install Checklist: Storage Cluster Information

Storage Cluster

Storage Cluster Name

Storage Cluster Management IP Address

Storage Cluster Data IP Address

BRKVIR-2700 178


HyperFlex Pre-Install Checklist: ESXi Information

Hypervisor Credentials Default

Username root

Password Cisco123

Hypervisor Credentials New

Username root

Password

BRKVIR-2700 179


HyperFlex Pre-Install Checklist: vCenter Information

vCenter Configuration

vCenter Server

https://<IP address>

(default ports are required)

vCenter admin user name

username@domain

vCenter Datacenter Name

Datacenter in vCenter where HX cluster will reside

Cluster Name

ESXi cluster name under the Datacenter where that the

HX nodes will be added to

BRKVIR-2700 180


HyperFlex Pre-Install Checklist: SSO Information

Single Sign On (SSO)

SSO Server URL

• Only required if SSO URL is not reachable

• Automatic for ESXi 6.0 and later

BRKVIR-2700 181


HyperFlex Pre-Install Checklist: Network Information

Network Services

DNS Server(s)

IP Address

NTP Server(s)

IP Address

Time Zone

Example: US/Eastern, US/Pacific

BRKVIR-2700 182


HyperFlex Pre-Install Checklist: Auto Support Information

Auto Support

Enable Auto Support

Yes/No

SMTP Server

Example: mailhost.company.com

Mail Sender

Example: [email protected]

BRKVIR-2700 183



Physical Requirements

• 2-4 uplink connections for FIs

• Each FI will need at least two 10 Gb connection to existing network

• Each HX node will need at least two 10 Gb connections

• Optical or Twinax cables are supported

• Do NOT mix optical and twinax cables

• Example: • HX node connects to FI-A with twinax and FI-B with optical

• FI A connects to Switch 1 with twinax and Switch 2 with optical

BRKVIR-2700 184


Physical Requirements: route network cables

• Connect server port 1 to FI-A (right port)

• Connect server port 2 to FI-B (left port)

• Use the same port numbers for each node

• Port 1 on FI-A and FI-B to server 1

• Port 2 on FI-A and FI-B to server 2

• …

• Connect uplink ports on each FI

• Suggest to use port 17 or higher

• 10Gb uplinks are strongly advised

BRKVIR-2700 185


Physical Requirements: Rack Space

• For HX220c Cluster

• 2RU for ToR switch pair

• 2RU for 6248 pair

• 4RU for 6296 pair

• 1RU for each HX220c node

• For HX240c Cluster

• 2RU for ToR switch pair

• 2RU for 6248 pair, 4RU for 6296 paisr

• 2RU for each HX240c node

• 4RU for 5108 chassis for up to eight B200 M4 ½ width blades

• 2 C13/C14 power connections to 15 AMP circuit per device

BRKVIR-2700 186


Verify datacenter space for the HX cluster (rack space, power, cooling, networking cables)

• UCS FI 6248UP = 1 RU

• UCS FI 6296UP = 2 RU

• HX220c / C220 M4 = 1 RU

• HX240c / C240 M4 = 2 RU

CISCO UCS 6248UP 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32

STAT

ID

2 3 4

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48

CISCO UCS 6296UP

STAT

ID

4

1

5

2

6

3

7 8HX220c

M4

Console!

HyperFlex

!

HX240cM4

Co

nso

le

1 6 12 18 24

HyperFlex

600W to

1400W

770W

950W

600W

Max power per component

BRKVIR-2700 187


Verify datacenter space for the HX cluster (rack space, power, cooling)

• 5108 Chassis = 4 RU

• B200 M4 = ½ width 1 RU

SLOT

1

SLOT

5

SLOT

3

SLOT

7

SLOT

2

SLOT

6

SLOT

4

SLOT

8

!

UCS 5108

OK FAIL OK FAIL OK FAIL OK FAIL

! ResetConsole

UCS B200 M4

! ResetConsole

UCS B200 M4

! ResetConsole

UCS B200 M4

! ResetConsole

UCS B200 M4

! ResetConsole

UCS B200 M4

! ResetConsole

UCS B200 M4

! ResetConsole

UCS B200 M4

! ResetConsole

UCS B200 M4

2500W

Max power output per chassis

BRKVIR-2700 188

Appendix: UCS


For new installs, download the UCS A-, B-, and C-firmware bundles and stage them in UCS Manager

A

B

C

BRKVIR-2700 190


For new installs, download the UCS A-, B-, and C-firmware bundles and stage them in UCS Manager

BRKVIR-2700 191


EXAMPLE – Use Auto-Install to upgrade the Infrastructure firmware – Only for new installs

BRKVIR-2700 192


EXAMPLE – Use Auto-Install to upgrade Server firmware for converged and compute nodes

BRKVIR-2700 193


Confirm the desired package version has been fully applied to all UCS components

BRKVIR-2700 194


HX installer configures QoS Policies that will be mapped to vNIC templates

BRKVIR-2700 195


HX installer also creates MAC pools with 100 MAC addresses for each vNIC template

!

HX240cM4

Co

nso

le

1 6 12 18 24

HyperFlex

!

HX240cM4

Co

nso

le

1 6 12 18 24

HyperFlex

!

HX240cM4

Co

nso

le

1 6 12 18 24

HyperFlex

BRKVIR-2700 196


To add new vNICs, run and complete the UCSM and ESXi configuration only in the HX installer

BRKVIR-2700 197


To add new vNICs, run and complete the UCSM and ESXi configuration only in the HX installer

BRKVIR-2700 198


After UCS and ESXi configurations have completed, create new MAC pools for each new vNIC

BRKVIR-2700 199


Create additional vNIC templates under the HX cluster sub-org

BRKVIR-2700 200


Create a pair of vNIC templates with 1:1 mapping to FI-A or FI-B

CISCO UCS 6248UP 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32

STAT

ID

CISCO UCS 6248UP 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32

STAT

ID

FI-A

FI-B

BRKVIR-2700 201


Add newly created vNIC templates to HyperFlexLAN connectivity policy

BRKVIR-2700 202


Name the vNIC, map the correct vNIC template, and select the HyperFlex adapter policy

BRKVIR-2700 203


Accept the changes to place the HX nodes into Pending Activities to prepare for staged reboots

Select the checkboxes for all the affected servers and click OK to apply the changes

BRKVIR-2700 204


Once the vNICs/vHBAs have been created, go back into HX installer and deploy HX

BRKVIR-2700 205


Once the vNICs/vHBAs have been created, go back into HX installer and deploy HX

BRKVIR-2700 206


If HX cluster is already deployed, verify that the HX cluster is healthy prior to rebooting the HX node

BRKVIR-2700 207


In vSphere web client, place the HX Node into HX Maintenance Mode

• If DRS is enabled, VMs will be migrated automatically and the stctlvm will be shut down gracefully

• If there is no DRS, manually migrate all the VMs off beforehand (except stctlvm)

• Confirm HX node has successfully entered maintenance mode

BRKVIR-2700 208


At UCSM, go to Pending Activities, select the server, click the checkbox for Reboot Now, and click OK

BRKVIR-2700 209


Once the HX node is back online, update its PCI Devices by adding the LSI3108_1 device

BRKVIR-2700 210


Reboot the HX node in order for the changes to take affect

BRKVIR-2700 211


Once the HX node is back online, exit HX maintenance mode

NOTE: If HX maintenance mode does not work, then simply exit maintenance mode the old fashioned way

BRKVIR-2700 212


Select the HX node’s stctlvm and edit its hardware settings to remove and re-add its PCI card

BRKVIR-2700 213


Select the HX node’s stctlvm and edit its hardware settings to remove and re-add its PCI card

BRKVIR-2700 214


Power on the stctlvm and monitor the HX storage cluster until it displays “healthy”

Once cluster is healthy, it is safe to move on to next HX node

BRKVIR-2700 215

Appendix: vSphere


vSwitch Settings vSwitch

NameVMNIC

Active

Standby

UCS

vNIC

MTU

Size

QoS

PolicyCoS

Packet

Drop

vswitch-hx-inband-mgmt vmnic0 Active hv-mgmt-a 1500 Silver 2 Yes

vswitch-hx-inband-mgmt vmnic1 Standby hv-mgmt-b 1500 Silver 2 Yes

vswitch-hx-storage-data vmnic2 Standby storage-data-a 9000 Platinum 5 No

vswitch-hx-storage-data vmnic3 Active storage-data-b 9000 Platinum 5 No

vswitch-hx-vm-network vmnic4 Active vm-network-a 1500 Gold 4 Yes

vswitch-hx-vm-network vmnic5 Active vm-network-b 1500 Gold 4 Yes

vmotion vmnic6 Active hv-vmotion-a 9000 Bronze 1 Yes

vmotion vmnic7 Standby hv-vmotion-b 9000 Bronze 1 Yes

BRKVIR-2700 217


Under vswitch-hx-vm-network, create additional port groups for other VM traffic for each HX node

BRKVIR-2700 218


Ensure VLANs are created and allowed in the northbound switches and in UCS ManagerInterface port-channel 201

switchport trunk allowed vlan 4-6, 10, 20, 67, 500, 1000, 2000, 3000

Interface port-channel 202

switchport trunk allowed vlan 4-6, 10, 20, 67, 500, 1000, 2000, 3000

BRKVIR-2700 219


For Disjoint L2 VMs, create a custom vSwitch in CLI, then add vmnics, and create port groupsSSH into the HX node and create a vSwitch with a custom name[root@esxi-01:~] esxcli network vswitch standard add -v vswitch-dmz-network

Repeat steps on every HX nodeBRKVIR-2700 220


ESXi Upgrade: Copy the bundle.zip file to a datastore that the HX node can access

FTP, SFTP,

TFTP, SCP

vSphere Client

HX Datastore

HX-Vmware-ESXi-60U2-4192238-Cisco-Custom-Bundle-6.0.2.3.zip

BRKVIR-2700 221


Enter the HX node into HX Maintenance Mode to prepare for the upgrade

BRKVIR-2700 222


SSH into the HX node, review the bundle, and then perform the upgrade[root@hx-esxi:~] esxcli software sources profile list -d /vmfs/volumes/hx-datastore/HX-

Vmware-ESXi-60U2-4192238-Cisco-Custom-Bundle-6.0.2.3.zip

Name Vendor Acceptance Level

------------------------------------------------------------------------

Vmware-ESXi-60U2-4192238-Cisco-Custom-HX Cisco PartnerSupported

[root@hx-esxi:~] esxcli software profile update -d /vmfs/volumes/hx-datastore/HX-Vmware-

ESXi-60U2-4192238-Cisco-Custom-Bundle-6.0.2.3.zip -p Vmware-ESXi-60U2-4192238-Cisco-

Custom-HX

Update Result Message: The update completed successfully, but the system needs to be

rebooted for the changes to be effective. Reboot Required

[root@hx-esxi:~] reboot

BRKVIR-2700 223


Once the HX node is back online, verify the ESXi version and then exit HX maintenance mode

BRKVIR-2700 224


Once the HX node is back online, verify the ESXi version and then exit HX maintenance mode

BRKVIR-2700 225


Check the health of the HX cluster

Repeat this process on each HX node until all are upgraded to the same ESXi version

BRKVIR-2700 226

Appendix: HX Data Platform


Sample output of post_install scriptroot@Cisco-HX-Data-Platform-Installer:~# post_install

Logging in to controller 10.255.255.253

HX CVM root Password for:

Getting ESX hosts from HX cluster...

vCenter URL: 10.255.255.254

Enter vCenter username (user@domain):

[email protected]

vCenter Password:

Found datacenter CNMI

Found cluster Saipan-HX-Cluster

Enable HA/DRS on cluster? (y/n) y

Disable SSH warning? (y/n) y

Configure ESXi logging onto HX datastore? (y/n) y

No datastores found

Creating datastore...

Name of datastore: Saipan-DS-1

Size (GB): 1024

Storing logs on datastore Saipan-DS-1

Creating folder [Saipan-DS-1]/esxi_logs

Add vmotion interfaces? (y/n) y

Netmask for vMotion: 255.255.255.0

VLAN ID: (0-4096) 4000

vMotion IP for 10.255.255.250: 192.168.255.250

Adding vmotion to 10.255.255.250

Adding vmkernel to 10.255.255.250

vMotion IP for 10.255.255.251: 192.168.255.250



vMotion IP for 10.255.255.252: 192.168.255.252



Add VM network VLANs? (y/n) y

Attempting to find UCSM IP

Found UCSM 10.255.255.200, logging with username admin. Org is

Saipan-HX-Cluster

UCSM Password:

Port Group Name to add: Saipan-Prod

VLAN ID: (0-4096) 100

Adding VLAN 100 to FI

Adding VLAN 100 to vm-network-a VNIC template

Adding VLAN 100 to vm-network-b VNIC template

Adding Saipan-Prod to 10.255.255.250



Add additional VM network VLANs? (y/n) n

BRKVIR-2700 228


Sample output of post_install script (cont’d)Enable NTP on ESX hosts? (y/n) y

Starting ntpd service on 10.0.0.251



Send test email? (y/n) y

Recipient email address: [email protected]

Sending test email to [email protected]

Verify if email recieved.

Validating cluster health and configuration...

Checking MTU settings

ESX password:

Pinging 192.168.100.251 from vmk1



Setting vnic2 to active and vmnic3 to standby


Pinging 192.168.100.253 with mtu 8972 from vmk1





Setting vmnic3 to active and vnic2 to standby




















BRKVIR-2700 229


Sample output of post_install script (cont’d)Network Summary:

Host: 10.0.0.251

vswitch: vswitch-hx-inband-mgmt - mtu: 1500 - policy:

loadbalance_srcid

vmnic0 - 1 - CNMI-FI-A - active

vmnic1 - 1 - CNMI-FI-B - standby

Portgroup Name - VLAN

Storage Controller Management Network - 10

Management Network - 10

vswitch: vswitch-hx-vm-network - mtu: 1500 - policy:

loadbalance_srcid


vmnic5 - 1 - CNMI-FI-B - active


Saipan-Prod-10 - 10

Saipan-Dev-20 - 20

vswitch: vmotion - mtu: 9000 - policy: loadbalance_srcid




vmotion - 101

vswitch: vswitch-hx-storage-data - mtu: 9000 - policy:

loadbalance_srcid

vmnic2 - 1 - CNMI-FI-A - standby



Storage Controller Data Network - 100

Storage Hypervisor Data Network - 100

Host: 10.0.0.252


loadbalance_srcid







loadbalance_srcid




Saipan-Prod-10 - 10

Saipan-Dev-20 - 20





vmotion - 101


loadbalance_srcid






BRKVIR-2700 230


Sample output of post_install script (cont’d)Host: 10.0.0.253


loadbalance_srcid







loadbalance_srcid




Saipan-Prod-10 - 10

Saipan-Dev-20 - 20





vmotion - 101


loadbalance_srcid






Host: 10.0.0.251

No errors found

Host: 10.0.0.252

No errors found

Host: 10.0.0.253

No errors found

Controller VM Clocks:

stCtlVM-FCH2017V1AB - 2016-12-25 12:25:10

stCtlVM-FCH2017V2CD - 2016-12-25 12:25:20

stCtlVM-FCH2017V3EF - 2016-12-25 12:25:30

Cluster:

Version - 1.8.1c-19725

Model - HX220C-M4S

Health - HEALTHY

Access Policy - LENIENT

ASUP enabled - True

SMTP Server - mail.cnmi.local

BRKVIR-2700 231

Appendix: Troubleshooting Scenarios


HX Upgrades: Configure Lenient Mode

Lenient mode is now default in 1.8(x)

1. SSH to any one of the controller VMs and login as root

2. Check if lenient mode is already configured

~/#stcli cluster get-cluster-access-policy

3. If set to strict, change to lenient. If already lenient, no further action is required

~/#stcli cluster set-cluster-access-policy --name lenient

4. Confirm the change by typing “stcli cluster info | grep -i policy”

BRKVIR-2700 233


HX Upgrades: Upgrading from 2.2(6f)

• Upgrade time will increase when upgrading from 2.2(6f) due to deep discovery of storage controller and VIC

• Add up to 60 additional minutes per node to your planning

• You might see node reboot multiple times while installing firmware

• The process is completely automated but can be monitored from FSM tab

• This must only be done once in a cluster’s lifetime and only if starting at 2.2(6f)

BRKVIR-2700 234


Create Validation fails due to DRIVER-Check mpt3sas and DRIVER-Check megaraid_sas

BRKVIR-2700 235



• Driver list in SCVM not updated

BRKVIR-2700 236




• Reboot the SCVMs to properly load the driver list

• Log into vSphere client and gracefully reboot all SCVMs

BRKVIR-2700 237




• Reboot the SCVMs to properly load the driver list

• Log into vSphere client and gracefully reboot all SCVMs

• Once all SCVMs are up and running, click “Retry Create Validation” to continue the HX installation

BRKVIR-2700 238


Deploy fails due to ‘disk /dev/… is formatted as a storfs disk’

BRKVIR-2700 239



• Occurs when ESXi is re-installed on the HX node

BRKVIR-2700 240




• Click Edit Configuration Continue

BRKVIR-2700 241





• Check the box “Clean up disk partitions” in the Advanced Configuration

BRKVIR-2700 242





• Check the box “Clean up disk partitions” in the Advanced Configuration

Click Continue Click “Retry

Deploy” at the top of the page

BRKVIR-2700 243

Documents

HyperFlex Configuration Best Practices and Troubleshootingclnv.s3.amazonaws.com/2017/eur/pdf/BRKVIR-2700.pdf · HyperFlex Configuration Best Practices and Troubleshooting ... Customer