Deploying and Troubleshooting the Nexus 1000v …d2zmdbbm9feqrf.cloudfront.net/2014/usa/pdf/BRKVIR-3013.pdfDeploying and Troubleshooting the Nexus 1000v Virtual Switch on vSphere BRKVIR-3013

Deploying and Troubleshooting the Nexus 1000v Virtual Switch on vSphere

BRKVIR-3013

Matthew Wronkowski – Technical Leader Virtualization Services

© 2014 Cisco and/or its affiliates. All rights reserved. BRKVIR-3013 Cisco Public

Agenda

• Current N1K Releases and New Features

• Licensing

• Virtual Supervisor Module (VSM) & VEM

• VSM High Availability

• Upgrades

• Port-Profiles & Port Channels

• VXLAN

• Cisco Cloud Services Platform / Nexus1x10

3


Cisco Nexus 1000V Virtual Switch | Build & Price

4


Cisco Virtual Networking and Cloud Network Services

Nexus 1000V

• Distributed Switch

• NX-OS consistency

VSG

• VM-level controls

• Zone-based FW

ASA 1000V

• Edge firewall, VPN

• Protocol Inspection

vWAAS

• WAN optimization

• Application traffic

WAN

Router

Servers

Tenant A ASA

1000V

Cloud

Firewall

Nexus 1000V Physical

Infrastructure

Virtualized/Cloud Data Center

vWAAS

Cisco

Virtual

Security

Gateway

Switches

Cloud Network Services

Citrix

NetScaler

VPX

Imperva

SecureSphere

WAF Cloud

Services

Router

1000V

Zone A

Zone B

vPath VXLAN

Multi-Hypervisor (VMware, Microsoft, Ubuntu, RedHat*)

Network

Analysis

Module

(vNAM)

vNAM

• App Visibility (L2-L7)

CSR 1k

• WAN GW

• Routing & VPN

Ecosystem

• Citrix NetScaler

• Imperva Web FW

5

“Name a feature we will not implement on Nexus 1000V.”

Saravan Rajendran, Cisco CNSG VP

6

Current Releases and New Features


Current Nexus 1000V Releases

• ESX – 5.2(1)SV3(1.1)*

– 256 VEMs, 12K vEth count

– VXLAN 2.0 (BGP)

– N1K Management Center

• ESX – 4.2(1)SV2(2.2)

– Dynamic Fabric Automation Leaf

– VDP – VSI Discovery Protocol

– Universal Licensing

• ESX - 4.2(1)SV2(2.1a)

– Scalability Release – 128 VEMs

– VXLAN 1.5, VXLAN GW

– Geographically Separated VSMs

– Removed ESX 4.1 support

• Hyper-V – 5.2(1)SM1(5.2a)

– SCVMM 2012 SP1 & R2

– Windows Server 2012 & R2

– VSG VM and Custom Attributes

– Universal Licensing

• InterCloud – 5.2(1)IC1(1.2)

– Simplified Platform Image

– Local License Server or Cisco PNSC

• Ubuntu KVM / OpenStack

– Initial Release

8

*Next Release


Evolution of VXLAN to version 1.5

• Unicast mode

– Simplifies VXLAN deployment

– Reduces network dependency (no multicast)

– Easier troubleshooting

– Flood directly to VXLAN Tunnel End Points (VTEP)

• Unicast Mac-address Distribution Mode

– Flooding is eliminated

– VSM learns all MACs and programs mappings to VEMs

– Faster response time

– Will not support VXLAN veth trunking(multi-mac)

– Requires static MACs (won’t work with MS NLB)

9


vTracker Feature

• Provides intuitive virtualization perspective to the network-admin

• Pulls data from vCenter and VEM

– Gives “cloud” view of connected objects

• Enabled with “feature vtracker”

• There are 5 view options

– module-view

– upstream-view

– vlan-view

– vm-view

– vmotion-view

10

SV2# show vtracker vm-view info vm win3

Module 5:

VM Name: win3

Guest Os: Microsoft Windows Server 2003

Standard (32-bit)

Power State: Powered On

VM Uuid: 423ca4df-26d0-50c1-d531-1a49b3a83aed

Virtual CPU Allocated: 1

CPU Usage: 0 %

Memory Allocated: 1024 MB

Memory Usage: 7 %

VM FT State: Unknown

Tools Running status: Running

Tools Version status: current

Data Store: datastore1 (2)

VM Uptime: 25 days 3 hours 56 minutes 15s


Nexus 1000V Manager – Installation Screenshot

• Zero CLI – full GUI interface

• Auto Host Selection

• Deploy Redundant VSMs

• Best Practices Auto-Implemented

• Automated prompts with suggestion for alternatives

• Customize Installation for Advanced Users

• *Available Summer 2014

Install / Migrate / Upgrade / Monitor

11

Licensing Info


Licensing – Essential Edition (No Expiration)

• Default mode for New Installs

• All features except…

– Cisco TrustSec (CTS)

– DHCP Snooping

– IP Source Guard / Dynamic ARP Inspection

– Virtual Security Gateway (VSG)

– VXLAN Gateway

• 128 modules with 4096 virtual ports

• Support Options

– Pay Nothing – support is through the communities site off cisco.com • https://communities.cisco.com/community/technology/datacenter/nexus1000v

– Pay for service contract

13

https://communities.cisco.com/community/technology/datacenter/nexus1000v


Licensing – Advanced Edition

• For customers that want more security features

• Customers with existing licenses will be considered Advanced

• Upgrade process will migrate VSM to Advanced Edition

• Required for VXLAN Gateway and VSG

• Licensed customers can get Virtual Security Gateway(VSG) for free

– Cisco Account Team can submit request

– VSG will no longer be sold separately

• 256 modules with 12k virtual ports (SV3)*

• 60-day Trial after which Advanced FeatureSet is disabled

14


Universal Licensing

• A common license is shared for both N1k & VSG.

• Cross Hypervisor portability.

• The license name is NEXUS1000V_LAN_SERVICES_PKG.

• Following upgrade, request a new Permanent license within 60 days.

15


Licensing – New Commands

• Display Current Edition switch# show switch edition

• To switch between Essential or Advanced switch(config)# svs switch edition [essential | advanced]

• VEM Licenses are Sticky – Removed & Offline VEMs hold a license switch# show module vem license-info

Licenses are Sticky

Mod Socket Count License Usage License Version License Status

--- ------------ ------------- --------------- --------------

3 2 2 1.0 licensed

• VEM license transfer to pool: switch(config)# svs license transfer src-vem <module> license_pool

16


Licensing – Overdraft Licenses

• Extra licenses to use in temporary situations

• 16 extra sockets

– Sometimes more depending on number of licenses you’ve purchased

• Can only be used after a valid license is installed

• No penalty

– Full TAC Support for Overdraft Modules

17

SV2# show license usage NEXUS1000V_LAN_SERVICES_PKG

----------------------------------------

Feature Usage Info

----------------------------------------

Installed Licenses : 16

Default Eval Licenses : 0

Max Overdraft Licenses : 16 <----

Installed Licenses in Use : 12

Overdraft Licenses in Use : 0 <----

Default Eval Lic in Use : 0

Default Eval days left : 0

Licenses Available : 20 <---- 4 + 16

Shortest Expiry : 04 Feb 2015

Virtual Supervisor Module Deployment and Troubleshooting


Cisco Nexus 1000V Architecture

Hypervisor Hypervisor Hypervisor

VEM-N VEM-1 VEM-2

VSM: Virtual Supervisor Module

VEM: Virtual Ethernet Module Server

Admin

NX-OS

Data Plane

VSM-1 (active)

VSM-2 (standby)

Virtual Appliance

NX-OS

Control Plane Network

Admin

Modular Switch

… Linecard-N

Supervisor-1 (Active)

Supervisor-2 (StandBy)

Linecard-1

Linecard-2

Ba

ck P

lan

e

19


Virtual Supervisor Module (VSM)

• VSM is a Virtual Machine

– On ESXi, Hyper-V, Ubuntu KVM / OpenStack

– On Nexus 1x10 / Cloud Services Platform

• Control plane for the Nexus 1000V solution

– VEM packet forwarding not impacted by reloads

• Responsible for

– Programming and Managing Virtual Ethernet Modules (VEM)

– Communicating with Management Applications • VMware vCenter, SCVMM, Horizon Dashboard

• 1 VSM HA pair can manage 128 VEMs

• Coexist with VMware vSwitch, vDS, Microsoft Logical, Native Switches

20


Nexus 1000V VSM Interfaces

• Control

– L2/L3 VEM (AIPC)

– VSM-VEM Heartbeats (L2/L3)

– VSM-VSM Synchronization (L2)

– VSM-VSM HA Heartbeats (L2/L3)

• Packet

– CDP, IGMP, NetFlow, SNMP

• L3 Mode

– Collapsed Ctrl, Pkt into mgmt0

– VSM-VEM flow from mgmt0

– Dedicated Control: svs mode L3 interface [control | mgmt0]

• Management

– SSH console access

– SNMP, HTTP

– vCenter Communication

– HA Heartbeat Backup

• Interface Order is always the same!

VSM-P eth0: control

eth1: mgmt0

eth2: packet

21


VSM Deployment Scenarios

• Supports the VSM on a VEM

• Supports the VSM on any hypervisor native, logical, or distributed switch

• Supports the VSM on any supported hypervisor (ESXi/Hyper-V/N1110)

• Keep VSMs on different physical hosts

– Use anti-affinity rules

• Storage wise we don’t care.

– VSM can be hosted on network storage

22


Stretched Nexus 1000V Model

• VSMs and VEMs spread across Datacenters

• VSMs can be split across DCs

– Requires L2 connectivity across DCI

– 10ms latency across DCI

• Not supported with Hyper-V

– Supported in a future release

23

VSM

VSM

hypervisor

VEM-1

VM VM VM

Local DC

hypervisor

VEM-2

hypervisor

VEM-4

VM VM VM

hypervisor

VEM-3

VM VM VM

Remote DC

VM VM

DCI

VM


VSM Control Modes

• L3 Mode

– L3 is the recommended & default • Easier to troubleshoot

• Flexible

– Requires an IP address be assigned to the VEM

– Uses UDP4785 for both source and destination

– Sourced from mgmt0 by default

• L2 mode

– Requires L2 connectivity through control0 interface to all VEM modules

– L2 still supported on ESX

– Not supported with Hyper-V or KVM

24


VSM L3 Configuration and Planning

• Two options for the L3 control interface

– mgmt0 (default)

– control0

• Use Control0 to separate control and management traffic

• Mgmt and Control use different VRF

– mgmt0 uses VRF management

– control0 uses VRF default

• Primary and Secondary VSM still need to be L2 adjacent!

– Test with mping broadcast command. 0x201 is control between VSMs

25

# mping broadcast

64 bytes from node 0x0201 (msg id = 0x030b1e 1) (time=0 sec, 1510 usec)


VSM Connectivity to VMware vCenter

• VSM connects to vCenter using SSL connection

– VC Extension contains the SSL cert

– Unique extension ID for the VSM

– Ability to generate own certificates

• VSM talks to vCenter using its API

– We push and pull data to/from vCenter

• VSMs get tied to a VMware Datacenter

– Multiple VSMs tied to same DC is allowed

– VSM can manage across clusters but not datacenters • Can get confusing

26


VSM Connectivity Errors - ESXi

• If you get “Extension key was not registered before it’s use”

– Re-register the Extension Key with VMware vCenter

• If you get “Connection refused. connect failed in tcp_connect()”

– Ping vCenter IP from VSM CLI

– VMware admin could have changed the http port

– API communication is through port 80 with VMware vCenter

– Find new port and change it on VSM

27


VSM and vMotion/Live Migration

• Manual vMotion/Live Migration is supported

• VMware DRS is NOT recommended for Primary & Secondary VSMs

• Aggressive settings could lead to excessive VSM-VEM heartbeat packet drops

• Best practice to keep Primary and Secondary VSM outside DRS control

• Use anti-affinity rules where possible

28


Backing up the VSM

• A running-config is not enough to restore

• VSM on ESXi

– Clone to a template

– You can restore from a template and saved-config

– Must be powered down

• VSM on Nexus 1x10

– Export a VSM to a file

– Import the saved VSM to restore

• VSM on ESXi Snapshots

– Not officially supported

– I/O latency cost associated with expanding the differential file

29


VSM Best Practices - Summary

• L3 control is the preferred method

• Use mgmt0 for control traffic

• Primary and Standby VSM in same L2 domain!!!

– Required even if VSMs are split between datacenters

• VSM on VEM is supported

• 10ms Latency between components: VSM-VSM, VSM-VEM

– 10ms even for VSMs split between datacenters

– For VEMs at branch locations 100ms

• Backup your config!!!

30

Nexus 1000V High Availability


VSM Redundancy Manager

• HA had to evolve to support split datacenter VSMs

• New Redundancy Manager process polls:

– VEM Manager – polls for number of active VEMs attached to VSM

– VMS process – retrieves which VSM has active VC connectivity

– SNMP Library – gets the last configuration time

• Runs on both primary and secondary VSM

• Heartbeats

– VSM-VSM every second. Drop after 6 missed

– VSM-VEM every second. Drop after 15 missed

32

SV2# show system internal redundancy trace

1 0s START_THREAD ST_NP ST_NP ST_INVALID

2 0s CP_STATUS_CHG ST_INIT ST_NP ST_INIT

3 0s SET_VER_RCVD ST_INIT ST_NP ST_INIT

4 0s STATE_TRANS ST_INIT ST_INIT ST_INIT EV_OS_INIT ST_AC_INIT

5 0s CP_STATUS_CHG ST_AC ST_INIT ST_AC_INIT

6 0s STATE_TRANS ST_AC ST_SB ST_AC_INIT EV_OS_SB ST_AC_SB


VSM Split Brain Recovery for ESXi

• Redundancy Manager in SV2(2.2)

– Module Count

– vCenter Status

– Last Configuration Time

– Last Standby-Active Switch(VSM with longer “primary” active time)

– Out-of-Sync / Split-Brain causes VSM to reload

33


When does a VEM switch VSMs?

• What if we have two active VSMs?

• What causes a VEM to switch?

– Standby VSM becomes active and broadcasts to all VEMs

– VEM will attach depending on • Connectivity between VEM and VSM

• VEM receives the “request to switch”

• VEM goes into headless mode after 15 seconds

• If a VEM is headless traffic forwarding continues!

– vMotion/Live Migration is blocked

34

Upgrades


Upgrades

• First always read and follow the upgrade guides

– Go in order

• Take a backup of the VSMs

– On ESXi use the clone to template option

– On Nexus 1x10s use the export function

– Backup the running-config

• Generate a Tech-Support before the upgrade

• If something goes wrong STOP and call TAC

• Use a maintenance window

– VEM upgrades require ESXi hosts to be in Maintenance Mode

36


Supported Upgrades

Starting Version 1.4 1.5 2.1 2.2 Combined

VMware Upgrade

Notes

1.3 Yes 1.4 first* 1.4 first* 1.4 first* No

1.4 Yes Yes Yes No 1.4 last version

supporting

ESX 4.0

1.5 Yes Yes Yes 1.5.2 for

combined

2.1 Yes Yes 2.1 last version

supporting

ESX 4.1

37

Upgrade matrix: http://www.cisco.com/web/techdoc/n1kv/upgrade/utility/n1kvmatrix.html

* Must upgrade to 1.4b first

http://www.cisco.com/web/techdoc/n1kv/upgrade/utility/n1kvmatrix.html





Upgrades to 2.2

• Scalability limits may require changes to the VM settings

• For full scalability support:

– CPU reservation to 2GHz

– Memory to 3GB

– VSMs do NOT support multiple vCPUs

• Steps

– Shutdown Secondary VSM

– Make VM changes

– Power Secondary on

– System Switchover

– Repeat steps on Primary VSM

• API can be upgraded individually now

– “show plugin status”

38


Upgrading the VSM

• Changes from 2.1

– VSMs can run newer software than VEMs. New features disabled until VEMs upgraded.

• ISSU upgrade is similar to other Nexus switches

– Copy new kickstart and system images to bootflash

– Run “install all” command • Verifies software compatibility

• Copies images to secondary’s bootflash.

• Upgrade/Reboot the Secondary VSM

• Switchover to Secondary VSM – It’s now the active VSM with VEMs attached

• Upgrade/Reboot the old-Primary VSM

• Requires no outage of the VSM

• Change CPU/Memory after the SV2(2.2) upgrade is complete

39


Troubleshooting VSM Upgrades

• If something is wrong after the VSM upgrade STOP

– Call TAC

– Rollback using backup method • Shutdown the VSM VMs

• Power-on the Clones (ESXi), Import the backup (Nexus 1x10)

• Changing boot variables to older image is not supported but often works

• Sometimes the VEM won’t connect to the Standby VSM

– Try a “system switchover” once the old primary is upgraded

• Might want to verify Standby VSM before upgrade

– Make sure VEMs can connect to standby

– Use “system switchover” command

40


Upgrading the VEMs

• VEM module upgrade kicked off on VSM

– If VUM is installed everything is automatic • VSM communicates with vCenter to manage the upgrade

• Host is placed in maintenance mode(if DRS is installed VMs are migrated off)

• VEM is upgraded and host exits maintenance mode

• Moves on to the next host

– If VUM is not installed • Still initiate the process on the VSM

• User manually places ESXi hosts in maintenance mode

• Upgrade the VEM with esxcli command

• Exit maintenance mode and move to the next host

• Always complete the upgrade

– Issue the “vmware vem upgrade complete” command

– Signals vCenter to use the new VEM VIB when hosts are added

41


Troubleshooting VEM Upgrades

• Remember the VMware admin has to acknowledge upgrade in vCenter

• Don’t upgrade the VEMs by pushing a baseline

• Make sure you have DRS capacity

– Need to be able to handle one ESXi host in maintenance mode

• If a particular ESXi host fails

– It’s usually because the host cannot go into maintenance mode

– From vCenter attempt to put the host in maintenance mode • Troubleshoot any issues that prevent it

– If an ESXi host is running a vCenter VM this can cause problems

• You can restart the VEM upgrade after it fails

– It will only upgrade hosts that did not succeed

42

Virtual Ethernet Module Deployment and Troubleshooting


VEM Deployment – Best Practices

• Again we recommend L3 Control

• L3 control requires a VMKernel NIC on N1K DVS

– We need an L3 interface to forward control traffic

– 10/100ms latency for local vs. branch office

• Recommend using the ESXi management VMKernel NIC

– Requires management interface to the VEM

– Doesn’t require static routes on ESXi hosts

• Don’t create an L3 vmk on same subnet as mgmt vmk

• Don’t use UCS “Dynamic vNICs” in Service-Profiles

– VEM and VM-FEX are mutually exclusive

44


VEM Deployment – vEth Port-Profile

• vmk0 interface needs to be migrated to this port-profile

• It must have capability l3control and system VLAN

• Each VMKernel VLAN needs a different port-profile

• VSM only permits VMKs to connect to this port-profile

45

port-profile type vethernet vmk-l3

capability l3control

vmware port-group

switchport mode access

switchport access vlan 119

capability vxlan

no shutdown

system vlan 119

state enabled


VEM Deployment – Uplink Port-Profile

• Typically a trunk – Verify upstream switch allowed VLAN list matches

• Must have system vlans & a port-channel defined

• MTU must match. Especially important when using OTV.

46

port-profile type ethernet system-uplink

vmware port-group

switchport mode trunk

switchport trunk allowed vlan 119,199,219,319

mtu 9000

channel-group auto mode on mac-pinning

no shutdown

system vlan 119,319

state enabled


VEM L3 Troubleshooting

1. VMK migrated behind VEM?

2. VSM-ESXi connectivity?

• Static route needed?

3. L3 vEth Port-Profile correct?

4. Uplink Port-Profile correct?

5. Check the Opaque Data

6. Check Heartbeats

47


VEM Troubleshooting – VSM Connectivity

• VEM adds in vCenter but does not show up on VSM “show module”

• With L3 its usually an IP routing problem

– If you can ping from VSM to VMK interface then VEM should connect.

– Troubleshoot as you would all VMware L3 issues

• With L2 most of the time its a Control VLAN issue

– Verify Control VLAN connectivity in upstream network

– Check upstream switches for VEM AIPC MAC address

• Additional Information in Appendix 2

48


VEM Deployment – VMKs on same subnet

• Don’t use multiple VMKs on the same subnet on different virtual switches

• VMware uses a single TCP/IP stack for all VMK interfaces

• No way to pin traffic to an uplink interface.

• One interface gets picked for all traffic on that subnet

– Check out VMware KB article 2010877

• Only one gateway per host

50

VMware ESX

VEM-1

VMK1

192.168.10.200

VMK0

192.168.10.100

vSwitch


VSM Setting Verification

• Verify the VRF

• Can the VSM ping the VEM

• Check SVS domain

52

SV2# show ip route vrf management

0.0.0.0/0, ubest/mbest: 1/0

*via 14.17.119.254, mgmt0,

[1/0], 6d20h, static

SV2# ping 14.17.219.22

PING 14.17.219.22 (14.17.219.22): 56 data bytes

64 bytes from 14.17.219.22: icmp_seq=0 ttl=62 time=1.254 ms



SV2# sh svs domain

SVS domain config:

Domain id: 1919

Control vlan: NA

Packet vlan: NA

L2/L3 Control mode: L3

L3 control interface: mgmt0

Status: Config push to

VC successful.


Check Opaque Data

• Opaque data is bootstrap information for the VEM

– Pushed via SCVMM or vCenter during “Host Add to DVS”

• Is the right Opaque data getting pushed to the ESXi host?

53

Should match VLAN defined

in vEth Port-Profile

Should match MAC of

control 0 or mgmt 0

~ # vemcmd show card Card UUID type 2: 9aed7c30-84f8-11e2-1234-ff987600005f

Card name:

Switch name: SV2

Switch alias: DvsPortset-0

Switch uuid: b2 40 3c 50 72 8e 15 f5-6a 3c 7f d1 c6 13 70 cd

Card domain: 1919

Card slot: 3

VEM Tunnel Mode: L3 Mode

L3 Ctrl Index: 49

L3 Ctrl VLAN: 119

VEM Control (AIPC) MAC: 00:02:3d:17:7f:02

VEM Packet (Inband) MAC: 00:02:3d:27:7f:02

VEM Control Agent (DPA) MAC: 00:02:3d:47:7f:02

VEM SPAN MAC: 00:02:3d:37:7f:02

Primary VSM MAC : 00:02:3d:70:1f:07

Primary VSM PKT MAC : 00:02:3d:70:1f:08

Primary VSM MGMT MAC : 00:02:3d:70:1f:06


View Heartbeat Messages on VEM

• Use vempkt on the ESXi host vempkt capture [egress|ingress] vlan 119 ltl 50

– Run for 10s to capture several heartbeat cycles

vempkt cancel capture all

vempkt display detail all

• vempkt can now export to a pcap file vempkt pcap export <filename>

• Look for heartbeat messages on VSM

54

SV2# show module vem counters

--------------------------------------------------------------------------------

Mod InNR OutMI InMI OutHBeats InHBeats InsCnt RemCnt Crit Tx Errs

--------------------------------------------------------------------------------

3 5086 2 2 593401 348535 2 1 0

4 5 4 4 593401 593296 4 3 0

5 0 0 0 593401 0 0 0 0

6 105 4 4 593401 591303 4 3 0


VEM Troubleshooting - vemlog

• Used for detailed debugging of programming and packet flows

• Executed on the Hypervisor Host

• Enable different debug options to help troubleshoot

– LACP

– QOS

– VXLAN

– IGMP

– VSM<-->VEM Data

• http://www.cisco.com/en/US/products/ps9902/products_tech_note09186a0080bed119.shtml

55

~ # vemlog show debug | grep lacp

Module Available Printing

sflacp ENWID PL (223) ( 0)

sf_lacp_pdu_utils ENWID PL (223) ( 0)

sflacp_hostdata ENWID PL (223) ( 0)

~ # vemlog debug sflacp all

~ # vemlog show debug | grep lacp

sflacp ENWID PL (223) ENWIDTPL (255)

sf_lacp_pdu_utils ENWID PL (223) ( 0)

sflacp_hostdata ENWID PL (223) ( 0)

http://www.cisco.com/en/US/products/ps9902/products_tech_note09186a0080bed119.shtml

http://www.cisco.com/en/US/products/ps9902/products_tech_note09186a0080bed119.shtml

Port-Profiles Deploying and Troubleshooting


Port-Profiles

Usage

<type>

Port-Profile Port-profiles

vEthernet

VM vmk l3control

/ vservice

Ethernet

UPLINK

vEthernet PP (default)

-Virtual Interfaces (vEth x/) (VMs, VMK)

-Typically Access Ports

-Configuration: VLAN, ACL, Pinning, QoS

Ethernet PP

-Physical Interfaces (Eth x/y)

-Typically Trunk (could also be access)

-Configuration: Port-Channel, ACLs, QoS

57


Switch Interface Types

• Ethernet Port (eth)

– Correspond to the physical NIC interfaces leaving the server

– Specific to each “module” or VEM

– VMware’s vmnicX == Cisco ethx/y

– Up to 32 physical ports supported per host

• Port Channel (port-channel)

– Aggregation of physical Ethernet ports

– Up to eight Port Channels per host

• Virtual Ethernet Port (vEth)

– One per virtual NIC interface (vNIC) including service console / vmknic

– Notation is VethX

– No module number is assigned to keep naming persistent as VMs move between modules (hosts/VEMs)

58

VM1 VM2

Eth3/1 Eth3/2

Po1

Veth2 Veth1


Loop Prevention without STP

59

Cisco VEM

VM1 VM2 VM3 VM4

Cisco VEM

VM5 VM6 VM7 VM7

Cisco VEM

VM9 VM10 VM11 VM12

BPDUs are Dropped

Eth4/1 Eth4/2

X

No Switching from

Physical NIC

to NIC

déjà vu check

Frames with local

MAC Dropped on

Ingress

X


Spanning-tree and BPDU – Best Practice

• Mandatory Spanning-Tree settings per port

– IOS set STP portfast • cat65k-1(config-if)# spanning-tree portfast trunk

– NXOS set port type edge • n5k-1(config-if)# spanning-tree port type edge trunk

• Highly Recommended Global BPDUFilter/BPDUGuard

– IOS • cat65k(config)# spanning-tree portfast bpdufilter

• cat65k(config)# spanning-tree portfast bpduguard

– NXOS • n5k-1(config)# spanning-tree port type edge bpduguard default

• n5k-1(config)# spanning-tree port type edge bpdufilter default

• BPDU Filter is mandatory for LACP port-channels

• Set per-port BPDU Guard when Global is not possible

60


Ethernet (uplink) Port-Profile Troubleshooting

• Port-Profiles with multiple NICs need a port-channel

– Causes duplicate packets

– Kicks in déjà vu driver • Requires extra CPU processing

• Fills the logs

– When in doubt, use mac-pinning

• Also same issue if you overlap VLANs in different Port-Profiles on same host

61

port-profile type ethernet uplink-nopc

vmware port-group


switchport trunk allowed vlan 1-3967,4048-4093

no shutdown

system vlan 11

state enabled

port-profile type ethernet uplink-nopc

vmware port-group


switchport trunk allowed vlan 1-3967,4048-4093


no shutdown

system vlan 11

state enabled

WRONG RIGHT


Cisco Nexus 1000V System VLANs

• System VLANs enable interface connectivity before an interface is programmed

• System port-profiles become part of the opaque data

– VEM will load system port-profiles and pass traffic even if VSM is not up

– Unprotected (No ACLs, VSG) before module registers for first time

• Addresses chicken and egg issue

– VEM needs to be programmed, but it needs a working network for this to happen

• Port profiles that contain system VLANs are “system port profiles”

– Allowed 32 port-profiles with system VLAN

62


System VLAN Guidelines

• The system VLAN must be a subset of the allowed VLAN list on trunk ports

• Only one system VLAN on an access port

• The ‘no system vlan’ command only when no interface is using the profile

• Once a system profile is in use by at least one interface

– Can add to the list of system VLANs

– Cannot delete VLANs from the list – reason to limit usage

• System vlans must be set on egress and ingress port-profiles

• Required System VLANs

– Control, Packet, IP Storage, VMKernel, vCenter, any Management Networks

63


VMware DVS Max-Port Issues

• Default to 32 max-ports per port-profile

• Counts toward the maximum number of VMware DVS ports

– 8192 by default

– Pre-Provisioned

– Some ports are consumed when you add an ESX host to the DVS

• Two methods to remedy:

– Max-ports under “svs connection <name>” • Allows you to increase the ports of the VMware DVS

– Port-binding “auto expand” in veth port-profiles • N1KV dynamically adds ports as VMs are added

• Set port-binding as default with “port-profile default port-binding static auto expand”

64


Microsoft Network Load Balancing Support

• Unicast mode is officially supported method

– “no mac auto-static-learn” in vEth port-profile

• Multicast Mode

– NLB virtual cluster address requires a static ARP entry on the edge router

– Works through flooding

• Multicast Mode IGMP

– Disable IGMP snooping on the N1KV

– Upstream switches enable IGMP snooping

– Enable IGMP Querier in the environment

– NLB virtual cluster address requires a static ARP entry on the edge router

– CSCue32210 - Add support for Microsoft NLB - Multicast+IGMP

65


Jumbo Frames Support

• System jumbo mtu 9000 – Enabled globally by default in SV1(4)+

– Sets the systemwide jumbo MTU size

– Generally do not need to change

• vEthernet ports are 9000 by default

• MTU setting for “ethernet” type port-profile

– Simply use “mtu size” in port-profile and nothing else

• Still need to configure upstream network devices

– UCS System QoS Class

– UCS vNIC QoS Policy

– Nexus 5k / 7k / etc

66


Port-Profile Using Weighted QOS

• Configuration Steps to limit vMotion traffic

68

n1kv-l3(config)# class-map type queuing match-all vmotion-class

n1kv-l3(config-cmap-que)# match protocol ?

n1k_control N1K control traffic

n1k_mgmt N1K management traffic

n1k_packet N1K inband traffic

vmw_ft VMware fault tolerance traffic

vmw_iscsi VMware iSCSI traffic

vmw_mgmt VMware management traffic

vmw_nfs VMware NFS traffic

vmw_vmotion VMware vmotion traffic

n1kv-l3(config-cmap-que)# match protocol vmw_vmotion

n1kv-l3(config-cmap-que)# policy-map type queuing vmotion-policy

n1kv-l3(config-pmap-que)# class type queuing vmotion-class

n1kv-l3(config-pmap-c-que)# bandwidth percent 50

n1kv-l3(config)# port-profile type eth uplink-vpc

n1kv-l3(config-port-prof)# service-policy type queuing output vmotion-policy

Port Channels


Port Channels

• LACP Port-Channels

– Requires upstream switch support and configuration

• VPC – MAC Pinning

– Works with any upstream switch

– Allows for pinning of vEths (VM) to specific links

• VPC – Host Mode CDP/Manual (deprecated)

– NIC association is either Manual or CDP

70


Port Channels

• Best Practice Configuration Guide

– http://www.cisco.com/en/US/products/ps9902/products_configuration_example09186a0080c1ee1e.shtml

• All Ethernet Port-Profiles must be configured in a Port-Channel

• LACP & MAC-Pinning are recommended modes

– Use Manual/Static Pin Group for granular traffic steering

– Use Manual/Static Pin Groups with multiple vMotion VMKs in ESX 5.x

• Same link-speed for all members. No mixing 1G+10GE+40GE interfaces.

71

http://www.cisco.com/en/US/products/ps9902/products_configuration_example09186a0080c1ee1e.shtml

http://www.cisco.com/en/US/products/ps9902/products_configuration_example09186a0080c1ee1e.shtml


Port Channels – Best Practice

• If the upstream switch can be clustered (VPC, VBS Stack, VSS) use LACP

• If you are using LACP also use LACP Offload

• UCS-B must use MAC-Pinning

• If the upstream switch can NOT be clustered use MAC-PINNING

• Create channel-groups in port-profile

– Let VSM build the interface port-channel & add physical NICs

• All physical switch ports in port-channel configured identical

72


Port Channels – MAC Pinning

• MAC Pinning provides the dynamism of vPC Host-Mode without requiring CDP to be configured on the upstream switch

vSphere

VM VM VM VM

sys-uplink

The VM MAC address is used to select link.

port-profile type ethernet uplink

vmware port-group


switchport trunk allowed vlan 1-10


no shut

state enable

system vlan 10

73


Port Channels – MAC Pinning (Link Failure)

• If a failover occurs, all the traffic pinned to an interface will be migrated to the other interfaces. VEM sends GARP to flush upstream CAM tables.

vSphere

VM VM VM VM

sys-uplink

The VM MAC address is used to select link.


vmware port-group




no shut

state enable

system vlan 10

74


Port Channels – MAC Pinning

• Use Network State Tracking (NST) to detect non-link failures

• Each Eth interface added is a unique Service Group

– SGID # assigned based off vmnic#

• Use “pinning id” command under vEthernet Port-Profile

– Pins the VM to a particular uplink

– Ordered list for backup

– n1kv(config-port-prof)# pinning id 0 backup 1 2

• Default assignment is Round Robin to an SGID

• New command to make SGID # relative

– n1kv(config-port-prof)# channel-group auto mode on mac-pinning relative

75


MAC Pinning (Host Pinning Tables) n1kv# sh port-channel summary

1 Po1(SU) Eth NONE Eth5/1(P) Eth5/2(P)

2 Po2(SU) Eth LACP Eth6/1(P) Eth6/2(P)

3 Po3(SD) Eth NONE Eth3/3(r)

[root@mw-esx15 ~]# vemcmd show channel type

LTL Channel_Type

------------------

17 MAC Pinning

18 MAC Pinning

76


MAC Pinning (Host Pinning Tables) [root@mw-esx15 ~]# vemcmd show port

LTL VSM Port Admin Link State PC-LTL SGID Vem Port Type

17 Eth3/1 UP UP F/B* 561 0 vmnic0


49 Veth1 UP UP FWD 0 1 vmk0

[root@mw-esx15 ~]# vemcmd show pc

pce_ind chan pc_ltl pce_in_pc LACP SG_ID NumVethsPinned mbrs

------- ---- ------ --------- ---- ----- -------------- ----

0 1 305 0 N 0 2 17,

1* 3 18,

[root@mw-esx15 ~]# vemcmd show pinning

LTL IfIndex PC_LTL VSM_SGID Eff_SGID iSCSI_LTL* Name

10 0 305 32 1 0

12 0 305 32 1 0

49 1c0000a0 305 32 1 0 vmk0

50 1c0000d0 305 32 0 0 vmk1

77


Port Channels – How to Tell Pinning

• Can run from the VSM now

• No need to run command on the VEM

78

n1kv-l3# show int virtual pinning module 5

------------------------------------------------------

Veth Pinned Associated PO List of

Sub Group id interface Eth interface(s)

------------------------------------------------------

Veth2 0 Po5 Eth5/1

Veth4 2 Po5 Eth5/3

Veth5 0 Po5 Eth5/1

Veth6 2 Po5 Eth5/3

Veth7 0 Po5 Eth5/1


Static Pinning to Sub-Group

• Static Pinning is similar to VMware’s vSwitch active/standby design.

vmk0 VMotion

Sub-group 0 Sub-group 2

Port-channel

C

P


channel-group auto mode on mac-

pinning relative

port-profile vmkernel

pinning sub-group id 0 backup 2 1

port-profile vmkernel

pinning sub-group id 0 backup 2 1

port-profile type ethernet vmotion

pinning sub-group id 2

vmk0 VMotion

Sub-group 2

Port-channel

C

P

After

failover

Sub-group 1 Sub-group 1

79


LACP Port Channels

• Use when single upstream or clustered (vPC,VSS, Catalyst Stack) switch

• Use “channel-group auto mode active” on N1KV

• Use “channel-group # mode active/passive” on upstream switch

• Switchports must be configured with

– spanning-tree portfast trunk

– spanning-tree bpdufilter enable

• Not compatible with Network State Tracking(NST) with LACP

80


Port-Channels - LACP

• LACP allows traffic from each VM to fully utilize multiple links simultaneously.

• Allows faster VMotion and faster VM connectivity by using flow based hashing


vmware port-group



channel-group auto mode active

no shut

state enable

vSphere

VM VM VM VM

Port-channel

LACP

Upstream switch clustered (vPC,VSS,VBS,Stack…)

81


LACP Troubleshooting

• Do not use Network State Tracking(NST) with LACP

• LACP Port-Channel configured on the upstream switches

• Port-profile created with “channel-group auto mode active”

• On the VEM

– vemcmd show lacp

• On the VSM and Upstream Switch

– show port-channel summary

– show lacp counters/neighbor • Are you seeing LACP PDUs?

82


LACP Debugging ~ # vemcmd show lacp

LACP Offload is Enabled

---------------------------------------------------

LACP Offload Config for LTL 17

---------------------------------------------------

Channel No : 8

Channel Mode : Active

Port Priority : 0x8000

LACP Bit Set : Yes

SV2# show lacp counters

LACPDUs Marker Marker Response LACPDUs

Port Sent Recv Sent Recv Sent Recv Pkts Err

---------------------------------------------------------------------

port-channel8

Ethernet10/3 8353 8356 0 0 0 0 0

Ethernet10/1 8353 8356 0 0 0 0 0

83


LACP Debugging ~ # vemlog show debug | grep lacp

sflacp ENWID P ( 95) ENW ( 7)

sf_lacp_pdu_utils ENWID P ( 95) ENW ( 7)

sflacp_hostdata ENWID P ( 95) ENW ( 7)

Debug (LTL 16, DIR TX) : Actorstate=7 agg=1 insync=0 coll=0 dis=0 active=1

short_timeout=1 Port ID (0x8000.0x602), Key (7)

Debug (LTL 16, DIR TX) :Partnerstate=2 agg=0 insync=0 coll=0 dis=0 active=0


Debug sf_lacp_tx_pdu_to_upstream: LTL = 18

Debug sf_lacp_tx_pdu_to_upstream, NEW LACP PKT : Src(1), Dst(18), VLAN(1),

FLAGS(1)

[…]

Debug (LTL 18, DIR RX) :Partnerstate=3d agg=1 insync=1 coll=1 dis=1

active=1 short_timeout=0 Port ID (0x8000.0x602), Key (7)

Debug (LTL 16, DIR TX) : Actorstate=3d agg=1 insync=1 coll=1 dis=1 active=1


84

Virtual Extensible LAN (VXLAN)


Virtual Extensible Local Area Network (VXLAN)

• Ethernet in IP overlay network

– Entire L2 frame encapsulated in UDP (port 4789)

– 50 bytes of overhead

• Include 24-bit VXLAN Identifier

– 16 M logical networks

– Mapped into local bridge domains

– Unique multicast group per segment

• VXLAN can cross Layer 3

• Tunnel between VEMs – VMs do NOT see VXLAN ID

• Egress to Non-VXLAN network

87

Outer

MAC

DA

Outer

MAC

SA

Outer

802.1Q Outer

IP DA

Outer

IP SA

Outer

UDP

VXLAN ID

(24 bits)

Inner

MAC

DA

InnerM

AC

SA

Optional

Inner

802.1Q

Original

Ethernet

Payload CRC

VXLAN Encapsulation Original Ethernet Frame


Virtual Extensible Local Area Network (VXLAN)

• Each overlay network is known as a VXLAN segment

• Each VXLAN segment identified by a 24-bit segment ID (VNI)

• VXLAN traffic carried between VXLAN Tunnel Endpoints (VTEP)

• VEM module acts as the VTEP

• VM traffic is carried over point to point tunnels between VTEPs

– VM to VM traffic is encapsulated in a VXLAN header

• 1550 MTU for encapsulation overhead

• Encapsulated multicast is always flooded – No IGMP in VXLAN

88


Deployment Modes: Multicast or Unicast?

• Multicast used to be required for unknown broadcast/unicast on VXLAN

• N1KV 2.2 introduced Unicast Mode and Unicast Mac Distribution Mode

• Multicast (VXLAN 1.0)

– Needs Multicast configured throughout complete network

– IGMP Querier in VLAN

– Multicast routing and proxy ARP across subnets

– VTEPs all join multicast group

– Interoperates with N9K, CSR1K, other Nexus products

• Unicast Mode (VXLAN 1.5)

– VEMs flood each other directly for unknown broadcast/unicast

– Keep a list of other VEMs in each VXLAN

89


Deployment Modes: When to use MAC Distribution?

• MAC distribution will provide best performance

• No Flooding & Learning

• Full MAC table distributed to each VEM

– VEMs report local MACs to VSM

– VSM distributes {MAC,VTEP} mapping to each VEM

• VXLAN traffic cannot span multiple Nexus 1000V switches*

• Two caveats

– No vEth VXLAN trunk mode support with MAC distribution

– Won’t work with Microsoft NLB

90


VXLAN Forwarding Basics

VEM 1 VEM 2

Forwarding mechanisms similar to Layer 2 bridge: Flood & Learn

VEM learns VM’s Source (MAC, Host VXLAN IP) tuple

Broadcast, Multicast, and Unknown Unicast Traffic

VM broadcast & unknown unicast traffic are sent as multicast

Unicast Traffic

Unicast packets are encapsulated and sent directly (not via multicast) to destination host VXLAN IP (Destination VEM)

VM VM VM VM

92


Enhanced VXLAN

VXLAN

(multicast mode)

Enhanced VXLAN

(unicast mode)

Enhanced VXLAN

MAC Distribution

Enhanced VXLAN

ARP Termination

Broadcast /

Multicast

Multicast

Encapsulation

Replication plus

Unicast Encap

Replication plus

Unicast Encap

Replication plus

Unicast Encap

Unknown Unicast

Multicast

Encapsulation

Replication plus

Unicast Encap

Drop Drop

Known Unicast Unicast

Encapsulation Unicast Encap Unicast Encap Unicast Encap

ARP Unicast

Encapsulation

Replication plus

Unicast Encap

Replication plus

Unicast Encap VEM ARP Reply

96


VXLAN Configuration: Unicast

• VMkernel interface acts as VTEP

• VSM Control Mode should be L3

• Bridge domain is configured as Unicast or Unicast Mac Distribution

97

feature segmentation

feature vxlan-gateway

port-profile type vethernet vmk-l3-vxlan-vtep

capability l3control

vmware port-group


switchport access vlan 119

capability vxlan

no shutdown

system vlan 119

state enabled


Bridge Domain Configuration: Unicast

• Create a bridge-domain in unicast mode

• Scenario 1:

• Scenario 2:

switch(config)# segment mode unicast-only (Global)

switch(config)# bridge-domain segment-cisco

switch(config-bd)# segment id 5000

switch(config-bd)# segment distribution mac

switch(config)# bridge-domain segment-cisco

switch(config-bd)# segment id 5000

switch(config-bd)# segment mode unicast-only (Per BD override)

switch(config-bd)# segment distribution mac

98


Port-Profile Configuration

• Create an Access Port-Profile with the VXLAN Bridge Domain

• Assign to VM’s in vCenter port-profile type vethernet bd-5000

vmware port-group


switchport access bridge-domain bd-5000

no shutdown

state enabled

99


VXLAN Debugging SV2# show bridge-domain bd-5000

Bridge-domain bd-5000 (2 ports in all)

Segment ID: 5000 (Manual/Active)

Mode: Unicast-only (override)

MAC Distribution: Disable (override)

Group IP: NULL

State: UP Mac learning: Enabled

Veth9, Veth45

SV2# show bridge-domain bd-5000 vteps

Bridge-domain: bd-5000

VTEP Table Version: 21

Port Module VTEP-IP Address VTEP-Flags

---------------------------------------------------------------------------

Veth1 3 14.17.119.21 (D) <---Designated VTEP (vmk)

Veth2 4 14.17.119.22 (D)

Veth13 10 14.17.119.36 (DI) <---VXGW

Veth15 11 14.17.119.36 (DI*)<---VXGW (Standby)

100


VXLAN Debugging ~ # vemcmd show vxlan-vteps

Bridge-Domain: bd-5000 Segment ID: 5000

Designated Remote VTEP IPs (*=forwarding publish incapable):

14.17.119.22(DSN: 1), 14.17.119.36(DSN: 1)*

~ # vemcmd show bd bd-name bd-5000

BD 31, vdc 1, segment id 5000, segment group IP 0.0.0.0,

encap VXLAN, vff_mode Anycast,swbd 4096, VLAN 0, 1 ports, "bd-5000"

Segment Mode: Unicast

VTEP DSN: 1 , MAC DSN: 0

Portlist:

52 win2k.eth0

Virtual Machine

in VXLAN 5000

101


VXLAN Debugging ~ # vemcmd show l2 segment 5000

Bridge domain 31 brtmax 4096, brtcnt 3, timeout 300

Segment ID 5000, swbd 4096, "bd-5000"

Flags: P - PVLAN S - Secure D - Drop

Type MAC Address LTL timeout PVLAN Remote IP DSN

Dynamic 00:50:56:bc:73:1a 561 121 14.17.119.22 0

Static 00:50:56:a9:00:2e 52 0 0.0.0.0 0

Dynamic 54:7f:ee:2f:33:81 561 2 14.17.119.36 0

ESXi Host #2

VXLAN

Gateway

102

Nexus 1010 and 1110


VSM Deployment Scenarios – Nexus 1110

• VSM on a Nexus 1010/X or 1110-S/X

– It’s still a Virtual Machine

– Up to 14 VSM pairs on one 1110-X cluster

• Always deploy in the appliance pairs!

• N110 allows for Network team to own the virtualization platform

• N110s should go in the Aggregation Layer

• Stretched Model requires

– L2 Connectivity

– 10ms latency

Cisco Cloud Services Platform

104 *Next Release


1110-S/X Deployment Scenario

105


Cisco Cloud Services Platform (CSP)

• Based off UCS C2x0 M3 server

– Same CIMC/BIOS/firmware

– Provide 6 x 1G network connections

– 1110-X 2 x 10G - SP1(7) • 10G available only on purchase. No upgrade available.

– Encryption Accelerator Card for Citrix VPX – SP1(7)

• Virtual Service Blade (VSB) Support

– 1010/1110-S supports up to 10

– 1010/1110-X supports up to 14

Nexus 1010/1010-X/1110-S/1110-X

106



• Current supported VSBs

– Nexus 1000V VSM (ESX/HyperV/KVM)

– Virtual Security Gateway (VSG)

– Network Analysis Module (NAM)

– Data Center Network Manager (DCNM)

– Citrix NetScaler VPX

107

VSB Minimum Version

HyperV SP(6.1)

VXLAN GW SP1(6.1)

Citrix Netscaler SP1(6.2)



• Must be deployed in pairs

– No option for standalone

• Deploy in the Aggregation Layer

• Must be in the same L2 domain for management and control

• Can be geographically diverse

• Uses same HA mechanism as VSM with domain-id and control vlan

– Do not overlap the domain-id between a 1x10 and a VSM

• What’s not supported?

– Primary and Secondary VSM on same 1x10

– Primary VSM on ESX and Secondary VSM on 1x10 or vice versa

108


VSB Backups using Import/Export

• Works with VSM, NAM, and VSG

• Can Import/Export both primary and secondary

• Export requires that VSB be shutdown

• Images are stored in “export-import/” dir on bootflash

– Can be manually copied off to remote storage

109

n1010-1# copy scp://[email protected]/root/Vdisk4.img.tar.00

bootflash:export-import vrf management

n1010-1(config)# virtual-service-blade training

n1010-1(config-vsb-config)# import primary Vdisk4.img.tar.00

Note: import started..

Note: please be patient..

Note: Import cli returns check VSB status for completion


Network Classes and Topologies

• Management

– Carries the mgmt0 interface of the 1x10

– Carries the mgmt0 traffic for all VSMs installed

• Control

– Carries all the control and packet traffic for the VSMs installed on the 1x10

– Carries control traffic for HA between primary and secondary 1x10

• Data

– Used by Virtual Service Blades other than VSM

• Passthrough

– Binds physical NIC to VSB

• 5 Network Topologies choices

110


Network Topologies

Uplink Type Management VLAN Control VLAN Data VLAN

1 Ports 1 and 2 Ports 1 and 2 Ports 1 and 2

2 Ports 1 and 2 Ports 1 and 2 (HA) Ports 3-6 (LACP)

3 Ports 1 and 2 Ports 3-6 (LACP) Ports 3-6 (LACP)

4 Ports 1 and 2 Ports 3 and 4 Ports 5 and 6

Flexible There is no traffic segregation based on traffic class.

*Must use for VXGW deployements.

111


Recommendations

• If you are not planning on using other VSBs

– Topology 3 gives best bandwidth and redundancy for control VLAN • Negative is that is harder to configure

• If using VXGW, Netscaler, or shared between production / lab network

– Topology 5 is Flexible

• Flexible allows any configuration

– Recommend port-channels

– Remember VSM latency is key over bandwidth

• Use VPC or VSS upstream if you have it

112


Participate in the “My Favorite Speaker” Contest

• Promote your favorite speaker through Twitter and you could win $200 of Cisco Press products (@CiscoPress)

• Send a tweet and include

– Your favorite speaker’s Twitter handle @juicyUCS

– Two hashtags: #CLUS #MyFavoriteSpeaker

• You can submit an entry for more than one of your “favorite” speakers

• Don’t forget to follow @CiscoLive and @CiscoPress

• View the official rules at http://bit.ly/CLUSwin

Promote Your Favorite Speaker and You Could be a Winner

113

http://bit.ly/CLUSwin


Complete Your Online Session Evaluation

• Give us your feedback and you could win fabulous prizes. Winners announced daily.

• Complete your session evaluation through the Cisco Live mobile app or visit one of the interactive kiosks located throughout the convention center.

Don’t forget: Cisco Live sessions will be available for viewing on-demand after the event at CiscoLive.com/Online

114

https://www.ciscolive.com/online


Continue Your Education

• Demos in the Cisco Campus

• Walk-in Self-Paced Labs

• Table Topics – Moscone Center West – 3rd Floor Lobby

– Discuss “Experiences with Cisco Services” with Distinguished Service Engineers

• Meet the Engineer 1:1 meetings

115

Appendix A – L2 Troubleshooting


L2 Control VEM – VSM Troubleshooting Steps

1. VSM MAC address

2. VSM is connected to vCenter

3. VSM has Control VLAN on right interface

4. Uplink port-profile has Control vlan

5. VEM sees control VLAN

6. VEM and VSM see each others MAC

7. Physical network sees VEM and VSM MAC

8. VSM sees heartbeat messages from VEM

119


n1kv-l2# show svs neighbors

Active Domain ID: 422

AIPC Interface MAC: 0050-56a9-2535

Inband Interface MAC: 0050-56a9-2537

Step 1: VSM MAC

• Need for L2 troubleshooting

• On VSM run show svs neighbors

• Its the AIPC Interface MAC

120


n1kv-l2# show svs connections

connection VC:

ip address: 172.18.217.241

remote port: 80

protocol: vmware-vim https

certificate: default

datacenter name: Harrington

admin:

max-ports: 8192

DVS uuid: 3e 80 29 50 ad 9f f9 7f-43 d6 9b 6d a2 af cb 3e

config status: Enabled

operational status: Connected

Step 2: VSM – vCenter Connectivity

• Verify VSM is connected to vCenter

121


Step 3: Verify VSM VM Control interface

• 1st interface listed is Control Interface

• Interface connected?

122


Step 4: Verify Uplink Port-Profile

• The first ESX interface added to the N1KV must have Control VLAN

• Verify uplink port-profile has Control VLAN defined and system VLAN

123

n1kv-l2# show run port-profile uplink version 4.2(1)SV1(5.1) port-profile type ethernet uplink vmware port-group switchport mode trunk switchport trunk allowed vlan 1-3967,4048-4093 no shutdown system vlan 2 state enabled


Step 5: Verify VEM Sees Control VLAN

• Verify VEM sees control VLAN with commands

– vemcmd show card

– vemcmd show port

– vemcmd show trunk

124


[~ # vemcmd show card

Card UUID type 2: 33393138-3335-5553-4537-31314e343636

Card name: cae-esx-154

Switch name: n1kv-l2

Switch alias: DvsPortset-0

Switch uuid: 3e 80 29 50 ad 9f f9 7f-43 d6 9b 6d a2 af cb 3e

Card domain: 422

Card slot: 5

VEM Tunnel Mode: L2 Mode

VEM Control (AIPC) MAC: 00:02:3d:11:a6:04

VEM Packet (Inband) MAC: 00:02:3d:21:a6:04

VEM Control Agent (DPA) MAC: 00:02:3d:41:a6:04

..

..

Card control VLAN: 2

Card packet VLAN: 2

Vemcmd show card

• Control, packet vlans and domain-ID match with VSM

125

MAC the VSM

should learn for

VEM


~ # vemcmd show port-old

LTL IfIndex Vlan/ Bndl SG_ID Pinned_SGID Type Admin State CBL Mode Name

SegId

6 0 1 T 0 32 32 VIRT UP UP 1 Trunk vns

8 0 3969 0 32 32 VIRT UP UP 1 Access

9 0 3969 0 32 32 VIRT UP UP 1 Access

10 0 2 0 32 32 VIRT UP UP 1 Access

11 0 3968 0 32 32 VIRT UP UP 1 Access



14 0 3971 0 32 32 VIRT UP UP 1 Access

15 0 3971 0 32 32 VIRT UP UP 1 Access

16 0 1 T 0 32 32 VIRT UP UP 1 Trunk ar

17 25010000 1 T 0 32 32 PHYS UP UP 1 Trunk vmnic0

Vemcmd show port-old

• Ports with LTLs 8, 9,10 are UP and CBL states are 1.

• ESX Physical ports are UP and CBL states 1.

126

Local Target Logic (LTL) is an index to address a port, or group of ports. Data path lookup engine takes LTL

as input, and gives LTL as output.

LTL scheme: [0-14: internal ports] [15-271: pNICs,VMs, etc…]


~ # vemcmd show trunk

Trunk port 6 native_vlan 1 CBL 1

vlan(1) cbl 1, vlan(3970) cbl 1, vlan(3969) cbl 1, vlan(3968) cbl 1, vlan(3971) cbl 1,


vlan(152) cbl 1, vlan(153) cbl 1, vlan(154) cbl 1, vlan(155) cbl 1,




vlan(152) cbl 1, vlan(153) cbl 1, vlan(154) cbl 1, vlan(155) cbl 1,




Vemcmd show trunk

• Control and packet are CBL states 1 on the physical ports.

127

~ # vemcmd show port vlans

Native VLAN Allowed

LTL VSM Port Mode VLAN State Vlans

17 Eth5/1 T 1 FWD 2,10-11,150-155

~ #

vemcmd show port vlans


~ # vemcmd show l2 2


VLAN 2, swbd 2, ""


Type MAC Address LTL timeout Flags PVLAN

Static 00:02:3d:21:a6:04 12 0

Dynamic 00:50:56:a9:25:35 17 1

Step 6: VEM and VSM See Each Other’s MAC

• Is the VEM learning the MAC of the VSM?

• On VEM “vemcmd show l2 <control-vlan>” do you see the mac of the VSM?

128


n1kv-l2# show mac address-table vlan 2

VLAN MAC Address Type Age Port Mod

---------+-----------------+-------+---------+------------------------------+---

2 0002.3d21.a604 static 0 N1KV Internal Port 5

2 0002.3d41.a604 static 0 N1KV Internal Port 5

VEM and VSM See Each Other’s MAC

• Is the VSM learning the MAC of the VEM?

129


cae-cat6k-1#show mac-address-table vlan 2

Legend: * - primary entry

age - seconds since last seen

n/a - not available

vlan mac address type learn age ports

------+----------------+--------+-----+----------+--------------------------

* 2 0050.5677.7770 dynamic Yes 360 Gi3/48

* 2 0050.56a9.2535 dynamic Yes 0 Gi4/9

* 2 3333.0000.0016 static Yes - Switch,Stby-Switch

* 2 0002.3d41.a604 dynamic Yes 0 Gi1/4

Step 7: Physical Switch Mac Table

• Check the physical switch MAC address table

• Are the MACs of the VEM and VSM getting learned by the physical switches in the right VLANs?

130


Step 8: VEM – VSM Heartbeat

• One Heartbeat per second per VEM from VSM

• Timeout for VEM from VSM is 6 seconds of missed heartbeats

• After 6 seconds VSM will drop VEM

• Use vempkt capture to view heartbeats

• SPAN physical switch ports for heartbeats

131

Appendix B – Miscellaneous Commands

Appendix C – VXLAN Multicast


VXLAN Configuration: Multicast

• VMkernel interface to act as VTEP

• VSM Control Mode should be L3

• Multicast for Broadcast traffic

• IP Multicast forwarding is required

– Multicast addresses

– Multiple segments can be mapped to a single multicast group

– If VXLAN transport is contained to a single VLAN, IGMP Querier must be enabled on that VLAN

– If VXLAN transport is traversing routers • Multicast routing must be enabled.

• Proxy ARP must also be enabled

• 1550 MTU for VXLAN encapsulation overhead

134



• Upstream Switch Configuration

– Enable IGMP Querier

– Set physical switch port MTU to 1550

– Enable proxy-arp on upstream SVI

• ESXi Host

– Create VMK interface for VXLAN

• Nexus 1000V

– Enable “feature segmentation”

– Create a Bridge Domain

– Create a port-profile for VTEP VMK interface

– Create a veth port-profile for the VMs

135



• Increase the MTU on your eth port-profile

n1kv-l3(config)# port-profile type eth uplink

n1kv-l3(config-port-prof)# mtu 1550

• Create veth port-profile for VXLAN VMK interface

n1kv-l3(config)# port-profile type vethernet VXLAN-VMK

n1kv-l3(config-port-prof)# switchport mode access

n1kv-l3(config-port-prof)# switchport access vlan 11

n1kv-l3(config-port-prof)# no shutdown

n1kv-l3(config-port-prof)# system vlan 11

n1kv-l3(config-port-prof)# vmware port-group

n1kv-l3(config-port-prof)# capability vxlan

n1kv-l3(config-port-prof)# state enabled

136



• Configure the Bridge Domain

Maps a segment ID to a multicast address

Segment ID >4096

n1kv-l3(config)# bridge-domain vxlan-1

n1kv-l3(config-bd)# segment id 5000

n1kv-l3(config-bd)# group 224.3.5.2

• Create VM port-profile

n1kv-l3(config)# port-profile type veth vm-vxlan-1

n1kv-l3(config-port-prof)# vmware port-group

n1kv-l3(config-port-prof)# switchport mode access

n1kv-l3(config-port-prof)# switchport access bridge-domain vxlan-1

n1kv-l3(config-port-prof)# no shut

n1kv-l3(config-port-prof)# state enabled

–

137


VXLAN Troubleshooting Tips

• Verify your Bridge Domains, VM port-profiles, and VXLAN VMK port-profiles

• Verify multicast on your upstream switches

– show ip igmp snooping

– Do you see the VTEPs

• Use vmkping on the ESXi host to verify network and MTU

– Use 1542 to cover the addition of the ICMP header

– ~ # vmkping -s 1542 -d 1.1.1.1

• Verify the VEM has the right VXLAN capability

~ # vemcmd show vxlan interfaces

LTL IP

---------------

69 1.1.1.2

138


VXLAN Troubleshooting Tips

~ # vemcmd show port vlans

LTL VSM Port Mode VLAN/ State Vlans/SegID

17 Eth4/1 T 1 FWD 25,626-640

18 Eth4/2 T 1 FWD 25,626-640

53 Veth19 A 6000 FWD 6000

• Verify the VEM was programmed correctly

~ # vemcmd show segment 6000

BD 23, vdc 1, segment id 6000, segment group IP 225.6.26.10, swbd 4096, 2 ports, "dvs.VCDVSvCDNI-6-26-vl634-backed-b69c1d1d-02bf-4581-9b7e-fa06c64e8c18"

Portlist:

53 vse-vCDNI-6-26-vl634-backed (b6

68 vCDNI-2 (5ac7d73c-d1d1-4877-8ef

139


VXLAN Other Useful Commands

• vemcmd show port

• vemcmd show igmp <vlan>

• vemcmd show l2 segment <segment-id>

• vemcmd show vxlan-encap [ltl/mac] <ltl/MAC address>

• vemcmd show vlxan-stats all

• Detailed slides in the Appendix

140

Appendix D - Additional VXLAN TShoot



• Verify Multicast Upstream Nexus 7K/5K

– Verify querier is enabled for vlan VMK interfaces are on switch# show run

vlan configuration 634

ip igmp snooping querier 1.1.1.161

142


VXLAN other Useful Commands

• Verify IGMP snooping is configured

CWD.35.04-7000-1# show ip igmp snooping vlan 634

IGMP Snooping information for vlan 634

IGMP snooping enabled

Optimised Multicast Flood (OMF) enabled

IGMP querier present, address: 1.1.1.161, version: 3, i/f Po1

Querier interval: 125 secs

Querier last member query interval: 1 secs

Querier robustness: 2

Switch-querier enabled, address 1.1.1.161, currently running

…..

IGMPv3 Report suppression disabled

Link Local Groups suppression disabled

Router port detection using PIM Hellos, IGMP Queries

Number of router-ports: 1

Number of groups: 1

VLAN vPC function enabled

Active ports:

Po1 Po9 Po17 Po25

Po31 Po52 Po100 Eth2/30

143


VXLN other Useful Commands

• Verify multicast IP address for the VXLAN is being learned

CWD.35.04-7000-1# show ip igmp snooping groups vlan 634

Type: S - Static, D - Dynamic, R - Router port, F - Fabricpath core

port

Vlan Group Address Ver Type Port list

634 */* - R Po1

634 225.6.26.10 v2 D Po100 Eth2/30

144



• vemcmd show port

– Will show ports that are on a vxlan ~ # vemcmd show port




49 Veth2 UP UP FWD 0 0 vmk0

...

53 Veth19 UP UP FWD 0 vse-vCDNI-6-26-vl634-backed (b6

54 Veth16 UP UP FWD 0 0 vse-vCDNI-6-26-vl634-backed (b6

...

68 Veth21 UP UP FWD 0 vCDNI-2 (5ac7d73c-d1d1-4877-8ef

69 Veth22 UP UP FWD 0 0 vmk1 vxlan

305 Po2 UP UP F/B* 0

145



• vemcmd show igmp <vlan>

– Verify that multicast is enabled ~ # vemcmd show igmp 634

IGMP is ENABLED on VLAN 634

Multicast Group Table:

Group */*, Multicast LTL: 4410

• vemcmd show l2 segment <segment-id>

– Verify the VEM is learning MAC addresses in the VXLAN ~ # vemcmd show l2 segment 6000


Segment ID 6000, swbd 4096, "dvs.VCDVSvCDNI-6-26-vl634-backed-b69c1d1d-02bf-4581-9b7e-fa06c64e8c18"


Type MAC Address LTL timeout Flags PVLAN Remote IP

Static 00:50:56:01:02:0a 68 0 0.0.0.0

Dynamic 00:50:56:01:02:09 305 1 1.1.1.1

Static 00:50:56:01:02:15 53 0 0.0.0.0

146



• vemcmd show vxlan-encap [ltl/mac] <ltl/MAC address>

– Identify the traffic path a MAC or LTL will utilize ~ # vemcmd show vxlan-encap ltl 68

Encapsulation details for LTL 68 in BD "dvs.VCDVSvCDNI-6-26-vl634-backed-b69c1d1d-02bf-4581-9b7e-fa06c64e8c18":

Source MAC: 00:50:56:01:02:0a

Segment ID: 6000

Multicast Group IP: 225.6.26.10

Encapsulating L2 LISP Interface LTL: 69

Encapsulating Source IP: 1.1.1.2

Encapsulating Source MAC: 00:50:56:7e:0e:b6

Pinning of L2 LISP Interface to the Uplink:

LTL IfIndex PC_LTL VSM_SGID Eff_SGID iSCSI_LTL* Name

69 1c000150 305 32 0 0 vmk1

147



• vemcmd show vlxan-stats all

– Show VXLAN traffic stats ~ # vemcmd show vxlan-stats all

LTL Ucast Mcast Ucast Mcast Total

Encaps Encaps Decaps Decaps Drops

53 67 300 47 0 23

68 11701 47135 11690 61 12

69 11768 125793 11737 61 0

148


VXLAN Load Balancing

• With LACP port-channel 5-tuple hash is used

– Use singe VMK VXLAN interface

– VEM does the hashing across all the links

– Remember to change load balancing to 5-tuple hashing • On the upstream switch and on the VSM

• With VPC MAC Pinning

– Create a VMK VXLAN interface for each available uplink

– VEM will pin an interface to each available link

– The VEM will distribute the VM's flows between the vmknics based on a hash of the source MAC.

149


Verification: Unicast

Verify the bridge-domain configuration on VSM switch# sho bridge-domain

Global Configuration:

Mode: Unicast-only

MAC Distribution: Disable

Bridge-domain segment-cisco (3 ports in all)

Segment ID: 9001 (Manual/Active)

Mode: Unicast-only (default)

MAC Distribution: Disable (default)

Group IP: NULL

State: UP Mac learning: Enabled

Veth2, Veth3, Veth5

Verify the bridge-domain configuration on VEM

switch# module vem 4 execute vemcmd show bd bd-name segment-cisco

BD 26, vdc 1, segment id 9001, segment group IP 0.0.0.0, swbd 4102, 2 ports, "segment-cisco"

Segment Mode: Unicast

VTEP DSN: 1 , MAC DSN: 1

Portlist:

53 RedHat_VM1_112.eth4

54 RedHat_VM1_112.eth5

~ #

If MAC Distribution is enabled this will be ‘Enable’

If MAC Distribution is enabled this will be

“Segment Mode: Unicast, Mac-Distribution”

VTEP and MAC download sequence numbers should

be checked against VTEP entries (vemcmd show

vxlan-vteps) and MAC entries (vemcmd show l2 bd

bd-name <>) respectively

150


Verification (Port configuration)

Verify the port configuration on VSM switch# sho int switchport | begin Vethernet2

Name: Vethernet2

Switchport: Enabled

Switchport Monitor: Not enabled

Operational Mode: access

Access Mode VLAN: 0 (none)

Access BD name: segment-cisco

[SNIP]

151


Verification (Port configuration)

switch# module vem 4 execute vemcmd show port



49 DOWN UP BLK 0 RedHat_VM1_112 ethernet7

50 Veth8 DOWN UP BLK 0 RedHat_VM1_112.eth8

51 Veth4 UP UP FWD 0 0 vmk1 VXLAN

52 DOWN UP BLK 0 RedHat_VM1_112.eth6

53 Veth2 UP UP FWD 0 RedHat_VM1_112.eth4

54 Veth3 UP UP FWD 0 RedHat_VM1_112.eth5

561 Po2 UP UP F/B* 0

* F/B: Port is BLOCKED on some of the vlans.

One or more vlans are either not created or

not in the list of allowed vlans for this port.

Please run "vemcmd show port vlans" to see the details.

~ #

Verify the port configuration on VEM

152


Verification (VTEP Distribution)

Verify the VTEP distribution on VEM

switch# sho bridge-domain segment-cisco vteps

D: Designated VTEP I:Forwarding Publish Incapable VTEP

Bridge-domain: segment-cisco

VTEP Table Version: 2

Ifindex Module VTEP-IP Address

-----------------------------------------------------------------

-------------

Veth4 4 10.106.199.116(D)

Veth1 5 10.106.199.117(D)

switch#

Verify the VTEP distribution on VSM

switch# module vem 4 execute vemcmd show vxlan-vteps

Bridge-Domain: segment-cisco Segment ID: 9001

Designated Remote VTEP IPs (*=forwarding publish incapable):

10.106.199.117(DSN: 1),

To be compared with

echo “show vxlan

version-table” output on

VEM

Compare against “vemcmd show bd bd-name <>”

VTEP DSN output

153


Verification (MAC table in unicast only mode)

switch# module vem 4 execute vemcmd show l2 bd-name segment-cisco


Segment ID 9001, swbd 4102, "segment-cisco"


Type MAC Address LTL timeout Flags PVLAN Remote IP DSN

Dynamic 00:50:56:83:01:4e 561 1 10.106.199.117 0

Static 00:50:56:83:01:61 54 0 0.0.0.0 0

Static 00:50:56:83:01:60 53 0 0.0.0.0 0

switch#

MAC address table will display remote IP learning in the segment-cisco bridge domain

154

Documents

Deploying and Troubleshooting the Nexus 1000v …d2zmdbbm9feqrf.cloudfront.net/2014/usa/pdf/BRKVIR-3013.pdfDeploying and Troubleshooting the Nexus 1000v Virtual Switch on vSphere BRKVIR-3013