© 2016 Cisco and/or its affiliates. All rights reserved ...d2zmdbbm9feqrf.cloudfront.net/2016/usa/pdf/BRKCOM-2007.pdf · © 2016 Cisco and/or its affiliates. All rights reserved

© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public


UCS Storage Integration, Technologies, and Topologies

Craig Ashapa, Technical Marketing Engineer

BRKCOM-2007


Agenda

Cisco UCS 3rd Generation Fabric Interconnect

FC/FCoE Uplink ConnectivitySAN Uplinks, FCoE Multi-Hop, Brocade, Hyper-VFC/FCoE

Modes of Operation, Direct Connectivity of Storage, Local Zoning

IP-Based Storage

Storage Profiles

UCS C3000 Series

Monitoring, Troubleshooting, Advanced CLI


Cisco UCS 3rd Generation Fabric Interconnect


FI 6332• 32 x 40GbE QSFP+ ports

• 2.56Tbps switching performance

• 1RU fixed form factor, two power supplies & four fans

FI 6332-16UP• 24 x 40GbE QSFP+ & 16 x UP ports (1/10GbE or 4/8/16G FC)

• 2.43Tbps switching performance

• 1RU fixed form factor, two power supplies & four fans

IOM 2304• 8 x 40GbE server links & 4 x 40GbE QSFP+ uplinks

• 960Gbps switching performance

• Modular IOM for UCS 5108

UCS FI & IOM ModelsFI 6300 Series and IOM 2304

BRKCOM-2007 5


FI 6300 Series Hardware Overview

FI 6332 (Front View) FI 6332-16UP (Front View)

26 x 40G QSFP+ *

or

98 x 10G SFP+ **

6 x 40G QSFP+

* QSA module required on ports 13-14 to provide 10G support

** Requires QSFP to 4xSFP breakout cable

18 x 40G QSFP+

Or

72 x 10G SFP+ *

6 x 40G QSFP+16 x UP

16 x 1/10G SFP+

or

16 x 4/8/16G FC

2 x Hot Swappable

Power Supplies (1+1)

4 x Hot Swappable

Fans (3+1)

6300 Series

(Rear View) Serial Ports

L1 & L2

High

avail

ports

L1 & L2

High

avail

ports

BRKCOM-2007 6


6332 Switch on Chip (SOC)

Storage Protocols

FCoE

iSCSI

Ethernet

40G only ports (Uplink Ports)

Storage Protocols

FCoE

iSCSI

Ethernet

6332-16UP Unified Port Controller (UPC)

Storage Protocols

FC

FCoE

iSCSI

Ethernet

Switch on Chip (SOC)

Same as 6332

40G only ports (Uplink Ports)

Same as 6332

6332 & 6332-16UP ASICs/Storage Protocols

BRKCOM-2007 7


6332-16UP ASIC Port Mapping

6332 ASIC Port Mapping

UPC SOC 40G Only

40G OnlySOC

BRKCOM-2007 8


6332/6332-16UP QSFP Breakout Ports

Each SOC QSFP Can be Breakout Port

Breakout Port Creates (4) 10Gb Ports

Port Modes Same as on Non-Breakout Ports

Each Port of Breakout Port Configured Individually

Change to/from Breakout Port Requires FI Reboot

Note: If 2nd No-Drop Class Configured with Jumbo MTU, QSFP

Breakout Ports limited to 4 (ie, 16 individual 10Gb ports) -

Remainder of QSFP Ports will be Non-Breakout 40Gb ports.

BRKCOM-2007 9


ASIC Type Server

Port

Network

Uplink

Port

FC

Uplink

Port

FCoE

Uplink

Port

FC

Storage

Port

FCoE

Storage

Port

Appliance

Port

Unified

Port

UPC

6332-16UP

SOC

6332-16UP/6332

40G Only

6332-16UP/6332

Port Type vs. ASIC Type

Port Types vs. ASIC Type(Green Means Qualified)

BRKCOM-2007 10


6332-16UP UP Port Groups

16 Unified SFP Ports

Configured as 6/6/4 Port Groups

FI Reboots Upon Change

All Groups FC/All Ethernet/ Or Mix

UCSM Configure Unified Ports

Slider - FC Left / Ethernet Right6 FC / 12 Ethernet

BRKCOM-2007 11


• Interface

• NIF

• 4 x 40G QSFP

• Connects only to FI63xx

• HIF

• 32 Interfaces

• Support 10G or 4 ports can combine to a single 40G

2304 IO Module

Templeton ASIC

Internal backplane ports to blades

No Local Switching – ever!

Traffic goes up to FI

Fabric Ports to FI63xx

1 32

BRKCOM-2007 12


`

VIC1340

23X

X-B

Mezz 1

(empty)

23X

X-A

CPU # 1 CPU # 0

Active KR Lane

Blade Server

PCIe Lanes

QPI

PCIe Lanes

20G (2x10G)

IOM 2304 and Adapter ConnectionVIC1340 Only

BRKCOM-2007 13


`

VIC1340

23X

X-B

Port Expander

Card

23X

X-A

Blade Server

CPU # 1 CPU # 0

PCIe Lanes

QPI

PCIe Lanes

40G (native)

2304 IOM and Adapter ConnectionVIC1340 Plus Port Expander

BRKCOM-2007 14


• Adapter Resiliency

• 2 independent Adapters

• vCon placement

• 4 20G connections

• 20G are 2x10

• 6 tuple hashing`

VIC1340

23X

X-B

VIC1380

23X

X-A

Blade Server

CPU # 1 CPU # 0

PCIe Lanes

QPI

PCIe Lanes

2304 IOM and Adapter ConnectionVIC1340 Plus VIC1380

BRKCOM-2007 15


FC/FCoE Uplink Connectivity,SAN Uplinks, FCoE Multi-Hop, Brocade, Hyper-V


Block Storage Uplink Versatility

Fibre Channel (FC) Up to 16Gb FC Connectivity with UCS Gen3

Cisco MDS

Cisco Nexus N5k

Brocade (See Cisco HCL)

Fibre Channel over Ethernet (FCoE) Up to 40Gb FCoE Connectivity with Gen3

Cisco MDS

97xx with 48-Port 10/40 Gbps FCoE Modules

9250i Multiservice Fabric Switch - 10 Gbps FCoE

Cisco Nexus

N5k - 10/40 Gbps FCoE

N7k with F-Series 10/40 Gbps Modules

17BRKCOM-2007


FC/FCoE Port Trunking / Channeling

Port Trunking

Multiple VSANs per Link

Help consolidate Infrastructure

vHBAs can be on different VSANs

All VSANs trunked on all FC/FCoE uplinks – No Pruning

Scalability Per UCS System:

Max of 32 VSANs (62xx/6332/6324)

Max of 15 VSANs (6332-16UP)

Port Channeling

Up to 16 FC/FCoE ports can be aggregated together in a

single FC/FCoE port channel (62xx//6332)

VSANs can be trunked over the port channel

Port Trunking/Channeling in NPV and FC Switch mode

Trunking/Channeling is not available for direct connect

FC/FCoE Storage Port types

18BRKCOM-2007


How FC/FCoE Trunking is Enabled

Per UCS Fabric Setting

Default is Not enabled

Enabling Flaps Ports

Maintenance Window

End Host Mode – TNP

FC Switch Mode – TE

Don’t Forget Northbound Device Configs!

19BRKCOM-2007


Multi-Hop FCoE

End-to-End 10/40Gb FCoE

MDS, N5K, N7K FCoE Capable Switches Supported Upstream

“Unified Uplink” Port Type

Carries FCoE and Normal Ethernet Traffic

Fabric Interconnect in NPV Mode Primary use Case

VNP Ports

FI in FC Switching Mode

VE Ports also Supported

MDS/N5K/N7K

Unified Uplinks

UCS B-Series

UCS FI UCS FI

FCoE STORAGE

MDS/N5K/N7K

NPV/EHM

FCoE FCoE

FCoE/Ethernet FCoE/Ethernet

20BRKCOM-2007


Multi-Hop FCoE

Pure FCoE or Converged FCoE/Ethernet Uplinks Northbound Switch Dependent

FCoE Uplinks Individual

Port Channels (62xx/6332): Max 12, Max 16 Members Per)

FCoE Uplink Trunking/Port Channels Trunking is Global

All VSANs Available on All Uplinks

FCoE Uplink Port is a “Border Port” NPV Mode – NP/TNP

FC Switching Mode – E/VE

21BRKCOM-2007


Multi-Hop FCoENorthbound Connectivity Comparison

MDS N5k N7k

FCoE Uplinks

Converged Uplinks N/A

Converged vPC N/A

FCoE Uplinks Carry Pure FCoE Storage Traffic

FCoE Uplinks Individual or Port-Channel

Converged Uplinks Carry FCoE AND Ethernet LAN traffic

Converged Uplinks Individual or Port-Channel

22BRKCOM-2007


UCS to Brocade FC Connectivity

Brocade NPIV Mode / Cisco UCS End-Host (NPV) Mode

Link Aggregation Not Possible

Cisco – Port Channeling / Brocade – Port Trunking

Cisco Trunking and VSANs Northbound of FIs Not Supported

See Cisco HCL for Latest Supported Brocade FOS Versions

http://www.cisco.com/en/US/products/ps10477/prod_technical_reference_list.html

Cisco to Brocade Traffic Engineering

Common VSANs

SAN Pin Groups

Unique VSANs

See Cisco UCS to Brocade Connectivity Guide for Additional Information

http://www.cisco.com/en/US/prod/collateral/ps10265/ps10281/whitepaper_C07-730016.pdf

23BRKCOM-2007

http://www.cisco.com/en/US/products/ps10477/prod_technical_reference_list.html

http://www.cisco.com/en/US/prod/collateral/ps10265/ps10281/whitepaper_C07-730016.pdf


UCS – Hyper-V Synthetic Fibre Channel

SAN-ANPIV Mode

UCS FI-ANPV Mode

vHBA – UCS Virtual HBAsHBA – Hyper-V Synthetic HBA

sHBA1

vHBA1

vHBA2

sHBA3

sHBA5

sHBA7

sHBA2

sHBA4

sHBA6

sHBA8

Hyper-V Server 2012

VM1

VM2

VM3

VM4

UCS FNIC NPIV Enabled

SAN-BNPIV Mode

UCS FI-BNPV Mode

sHBA WWPNs: C0:03:FF:XX:XX:XX:XX:XX

vHBA WWPNs: 20:00:00:25:b5:XX:XX:XX

20:00:00:25:b5:XX:XX:XXC0:03:FF:XX:XX:XX:XX:XX

UCS FNIC NPIV Enabled

24BRKCOM-2007



Benefits Security – FC LUN Masked Direct to VM

Multi-Pathing – Individual Multi-Pathing Policies per VM

Shared Clustered FC storage

Live Migration Resiliency – Uninterrupted FC Access via Microsoft’s Synthetic HBAs

Details NPIV on the UCS FNIC

Enabled by Default, Not Configurable

Requires NPIV Enabled Upstream SAN (Pre-UCSM 3.1.2)

Supports Windows Server 2008, 2008R2, 2012, 2012R2 VMs

4 Microsoft Synthetic HBAs per VM Maximum

128 Maximum NPIV Ports per Hyper-V Server Example: Microsoft Hyper-V (32) virtual machines each with (4) Virtual Fibre Channel

Adapters = 128 NPIV ports (128 is the maximum)

25BRKCOM-2007



Caveats Not supported with UCS Local Zoning (Pre-UCSM 3.1.2)

No Persistent Binding

No Boot from SAN from VM via Microsoft Synthetic FC Adapters

26BRKCOM-2007


Modes of Operation, Direct Connectivity of Storage,Port Types, Local Zoning

© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 28BRKCOM-2007

Operation Mode for FC/FCoE

End-Host (NPV) Mode - Default

UCS Functions as Node Port (initiator)

Required for Connecting FC to Non-MDS FC Switches

FC Switching Mode

Upstream MDS or Nexus FC Switch Supported

Required for UCS Local Zoning Feature

Direct Connect from Fabric Interconnect to FC/FCoE Storage Target

Operation Mode for Ethernet/iSCSI/NAS

End-Host Mode - Default

Appliance Ports which allow Direct Connect Ethernet/iSCSI/NAS Storage Targets

Ethernet Switch

No Storage Based Reasons to use this Mode

Operation Mode vs. Features


Four UCS port types related to direct storage capability

Storage FC Port – direct connect this port to FC port on array

Storage FCoE Port – direct connect this port to FCoE port on array

Appliance Port - direct connect this port to 10G/40G Ethernet port on array

Unified Storage Port (Combined Appliance and FCoE) – direct connect this port to NetApp Unified Target Adapter on array

UCS Direct Connect Storage

29BRKCOM-2007


Port Types vs. FI Operating Mode (Green Means Valid in Mode of Operation)

FI Operational Role Server

Port

Uplink

Port

Uplink

FC Port

FCoE

Uplink

Port

FC

Storage

Port

FCoE

Storage

Port

Appliance

Port

Unified

Port

Ethernet: EHM

FC: EHM

Ethernet: EHM

FC: Switching

Ethernet: Switching

FC: EHM

Ethernet: Switching

FC: Switching

FI Mode vs. Valid Port Type in that Mode

Valid Port Types Available with UCS 2.1 and Above Release

End Host Mode is the Default Mode for the Fabric Interconnects

30BRKCOM-2007


Local Zoning OR Upstream Zoning

UCS Configured in FC Switch Mode

Ethernet and FC Switching modes are Independent

Hybrid Topology - Upstream Zoning Provided by MDS/Nxk Switches

See UCS HCL for Supported Arrays

Support to Directly Connect FC/ FCoE Storage to Fabric Interconnects

End to End FC/FCoE Topologies Possible

Lower Cost Point for Small Deployments (no Access Layer FC/FCoE Switches Required)

Customer benefits

Feature details

FCoE Storage FC Storage

Direct Connection of FC/FCoE Storage

UCS B-Series

UCS 62xx/63xx UCS 62xx/63xx

31BRKCOM-2007


FC/FCoE Local Zoning

Direct connect FC/FCoE storage to FI with noupstream SAN switch

FIs in FC Switch Mode

Implicit zone creation No need to create zones

manually

Single-Initiator-Single-Target and Multi-Target Zones

Migrating from Upstream to Local Zoning

CLI Commands to Remove Residual Upstream Zoning

FCoE Storage FC Storage

UCS B-Series


32BRKCOM-2007


UCSM User Defined Zoning

New to UCSM 3.1.2

Custom Zoning Creation without Upstream FC Switch FC Target to FC Target Zoning

Replication

Microsoft Hyper-V Virtual Fibre Channel Zoning of Microsoft Synthetic HBAs to UCS

Direct Attached Storage

Live Migration

New FC Zone Profile Construct Logical Container for FC Zones

Each Zone can be in Different VSAN

Enable/Disable FC Zone Profile Enables/Disables all Zones within that Profile

33BRKCOM-2007

sHBA to FC Storage

Target to Target


UCSM User Defined Zoning

34BRKCOM-2007

CC25-Antwerp-A# scope fc-storage

CC25-Antwerp-A /fc-storage # show fc-zone-profile

FC Zone Profile:

Name Descr Admin State Oper State

---------- ---------- ----------- -----------------------

hyper-v-zone Enabled Ok

NetApp_Repl Enabled Ok

CC25-Antwerp-A /fc-storage # scope fc-zone-profile NetApp_Repl

CC25-Antwerp-A /fc-storage/fc-zone-profile # show fc-user-zone detail expand

FC User Zone:

Name: Net-T1_Net-T2

Zone Name: ucs_CC25-Antwerp_A_6_NetApp_Repl_Net-T1_Net-T2

Vnet: 1500

Path: A

Config State: Applied

Oper State: Active

Zone Member:

Member wwpn address: 50:0A:09:84:8F:9B:4C:B2

Member wwpn address: 50:0A:09:88:8F:3B:02:C5

Target VSAN:

Name: 1500

VSAN ID: 1500

Operational VSAN: fabric/fc-estc/net-1500

Fabric ID: Dual

FC Zoning: Enabled

Zone Name: ucs_<system

name>_<switch id>_<zone_id>_<Zone

Profile Name>_<FC Zone name>


IP-based StorageAppliance Ports, iSCSI


UCS Manager Appliance Ports

Appliance Port

Utilized for:

Direct Connect IP Storage

Direct Connect Unmanaged non-Switch/non-Router IP Devices (Servers, Appliances)

iSCSI/NFS/SMB

Qualified Scale – 4 Ports per FI

Ethernet End-Host and Switching Mode

Does Not Run STP

36BRKCOM-2007


Appliance Port Exposed Settings

QoS per port settings,

normal UCS QoS

constructs

Manual (static) pinning

using pin groups for border

port selection

Select which VLANs can

traverse this port

Optionally specify the

destination MAC address

of the filer . Some Filers

do not broadcast their

MAC address

QoS Settings

VLAN Trunking

Optional – If Not

then MAC

Learning

37BRKCOM-2007


VLANs and Appliance Ports

Similar to VSAN concept, there are two scopes

Traditional, LAN Cloud

Appliance Cloud with scope restricted to appliance ports and associated VLANs

Use the same VLAN ID in both scopes

LAN Scope

Appliance

Scope

38BRKCOM-2007


Appliance Ports Have Uplink Ports

Appliance ports like server ports have an uplink/border port assigned by static or dynamic pinning

Loss of last uplink port results in UCS appliance port being taken down

Default Network Control policy

Modifiable to Warning

39BRKCOM-2007


UCSM iSCSI Feature Overview

Primary purpose is to support booting via iSCSI

Object called an “iSCSI vNIC” is created as a child object to the parent vNIC

IP Assignment – DHCP, Pools, Static

IQN Assignment – Manual, Pools, Per vNIC, Single Profile IQN (RFC3720)

CHAP / Mutual-CHAP Security

Initiator Authentication (CHAP), Target Authentication (M-CHAP)

Password (CHAP Secret) 12-16 Characters, $ = ? Invalid

iSCSI Failover and Redundancy

MPIO / Host Multi-Pathing Drivers Similar to FC – No Network Link Aggregation

40BRKCOM-2007


iSCSI Boot Flow

Create iSCSI vNICs

Create iSCSI boot policy

Provide UCSM with iSCSI boot information Target ip, iqn

Initiator ip/mask/gw, iqn

vMedia map the OS and drivers

Adapter successfully initializes

Install OS and Drivers (if required)

41BRKCOM-2007


Unified Appliance Support

File and block data over a single

port/cable

FCoE, iSCSI, NFS, SMB

Port and cable consolidation

New port type: Unified Appliance Port

Appliance port of today + FCoE

Initial support for NetApp storage and

their Unified Target Adapter

Storage

FCoEiSCSINFSCIFSUnified Appliance Port

UCS B-Series


42BRKCOM-2007


Storage Profiles


Storage Profiles – Local Storage

44

Legacy – Initial Disk Configuration and Virtual Drive Creation

BRKCOM-2007

© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 45

Storage Profiles / MultiLUN

Today - UCSM Profile Driven Local Storage

Local LUN Provisioning from Within UCSM

Storage Profiles

Virtual Drives

Advanced Disk Geometries and Configurations

Disk Group Policies

Direct Communication to Local PCIe Storage Controller

PCH Controller Integration

C240M4 – Internal SSDs

Modification of Local Storage Without Reboot

Consumed by Service Profiles and Templates

UCSM Managed M3 and M4 (B,C)

BRKCOM-2007



Disk Group Policies

Each Storage Profile Defined LUN Mapped to a Disk Group

Disk Group Configuration Defines RAID Level, Disks and

Quantities

Automatic Configuration

Chooses from Available Disks

Disk Qualifiers

Manual Configuration – Choose Specific Disks

Virtual Drive Configuration Defines LUN Parameters Within

Disk Group

46BRKCOM-2007



Local LUN Creation

Define Individual LUNs

Claim Existing Local LUN (Orphan LUN)

Expand To Available

Use All Remaining Space in Disk Group

Auto / No Auto Deploy

Automatic or Manual LUN Deployment

Disk Group

Consume or Create Disk Group for LUN

BRKCOM-2007 47


Storage Profiles – Local Storage Advanced Disk Geometries and Configurations

Multiple Disk Groups and Virtual Drives (LUNs

48

Virtual Drives on Server

Physical Disks in Selected Virtual Drive

BRKCOM-2007


Storage Profiles – Provisioning

BRKCOM-2007 49

Local Storage Provisioning Flexibility

Specific Storage Profiles

Single System Use

Legacy Local Disk Configuration

Storage Profile Policies

Multi-Use


UCSM Control of PCH Wellsburg Controller

Responsible for the (2) Internal C240M4 SSDs

Models UCSC-C240-M4L/M4SXX

AHCI and SWRAID Modes

RAID0 and RAID1

Integrated into Storage Profiles Construct

Scrub Policy and Protect Configuration Support

UCSM PCH/Internal SSD Support



51

Boot Policies Boot From Local LUN

Enter Defined Boot LUN Name

PCH Controller Support C240M4 Internal SSDs

SWRAID - Add Embedded Local LUN

AHCI – Add Embedded Local Disk

Local LUN Image Path Secondary Option – M Series Only

BRKCOM-2007


UCS C3000 High Density Server


Compute Resources

Storage Resources

Shared Local Resources Designed for large unstructured data repositories, backup and

archiving, cloud storage and long term retention

Full-Featured Modular Two-Socket Xeon Server

Stand-alone CIMC, IMC Supervisor and UCSM

Massive Local Storage

Up to 560TB of dense storage in a compact 4U Form Factor that fits in a standard rack

Network Resources

High I/O Bandwidth

Powered by Cisco VIC and the latest LSI 12-SAS Lane Technology

UCS C3000 Storage

BRKCOM-2007 53


UCS C3000 Chassis ArchitectureDrives

4 Rows of hot-swappable HDD

4/6/8/10TB with up to 2 rows

of 400/800GB/1.6TB SSD

Total top load: 56 drives

FAN8 hot-pluggable

fans

Server Node2x E5-2600 V2 or v4 CPUs

128/256/512GB RAM

1GB/4GB RAID Cache

SAS HBA Node

Optional Slot 2Server Node, I/O Expansion or

Drive Expansion

Four 120GB/480GB

SSDsHot-Plug OS/Boot

System I/O Controller (SIOC)Cisco mLOM Slot or VIC 1300 on Chip

Power Supply4 hot-pluggable PSUs

Form Factor4U / Standard

Depth Rack <32”

BRKCOM-2007 54


Broadwell Refresh Server Node SPECS:

1 or 2 CPU per Server Node: E5-2620v4: 8C/2.1G/85W,

E5-2650v4: 12C/2.2G/105W,

E5-2680v4: 14C/2.4G/120W

E5-2695v4: 18C/2.1G/120W

4 or 8 DDR4 2400Mhz DIMMs per CPU: 16G, 32G, 64G (Post-FCS)

RAID with 4G Cache: LSI SAS3316/24(Intruder) 12-CH

Up to 1, 15mm NVMe on board: 800G or 1.6TB

HW RAID for Boot Drives

BRKCOM-2007 55


Extending UCS C3260 Infrastructure with 2nd Slot

Storage

Optional 2nd Server Node

Network

Optional 2X PCIe Expansion Board

Compute

Optional 4X LFF Disk Expansion

BRKCOM-2007 56


PCIe Expansion Board Stonycreek

Occupies Top Slot

2x PCIe Slots: Half Height/Half Width

Ethernet: Intel x520, Intel Quad port

Fiber Channel: Emulex, Qlogic

PCIe Flash Storage: Fusion I/O

Coming Soon:

2nd RAID/HBA Controller in single server node

2x NVMe On board

BRKCOM-2007 57


UCS C3260 Management

Standalone

CIMC WebUI - HTML5 Interface

CLI

XML API

IPM

Dynamic Storage RAID Card – Dedicated

Disks

HBA Card – Dedicated and Shared Disks

Global Hot Spares

58BRKCOM-2007


UCS C3260 Management

• UCSM 3.1.2 Integration

Fully Managed by 2nd and 3rd

Generation Fabric Interconnects

Connects via FI Server Ports to 3260 SIOC Ports

Each 3260 Physical Box is a Chassis

Chassis-Wide and Per Server Node Management

Inventory, Compute and Storage Configuration, FW Mgmt, Pools, Policies, Profiles, Templates, vNICs/vHBAs, and Much More

Storage Profiles – Disk Group (RAID) and LUN Configuration

59BRKCOM-2007


UCS C3260 UCSM Integration

New Chassis Profile Object

Container for Chassis Firmware Package and Disk Zoning Policy

Assigned to Each 3260 Chassis to Apply Specified FW Package and Disk Zoning Policy

Initial and Updating Templates

New Parent Navigation Bar Chassis Icon


UCS 3260 UCSM Integration

Disk Zoning Policy

Defines Disk Ownership from Chassis Disk Slots for Each Server Node

Unassigned, Dedicated, Shared (with HBA Card), Global Hot Spares

One Policy per Chassis

Consumed by Chassis Profile


Monitoring, Troubleshooting and Advanced CLI


Monitoring the Storage Path - Identification

Adapter Port:

VIC to IOM

HIF Port:

IOM to VIC

NIF Port:

IOM to FI

Server Port:

FI to IOM

Uplink Port:

FI to SAN

63BRKCOM-2007


Monitoring the Storage Path - GUI

GUI – VIF Paths Tab

vHBA to IOM to FI to Uplink

GUI – Statistics Tab

vHBA

IO Module Backplane Port

Fabric Interconnect Server Port

Fabric Interconnect Uplink Port

64BRKCOM-2007


Monitoring the Storage Path – GUIVIF Screen – With Gen2/3 VIC to FI Port Channels (220x/2304)

Adapter Port: Statistics Under Service Profile vHBA

Fex/Network/FI Server Port: Find Port Channel in Equipment/IOM/Fabric Ports/, Click on one of the Ports and follow

to Peer which is the Fabric Interconnect Server Port – View Statistics

FI Uplink Port: Find Port Channel in SAN\SAN Cloud\Fabric A or B\FC Port Channels – View Statistics

65BRKCOM-2007


Monitoring the Storage Path - CLI

CLI

vHBA - /org/service-profile/vhba # show stats

IOM Module “HIF” Port (Backplane Port)

connect IOM x (x designates IOM number)

For 2104 IOM: show platform software redwood sts

show platform software redwood rmon 0 hifx (x designates hif number)

For 220x IOM: show platform software woodside sts

show platform software woodside rmon 0 hifx (x designates hif number)

For 2304 IOM: show platform software tiburon sts

show platform software tiburon rmon 0 hifx (x designates hif number)

66BRKCOM-2007



Map “HIF” to “NIF” (Not Possible if IOM to FI Port Channels being Used)

connect nxos

show fex x detail (replace x with fex #)

show interface fex-fabric

IOM Module “NIF” Port (Fabric Port)

Connect IOM x (x designates IOM number)

For 2104 IOM: show platform software redwood rmon 0 nifx (x designates nif number)

For 220x IOM: show platform software woodside rmon 0 nifx (x designates nif number)

For 2304 IOM: show platform software tiburon rmon 0 nifx (x designates nif number)

67BRKCOM-2007



Map Backplane Port to FI Server Ports Interface/Port Channel

Connect NXOS

Find IOM/FI Port Channel associated with FEX Port(s)

show fex # detail (replace x with fex #)

Find interfaces associated with IOM/FI Port Channel and Statistics

show interface po # (replace x with po #)

Find FI FC PO /Uplinks and Show PO/Uplink Statistics

connect nxos

show npv flogi-table (match vfc/fcid/wwpn of vHBA to External Interface/po)

show interface san-port-channel # (replace # with po #)

or

From UCS CLI prompt: show service-profile circuit name x (replace x with SP name)

connect nxos

show interface san-port-channel # (replace # with po # found in show circuit output)

68BRKCOM-2007


Monitoring the Storage Path - CLIFor 220x IOM: show platform software woodside sts

69BRKCOM-2007



For 2304 IOM: show platform software tiburon sts

70BRKCOM-2007


Monitoring the Storage Path - CLIFor 220x IOM: show platform software woodside/tiburon rmon 0 hifx (x designates hif number)

71BRKCOM-2007



From UCS CLI prompt: show service-profile circuit name x (replace x with SP name)

72BRKCOM-2007


UCS Internal FC/FCoE Port Channel HashingIOM to FI / FI to IOM

220x/2304 IO Modules – Port Channels Enabled

IOM Inherits Load-Balance Hash from FI

To Display:

connect IOM x (x designates IOM number)

show platform software fwmctrl nifport

Scroll down a bit to see the hash parameters:

l2_da: 1 l2_sa: 1 l2_vlan: 0l3_da: 1 l3_sa: 1l4_da: 1 l4_sa: 1FCoE l2_da: 1 l2_sa: 1 l2_vlan: 0FCoE l3_did: 1 l3_sid: 1 l3_oxid: 1

Transmit (I’ve left MAC addresses of my system as an

example)

Source = 00:05:73:b8:fa:1d FCF-MAC of FC NP uplink port

Destination = 0e:fc:00:7b:09:0f = fpma (0e:fc:00 = FC-MAP

, 7b:09:0f = FC_ID of VFC of FC NP uplink port)

Did = FC_ID of VFC of FC NP Uplink Port

Sid = FC Target FC_ID

OX_ID = Unique Exchange ID

Receive (I’ve left MAC addresses of my system as an

example)

Source = 0e:fc:00:7b:09:0f = fpma (0e:fc:00 = FC-MAP ,

7b:09:0f = FC_ID of VFC of FC NP uplink port)

Destination = 00:05:73:b8:fa:1d = FCF-MAC of FC NP

uplink port

Did = FC Target FC_ID

Sid = FC_ID of VFC of FC NP Uplink Port

OX_ID = Unique Exchange ID

73BRKCOM-2007


UCS Internal FC/FCoE Port Channel HashingIOM to FI / FI to IOM

74BRKCOM-2007


UCS Internal FC/FCoE Port Channel HashingVIC to IOM / IOM to VIC

Sereno/Cruz ASIC – 12xx/13xx Port Channels

To Display Hash:

connect adapter x/y/z (x=chassis, y=blade, z=adapter)

connect

attach-mcp

echash_cfg

UIF: 0

eth fields: MAC_SA, MAC_DA, SRC_PORT, DST_PORT, SRC_IP, DST_IP

fcoe fields: MAC_SA, MAC_DA, OX_ID, RX_ID, S_ID, D_ID

hash bins: 1,3,1,3, 1,3,1,3, 1,3,1,3, 1,3,1,3

crc8 hash poly: polynomial 0x12F

75BRKCOM-2007


UCS Internal FC/FCoE Port Channel HashingVIC to IOM / IOM to VIC

76BRKCOM-2007


UCS Internal vHBA and VNIC ExposedShow vHBA/vNIC State, Speed, MTU

granada-A# connect adapter 1/6/1 (chassis/server/adapter)

adapter 1/6/1 # connect

adapter 1/6/1 (top):6# attach-mcp

adapter 1/6/1 (mcp):3# vnic

Fibre Channel and Ethernet vnics configured on VIC will display13 vnic_1 enet 0 UP 2 UP =>0 746 57 147 UP

14 vnic_2 fc 0 UP 3 UP =>0 747 59 300 UP

15 vnic_3 enet 1 UP 4 UP =>0 751 58 25 UP

16 vnic_4 fc 1 UP 5 UP =>1 748 43 301 UP

adapter 1/6/1 (mcp):4# vnic 14 notify

.

.

link_state: UP

port_speed: 40000

mtu: 2158

77BRKCOM-2007


iSCSI Boot Troubleshooting – Cisco VIC

78BRKCOM-2007

If your SP and iSCSI config is correct,

you will see this during POST

cae-sj-ca1-A# conn adapter 1/8/1


adapter 1/8/1 (top):1# attach-mcp

adapter 1/8/1 (mcp):1# iscsi_get_config

vnic iSCSI Configuration:

----------------------------

vnic_id: 5

link_state: Up

Initiator Cfg:

initiator_state: ISCSI_INITIATOR_READY

initiator_error_code: ISCSI_BOOT_NIC_NO_ERROR

vlan: 0

dhcp status: false

IQN:iqn.2012-03.com.cisco.cma:palo-iscsi-boot:0

IP Addr: 172.25.183.142

Subnet Mask: 255.255.255.0

Gateway: 172.25.183.1Target Cfg:

Target Idx: 0State: ISCSI_TARGET_READY

Prev State: ISCSI_TARGET_DISABLEDTarget Error: ISCSI_TARGET_NO_ERROR

IQN:iqn.199208.com.netapp:sn.101202278IP Addr: 172.25.183.49

Port: 3260Boot Lun: 0

Ping Stats: Success (9.698ms)Session Info: session_id: 0host_number: 0bus_number: 0target_id: 0

Cisco VIC

Initialization


FC Boot Troubleshooting – Cisco VIC

Looking Good – Assigned Service Policy

can see Targets and LUNs on Boot

Using LUNLIST to Troubleshoot

FIELD-TME-EL-CAP-A# connect adapter 3/7/1


adapter 3/7/1 (top):3# attach-fls

adapter 3/7/1 (fls):1# vnic

---- ---- ---- ------- -------

vnic ecpu type state lif

---- ---- ---- ------- -------

15 1 fc active 4

16 2 fc active 5

adapter 3/7/1 (fls):2# lunlist 15

vnic : 15 lifid: 4

- FLOGI State : flogi est (fc_id 0x340005)

- PLOGI Sessions

- WWNN 50:06:01:69:08:64:08:12 fc_id 0x3403ef

- LUN's configured (SCSI Type, Version, Vendor, Serial No.)

(0x0, 0x4, DGC , FNM00130800204)

- REPORT LUNs Query Response

LUN ID : 0x0000000000000000

LUN ID : 0x0001000000000000

- WWNN 50:06:01:60:08:64:08:12 fc_id 0x3402ef

- LUN's configured (SCSI Type, Version, Vendor, Serial No.)

(0x0, 0x4, DGC , FNM00130800204)

- REPORT LUNs Query Response

LUN ID : 0x0000000000000000

LUN ID : 0x0001000000000000

- Nameserver Query Response

- WWPN : 50:06:01:60:08:64:08:12

- WWPN : 50:06:01:69:08:64:08:12

79BRKCOM-2007


Demo


Complete Your Online Session Evaluation

Don’t forget: Cisco Live sessions will be available for viewing on-demand after the event at CiscoLive.com/Online

• Give us your feedback to be entered into a Daily Survey Drawing. A daily winner will receive a $750 Amazon gift card.

• Complete your session surveys through the Cisco Live mobile app or from the Session Catalog on CiscoLive.com/us.

81BRKCOM-2007

CiscoLive.com/Online

http://ciscolive.com/Online

http://ciscolive.com/Online

http://ciscolive.com/us

http://ciscolive.com/us


Continue Your Education

• Demos in the Cisco campus

• Walk-in Self-Paced Labs

• Lunch & Learn

• Meet the Engineer 1:1 meetings

• Related sessions

82BRKCOM-2007


Please join us for the Service Provider Innovation Talk featuring:

Yvette Kanouff | Senior Vice President and General Manager, SP Business

Joe Cozzolino | Senior Vice President, Cisco Services

Thursday, July 14th, 2016

11:30 am - 12:30 pm, In the Oceanside A room

What to expect from this innovation talk

• Insights on market trends and forecasts

• Preview of key technologies and capabilities

• Innovative demonstrations of the latest and greatest products

• Better understanding of how Cisco can help you succeed

Register to attend the session live now or

watch the broadcast on cisco.com


Thank you


Documents

© 2016 Cisco and/or its affiliates. All rights reserved ...d2zmdbbm9feqrf.cloudfront.net/2016/usa/pdf/BRKCOM-2007.pdf · © 2016 Cisco and/or its affiliates. All rights reserved