Traffic Monitoring and Management for UCSd2zmdbbm9feqrf.cloudfront.net/2012/usa/pdf/BRKCOM-2004.pdfTraffic Monitoring and Management for UCS ... ‒Devices tab ( server ports, network

© 2012 Cisco and/or its affiliates. All rights reserved. BRKCOM-2004 Cisco Public

Traffic Monitoring and Management for UCS Session ID-BRKCOM-2004

www.ciscolivevirtual.com

Steve McQuerry, CCIE # 6108, UCS – Technical Marketing @smcquerry


Agenda

UCS Networking Overview

Network Statistics in UCSM

Understanding Collection Policies

Hotspot Detection

Engineering to Avoid Hotspots

3



System Components: High-level Overview

• 6120 & 6140 • 6248UP

• 2104XP • 2204XP • 2208XP

• Cisco (M81KR; VIC1280) • 3rd party

FABRIC INTERCONNECT

CHASSIS IO MODULE (FEX)

INTERFACE CARDS

5



Top of Rack Controller (Fabric Interconnect)

‒(10GE ports) + (1 or 2 Slots for expandability)

Chassis ‒Up to 8 half width blades or 4 full width blades

Fabric Extender (FEX or I/O Module) ‒Host to uplink traffic engineering ‒Up to 80Gb Flexible bandwidth allocation (Gen 1)

Mezzanine Card Adapter ‒Virtualized adapter for single OS and hypervisor systems ‒Dual connected

Compute Blade

Compute Chassis

x86 Computer x86 Computer

X

I I x8 x8 x8 x8

B

MGMT

S S

B

X X X X X

C C

A

G G

G G

SAN

G

R

A

G

G G

G

R

G

P M P

SAN LAN

Fabric Interconnect

Fabric Interconnect

Fabric Extender

Fabric Extender

Compute Blade (Half slot)

Adapter

Compute Blade (Full slot)

Adapter Adapter

6


UCS Networking Overview 1st Generation Hardware

vpc

Mezz

Fabric A Fabric B

Mezz Mezz

FEX A FEX B

ni

c

ni

c

hb

a

hb

a

ni

c

ni

c

hb

a

hb

a

ni

c

ni

c

hb

a

hb

a

SAN A SAN B

Port-Channel Port-Channel

7


UCS 2104 (1st Gen FEX) — Server to Fabric Pinning

Fabric Interconnect 1 link

2 links

4 links

Server slots pinned to uplink

Uplink: slots 1,2,3,4,5,6,7,8

Uplink 1: slots 1,3,5,7 Uplink 2: slots 2,4,6,8

Uplink 1: slots 1,5 Uplink 2: slots 2,6 Uplink 3: slots 3,7 Uplink 4: slots 4,8

slot 1 slot 2 slot 3 slot 4 slot 5 slot 6 slot 7 slot 8



NIF

NIF

NIF

F E X

F E X

Fabric Interconnect

Fabric Interconnect

F E X

8


UCS Networking Overview 1st Generation Hardware

Fabric A Fabric B

Mezz

IOM A IOM B

ni

c

ni

c

hb

a

hb

a

9


UCS Networking Overview 2nd Generation Hardware

Mezz

Fabric A Fabric B

Mezz Mezz

IOM A IOM B

ni

c

ni

c

hb

a

hb

a

ni

c

ni

c

hb

a

hb

a

ni

c

ni

c

hb

a

hb

a

SAN A SAN B

vpc


10


UCS 2204 (2nd Gen FEX) — Sever to Fabric Pinning


2 links

4 links


Uplink: slots 1,2,3,4,5,6,7,8






NIF

NIF

NIF

F E X

F E X

Fabric Interconnect

Fabric Interconnect

F E X

11


UCS 2204 (2nd Gen FEX) — Sever to Fabric Pinning

4 links

Server slots channeled across all uplinks

Uplink 1: slots 1-8 Uplink 2: slots 1-8 Uplink 3: slots 1-8 Uplink 4: slots 1-8


NIF

F E X

Fabric Interconnect

12


UCS 2208 (2nd Gen FEX) — Server to Fabric Pinning


2 links

4 links


Uplink: slots 1,2,3,4,5,6,7,8






NIF

NIF

NIF

F E X

F E X

Fabric Interconnect

Fabric Interconnect

F E X

13


UCS 2208 (2nd Gen FEX) — Server to Fabric Pinning Server slots pinned to uplink

8 links Uplink 1: slot 1 Uplink 2: slot 2 Uplink 3: slot 3 Uplink 4: slot 4 Uplink 5: slot 5 Uplink 6: slot 6 Uplink 7: slot 7 Uplink 8: slot 8


NIF

F E X

Fabric Interconnect

8 links Uplink 1: slots 1-8 Uplink 2: slots 1-8 Uplink 3: slots 1-8 Uplink 4: slots 1-8 Uplink 5: slots 1-8 Uplink 6: slots 1-8 Uplink 7: slots 1-8 Uplink 8: slots 1-8


NIF

F E X

Fabric Interconnect

Server slots channeled across all uplinks

14


UCS Networking Overview 2nd Generation Fabric Interconnect

Fabric A Fabric B

Mezz

IOM A IOM B

ni

c

ni

c

hb

a

hb

a

15




Network statistics are collected by UCSM from the NX-OS software in the Fabric Interconnects.

These are counters that are available for networking components

Because of NIV technology the Fabric has visibility to the Cloud (LAN/SAN uplinks), the IOM and the server NIC.

17


Network Statistics in UCSM Access Statistics through:

‒ LAN or SAN tab (port-group)

‒ Devices tab ( server ports, network uplink, storage ports, and mezz ports,)

‒ Server Tab (vNIC)

18


Network Statistics in UCSM FI to LAN Network Uplink (Cloud)

Fabric A Fabric B

SAN A

vpc


19



Port Channel is the Aggregate of all interfaces in the channel

Statistic for the channel are the sum of the statistics of the members

Individual member statistics are also visible in the system

Network usage is measured against TX Total bytes and RX Total bytes

20


Network Statistics in UCSM FI to Uplink Port Channel Statistics

21


Network Statistics in UCSM FI to Uplink Individual Port Statistics

22


Network Statistics in UCSM FI to SAN Port Statistics

Fabric A Fabric B

SAN A SAN B

23



A FC Port Channel is the Aggregate of all interfaces in the channel

Statistic for the channel are the sum of the statistics of the members

Individual member statistics are also visible in the system

FC usage is measured against Bytes RX and Bytes TX

FI to SAN Uplink (Cloud) Port Statistics

24


Network Statistics in UCSM FI to SAN Port Statistics

25


Network Statistics in UCSM FI to IOM (Internal LAN)

Fabric A Fabric B

IOM A IOM B

26


Network Statistics in UCSM FI to IOM (Internal LAN)

27


UCS Networking Overview Server to IOM

Mezz

ni

c

ni

c

hb

a

hb

a

28


Network Statistics in UCSM Server to IOM vNIC Port Statistics

29




A collection Policy consist of a collection interval and a reporting interval

The collection policy is set under the admin tab in Stats Management -> Collection Policies -> Collection Policy name

A unique policy can be set for, Adapters, Chassis, FEX, Port, Server, and Host*

Not all policies involve network components

*Host is an unused policy in UCSM at present

31



32



The collection interval is how often the system will query a device for statistics.

The default collection interval is 60 seconds

The more frequent the interval the more granular the data. We will use 30 seconds.

The timing of the collection interval is important because it will be used in BW calculations for hotspot detection

Collection Interval

33



The reporting interval is internal to UCSM and determines how often UCSM will store data from the collection interval.

This data is stored in tables and the last 5 reporting intervals are available for inspection in the system

Reporting interval data is used to calculate minimum, maximum and average values shown in the statics view.

Reporting Interval

34



Select the Policy you want to change.

Make selections and press the save changes button

35

Hotspot Detection


Hotspot Areas Fabric Interconnects to LAN/SAN

FEX to Host

FEX to FI

Mezz

Fabric A Fabric B

Mezz Mezz

FEX A FEX B

ni

c

ni

c

hb

a

hb

a

ni

c

ni

c

hb

a

hb

a

ni

c

ni

c

hb

a

hb

a

SAN A SAN B


There are three potential hotspot locations for UCS network connectivity.

1.) Send and Receive between Fabric Interconnect and LAN/SAN

2.) Send and Receive between FEX and Fabric Interconnect

3.) Send and Receive between Host and FEX

37


Hotspot Detection

To identify hot spots we will use Threshold policies in conjunction with collection policies to alert as we pass thresholds.

A threshold is calculated by measuring a statistic against a policy.

The policy measures change against a user defined normal value and turns on the alert between a users set high/low threshold and turns off the alert below the user set low threshold.

Threshold Policies

High - Up

Low - Up

Normal

High - Up

Low - Up

Normal

High - Up

Low - Up

Normal

38


Hotspot Detection

The statistic we use to calculate bandwidth is the delta in bytes.

This should be measured for both TX and RX

This delta is calculated in bytes changed over a period of time defined by the collection interval, for example 30 seconds

Calculating BW Threshold Limits for an Element

39


Hotspot Detection

For Ethernet the BW of a single link is 10Gbps

First we determine our desired threshold for example 8Gbps

We need to calculate the expected change in bytes over the collection interval.

To calculate divide the desired BW by 8 bits per byte and then multiply by the collection interval by the time to get the expected delta in bytes for our collection period.

Example 8 Gbps over 30 seconds = 30,000,000,000 bytes

‒ 8Gbps / 8bits per byte = 1,000,000,000 bytes per second

‒ 1,000,000,000 bytes per second * 30 seconds = 30,000,000,000 bytes

Calculating BW Threshold Limits for an Element

40


Hotspot Detection Threshold Calculations

Speed in Gbps Percentage of BW Conversion to Bytes Delta expected over 30 second collection interval

10 100% 1,250,000,000 37,500,000,000

9 90% 1,125,000,000 33,750,000,000

8.5 85% 1,062,500,000 31,875,000,000

8 80% 1,000,000,000 30,000,000,000

7.5 75% 937,500,000 28,125,000,000

7 70% 875,000,000 26,250,000,000

6.5 65% 812,500,000 24,375,000,000

6 60% 750,000,000 22,500,000,000

5 50% 625,000,000 18,750,000,000

4 40% 500,000,000 15,000,000,000

3 30% 375,000,000 11,250,000,000

41


Hotspot Detection Threshold Alerts

42


Hotspot Detection

Threshold policies can be configured for the following: ‒ Internal LAN – IOM to FI

‒ LAN Cloud – FI to Upstream Ethernet switches

‒ SAN Cloud – FI to Upstream SAN switches

‒ Server – Between the server NIC and the IOM

Threshold Policies Placement

43


Hotspot Detection

Navigate to admin->stats management and expand the fabric.

Select thr-policy-default and create a threshold class

Choose Ether Tx Stats from the stat class and click next.

Threshold Policies Internal LAN

44


Hotspot Detection Threshold Policies Internal LAN

45



46


Hotspot Detection

Click the add button and select Ether Tx Stats Total Bytes Delta as the property type enter 0.0 as the normal value

Select the Alarm triggers you want to get and enter your values and click OK

Click finish to be returned to the policy.

Click the classes tab to see your policy.

Threshold Policies

47



48


Hotspot Detection

You will need to add another class for RX traffic.

Click the + bottom to the right and repeat the steps from the previous policy choosing Eter Rx Stats as the stats class this time

Click save changes once you have completed the steps

Threshold Policies

49


Hotspot Detection Threshold Policies

50


Hotspot Detection

For Uplinks you can repeat this process for the LAN cloud.

For SAN use a single Stats class fcstats and create a definition for rx and tx stas under the same stats class

Threshold Policies

51



52


Hotspot Detection

For the vNIC port you will need to create a threshold policy to be used with a service profile.

Go the the appropriate organization level and select create threshold policy

Threshold Policies

53



54


Hotspot Detection

Give the policy an applicable name and description and press next

Choose the vnic stats class and create a single threshold with the rx bytes delta and the tx bytes delta.

Threshold Policies

55



56


Hotspot Detection

Apply the threshold policy to the Service profile of the servers you want to monitor

Threshold Policies

57


Hotspot Detection

When a server reaches a threshold you will receive an alert on UCSM.

This will exist while threshold is exceeded once it drops below the definition the alert will disappear

Alerts show as system faults as defined by the threshold policy

Currently UCSM does not send traps for this fault

Threshold Policies

58



59

Engineering to Avoid Hotspots


QoS Architecture

Compute Chassis

x86 Computer x86 Computer

X

I I

x8 x8 x8 x8

B

MGMT

S S

B

X X X X X

C C

A

G G

G G

SAN

G

R

A

G

G G

G

R

G

P M P

SAN LAN

Fabric Switch

Fabric Switch

Fabric Extender

Fabric Extender

Compute Blade (Half slot)

Adapter

Compute Blade (Full slot)

Adapter Adapter

No packet drops within the array

Largest buffers are on switch and host memory, so congestion pushed to edges

Priority Flow Control (PFC) used to ensure packet drops are at vNIC or Switch

All traffic in a CA system belongs to 1 of 6 System Classes

Four are user configurable while the other two are for FCoE and standard Ethernet

QoS parameters can be configured at a per system class level, or a per vNIC level.

61


System Buffering/Queuing

62


User Configuration

Globally for each System Class

COS value for packets in this class

Drop/No-drop behavior

Strict Priority

Bandwidth/Weight

Users configure QoS parameters at two levels

Class Name FC Bronze

COSValue 3 0

Drop/No-Drop No-Drop Drop

Strict Priority No No

Bandwidth/Weight 20% 30%

Example

63


User Configuration

For each vNIC (Egress properties)

System Class for traffic from this vNIC

Rate limit (Mbps)

Burst Size (Kbytes)

Users configure QoS parameters at two levels

vNIC1 vNIC2 vNIC3

Class FC FC Bronze

Rate 4000 4000 5000

Burst 300 400 100

Example: Logical Server A

64


User Configuration – Example

Class Name FC Gold Ethernet BE

COS Value 3 1 0

Drop/No-Drop No-Drop Drop Drop

Strict Priority No No No

Bandwidth/Weight 1 (20%) 3 (60%) 1 (20%)

vNIC1 vNIC2 vNIC3

Class FC FC Eth. BE

Rate 4000 4000 5000

Burst 300 400 100

Logical Server A

Global System Class Definitions

vNIC1 vNIC2

Class Gold Eth. BE

Rate 600 4000

Burst 100 300

Logical Server B

FC Traffic High Priority Ethernet

Best Effort Ethernet

65


QoS Tools

Priority Flow Control

• Enables lossless Fabrics for each class of service • PAUSE sent per virtual lane when buffers limit exceeded

Transmit Queues Ethernet Link

Receive Buffers

Eight Virtual Lanes

One One Two Two

Three Three Four Four Five Five

Seven Seven Eight Eight

Six Six STOP PAUSE

COS based Bandwidth Management

• Enables Intelligent sharing of bandwidth between traffic classes control of bandwidth •802.1Qaz Enhanced Transmission

10 GE Link Realized Traffic Utilization

3G/s HPC Traffic 3G/s

2G/s

3G/s Storage Traffic 3G/s

3G/s

LAN Traffic 4G/s

5G/s 3G/s

t1 t2 t3

Offered Traffic

t1 t2 t3

3G/s 3G/s

3G/s 3G/s 3G/s

2G/s

3G/s 4G/s 6G/s

Among the tools used are aggregate shapers at the vNICs (VIC Adapter), ETS, Policers at the switch for each vNIC.

66


QoS Configuration in UCSM Enable QoS Classes in UCSM

67


QoS Configuration in UCSM Create a QoS Policy

68


Applying QoS to a Policy Apply Policy to Adapter

69


Traffic Engineering

FI-1

Class-A Class-B Class-C

vNIC-1

vNIC-2

vNIC-3

FEX-1

FI-2

Blade-2, VIC-1 vNIC-1

vNIC-2

vNIC-3

FEX-2

VIC with 3 vNICs

2 Fabric Extenders in chassis, each with 1 link to the FI.

2 FI, both with 1 connection to each FEX

Blade-1, VIC-1

vNICs can be pinned to a specific FI when created (with configurable failover to other switch)

Depending on requirements, vNICs could be pinned to one interconnect or distributed evenly

vNICs in System Class C pinned to one interconnect

70


Controlling Pinning in Profile From the each server a fabric interconnect can be chosen to balance the traffic

71


Traffic Engineering

FI-1

Class-A Class-B Class-C

vNIC-1

vNIC-2

vNIC-3

FEX-1

FI-2

Blade-2, VIC-1 vNIC-1

vNIC-2

vNIC-3

FEX-2

Blade-1, VIC-1

vNICs can be pinned to a specific FI when created (with configurable failover to other switch)

Depending on requirements, vNICs could be pinned to one interconnect or distributed evenly

vNICs in System Class C distributed across interconnects

72


Summary

UCSM is designed for optimized traffic flow

Stats Management and Threshold policies allow you to monitor traffic levels

QoS and Traffic engineering tools allow you to manage potential bottlenecks in UCS

73


Complete Your Online Session Evaluation Give us your feedback and you

could win fabulous prizes. Winners announced daily.

Receive 20 Passport points for each session evaluation you complete.

Complete your session evaluation online now (open a browser through our wireless network to access our portal) or visit one of the Internet stations throughout the Convention Center.

Don’t forget to activate your Cisco Live Virtual account for access to all session material, communities, and on-demand and live activities throughout the year. Activate your account at the Cisco booth in the World of Solutions or visit www.ciscolive.com.

74


Final Thoughts

Get hands-on experience with the Walk-in Labs located in World of Solutions, booth 1042

Come see demos of many key solutions and products in the main Cisco booth 2924

Visit www.ciscoLive365.com after the event for updated PDFs, on-demand session videos, networking, and more!

Follow Cisco Live! using social media: ‒ Facebook: https://www.facebook.com/ciscoliveus

‒ Twitter: https://twitter.com/#!/CiscoLive

‒ LinkedIn Group: http://linkd.in/CiscoLI

75


Documents

Traffic Monitoring and Management for UCSd2zmdbbm9feqrf.cloudfront.net/2012/usa/pdf/BRKCOM-2004.pdfTraffic Monitoring and Management for UCS ... ‒Devices tab ( server ports, network