View
221
Download
2
Category
Preview:
Citation preview
Network Visibility using Advanced Analytics in Nexus Switches
Oliver Ziltener - Technical Marketing Engineer
BRKDCT-1890
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Session Abstract
• Session ID : BRKDCT – 1890
• Title : Network visibility using advanced Analytics in Nexus switches
• Abstract:• Learn how to get the most visibility from your Nexus-based network with new monitoring
capabilities and advanced enhancements to traditional features like SPAN, ERSPAN and NetFlow. We will delve deeper into platform specific features like ERSPAN, Virtual SPAN to leverage multi destination SPAN, SPAN filters, In-Band SPAN, Extended SPAN/ERSPAN, Rule based SPAN, SPAN with MTU truncation, SPAN rate limiting, Exception SPAN on Nexus 7000 & Microburst monitoring, latency monitoring, line rate SPAN, SPAN on drop, SPAN on latency, buffer usage histogram etc. in Nexus 5000/6000. NetFlow and its unique aspects on Nexus switches will be discussed as well. These features help you understand the network and the applications running on the network better, and quickly pinpoint trouble spots in the network. We will go over what each feature is capable of, what proper real world use cases are, proper configurations, how to interpret the outputs and use the data collected. This session will focus on analytics and monitoring. It will not focus on other management aspects like SNMP, Syslog, RMON etc.
BRKDCT-1890 3
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Session Goal
• Create awareness of the Analytics and Monitoring tools available in the Nexus family (N3k, N5K, N6K, N7K and N9K) in NX-OS standalone mode
• Provide the ability to choose the right tool to analyze, which helps in timely resolution of the problem
• It will NOT focus on other management aspects like SNMP, Syslog, RMON, troubleshooting, QOS, architecture and packet flows
Reference Slide
BRKDCT-1890 4
• Introduction
• Quick Product Overview
• Advanced Visibility
• SPAN / ERSPAN
• Flexible NetFlow
• Conclusion
Agenda
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Network Bandwidth Explosion
M2M
Trillions of new
“connected
events” will
occur over IP
networks
throughout the
next decade
Cloud
Global cloud
traffic will grow
6X by 2016
4G Mobile Adoption
4G will account
for 45% of
global mobile
data traffic
Global IP traffic
will grow 3X to
1.4 zettabytes
annually by
2017
IP Traffic
By 2017, the
world will reach
3 trillion Internet
video minutes
per month
Video
BRKDCT-1890 6
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
If not handled well....
• Degrading performance
• Difficulty to troubleshoot
• Improper planning of resources
BRKDCT-1890 7
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
What is Analytics?
The systematic
computational analysis
of data or statistics
statistics.
Discovery and
communication of
meaningful patterns in
data
Studying past historical
data to research potential
trends
BRKDCT-1890 8
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Advanced Analytics on Nexus Switches
• Collection of various features and enhancements to the traditional monitoring tools
• Latency Monitoring, Buffer Monitoring, SPAN-on-drop, Exception SPAN, SPAN filters, Microburst Monitoring and a LOT MORE!
• Advantages: Microbursts, Congestion, find malicious source, filter SPAN packets etc...
BRKDCT-1890 9
• Introduction
• Quick Product Overview
• Advanced Visibility
• SPAN / ERSPAN
• Flexible NetFlow
• Conclusion
Agenda
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Nexus Switches Family
Nexus 5000
Nexus 7000
Nexus 3000
Nexus 2000
Nexus 3100
Nexus1000V
Nexus 9000Nexus 5600/6000
Nexus 2300Nexus 7700
Nexus 3500
BRKDCT-1890 11
• Introduction
• Quick Product Overview
• Advanced Visibility
• SPAN / ERSPAN
• Flexible NetFlow
• Conclusion
Agenda
Latency Monitoring
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
• Many applications can get impacted because of high latency
• Website download
• Video streaming
• Video conferencing
• Online gaming
• Banking
• Airline reservation
• Stock Market
• Web hosting
Why do we need to correct latency problems?
BRKDCT-1890 14
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
How does Latency Monitoring work?
Packet
Packet
INGRESS TIMESTAMPING
EGRESS TIMESTAMPING
Latency Monitoring Feature measure: T2 – T1 in ns
Packet Time T1
Packet Time T2
BRKDCT-1890 15
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
• Latency Monitoring provides {min, average, max} latency between a specified port pair and also maintains latency histogram (accuracy in few nanoseconds)
• By default instantaneous Latency Monitoring is enabled between pair of ports
• Latency Histogram can be enabled for specific port-pair to provide histogram instead of instantaneous mode
• Measures switch latency for each packet, no sampling required
• Fully implemented in HW, no CPU impact, no traffic impact
How does Latency Monitoring work?
BRKDCT-1890 16
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Modes of Latency monitoring
• Instantaneous - Enabled by default on all pairs of ports
• No configuration required
• The latency measured is after the packet enters Port ASIC (Bigsur)
NEXUS# show hardware profile latency monitor interface e1/7 interface ethernet 1/14
--------------------------------------------------------------------------------
Egress Port: Ethernet1/7 Ingress Port: Ethernet1/14 Mode: Inst
--------------------------------------------------------------------------------
| | Minimum | Maximum | Average |
--------------------------------------------------------------------------------
| cnt | 912| 936| 923|
--------------------------------------------------------------------------------
Egress Interface Ingress Interface
cnt denotes the latency of packets entering e1/14 and egressing e1/7
BRKDCT-1890 17
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Modes of Latency Monitoring
• Custom histogram – Counts packets in defined range. Needs below configurations
NEXUS# show hardware profile latency monitor interface e1/3 interface e1/1
--------------------------------------------------------------------------------
Egress Port: Ethernet1/3 Ingress Port: Ethernet1/1 Mode: Custom Histogram
--------------------------------------------------------------------------------
| Range| 800 <= Latency < 10000| 800 > Latency >= 10000|
--------------------------------------------------------------------------------
| cnt | 3542903| 56792|
--------------------------------------------------------------------------------
NEXUS(config)# interface e1/3
NEXUS(config-if)# packet latency interface e1/1 mode custom low-latency 800 high-latency 10000
Ingress Interface
cnt denotes the number of packet in the specific range
Egress Interface Time in nano seconds
BRKDCT-1890 18
Microburst monitoring
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
• Spike of high activity
• Passes under the radar of traditional load-monitoring tools
• Traffic spike that causes that system to saturate
• How short and how high? – Capacity of worst system in N/W
Microburst – A Concern
BRKDCT-1890 20
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Microburst in Reality
• UW-Madison & Microsoft Research Paper: “Understanding Data Center Traffic Characteristics” http://research.microsoft.com/pubs/136788/wren09.pdf
• Results: “we find only a small fraction of losses do not belong to any microburst. This indicates that, more often that not, when losses happen at the edge or aggregation links, they happen in bursts.”
BRKDCT-1890 21
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Challenge: It’s Very Hard to see Microbursts
NEXUS# show interface ethernet 1/2
Ethernet1/2 is up
[…]
Last clearing of "show interface" counters 00:00:58
0 interface resets
30 seconds input rate 96315720 bits/sec, 1331 packets/sec
30 seconds output rate 0 bits/sec, 0 packets/sec
Load-Interval #2: 5 minute (300 seconds)
input rate 77.00 Mbps, 1.05 Kpps; output rate 0 bps, 0 pps
RX
200000 unicast packets 0 multicast packets 0 broadcast packets
200000 input packets 1800000000 bytes
200000 jumbo packets 0 storm suppression bytes
0 runts 0 giants 0 CRC 0 no buffer
[…]
BRKDCT-1890 22
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Solution: Burst Monitoring
• Configure your own burst filter per port per direction
• This command essentially enables micro burst detection on a port
• This command defines the maximum number of bursts that should happen over a time window before firing an syslog
burst threshold {ingress | egress} {limit percent | size max_bytes} interval
interval_time
[no] burst maximum {ingress | egress} burst-count max-burst
BRKDCT-1890 23
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Feature guideline
• Supported on physical ports, port-channel members, and FEX fabric ports
• Not supported on sub interfaces, FEX HIF ports and port-channels
BRKDCT-1890 24
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Burst Monitoring CLI
• To monitor bursts
• Example:
• To clear counters:
clear burst-counters [interface {all | ethernet interface}] {both | egress |
ingress }
show interface [ethernet slot/port]] burst-counters
NEXUS# show interface e1/14 burst-counters
--------------------------------------------------------------------
| Interface | Ingress Bursts | Egress Bursts | Total Bursts |
--------------------------------------------------------------------
| Ethernet1/14| 2| 0| 2|
BRKDCT-1890 25
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Real World Example
• Troubleshooting Methodology: Detect micro bursty traffic
• Enable Micro burst detection to provide syslog notification
interface Ethernet1/13
burst threshold ingress size 10000 interval 100
burst maximum ingress burst-count 100
burst threshold egress size 10000 interval 100
burst maximum egress burst-count 100
!
2016 Feb 8 12:10:05 NEXUS %$ VDC-1 %$ %USER-2-SYSTEM_MSG: Micro
Burst has been detected on ingress side on Ethernet1/13 - bigsurusd
Time in micro seconds
Define how many bursts to
be detected, before send
syslog
Both commands are recommend per direction
limit: Threshold size as percentage of link speed
size: Threshold size in bytes
BRKDCT-1890 26
Buffer monitoring
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Why do we need to monitor buffers?
• Is my network congested?
• Can I add a new server?
• Will the performance be impacted?
• Why are the drops happening?
BRKDCT-1890 28
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
What is Buffer monitoring on Nexus?
• Buffer utilization is on a per port basis
• Buffer utilization shows buffer for unicast traffic in ingress and unicast and multicast in egress directions
• Histogram mode – slow (1sec) or fast (250ms) sampling
BRKDCT-1890 29
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
• Supported on physical ports, port-channel members, and FEX fabric ports
• Not supported on sub interfaces, FEX HIF ports and port-channels
Feature Guideline
BRKDCT-1890 30
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Configuration
• Buffer utilization must be enabled on interface
• Fast sampling must be enabled in global configuration mode
• Default sampling is slow = 1 second
NEXUS(config)# inter e1/10
NEXUS(config-if)# hardware profile buffer monitor
NEXUS(config-if)#
NEXUS(config)# hardware profile buffer monitor sampling fast
NEXUS(config)#
BRKDCT-1890 31
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Configuration
• To see buffer utilization and/or the buffer utilization histogram*, the next command must be executed
• To clear buffer utilization history use
show hardware profile buffer monitor { interface <ifid> | all } history {
brief | detail }
clear hardware profile buffer monitor [ interface <ifid> ]
*History up to 1 hour
BRKDCT-1890 32
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Output of Buffer Monitoring tool
NEXUS# show hardware profile buffer monitor interface ethernet 1/21 history brief
--------------------------------------------------------------------------------
Interface : Eth1/21
--------------------------------------------------------------------------------
--------------------------------------------------------------------------------
Sampling Mode : Slow (1 second)
--------------------------------------------------------------------------------
--------------------------------------------------------------------------------
Ingress Buffer Utilization Detected(in KB)
Per asic Ingress Total Usage (15.628800MB)
--------------------------------------------------------------------------------
1 sec | 5 sec | 1 min | 5 min | 1 hour |
--------------------------------------------------------------------------------
0.6| 0.6| 0.6| 0.6| 0.6|
--------------------------------------------------------------------------------
Egress Buffer Utilization Detected(Unicast|Multicast)(in KB)
Per asic Egress Total Usage (8.611850MB)
--------------------------------------------------------------------------------
1 sec | 5 sec | 1 min | 5 min | 1 hour |
--------------------------------------------------------------------------------
112.6| 0.0| 177.2| 0.0| 158.0| 0.0| 164.1| 0.0| 164.3| 0.0|
--------------------------------------------------------------------------------
BRKDCT-1890 33
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Real World Example
• Slow Application Response – Port Oversubscription
• Interface and queueing statistics verification ingress discards due oversubscription egress port
• Check buffer utilization
• Determine the egress port that is congested using virtual output queue (VoQ) statistics
e1/25
10G
e1/5
10G
e1/4
10G
Destination
BRKDCT-1890 34
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Real World Example
• We spot input discards on interfaces
Slow Application Response – Port Oversubscription
NEXUS#show inter e1/5
Ethernet1/5 is up
---snip---
RX
112068891 unicast packets 0 multicast packets 0 broadcast packets
112068891 input packets 14344818048 bytes
0 jumbo packets 0 storm suppression bytes
0 runts 0 giants 0 CRC 0 no buffer
0 input error 0 short frame 0 overrun 0 underrun 0 ignored
0 watchdog 0 bad etype drop 0 bad proto drop 0 if down drop
0 input with dribble 57491175 input discard
0 Rx pause...
NEXUS#show inter e1/25
Ethernet1/25 is up
---snip---
RX
176069516 unicast packets 0 multicast packets 0 broadcast packets
176069516 input packets 22536898048 bytes
0 jumbo packets 0 storm suppression bytes
0 runts 0 giants 0 CRC 0 no buffer
0 input error 0 short frame 0 overrun 0 underrun 0 ignored
0 watchdog 0 bad etype drop 0 bad proto drop 0 if down drop
0 input with dribble 1457036 input discard
0 Rx pause...
BRKDCT-1890 35
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Real World Example
• We spot ingress discards on interface (RX queuing)
Slow Application Response – Port Oversubscription
NEXUS#show queuing interface e1/5
Ethernet1/5 queuing information:
TX Queuing
qos-group sched-type oper-bandwidth
0 WRR 100
RX Queuing
qos-group 0
q-size: 100160, HW MTU: 1500 (1500 configured)
drop-type: drop, xon: 0, xoff: 0
Statistics:
Pkts received over the port : 112068891
Ucast pkts sent to the cross-bar : 54577716
Mcast pkts sent to the cross-bar : 0
Ucast pkts received from the cross-bar : 0
Pkts sent to the port : 0
Pkts discarded on ingress : 57491175
Per-priority-pause status : Rx
(Inactive),Tx(Inactive)
Only default queue is used
(default) here
BRKDCT-1890 36
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Real World Example
• Egress interface is ok
Slow Application Response – Port Oversubscription
NEXUS#show inter e1/4
Ethernet1/4 is up
---snip---
TX
228498012 unicast packets 262 multicast packets 3 broadcast packets
228498277 output packets 30161765824 bytes
0 jumbo packets
0 output error 0 collision 0 deferred 0 late collision
0 lost carrier 0 no carrier 0 babble 0 output discard
0 Tx pause
BRKDCT-1890 37
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Real World Example
• What is about buffer utilization?
Slow Application Response – Port Oversubscription
NEXUS# show hardware profile buffer monitor interface e1/5
+---------------------------------------------------------------------------+
| Instant Ingress Buffer utilization per class per port. Every line |
| displays the number of cells utilized for a given port for each class |
| One cell represents 320 bytes |
+---------------------------------------------------------------------------+
-----------------------------------------------------------------------------
Interface : Eth1/5
-----------------------------------------------------------------------------
Total Port Instant Usage 17744 (5.678080MB)
Remaining Asic Instant Usage 31096 (9.950720MB)
Per asic ingress cell count 48840 (15.628800MB)
+----------+-------+-------+--------+-------+-------+-------+-------+-------+
port| class0| class1| class2| class3| class4| class5| class6| class7|
+----------+-------+-------+--------+-------+-------+-------+-------+-------+
Eth1/5| 0| 0| 0| 17744| 0| 0| 0| 0|
+----------+-------+-------+--------+-------+-------+-------+-------+-------+
---snip---
Note:
Class0 is control
traffic
Class1 is internetwork
control traffic
Class2 is FCoE traffic
Class3 is QoS group
0 (default queue)
Class4-7 are QoS
group 2-5 sequentially
Real-time buffer/cell allocation
of buffer/cell at ingressBRKDCT-1890 38
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Real World Example
• To get additional information about the oversubscribed port, we will need to look at the virtual output queue (VoQ) statistics for the ingress ASIC group
Slow Application Response – Port Oversubscription
NEXUS# show platform fwm info pif e1/5 | grep global_asic_num
Eth1/5 pd: slot 0 logical port num 4 slot_asic_num 1 global_asic_num 1 fw_inst 4 phy_fw_inst 1 fc 0
NEXUS# show platform fwm info pif e1/25 | grep global_asic_num
Eth1/25 pd: slot 0 logical port num 24 slot_asic_num 3 global_asic_num 3 fw_inst 0 phy_fw_inst 0 fc 0
ASIC group is the
«global_asic_num»
BRKDCT-1890 39
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Real World ExampleSlow Application Response – Port Oversubscription
NEXUS# show platform software qd info counters voq asic-num 1
+----------+------------------------------+------------------------+-----------+
| port| TRANSMIT| TAIL DROP| HEAD DROP|
+----------+------------------------------+------------------------+-----------+
Eth1/4
QUEUE-3 54577716 57491175 0
---snip--
+----------+------------------------------+------------------------+-----------+
NEXUS# show platform software qd info counters voq asic-num 3
+----------+------------------------------+------------------------+-----------+
| port| TRANSMIT| TAIL DROP| HEAD DROP|
+----------+------------------------------+------------------------+-----------+
Eth1/4
QUEUE-3 173917190 1457036 0
---snip---
+----------+------------------------------+------------------------+-----------+
• VoQ statistics indicates that QUEUE-3 of Eth1/4 is oversubscribed (tail drops)
QoS Group 0
Egress Interface
Note: Internal queue
numbers are mapped
as follows:
QUEUE-0 is control
traffic
QUEUE-1 is
internetwork control
traffic
QUEUE-2 is FCoE
traffic
QUEUE-3 is QoS
group 0 (default
queue)
QUEUE 4-7 are QoS
group 2-5 sequentially
BRKDCT-1890 40
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Real World Example
• The same drops on the egress interface using the following command:
• The output indicates that ASIC1 and ASIC3 are dropping traffic destined to Eth1/4 as seen in the initial VoQ output
Slow Application Response – Port Oversubscription
NEXUS# show platform software qd info counters voq interface e1/4
+----------+------------------------------+------------------------+-----------+
|slot asic| TRANSMIT| TAIL DROP| HEAD DROP|
+----------+------------------------------+------------------------+-----------+
---snip--
0 1
QUEUE-3 54577716 57491175 0
0 3
QUEUE-3 173917190 1457036 0
+----------+------------------------------+------------------------+-----------+
BRKDCT-1890 41
• Introduction
• Quick Product Overview
• Advanced Visibility
• SPAN / ERSPAN
• Flexible NetFlow
• Conclusion
Agenda
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Switch Port Analyzer (SPAN)
• A SPAN session is an association of source ports/vlans to one or more destination ports
• Once the traffic is identified for replication, switch copies the matching traffic to the SPAN destination port(s)
• The SPAN (copied) packets are created in hardware without overloading the CPU
SPAN Source
Host B
SPAN Destination
Sniffer Device
e1/1 e5/1
e2/1
SPAN all the packets
ingressing e1/1
Spanned (copied)
traffic
Host A
BRKDCT-1890 43
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
SPAN Sources
• Switchports
• Access ports
• Trunk ports
• Private VLAN ports
• Port-channels
• Routed interfaces
• Physical interfaces
• Port-channels
• VLANs and PVLANs
• Supervisor inband interface
• Up to 128 physical interfaces and/or up to 32 VLANs per session
• Mix of interface types allowed in single session
• For example, SPAN source of VLAN 10 and interface e1/1 in same session
• Individual subinterfaces cannot be SPAN source
BRKDCT-1890 44
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Layer 3 Network
Encapsulated Remote SPAN (ERSPAN)
• ERSPAN supports source and destinations on different switches*
• It uses a GRE tunnel to carry traffic
• Packets replicated in hardware
ERSPAN Source
Sniffer Device
Packets are replicated and
GRE encapsulated at
ERSPAN source device
ERSPAN
Destination
At ERSPAN Destination
device, GRE packet is
decapsulated
*Not all HW supports ERSPAN destination, e.g. N9272 BRKDCT-1890 45
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
ERSPAN with IEEE1588 timestamp – Find Network Latency
ID
N5K-C56-72UP
STAT
2
5 6 7 8
1 3 4 10
13 14 15 16
9 11 12 18
21 22 23 24
17 19 201 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 4825 26 27 28 29 30 31 32
ID
N5K-C56-72UP
STAT
2
5 6 7 8
1 3 4 10
13 14 15 16
9 11 12 18
21 22 23 24
17 19 201 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 4825 26 27 28 29 30 31 32
PTP messages
Data
Switch A
Switch N
ERSPAN type III
Sniffer
Latency from Switch A
to Switch N = T2 – T1
GPS
PTP grandmaster
BRKDCT-1890 46
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
ERSPAN Type III – Packet Capture Example
• Timestamp information in the Type III header to be used to calculate the packet latency
monitor session 1 type erspan-source
header-type 3
erspan-id 1
vrf default
destination ip 104.104.104.21
source interface ethernet1/1 both
rate-limit auto
no shut
!
monitor erspan granularity 1588
ERSPAN Type III use a new GRE
Protocol Type 0x22EB
--------
ERSPAN II use 0x88BE
Direction (0xxx) and Granularity x10x = 1588*
Time Stamps
*This wireshark version does not decode properly BRKDCT-1890 47
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
ERSPAN Type III – Configuration Example (N7700)
hostname Node1
interface loopback0
ip address 1.1.1.1/32
!
monitor session 2 type erspan-source
header-type 3
erspan-id 1
vrf default
destination ip 3.3.3.3
source interface Ethernet1/3 both
rate-limit auto
no shut
!
! Admin VDC
monitor erspan origin ip-address 1.1.1.1 global
monitor erspan granularity 1588
hostname Node3
interface loopback0
ip address 3.3.3.3/32
!
monitor session 2 type erspan-destination
erspan-id 1
vrf default
source ip 3.3.3.3
destination interface Ethernet1/19
no shut
e1/3 e1/19
*only the relevant configuration is shown
Layer 3 Network
ERSPAN
BRKDCT-1890 48
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Nexus 2000 (FEX) with Nexus Parent Switch
• Parent Switch support: please check latest release notes
++ x F
EX
Parent Switch Nexus 2000 Fabric Extenders
BRKDCT-1890 49
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
SPAN with FEX
• FEX Host ports can be SPAN source ports*
• FEX Fabric ports can be SPAN source ports with all parent switches
• FEX Host ports can be SPAN destination port with Nexus 5K**
Nexus Fabric Extender
*Except Nexus 7000 parent switch with F2/F2e Line card
Sniffer Device
Host A
Nexus Parent Switch
**Supported since NX-OS 7.2
BRKDCT-1890 50
Nexus 5600/6000 SPAN Features
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Use Case - Packet Drops
• What packets are dropped?e1/5
10G
NEXUS#show inter e1/5
Ethernet1/5 is up
---snip---
RX
112068891 unicast packets 0 multicast packets 0 broadcast packets
112068891 input packets 14344818048 bytes
0 jumbo packets 0 storm suppression bytes
0 runts 0 giants 0 CRC 0 no buffer
0 input error 0 short frame 0 overrun 0 underrun 0 ignored
0 watchdog 0 bad etype drop 0 bad proto drop 0 if down drop
0 input with dribble 57491175 input discard
0 Rx pause...
BRKDCT-1890 52
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
SPAN-on-Drop
• SPAN-on-Drop allows SPAN’ning of the packets which were dropped due to unavailable buffer on ingress
Dedicated
SPAN Buffer
Ingress
Data Buffer
SP
AN
-on
-Dro
pTail-
Dro
p
Port 3 is
congested
Sniffer Device
N5600/N6000
BRKDCT-1890 53
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
SPAN-on-Drop Information
• Works for unicast packets only
• Supports both local SPAN and ERSPAN
• One SPAN-on-Drop session is supported
• Can have multiple source ports, and multiple destination ports
• Source port(s) can be a part of a SPAN-on-Drop session, and a local SPAN session simultaneously
• Note: SPAN-on-Drop is supported on N9K since 7.0(3)I4(1) on 2nd Generation N9K (N9200-X and N9200-Q/C). Hardware support N9300-EX, as Standalone NX-OS is not shipped yet.
BRKDCT-1890 54
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
SPAN-on-Drop Configuration
• The source interface is the ingress port for which we want to monitor drops
NEXUS(config)# monitor session 1 type span-on-drop
NEXUS(config-span-on-drop)# source interface e1/1 rx
NEXUS(config-span-on-drop)# source interface e1/2 rx
NEXUS(config-span-on-drop)# destination interface e1/4
NEXUS(config)# monitor session 2 type span-on-drop-erspan
NEXUS(config-span-on-drop-erspan)# source interface e1/1 rx
NEXUS(config-span-on-drop-erpsan)# source interface e1/2 rx
NEXUS(config-span-on-drop-erspan)# destination ip 100.1.1.2
Always Rx :
Ingress interface
– Packets
dropped at
ingress
BRKDCT-1890 55
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
SPAN-on-Drop Guidelines
• The source interfaces can only be Ethernet. They can be port-channel members, but port-channel as source is not supported
• Fabric extender (HIF) interfaces are not supported as sources; however, fabric (NIF) interfaces are supported. Setting a fabric interface as a source allows SPAN-on-Drop to be enabled on all fabric extender ports associated with that fabric interface.
• One SPAN-on-drop or SPAN-on-drop ERSPAN session can be active at a time
BRKDCT-1890 56
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Use Case – Identify delayed flows
• Is a packet delayed?
e1/14
10G
e1/7
10G
BRKDCT-1890 57
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
SPAN-on-Latency
Sniffer Device
Data
Timestamp
Data
Port 3 is congested
Latency monitoring
If Latency Threshold
> 10 usec:
SPAN to 4
N5600/N6000
BRKDCT-1890 58
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
SPAN-on-Latency Information
• Replicated traffic uses the SPAN buffer so it doesn't impact the production traffic
• Supports both local SPAN and ERSPAN
• Latency threshold is per-port
• One SPAN-on-Latency session is supported in hardware
BRKDCT-1890 59
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
SPAN-on-Latency Configuration
• SPAN-on-Latency session makes a copy of all high-latency packets egressing on this port, coming from any ingress port
NEXUS(config)# monitor session 1 type span-on-latency
NEXUS(config-span-on-latency)# source interface Ethernet1/7 tx
NEXUS(config-span-on-latency)# destination interface Ethernet1/23
interface Ethernet1/7
packet latency threshold 10000
interface Ethernet1/23
switchport monitor
Always Tx:
packets
egressing on 1/7
(any source) with
latency >10us
will be replicated
to the SPAN
destination 1/23
BRKDCT-1890 60
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
SPAN-on-Latency Guideslines
• Support for one SPAN-on-latency session
• Multiple sources can be configured – latency threshold is per SPAN-on-drop TX source port
• A SPAN-on-Latency source port cannot be in another SPAN session
BRKDCT-1890 61
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
SPAN-on-Latency Guideslines
• Source port can be an regular Ethernet port, not a port-channel. Can be a port-channel member
• Source port cannot be FEX HIF port. But FEX fabric port is supported
• Destination is only a single Ethernet port, not port-channel
BRKDCT-1890 62
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Real World Example
• Troubleshooting Methodology
• Verification of interface errors and Switch CPU
• Maybe congestion?
• Use Analytics Latency monitoring & Span-on-Latency
Slow Download Rate
e1/14
10Ge1/7
10G
BRKDCT-1890 63
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Real World Example
• Instantaneous Latency Monitoring (no configuration required)
Slow Download Rate
NEXUS# show hardware profile latency monitor interface e1/7 interface ethernet 1/14
--------------------------------------------------------------------------------
Egress Port: Ethernet1/7 Ingress Port: Ethernet1/14 Mode: Inst
--------------------------------------------------------------------------------
| | Minimum | Maximum | Average |
--------------------------------------------------------------------------------
| cnt | 912| 936| 923|
--------------------------------------------------------------------------------
NEXUS# show hardware profile latency monitor interface e1/7 interface ethernet 1/14
--------------------------------------------------------------------------------
Egress Port: Ethernet1/7 Ingress Port: Ethernet1/14 Mode: Inst
--------------------------------------------------------------------------------
| | Minimum | Maximum | Average |
--------------------------------------------------------------------------------
| cnt | 904| 7526784| 4047543|
--------------------------------------------------------------------------------
When no
congestion
on e1/7
When heavy
congestion
on e1/7
BRKDCT-1890 64
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Real World Example
• Optional: Configure Latency Monitoring
Slow Download Rate
NEXUS(config)#interface Ethernet1/7
NEXUS(config-if)# packet latency int e1/14 mode custom low-latency 800 high-latency 10000
NEXUS#show hardware profile latency monitor interface e1/7 interface e1/14
--------------------------------------------------------------------------------
Egress Port: Ethernet1/7 Ingress Port: Ethernet1/14 Mode: Custom Histogram
--------------------------------------------------------------------------------
| Range| 800 <= Latency < 10000| 800 > Latency >= 10000|
--------------------------------------------------------------------------------
| cnt | 203029| 8193520|
--------------------------------------------------------------------------------
Ingress
Interface
Egress
Interface
Out of the required
latency > 10us
BRKDCT-1890 65
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Real World Example
• Find which application is impacted - SPAN-on-Latency
Slow Download Rate
NEXUS(config)# monitor session 1 type span-on-latency
NEXUS(config-span-on-latency)# source interface Ethernet1/7 tx
NEXUS(config-span-on-latency)# destination interface Ethernet1/23
interface Ethernet1/7
packet latency threshold 10000
interface Ethernet1/23
switchport monitor
Always Tx:
packets
egressing on 1/7
(any source) with
latency >10us
will be replicated
to the SPAN
destination 1/23
BRKDCT-1890 66
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
SPAN with ACL filter
• Selectively monitor traffic in a SPAN session using Access-Control-List (ACL) to avoid destination sniffer overload
• SPAN session ignores any permit/deny actions specified in the ACL
• SPANs packets that match (permit) the ACL filter criteria
NEXUS(config)# ip access-list ACL-IP-01
NEXUS(config-acl)# 10 permit ip host 192.168.111.11 host 192.168.112.12
NEXUS(config-acl)# end
NEXUS(config)# monitor session 1
NEXUS(config-monitor)# source interface ethernet 1/3
NEXUS(config-monitor)# destination interface ethernet 1/9
NEXUS(config-monitor)# filter access-group ACL-IP-01
NEXUS(config-monitor)# no shut
*Supported with SPAN local and ERSPAN BRKDCT-1890 67
Nexus 7000/9000 SPAN Features
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
SPAN VLAN Filters
• VLAN filters allow monitoring subset of VLANs on trunk ports
• Filter specifies list of VLANs to capture
• Traffic for other VLANs not sent to SPAN destination
NEXUS(config)# monitor session 1
NEXUS(config-monitor)# source interface e1/17 both
NEXUS(config-monitor)# destination interface e1/32
NEXUS(config-monitor)# filter vlan 77,88
BRKDCT-1890 69
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
SPAN ACL Filtering Nexus 9000
• Configuration Example
NEXUS(config)# ip access-list match_my_pkts
NEXUS(config-acl)# permit ip 11.0.0.0 0.255.255.255 any
NEXUS(config)# vlan access-map span_filter 5
NEXUS(config-access-map)# match ip address match_my_pkts
NEXUS(config-access-map)# action forward
NEXUS(config)# monitor session 1
NEXUS(config-monitor)# filter access-group span_filter
BRKDCT-1890 70
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Multi-Destination Virtual SPAN
• Use Case:«Breakout» high speed source
• To monitor multiple VLAN sources and choose only VLANs of interest to transmit on multiple destination ports
SPAN Destination802.1Q, Allowed VLAN 10
e2/1
802.1Q Trunk
VLANs 10-20
SPAN Destination802.1Q, Allowed VLAN 11
SPAN Destination802.1Q, Allowed VLAN 12
SPAN Destination802.1Q, Allowed VLAN 13
e1/1
e1/2
e1/3
e1/4
monitor session 1
source interface e2/1 both
destination interface e1/1
destination interface e1/2
destination interface e1/3
destination interface e1/4
filter vlan 10-13
High-speed
Interface
Multiple SPAN destination configured
as trunk and allow vlan list
BRKDCT-1890 71
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
SPAN with ACL Capture
• Selectively monitor traffic on an interface or VLAN
• Packets that match ACL rule are permitted or denied and/or sent to an monitor destination
NEXUS(config)# monitor session 1 type acl-capture
NEXUS(config-acl-capture)#destination interface Ethernet1/32
NEXUS(config-acl-capture)#no shut
NEXUS# show monitor session 1
session 1
---------------
type : acl-capture
state : up
destination ports : Eth1/32
BRKDCT-1890 72
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
SPAN with ACL Capture
• Enable a capture session for an ACL's access control entries (ACEs) and then apply the ACL to an interface or VLAN filter-list (VACL)
• Capture session ID matches with the monitor session ID
• An example with the capture option applied to a VLAN-List
• Note: The ACL rule with the capture option can be also applied to an interfaces
ip access-list MY-ACL
10 permit udp any any capture session 1
vlan access-map MY-VACL 10
match ip address MY-ACL
action forward
vlan filter MY-VACL vlan-list 77
BRKDCT-1890 73
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
SPAN with ACL Capture
• The ACL Capture filter feature requires to enable hardware access-list command in the admin VDC or default VDC*
hardware access-list capture
*ACL capture is not supported with ACL logging
BRKDCT-1890 74
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Packet Injection
• Allows device connected to SPAN destination interface to inject traffic into the network
• Specify input packets option when configuring SPAN destination:
e1/3
switchport monitor ingress
SMAC: 0000.0000.2222
e1/4
switchport monitor ingress learning
SMAC: 0000.0000.3333
e1/3 –
e1/4 0000.0000.3333
interface Learned MAC
MAC Table
e1/2
SMAC: 0000.0000.1111
switchport monitor
X
NEXUS(config)# interface ethernet 1/2
NEXUS(config-if)# switchport monitor
NEXUS(config-if)# interface ethernet 1/3
NEXUS(config-if)# switchport monitor ingress
NEXUS(config-if)# interface ethernet 1/4
NEXUS(config-if)# switchport monitor ingress learning
Allow inject packets, but do not learn the MAC
Allow inject packets and learn MAC
Normal SPAN session
BRKDCT-1890 75
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Inband SPAN – Monitor control traffic
• Supervisor CPU sends/receives traffic via dedicated interface to Fabric using inband interface
• Monitoring direction is from perspective of switch fabric, not CPU
• Tx SPAN monitors traffic from switch fabric to CPU
• Rx SPAN monitors traffic from CPU to switch fabric
• One Inband SPAN session per switch supported
NEXUS(config)# monitor session 1
NEXUS(config-monitor)# source interface sup-eth 0
BRKDCT-1890 76
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Inband SPAN – Monitor control traffic
• Inband SPAN Packet Trace Example
monitor session 1
source interface sup-eth0 both
rate-limit auto
destination interface Ethernet1/32
no shut
interface Ethernet1/32
switchport
switchport monitor
speed 1000
no shutdown
BRKDCT-1890 77
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Real World ExampleHigh CPU – Use INBAND SPAN to find out!
NEXUS# show processes cpu sort
CPU utilization for five seconds: 100%/100%; one minute: 99%; five minutes:98%
PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
6131 11367100 1497150 7 78.02% 77.12% 76.35% - X
5615 44622720 3059816 14 15.121% 14.13% 14.59% - Y
NEXUS(config)# monitor session 1
NEXUS(config-monitor)# source interface sup-eth 0
BRKDCT-1890 78
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Rule Based SPAN – SPAN-Filter
• Filter applied selectively on a session results in desired subset of traffic
• Filter by L2/L3/L4 fields
BRKDCT-1890 79
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Rule Based SPAN – SPAN-Filter
• Configure a filter within the session configuration mode
• Simple SPAN-Filter Configuration:
monitor session 1
source interface Ethernet1/17 both
rate-limit auto
destination interface Ethernet1/32
filter frame-type ipv4 src-ip 10.10.77.113/32
filter frame-type ipv4 dest-ip 10.10.77.114/32
no shut
Boolean AND between
different filters
BRKDCT-1890 80
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Rule Based SPAN – SPAN-Filter
• How is ACL different from SPAN-Filter?
• ACL
• Applied on interfaces & vlans
• Requires large TCAM size
• SPAN-Filter
• Applied on a SPAN session
• Limited TCAM space
BRKDCT-1890 81
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
SPAN – Filters, ACL etc…Confused?
• Nexus 5600/6000 :
• ACL Filters for SPAN (Use Access lists to filter SPAN)
• Nexus 7000/7700 :
• VLAN filters (Filter by VLAN)
• Rule based SPAN (Filter by L2/L3/L4 fields)
• Nexus 3100/9200/9300/9500 :
• VLAN filters (Filter by VLAN)
• ACL Filters for SPAN (Use Access lists to filter SPAN)
BRKDCT-1890 82
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
SPAN Rate Limiting
• Limits the number of SPAN copies made on ingress
• In manual mode, the rate limit will be in 1-100 range, i.e., 1%, 2%, 3% …100% of 10G SPAN rate
• In auto mode, the rate limit will automatically calculated as follows:
• Rate limit = Destination Bandwidth / Source Bandwidth
• Rate Limiting is by default in auto-modeNEXUS(config-monitor)# [no] rate-limit [auto | manual [1..100]]
BRKDCT-1890 83
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Sampled SPAN
• Used to provide an accurate count of the SPAN source packets
• Sampling and MTU truncation can be enabled at the same time and have no precedence over each other because they are applied to different aspects of the source packet (packet count versus size)
• Sampling takes precedence over SPAN source rate limiting. Rate limiting takes effect after sampling is completed on SPAN source packets
NEXUS(config-monitor)# sampling [2..1023]
BRKDCT-1890 84
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Exception SPAN
• Exception SPAN enables you to span exception packets
• Packets that have failed an build-in Nexus 7x00 intrusion detection system (IDS); as example for Layer 3 IP verification
• Rate limiters, MTU truncation, and sampling are supported in the exception SPAN session
• Exception SPAN is supported in TX direction only
BRKDCT-1890 85
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Exception SPAN – Verify CLI (IP IDS)
NEXUS# show hardware ip verify
IPv4 IDS Checks Status Packets Failed
-----------------------------+---------+------------------
address source broadcast Enabled 65536
address source multicast Enabled 65536
address destination zero Enabled 65536
address identical Enabled 65536
checksum Enabled 768
protocol Enabled 0
fragment Enabled 0
length minimum Enabled 0
length consistent Enabled 0
length maximum max-frag Enabled 0
length maximum max-tcp Enabled 0
tcp flags Enabled 0
tcp tiny-frag Enabled 0
version Enabled 0
BRKDCT-1890 86
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Exception which lead to SPAN
• length minimum = if the packet length is smaller than 64 bytes
• length consistent = when L2 frame size is shorter than the expected length to include the IP packet + MAC header.
• length maximum max-frag = if the packet fragment exceeds allowed fragmentation count
• length maximum udp = if the UDP payload is larger than specified
• length maximum max-tcp = if the TCP payload is larger than specified
• tcp flags = if incorrect flags are set in the TCP packet
• tcp tiny-frag = if TCP payload is smaller or is fragmented unexpectedly
• Version = if the IP header version is incorrect
BRKDCT-1890 87
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Exception SPAN
• Each VDC supports one exception SPAN session
• Configuration Example
NEXUS(config)# monitor session 3
NEXUS(config-monitor)# source exception all
NEXUS(config-monitor)# destination interface ethernet 2/5
NEXUS(config-monitor)# no shut
BRKDCT-1890 88
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Exception which lead to SPAN Exceptions Brief explanation
No route in hardware This is seen when adjacency is not yet formed
Unicast/Multicast route error
(incoming/outgoing interface)
This is seen when the outgoing interface is not available (say,
when the LC is reloaded)
Multicast DF failure Seen when the designated forwarder is not available.
SMAC IP check failure Incorrect SMAC / DMAC combinations, like multicast SRC
MAC or SRC.IP = DST.IP or SRC.IP is a broadcast address or
DST.IP is all zeros
Protocol field failure Incorrect IP protocol specified in the IP header
FCS / CRC errors Errors related to incorrect FCS or CRC
BRKDCT-1890 89
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Exception which lead to SPAN Exceptions Brief explanation
TTL expiry When the number of hops in the header exceeds TTL
configured
SPAN replication before L2/L3 ACL deny If the copy is made before the decision engine takes a
decision, it is Ingress replication
IPV6 scope check fail Seen when there are multiple link-local addresses tied to an
interface and the route does not exist for the packet through
either one of them,.
MTU fail When packet size exceeds the link MTU
Stale adjacency When the adjacency does not exist / is not updated for a long
time / fails refresh
CoPP violations Any packets that violated CoPP rate-limits
BRKDCT-1890 90
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Real World ExampleCRC errors – Use Exception SPAN
Packet dropped in hardware
Packet which came in didn't
make it to the egress
Use Exception SPAN
– Find reason for
drop and what was
dropped!
Packets coming into interface
were mishandled by Transceiver
leading to CRC errorsReceive packet from
wire
BRKDCT-1890 91
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Real World ExampleMalfunction NIC – Use Exception SPAN
Use Exception SPAN
– Find reason for
drop and what was
dropped!
Receive packet from
wire
Packet dropped in hardware
Packets from the server were
sent with BROADCAST
SOURCE IP because of
MALFUNCTIONING NIC
I didn’t
receive the
data!
BRKDCT-1890 92
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
SPAN Sessions on Nexus 7000
• 14 active unidirectional SPAN session
• 2 bidirectional local SPAN sessions per system
• 11 unidirectional extended sessions with F2/F2E/F3 modules present in addition to 2 traditional SPAN sessions
• 12 unidirectional extended sessions with F1/M2 modules present in addition to 2 traditional SPAN sessions
• M1 supports only 2 bidirectional local SPAN sessions per system
BRKDCT-1890 93
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
SPAN Sessions on Nexus 7700
• 16 active unidirectional SPAN session (F2E, F3 and M3)
• All SPAN sessions are unidirectional and any two can be combined to create a bidirectional session
• The Cisco Nexus 7700 switch does not have standard and extended sessions
BRKDCT-1890 94
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
SPAN Sessions on Nexus 9000
• Nexus 9500: up to 32 active SPAN session
• Scale is based on the number of linecards and the SPAN source interface to ASIC mapping
• Nexus N9200-X / N9200-Q/C / N9300 / N9300-EX): 4 active SPAN session
• Up to 3 bidirectional session plus 1 unidirectional
BRKDCT-1890 95
Review SPAN
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
SPAN Overview Nexus 5600/6000 and 7x00
SPAN Features* Nexus 5600/6000 Nexus 7000 Nexus 7700
ERSPAN destination session Yes All except F1 All LC’s
Prioritize data over SPAN Yes Yes (F2E/F3/M1/M2) Yes (F2E/F3/M3)
Line-rate SPAN throughput Yes No No
ERSPAN (v3) with 1588 PTP
timestampYes M2/F2/F2E/F3 F2E/F3/M3**
Number of SPAN destinations 16 32 N/A
SPAN with MTU truncation Yes Yes (Except M1) Yes
Virtual SPAN Yes Yes Yes
ACL filters Yes Rule based SPAN Rule based SPAN
SPAN source as VLAN Receive only Bidirectional Bidirectional
*Please check release notes for additional details and support
**M3 has HW support for ERSPAN III with IEEE15888, SW
support is pending BRKDCT-1890 97
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
SPAN Overview – Nexus 3000/9200/9300
SPAN Features* Nexus 3100Nexus 9300
1st Generation
Nexus 9200-X
Nexus 9200-Q/CNexus 9300-EX**
SPAN source as VLAN Receive only Receive only Receive/Transmit Receive/Transmit
ERSPAN destination
session (V2 and V3)Yes No HW support HW support
ERSPAN with V2 header Yes Yes-Only on
uplink portsYes Yes
Prioritize data over SPAN Yes Yes Yes Yes
Line-rate SPAN throughput Yes Yes Yes Yes
ERSPAN V3 with 1588 PTP
timestampsNo
Yes-Only on
uplink portsYes Yes
Number of SPAN session 1 1 4 4
ACL filters for SPAN Yes Yes Yes Yes
*Check SPAN/ERSPAN Configuration Documentation for details on CCO
** NX-OS Standalone will be available Q3 CY2016 BRKDCT-1890 98
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
SPAN Overview – Nexus 9500
SPAN Features*Nexus 9500
Linecards 9400/9500/9600
Nexus 9500
Linecard 9700-EX**
SPAN source as VLAN Receive only Receive/Transmit
ERSPAN destination session No HW support
ERSPAN with V2 header No Yes
Prioritize data over SPAN Yes Yes
Line-rate SPAN throughput Yes Yes
ERSPAN V3 with 1588 PTP timestamps No Yes
Number of SPAN destinations per session 32 32
ACL filters for SPAN Yes Yes
*Check SPAN/ERSPAN Configuration Documentation for
details on CCO
** NX-OS Standalone will be available Q3 CY2016 BRKDCT-1890 99
• Introduction
• Quick Product Overview
• Advanced Visibility
• SPAN / ERSPAN
• Flexible NetFlow
• Conclusion
Agenda
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Flexible NetFlow
• Enhanced network anomaly
• Customized user configurable flow (records)
• Monitor a wider range of packet information
BRKDCT-1890 101
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
NetFlow = Visibility A single NetFlow Record provides a wealth of information
switch# show flow monitor MONITOR-1 cache
…
IPV4 SOURCE ADDRESS: 192.168.100.100
IPV4 DESTINATION ADDRESS: 192.168.20.6
TRNS SOURCE PORT: 47321
TRNS DESTINATION PORT: 443
INTERFACE INPUT: E1/1
IP TOS: 0x00
IP PROTOCOL: 6
ipv4 next hop address: 192.168.20.6
tcp flags: 0x1A
interface output: Gi0/1.20
counter bytes: 1482
counter packets: 23
timestamp first: 12:33:53.358
timestamp last: 12:33:53.370
ip dscp: 0x00
ip ttl min: 127
ip ttl max: 127
application name: nbar secure-http
…
BRKDCT-1890 102
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Layer 2 NetFlow versus Bridged NetFlow
• Layer 2 NetFlow - ability to collect IP traffic statistics based on the packet’s Layer 2 header and thus allowing for SRC/DST MAC accounting
• Bridged NetFlow - ability to collect statistic for IP traffic being bridged within a given VLAN. The flow information will be based on the packet’s Layer 3 and Layer 4 headers, allowing for applications visibility
DMAC SMAC VLAN Ethertype
000A:ABCD:00EF 001E:A12D:1287 16 0x86DD
Layer2 NetFlow
Flow Information
IP SA IP DA IP ProtoLayer4 SRC
PortLayer4 DST
Port
115.12.34.2 115.12.34.3 6 1023 5230
Bridged NetFlow
Flow Information
BRKDCT-1890 103
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Seven Steps of Flow Creation
1 Packet
Extract relevant fieldsFlow
Flow
Flow
Flow
Flow Statistics
Statistics
Statistics
Statistics
Statistics
DMAC SMAC VLAN Ethertype
000A:ABCD:00EF 001E:A12D:1287 16 0x86DD
43
I/O module collects the flows and their
statisticsonce the flow ages out
5
2
6
Formatted into
NetFlow Export
7 Collector
I/O Module
BRKDCT-1890 104
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Full versus Sampled NetFlow
• NetFlow collects full or sampled flow data
• Full NetFlow: Accounts for every packet of every flow on interface
• Available on M1/M2 modules only on Nexus 7000
• Flow data collection up to capacity of hardware NetFlow table
• Sampled NetFlow: Accounts for M in N packets on interface
• Available on M1/M2 and F3/M3 in Nexus 7x00 and Nexus 5600/6000
• M2: Flow data collection up to capacity of hardware NetFlow table
• F3: Flow data collection for up to ~500pps per ASIC(SOC) module before NX-OS 7.2
• F3/M3: Increased per-module sampling rate leveraging on-board Fabric Services Accelerator (FSA) complex to ~50kpps with NX-OS 7.2
• Nexus 5600/6000: Flow data collection for up to ~120kpps per chassis
BRKDCT-1890 105
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
NetFlow on M2 Modules
Fabric
ASIC
VOQs
MgmtEnet
Supervisor
Engine
Forwarding
Engine
LC
CPU
NetFlow
Table
M2 Module
Forwarding
Engine
LC
CPU
NetFlow
Table
M2 Module
Forwarding
Engine
LC
CPU
NetFlow
Table
M2 Module
Hardware
Flow Creation
Hardware
Flow Creation
Hardware
Flow Creation
Aged Flow Info
Aged Flow Info
Aged Flow Info
Generate NetFlow v5
or v9 export packets
Main
CPU
To NetFlow Collector
To NetFlow Collector
Switched
EOBC
via Supervisor
Inband
via mgmt0
BRKDCT-1890 106
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
NetFlow on F3/M3 Modules
F3/M3 Module
FSA
CPU
SoC
Decision
Engine
DRAM
NetFlow
Cache
F3/M3 Module
Fabric
ASIC
VOQs
MgmtEnet
Supervisor
Engine
FSA
CPU
SoC
Decision
Engine
Main
CPU
To NetFlow Collector
To NetFlow Collector
Switched
EOBC
via mgmt0
DRAM
NetFlow
Cache
Populate cache based
on received samples
Age flows and
generate NetFlow v5
or v9 export packets
F3/M3 Module
FSA
CPU
SoC
Decision
Engine
DRAM
NetFlow
Cache
Data Flow
Data Flow
Data Flow
via Module
Inband
via Module
Inband
via Module
Inband
Sampled
Packets
Sampled
Packets
Sampled
Packets
Aged
Flows
Aged
Flows
Aged
Flows
BRKDCT-1890 107
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
NetFlow - Traffic StatisticsConfiguration Steps for Full Netflow*
flow exporter FLOW-EXPORT
description NetFlow v9 Exporter
destination 11.1.1.1 use-vrf management
source Loopback0
transport udp 2055
version 9
flow monitor FLOW-MONITOR
description NetFlow v9 Monitor
record FLOW-RECORD
exporter FLOW-EXPORT
interface eth 1/1
ip address 172.16.0.1 255.255.255.0
ip flow monitor FLOW-MONITOR input
ip flow monitor FLOW-MONITOR output
flow record FLOW-RECORD
match ipv4 source address
match ipv4 destination address
match transport source-port
match transport destination-port
collect counter bytes
collect counter packets
collect timestamp sys-uptime first
collect timestamp sys-uptime last
NON-KEY
Flexible NetFlow
KEY
1. Create Flow Record
2. Create Flow Exporter
3. Associate Record and Exporter to a Flow Monitor
4. Apply to the interface
*command “feature netflow” is not shown BRKDCT-1890 108
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
NetFlow - Traffic StatisticsConfiguration Steps for Sampled Netflow*
flow exporter FLOW-EXPORT
description NetFlow v9 Exporter
destination 11.1.1.1 use-vrf management
source Loopback0
transport udp 2055
version 9
flow monitor FLOW-MONITOR
description NetFlow v9 Monitor
record FLOW-RECORD
exporter FLOW-EXPORT
interface eth 1/1
ip address 172.16.0.1 255.255.255.0
ip flow monitor FLOW-MONITOR input sampler FLOW-SAMPLER
ip flow monitor FLOW-MONITOR output sampler FLOW-SAMPLER
flow record FLOW-RECORD
match ipv4 source address
match ipv4 destination address
match transport source-port
match transport destination-port
collect counter bytes
collect counter packets
collect timestamp sys-uptime first
collect timestamp sys-uptime last
NON-KEY
Flexible NetFlow
KEY
1. Create Flow Record
2. Create Flow Exporter
3. Associate Record and Exporter to a Flow Monitor
4. Create Flow Sampler
5. Apply Flow Monitor and Flow Sampler to the
interface
*Command “feature netflow” is not shown
sampler FLOW-SAMPLER
description Netflow v9 Sampler
mode 1 out-of 1200
BRKDCT-1890 109
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
NetFlow – Traffic Statistics
• Example with a N77-F348XP-23 linecard (6 SOC/ASICs per Linecard), FSA enabled
• Assumption: average packetsize 512bytes, average traffic rate max 50%
• Per F3 Linecard sampling rate (FSA) = 50kpps, per SOC/ASIC = 8’333pps*
• Calculation
• 50% x 10Gbps / 512 Bytes = ~1’221kpps
• 8 Ports per SOC: 8 x 1’221kpps = 9’768kpps
• Calculated Sampling Rate: 9’768kpps / 8’333pps = ~ 1172
• Recommended sampling rate to be configured = 1 : 1’200
Calculating the sampling rate
*Exceeding per SOC sampling rate will result tail-dropping
packets due the rate limiter BRKDCT-1890 110
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Nexus 7000 Control-Plane NetFlow
• Creates NetFlow records for control-traffic* punted to the CPU on the Supervisor
• Traffic flows from Linecard to CPU, hence the NetFlow monitor could be applied in egress (output) direction only
• Flow monitor is applied on control-plane interface
• Only sampled Netflow is supported
• Configuration applied in the default VDC
• Linecard specific NetFlow capabilities and resources are used for creating this internal control-plane flows
NX-OS
7.3
*today unicast control-plan traffic, only multicast control-
traffic should be supported from the next 7.3
Maintenance Release (Q3/4 CY2016) BRKDCT-1890 111
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Nexus 7000 Control-Plane NetFlow Resolving High CPU using CoPP NetFlowNEXUS# show processes cpu sort
CPU utilization for five seconds: 65%/8%; one minute: 63%; five minutes: 61%
PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
310 30544 189234 81 47.12% 45.11% 45.23% 0 IP Input
flow record NF-RECORD
match ipv4 source address
match ipv4 destination address
match transport source-port
match transport destination-port
collect counter packets
High CPU due to process “IP Input”
Building a NetFlow record, matching L3 and L4
parameters (key fields) and collecting details on
Input interface and packet count (non-key fields)
NX-OS
7.3
flow exporter NF-EXPORT-1
destination 11.1.1.1 use-vrf management
transport udp 2055
source mgmt0
version 9
Optional: Building a NetFlow exporter
sampler NF-SAMPLER
mode 1 out-of 1Create a sampler
BRKDCT-1890 112
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Nexus 7000 Control-Plane NetFlow Resolving High CPU using CoPP NetFlow
NX-OS
7.3
flow monitor NF-MONITOR
record NF-RECORD
exporter NF-EXPORT-1
Create the flow monitor and associate Record
and Exporter
control-plane
ip flow monitor NF-MONITOR output sampler NF-SAMPLER
NEXUS# show hardware flow ip
---snip---
D - Direction; L4 Info - Protocol:Source Port:Destination Port
IF - Interface: (Eth)ernet, (S)vi, (V)lan, (P)ortchannel, (T)unnel
TCP Flags: Ack, Flush, Push, Reset, Syn, Urgent
D IF SrcAddr DstAddr L4 Info PktCnt TCP Fl
--+-----------+---------------+---------------+---------------+----------+------
CP sup-eth1 104.104.104.011 104.104.104.021 000:00000:00000 0000000100 ......
Applying to the control-plane interface the NetFlow
monitor in egress direction with a sampler
Check your control-plan flow entries
Troubleshooting Methodology:
Once the flow is identified, further action could be (1) blocking the flow with an
Access List (ACL) (Infrastructure or CoPP) or/and (2) rate-limiting the flow using
CoPP depending on the criticality of the flow to the production
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
NetFlow Overview
M2 (N7000) F3 (Nexus 7x00) M3 (Nexus 7700) Nexus 5600/6000
Per-interface NetFlow Yes Yes Yes Yes
NetFlow direction Ingress/Egress Ingress / Egress* Ingress / Egress Ingress only
Full NetFlow Yes No No No
Sampled NetFlow Yes Yes Yes Yes
FSA assisted for Sampled
NetFlowNo Yes* Yes No
Bridged NetFlow Yes Yes Yes Yes
Hardware Cache Yes No No No
Software Cache No Yes Yes Yes
Hardware Cache Size512K entries per
forwarding engineN/A N/A N/A
NDE (v5/v9) Yes Yes Yes Yes
*supported since NX-OS 7.2
Note: Nexus 9K (N9200-X/N9300-EX/N9700-EX) supports full NetFlow; software support is on the roadmap
BRKDCT-1890 114
• Introduction
• Quick Product Overview
• Advanced Visibility
• SPAN/ ERSPAN
• NetFlow
• Conclusion
Agenda
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Tools designed with you in mind
• Advanced feature rich analytics tools
• Visibility into the products helping to validate the path-of-the-packet
• Analytics tools can help in isolating problems we see in Datacenters today
• Reduce the time to resolution of network issues
Netflow
SPAN ERSPAN
ACL CaptureLatency
monitoringSPAN-on-drop
Microburst
monitoring
BRKDCT-1890 116
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Call to Action
• Attend the following related sessions
• BRKDCN-3020 - Network Analytics using Nexus 3000/9000 Switches
• BRKARC-3452 - Cisco Nexus 5600/6000 Switch Architecture
• BRKARC-3470 - Cisco Nexus 7000/7700 Switch Architecture
• BRKARC-2222 - Cisco Nexus 9000 Architecture
• BRKARC-2011 - Overview of Packet Capturing Tools in Cisco Switches and Routers
• Visit the World of Solutions for
• Cisco Campus | Walk in Labs | Technical Solution Clinics
• Meet the Engineer
• Lunch and Learn Topics
• DevNet zone related sessions -
BRKDCT-1890 117
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Complete Your Online Session Evaluation
Don’t forget: Cisco Live sessions will be available for viewing on-demand after the event at CiscoLive.com/Online
• Give us your feedback to be entered into a Daily Survey Drawing. A daily winner will receive a $750 Amazon gift card.
• Complete your session surveys through the Cisco Live mobile app or from the Session Catalog on CiscoLive.com/us.
BRKDCT-1890 118
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Continue Your Education
• Demos in the Cisco campus
• Walk-in Self-Paced Labs
• Lunch & Learn
• Meet the Engineer 1:1 meetings
• Related sessions
BRKDCT-1890 119
Please join us for the Service Provider Innovation Talk featuring:
Yvette Kanouff | Senior Vice President and General Manager, SP Business
Joe Cozzolino | Senior Vice President, Cisco Services
Thursday, July 14th, 2016
11:30 am - 12:30pm, In the Oceanside A room
What to expect from this innovation talk
• Insights on market trends and forecasts
• Preview of key technologies and capabilities
• Innovative demonstrations of the latest and greatest products
• Better understanding of how Cisco can help you succeed
Register to attend the session live now or
watch the broadcast on cisco.com
Thank you
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
White Papers on Cisco Connection Online (CCO)
• Monitor Microbursts on Cisco Nexus 5600 Platform and Cisco Nexus 6000 Series Switches
• http://www.cisco.com/c/en/us/products/collateral/switches/nexus-5000-series-switches/white-paper-c11-733020.html
• SPAN-on-Latency Feature on Cisco Nexus Switches: Troubleshoot Network Latency
• http://www.cisco.com/c/en/us/products/collateral/switches/nexus-5000-series-switches/white-paper-c11-733021.html
BRKDCT-1890 123
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
White Papers on Cisco Connection Online (CCO)
• SPAN-on-Drop Feature on Cisco Nexus Switches: Troubleshoot Network Congestion
• http://www.cisco.com/c/en/us/products/collateral/switches/nexus-5000-series-switches/white-paper-c11-733022.html
• Latency Monitoring Tool on Cisco Nexus Switches: Troubleshoot Network Latency
• http://www.cisco.com/c/en/us/products/collateral/switches/nexus-5000-series-switches/white-paper-c11-733025.html
BRKDCT-1890 124
Recommended