Upload
tranxuyen
View
233
Download
2
Embed Size (px)
Citation preview
Troubleshooting Cisco CMTS Based Services
BRKSPG-2501
Tejal Patel [email protected]
Jack Yu [email protected]
• CMTS Based Services Evolution
• Troubleshooting High Speed Data• DOCSIS 3.0 DS and US Channel Bonding Issues
• Troubleshooting DOCSIS Timing Issues
• Troubleshooting DOCSIS Load Balancing
• Troubleshooting Voice Services• Troubleshooting Voice Subscriber Issues
• Troubleshooting SNMP Services
• Troubleshooting DSG Services
• Summary
• Q & A
Agenda
• CMTS Based Services Evolution
• Troubleshooting High Speed Data• DOCSIS 3.0 DS and US Channel Bonding Issues
• Troubleshooting DOCSIS Timing Issues
• Troubleshooting DOCSIS Load Balancing
• Troubleshooting Voice Services• Troubleshooting Voice Subscriber Issues
• Troubleshooting SNMP Services
• Troubleshooting DSG Services
• Summary
• Q & A
Agenda
Outside
Plant
Cable Multi-Service Networks
Aggregation NetworkAccess/Hub
Backbone Network
MPLS PE
HR
AR
Corporate
Business
Residential
VoD
Content Network
TV SIP
CRS-x Core
VoD
Content Network
TV SIPMultiservice Backbone
Core Network
IP / MPLS
Regional Network
CR
Regional IP /
MPLS Network
CRCR
IPoDWDM Optical Network
CR
AR
Deep Packet
Inspection
CMTS
EQAM
HR
ASR901/903
FTTP
Corporate
Business
ME3400
STB
CM/
eMTA
Corporate
Business
ISR
Corporate
Business
L2VPN CM
L3 (IP/MPLS)L1/L2L2/L3
Video Complex
SDV VoD
PEG DS/DB
SDC (System/Div)
HR
Voice /Video Complex
Network Mgt Complex
IP Security Complex
SDC (Regional) SDC (National)
HFC
GPON OLT
• CMTS Based Services Evolution
• Troubleshooting High Speed Data• DOCSIS 3.0 DS and US Channel Bonding Issues
• Troubleshooting DOCSIS Timing Issues
• Troubleshooting DOCSIS Load Balancing
• Troubleshooting Voice Services• Troubleshooting Voice Subscriber Issues
• Troubleshooting SNMP Services
• Troubleshooting DSG Services
• Summary
• Q & A
Agenda
Troubleshooting High Speed Data Services
• DOCSIS 3.0 DS/US Channel Bonding Issues
• CMs failing to register as W-Online or Bonding Capable
• DOCSIS 3.0 performance issues
• Poor DS throughput on DSCB modem
• Poor US throughput issues on USCB modem
Common Issues
DOCSIS 3.0 DS & US Channel Bonding Issues
DOCSIS 3.0 M-CMTS System
DOCSIS 3.0 Registration Diagram
WCM performs usual US
channel selection, but does not
start initial rangingMDD message
WCM performs bonded service
group selection, and indicates
via initial rangingB-INIT-RNG-REQ message
WCM acquires QAM/FEC lock
of DOCSIS DS channel
Usual DOCSIS initial ranging sequence
DHCP DISCOVER packet
DHCP RESPONSE packetDHCP REQUEST packetDHCP OFFER packet
WCM transitions to ranging
station maintenance as usual
REG-ACK message
REG-RSP-MP message
Usual BPI init. If configured
TOD Request/Response messages
TFTP Request/Response messages
REG-REQ-MP message WCM provides Rx-Chan(s)-Prof
WCM receives Rx-Chan(s)-Config
WCM confirms all Rx Channels
CMTS
SYNC, UCD, MAP messages
D 3.0 CM
CMs Not Coming Up As W-Online/UB
• 1. Improper configuration1.a - Verify Integrated controller and interface, WB interface, CGD and
Fiber-node configuration and verification
1.b - MDD issues
1.c - Real time debugs and show commands verification
• 2. Connectivity issues between the M-CMTS components
2.a - Between M-CMTS components and DTI server
2.b - Between the CMTS and EQAM
• 3. RF side combining issues at the headend
Possible Reasons
Controller Modular-Cable 8/1/1
rf-channel 0 cable downstream channel-id 200
rf-channel 0 frequency 615000000 annex B modulation 256qam interleave 32
rf-channel 0 depi-tunnel UBR10K_05_GE812 tsid 9101
rf-channel 0 rf-power 40.0
no rf-channel 0 rf-shutdown
rf-channel 1 cable downstream channel-id 201
rf-channel 1 frequency 621000000 annex B modulation 256qam interleave 32
rf-channel 0 depi-tunnel UBR10K_05_GE812 tsid 9102
rf-channel 1 rf-power 40.0
no rf-channel 1 rf-shutdown
…
rf-channel 15 cable downstream channel-id 215
rf-channel 15 frequency 705000000 annex B modulation 256qam interleave 32
rf-channel 0 depi-tunnel UBR10K_05_GE812 tsid 9116
rf-channel 15 rf-power 40.0
no rf-channel 15 rf-shutdown
Controller configuration verification
interface Modular-Cable8/1/0:0
cable bundle 1
cable rf-bandwidth-percent 1
Bundle ID inherited from LC, need to match with
wideband interface
BW has to be configured
Modular-Cable interface configuration verification
Unique DS channel-id
1.a – Downstream Configuration Checklist
DEPI Tunnel Name and
TSID
RF-Power within range
for 16 Channel output
16 Channel
Configuration
M-CMTS615Mhz621Mhz
627Mhz
705Mhz
………
……
US
DS
D 3.0
CMIP Core
HFC Node
1.a – Downstream/Upstream Configuration Checklist
Attributes value:
8=1000
bit0 = Bonding,
bit1= Low Latency,
bit2=High Availability,
bit3-15=Reserved for
future use
interface Cable8/1/1
downstream Modular-Cable 8/1/1 rf-channel 0-15
cable mtc-mode required-attribute
no cable packet-cache
cable bundle 1
cable upstream max-ports 4
cable upstream bonding-group 1
upstream 0
upstream 1
upstream 2
upstream 3
attributes 80000000
cable upstream balance-scheduling
Cable Interface Configuration
US BG definition with US channels
Attribute needs to match in CM config file
Default CLI. Need to
have attribute in CM
config file
Optional –USBG scheduler to send MAPs on
all upstreams of the MD.
interface Wideband-Cable8/1/1:0
cable bundle 1
cable rf-channel 0 bandwidth-percent 1
cable rf-channel 1 bandwidth-percent 1
…
cable rf-channel 14 bandwidth-percent 1
cable rf-channel 15 bandwidth-percent 1
Bundle ID that should match under modular/IC interface
Wideband interface verification
Cisco ACFE - Adaptive CIR and
Fair EIR - allows for dynamic
bandwidth allocation
1.a – Fiber-node Configuration and Verification
show cable cgd-associations
CGD Host Resource DS Channels Upstreams (AllUS) Active DS
Ca8/1/1 8/1/1 0-15 0-3 Yes
1-15
Configured DS
CGD association verification
Show cable fiber-node
Fiber-Node 101
Channel(s) : downstream Modular-Cable 8/1/1: 0-15
Channel ID(s): 200, 201, 202, 203, 204, 205, 206, 207, 208, 209, 210, 211, 212, 213,
214, 215
upstream Cable 8/1: 1
FN Config Status: Configured (status flags = 0x01)
MDD Status: Valid MDD has to be valid
Fiber Node with RF channels(DS Does not have to be unique)
Fiber-node configuration verification
US Connectors Defined(Unique per FN)
cable fiber-node 101
downstream Modular-Cable 8/1/1 rf-channel 0-15
upstream Cable 8/1 connector 1
US connector addition under fiber-node
Fiber-node configuration verification
1.a – Downstream Configuration Checklist
Primary channel SYNC and MAP verification
F241-38-05-uBR10K-01#show controller Modular-Cable 8/1/1 mapping rf-chan
Ctrlr RF MC MC Rem. WB WB WB Rem.
channel BW % Ratio channel BW % Ratio
8/1/1 0 1 1 8/1/1:0 1 1
8/1/1:1 1 1
<SNIP>
8/1/1:14 1 1
8/1/1:15 1 1
…
8/1/1 1 1 1 8/1/1:0 1 1
8/1/1:1 1 1
…
WB interface BW
(1% with ACFE)
RF-Channel mapping verification
RF Channels that
make this up
Bonding Group
Modular-Cable
(Narrowband and
Primary) BW
F241-38-05-uBR10K-01#show controller Modular-Cable 8/1/1 counters rf-channel
Controller RF MPEG MPEG MPEG Sync MAP Queue
Chan Packets Tx bps Mbps Packets Tx Packets Tx
8/1/1 0 5406341508 29337376 29.743 1105170888 20849541696
8/1/1 1 5352261326 29280123 29.280 1105170887 20849541695
…
8/1/1 15 5352276150 29251244 29.251 1105170887 20849541695
Packet CounterThroughput Rate (~37.5 Max)
Show hw-
module bay for
WB/6G-SPA
Sync and MAP packets on
PC channel
1.b – MDD Verification• Debugs to verify MDD generation on CMTS
• debug cable interface cable {slot/subslot/port} [verbose]
• debug cable mdd
• Debugs needed for ranging and registration—For DSCB/USCB modem• debug cable mac-address {cable-modem-mac-address} verbose
• debug cable mdd
• debug cable ranging
• debug cable registration
• debug cable tlv
• debug cable ipv6 dhcp
• debug cable service-ds-selection
• debug cable rcc
• debug cable wbcmts
• debug cable range initial
• debug cable md-sg
• debug cable ubg
1.b – Sample MDD Debug from PC ChannelCable8/1/1: size 592 mdd_tlv_size 562 num_frag 1 seq_num 1
test_mdd_tlv_length 0
Cable8/1/1 MDD datagramsize 592, msg len 590, ehdr type_or_len
572, tlv_size 562 max_pak_size 1518
MDD MESSAGE
FRAME HEADER
FC, MAC_PARM, LEN - 0xC2, 0x00, 0x024E
MAC MANAGEMENT MESSAGE HEADER
DA, SA - 01E0.2F00.0001, 0012.001A.689B
msg LEN - 0x023C
DSAP, SSAP - 0, 0
control, version, type - 0x03, 0x04, 0x21
change_count - 0x9F
num_fraqment, seq_num - 0x01, 0x01
dcid - 203
MDD TLV, Total TLV size - 562
MDD TLV
Downstream Active Channel List
Channel ID: 200
Frequency: 615000000Hz
Modulation Order/Annex: 256 QAM/Annex B
Primary Capable: Primary-Capable
CM-STATUS Event Bitmask:0x36
MDD Timeout
QAM FEC failure
MDD Recovery
QAM FEC recovery
Downstream Active Channel List
Channel ID: 201
Frequency: 621000000Hz
Modulation Order/Annex: 256 QAM/Annex B
Primary Capable: Not Primary-Capable
<SNIP>
MAC Domain Downstream Service Group
MD-DS-SG ID: 1
Channel IDs: 200
201
202
203
204
205
<SNIP>
211
212
213
214
215
Downstream Ambiguity Resolution Frequency List
Frequencies: 615000000Hz
621000000Hz
627000000Hz
633000000Hz
<SNIP>
687000000Hz
693000000Hz
699000000Hz
705000000Hz
IP Initialization Parameters
IP Provisioning Mode: IPv6
Receive Channel Profile Reporting Control
Center Freq spacing: 6 MHz
Verbose Reporting: No
Fragmented RCP accept: Yes
Early Authentication and Encryption (EAE)
Early Authentication: Disabled
Symbol Clock Locking Indicator
1.b – Continued MDD with US Channels
<snip> Downstream Ambiguity Resolution Frequency List
Frequencies: 615000000Hz
621000000Hz
627000000Hz
633000000Hz
…
675000000Hz
<snip>
IP Initialization Parameters
IP Provisioning Mode: IPv6
Receive Channel Profile Reporting Control
Center Freq spacing: 6 MHz
Verbose Reporting: No
<snip>
MAC Domain Upstream active chan list
MD-US Chan ID/CM-STATUS: 1/0x0000
MAC Domain Upstream active chan list
MD-US Chan ID/CM-STATUS: 2/0x0000
MAC Domain Upstream active chan list
MD-US Chan ID/CM-STATUS: 3/0x0000
<snip>
Upstream Ambiguity Resolution Channel List
MD-US Chan ID: 0 1 2 3
Upstream Frequency Range 0
Upstream Transmit Power Reporting: On
CM-STATUS non-channel-specific events : Seq out of range
Active US channel list
US Ambiguity channel
list
Debug cable mdd will have active US channels
1.c – Debug CM’s Ranging to Registration
Mar 24 13:30:19.684 EDT: Bonding Initial Ranging request from 001d.d4d3.31d2, SID 0 [16383] on Interface Cable8/1/1/U1:
MD-DS-SG-ID 1, Cap flags 192 Src sap 173
Mar 24 13:30:19.684 EDT: Initial Ranging: Downstream channel ID is 205 (CGD host DS chan Id 173)
Mar 24 13:30:19.684 EDT: CM mac address found. Assigned Primary SID 3.
Mar 24 13:30:19.684 EDT: cmts_tcc_uschan_add: CM 001d.d4d3.31d2 tcs 0 chan 1 tech 2 ref 0
Mar 24 13:30:19.684 EDT: CMTS IPV6DB: Not found, cmts_ipv6_lookup_entry_by_mac: MAC 001d.d4d3.31d2 in Cable8/1/1
Mar 24 13:30:19.684 EDT: Modem 001d.d4d3.31d2: Host Ca8/1/1/U1: ds_channel_id 205, rfid 5213
Initial Ranging
Mar 24 13:30:19.684 EDT: CM Ambiguity Resolution Done SG_ID=1, upstream 2
Mar 24 13:30:19.684 EDT: Timing error 1799, power error 0.00dB, freq error -100(thres 1280 adj 0) [sm per 20.0 sec]
Mar 24 13:30:19.684 EDT: Equalization coeff size 48
Mar 24 13:30:19.688 EDT: Ca8/1/1/U1: Send RNG-RSP (1) for 001d.d4d3.31d2, SID 3, DS RFID 5213
Mar 24 13:30:20.684 EDT: Ranging CM 001d.d4d3.31d2, SID 3 iuc 4 mslot 7 on I/f Cable8/1/1/U1
Mar 24 13:30:20.684 EDT: 001d.d4d3.31d2: dyn pwr status 0 tx lvl 173 ch load 255 min load 0
Mar 24 13:30:20.684 EDT: Ranging request from 001d.d4d3.31d2, SID 3 [3/205/5213] on Interface Cable8/1/1/U1
Mar 24 13:30:20.684 EDT: Timing error 0, power error 0.00dB, freq error -102(thres 1280 adj 0) [sm per 20.0 sec]
Mar 24 13:30:20.684 EDT: Ranging adjustments needed.
Ambiguity Resolution and Ranging Response
IPv6 Neighbor Discovery & DAD InitialMar 24 13:30:25.404 EDT: Proxy DAD: Initiating DAD for FE80::21D:D4FF:FED3:31D2 for sid 3 prim_sid 3 in Cable8/1/1
Mar 24 13:30:25.408 EDT: Proxy DAD: Learnt new LL address FE80::21D:D4FF:FED3:31D2 for 001d.d4d3.31d2
Mar 24 13:30:25.408 EDT: CMTS IPV6DB: cmts_ipv6_lookup_entry_by_v6addr: FE80::21D:D4FF:FED3:31D2 not found in Cable8/1/1
Mar 24 13:30:25.408 EDT: IPV6 ND: IPv6 addr FE80::21D:D4FF:FED3:31D2 added to new host 001d.d4d3.31d2, SID 3, Mar 24
13:30:25.408 EDT: Proxy DAD : Succeeded manual ND cache add for CPE FE80::21D:D4FF:FED3:31D2 001d.d4d3.31d2
Mar 24 13:30:25.408 EDT: Proxy DAD: << Proxy DAD FINAL exit for FE80::21D:D4FF:FED3:31D2 for prim_sid 3 in Cable8/1/1
Bonding Initial Request on MD-DS-SG 1
Ambiguity resolution done with CM on US-SG 1
1.c – Debug CM’s Ranging to RegistrationIPv6 DHCPv6 Addressing AssignmentMar 24 13:30:31.034 EDT: IPv6 UDP Packet from FE80::21D:D4FF:FED3:31D2 to dport 547
Mar 24 13:30:31.034 EDT: CMTS DHCPV6: Incoming DHCPv6 SOLICIT from 001d.d4d3.31d2 (sid 3 prim_sid 3) in Cable8/1/1.
Mar 24 13:30:31.034 EDT: CMTS IPV6DB: cmts_ipv6_lookup_entry_by_v6addr: bucket = 449 FOUND for
FE80::21D:D4FF:FED3:31D2 Flag 0x4200 in Cable8/1/1
Mar 24 13:30:31.034 EDT: IPv6 DHCP: detailed packet contents
Mar 24 13:30:31.034 EDT: src FE80::21D:D4FF:FED3:31D2 (Bundle99)
Mar 24 13:30:31.034 EDT: dst FF02::1:2
Mar 24 13:30:31.034 EDT: type SOLICIT(1), xid 6374736
Mar 24 13:30:31.034 EDT: option CLIENTID(1), len 10
Mar 24 13:30:31.034 EDT: 00030001001DD4D331D2
Mar 24 13:30:31.034 EDT: option IA-NA(3), len 40
Mar 24 13:30:31.034 EDT: IAID 0xD4D331D2, T1 0, T2 0
Mar 24 13:30:31.034 EDT: option IAADDR(5), len 24
Mar 24 13:30:31.034 EDT: IPv6 address ::
Mar 24 13:30:31.034 EDT: preferred 0, valid 0
Mar 24 13:30:31.034 EDT: option RAPID-COMMIT(14), len 0
Mar 24 13:30:31.034 EDT: option ELAPSED-TIME(8), len 2
Mar 24 13:30:31.034 EDT: elapsed-time 0
Mar 24 13:30:31.034 EDT: option ORO(6), len 4
Mar 24 13:30:31.034 EDT: VENDOR-OPTS,DNS-SERVERS
Mar 24 13:30:31.034 EDT: option RECONF-ACCEPT(20), len 0
Mar 24 13:30:31.034 EDT: option VENDOR-CLASS(16), len 15
Mar 24 13:30:31.034 EDT: option VENDOR-OPTS(17), len 300
Mar 24 13:30:31.034 EDT: Removing dhcpv6 GUA addrs from host 001d.d4d3.31d2
Mar 24 13:30:31.034 EDT: cmts_dhcpv6_cm_set_device_class_list: Setting device_class_list CM-mac_addr 001d.d4d3.31d2,
esafe option ECM:EROUTER, device_class_list in hex = 41
Mar 24 13:30:31.034 EDT: DHCPv6 SOLICIT After calling cmts_dhcpv6_cm_set_device_class_list, mac = 001d.d4d3.31d2
cminstp->device_class_list = 41
DHCP v6 SOLICIT from CM
RAPID COMMIT
Device class information
Need IPv6 Address
1.c – Debug CM’s Ranging to Registration
IPv6 CM NS and DAD
Mar 24 13:30:31.038 EDT: CMTS IPV6DB: Host entry is a CM, SID 3, if_number 583.
Mar 24 13:30:31.038 EDT: CMTS IPV6DB: Created an address entry(0xFBB484E8) IP(0xFDDA7048) in host(0xFDDA73A8)
001d.d4d3.31d2, intf Cable8/1/1, Sid 3.
0:F241:1300:18:21D:D4FF:FED3:31D2/128 flag = 0x6214
Mar 24 13:30:31.038 EDT: CMTS IPV6DB: DHCPv6 Reply, calling cmts_entry_add... 001d.d4d3.31d2,
FD00:F241:1300:18:21D:D4FF:FED3:31D2
Mar 24 13:30:31.565 EDT: CMTS IPV6DB: Sid 3 not found while processing NS
Mar 24 13:30:31.565 EDT: Proxy DAD: Initiating DAD for FD00:F241:1300:18:21D:D4FF:FED3:31D2 for sid 3 prim_sid 3 in
Cable8/1/1
Mar 24 13:30:31.565 EDT: Proxy DAD: << Proxy DAD v6addr exit for FD00:F241:1300:18:21D:D4FF:FED3:31D2 for prim_sid 3
in Cable8/1/1
Mar 24 13:30:31.565 EDT: Proxy DAD : GUA FD00:F241:1300:18:21D:D4FF:FED3:31D2 not added for CM 001d.d4d3.31d2
Mar 24 13:30:31.565 EDT: Proxy DAD: << Proxy DAD FINAL exit for FD00:F241:1300:18:21D:D4FF:FED3:31D2 for prim_sid 3
in Cable8/1/1
SLOT 8/1: Mar 24 13:30:31.692 EDT: Receive REG-REQ-MP from 001d.d4d3.31d2, SID 1 on Cable8/1/1
SLOT 8/1: Mar 24 13:30:31.692 EDT: CM 001d.d4d3.31d2 on Cable8/1/1: Allocate 255 bytes REG-REQ-MP TLV buffer at
SLOT 8/1: Mar 24 13:30:31.692 EDT: Now parse REG-REQ-MP 1/1 for CM 001d.d4d3.31d2
SLOT 8/1: Mar 24 13:30:31.692 EDT: Selected RCC 12 with total current cms 1 for Cable8/1/1 16 channel modem
001d.d4d3.31d2, (16 rcc rfs) BG 6952
SLOT 8/1: Mar 24 13:30:31.692 EDT: Performing admission control check for MTC CM 001d.d4d3.31d2
SLOT 8/1: Mar 24 13:30:31.692 EDT: LB 60004 assigned to 001d.d4d3.31d2, so limit the TCS to 0xF
SLOT 8/1: Mar 24 13:30:31.692 EDT: not DEVICE_CLASS_STB, not RNG_HOLDOFF_ESTB, mac_addr = 001d.d4d3.31d2
device_class_list = 0x00000041, rng_holdoff_supp = 0x00000002
Registration Request Registration Request received from CM RCC12 with 16 ch.
Selected for a CM
1.c – Debug CM’s Ranging to Registration
Mar 24 13:30:31.692 EDT: Found Vendor Capability Information TLV
Mar 24 13:30:31.692 EDT: Found Vendor Id Sub-TLV {00 00 0C}
Mar 24 13:30:31.692 EDT: This Is A Cisco Vendor Id
Mar 24 13:30:31.692 EDT: RCP index : 3
Mar 24 13:30:31.692 EDT: BG ID : 6952
Mar 24 13:30:31.692 EDT: RCC ID : 12
Mar 24 13:30:31.692 EDT: Found Vendor Specific Information TLV
Mar 24 13:30:31.692 EDT: Found Vendor Id Sub-TLV {00 00 0C}
Mar 24 13:30:31.692 EDT: This Is A Cisco Vendor Id
Mar 24 13:30:31.692 EDT: Found Vendor Specific Information TLV
Mar 24 13:30:31.692 EDT: Found Vendor Id Sub-TLV {00 00 0C}
Mar 24 13:30:31.692 EDT: This Is A Cisco Vendor Id
Mar 24 13:30:31.692 EDT: SFAC Bucket Number TLV, 1 entries
Mar 24 13:30:31.692 EDT: SFID 46 SFAC bucket number is 7
Mar 24 13:30:31.692 EDT: Parsing CM's Reg Info Complete
<SNIP>
SLOT 8/1: Mar 24 13:30:31.752 EDT: Now sending 2 REG-RSP-MP fragment(s) for CM 001d.d4d3.31d2
SLOT 8/1: Mar 24 13:30:31.752 EDT: REG-RSP-MP Status : ok (0), REG-ACK required from CM (0)
<SNIP>
Mar 24 13:30:31.692 EDT: Response fragment 1/1 (558 bytes) Transmitted
Mar 24 13:30:31.692 EDT: Registration acknowledgement (0) from 001d.d4d3.31d2, SID 3 on Cable8/1/1/U1
SLOT 8/1: Mar 24 13:30:32.845 EDT: CM is waiting REG-ACK, now handle confirmation code : 0
RCPs and Registration Response
CMTS selected
Bonding Group
RCP from CM
RCC 12
selected for CM
Remaining USs RangingSLOT 8/1: Mar 24 13:31:32.909 EDT: Ca8/1/1/U1: Send RNG-RSP (0) for 001d.d4d3.31d2, SID 1, DS RFID 5212 <snip>
SLOT 8/1: Mar 24 13:31:32.997 EDT: Ca8/1/1/U3: Send RNG-RSP (0) for 001d.d4d3.31d2, SID 1, DS RFID 5212 <snip>
SLOT 8/1: Mar 24 13:31:33.158 EDT: Ca8/1/1/U0: Send RNG-RSP (0) for 001d.d4d3.31d2, SID 1, DS RFID 5212
1.c – Expanded Show Commands
F241-38-05-uBR10K-01#show cable modem 001d.d4d3.3122 verbose
MAC Address : 001d.d4d3.3122
Multi-Transmit Channel Mode : Y
IPv6 Address : FD00:F241:1300:18:21D:D4FF:FED3:31D2
Number of US in UBG : 4
Upstream Channel : US0 US1 US2 US3
Ranging Status : sta sta sta sta
Upstream SNR (dB) : 39.63 35.95 37.65 38.6
Upstream Data SNR (dB) : -- -- -- --
Received Power (dBmV) : 0.00 -0.50 0.00 -1.00
Reported Transmit Power (dBmV) : 42.25 42.75 42.75 42.75
Peak Transmit Power (dBmV) : 56.00 51.00 51.00 56.00
Phy Max Power (dBmV) : 56.00 51.00 51.00 56.00
Minimum Transmit Power (dBmV) : 21.00 21.00 21.00 21.00
Timing Offset (97.6 ns): 1803 1804 1804 1804
Initial Timing Offset : 1548 1804 1548 1548
Pre-EQ Good : 14049 14057 14048 14049
Good Codewords rx : 14545 14635 14508 14575
Corrected Codewords rx : 0 211 0 0
Uncorrectable Codewords rx : 0 156 0 0
Phy Operating Mode : tdma* atdma* atdma* tdma*
Ranging Status in
Station
Maintenance. Not
in Partial Service
Deep look in to US stats
Show cable modem with UB for MTC modems
UB Added for USCB modem
SNR Readings
Pre-EQ Counters
Codeword are a good indication
of plant health
F241-38-05-uBR10K-01#show cable modem 001d.d4d3.3122 ipv6
MAC Address Type Interface Mac State D/IP IP Address
001d.d4d3.31d2 B/D C8/1/1/UB w-online(pt) N FD00:F241:1300:18:21D:D4FF:FED3:31D2
1.c - CM’s Partial Mode Registration
Feb 17 11:25:40.930: Cable8/1/1: CM 001d.d4d3.3122 REG-ACK response
Feb 17 11:25:40.930: partial-service:
Feb 17 11:25:40.930: 0x0000: 31 04 06 02 03 02
Feb 17 11:25:40.930: Cable8/1/1: CM 001d.d4d3.3122 ds-chid 201 is impaired
Feb 17 11:25:40.930: Cable8/1/1: CM 001d.d4d3.3122 ds-chid 202 is impaired
Partial Service indicating
impaired channels
Sample Registration ACK with Partial Service
UBR10K2#show cab mode 001d.d4d3.3122 wideband rcs-status
RF : 8/1/1 1
Status : UP
FEC/QAM Failure : 0
MDD Failure : 0
<snip>
Flaps : 0
Flap Duration : 00:00
RF : 8/1/1 0
Status : UP
RF : 8/1/1 3
Status : UP
Status reported by CM for individual DS
channel
No MDD failure on this channel
No Flaps on this channel
Modem Status the way modem reports
DS Channel from controller 6/0/1
HFC Node
M-CMTS615Mhz621Mhz
627Mhz
705Mhz
………
……
US
DS
D 3.0
CMIP CoreRF Impairment
CM Status Msg
2 – Connectivity Between M-CMTS Components
DTI in M-CMTS system
DTCC
MC20x20 LC
Edge
QAMs
CM1 CM3 CM4
DTI Server
DS0
DS1
DS2
MC20x20 upstream channel
GigE link with Downstream channels
MC20x20 downstream channel
CM2
6G
SPA/3G60
TCC
2.a – Connectivity Between M-CMTS and DTI
• Verify that the CMTS and EQAM are locked to the same DTI clock source
• Issue show cable clock
F241-38-05-uBR10K-01#show cable clock
Number of TCC Cards in the Chassis: 2
Active TCC Card is in slot: 1 subslot: 0,(DTCC Eightbells card)
Clock reference used by the active card is DTI
Active TCC card in slot 1/0
TCC Card 1/0 DTI status:
-------------------------------------------
Active Client port : 2
Active Client status : normal
Active Client Server status : freerun
Active Client frame error rate : < 2%
Active Client CRC error count : 0x02
Standby Client Signal detected : no
Status has to be
normal
<2% error rate is normal
DTI used for Clocking
2.b – Connectivity Between CMTS and EQAM • The effect
• Link flap may cause CMs to re-initialize
• Interface up/down message for MC and WB interface in the logs
• D3.0 CM may be Narrowband Online on Primary Capable Channel off MC3G60
• With cable wideband auto-reset, 3.0 CMs will bounced on re-establishment
F241-38-05-uBR10K-01#sh controllers modular-Cable 8/1/1 ge_phy
Ethernet Quad Phy 0: reg_addr = F8800C8C, contents = 1851AC10
MV88E1146C FIBER PHY registers for phy:0, port 2
Port 2 Status
Autonegotiation : ENABLED
Duplex : FULL
Speed : 1000 Mbps
Link : UP
Port 2 Configuration
Auto-negotiation ENABLED
<SNIP>
Transmitter Mode - Class B
Frame Count: 65280, CRC Error Count: 0
<SNIP>
SFP Module Status: UP
<SNIP>
Phy 0 / Port 2 Interrupt Counts:
Speed Change Intr - 0
Duplex Change Intr - 0
Page Received Intr - 1
AutoNeg Complete Intr - 1
Link Status Changed Intr - 2
Symbol Error Intr - 0
False Carrier Intr - 0
Fifo Overflow/Underflow Intr - 0
Fiber Energy Detect Intr - 0
Frame Count
Incrementing /
Error Counts not
incrementing
Status Counters –
Check for unexpected
counters
2.b – DEPI Sessions Between CMTS and EQAM
• Leverage the following to troubleshoot DEPI tunnels and sessions
• DEPI Tunnels via Control Plane, not manual DEPI
F241-38-05-uBR10K-01#show depi session configured
Session Name State Reason Time
Modular-Cable8/1/0:0 ACTIVE -
Modular-Cable8/1/0:1 ACTIVE -
<SNIP>
Modular-Cable8/1/0:8 IDLE<4> recv CDN Mar 25 21:58:34
Modular-Cable8/1/0:9 IDLE<4> recv CDN Mar 25 21:58:59
<SNIP>
Modular-Cable8/1/1:0 ACTIVE -
Modular-Cable8/1/1:1 ACTIVE –
<SNIP>
Modular-Cable8/1/1:15 ACTIVE -
show depi session configured
F241-38-05-uBR10K-01#show depi session endpoints
DEPI Tunnel RF Channel EQAM RF-Port Tsid State Type
UBR10K_05-GE810 Mod8/1/0:0 Qam1/1.1 8016 est P
UBR10K_05-GE810 Mod8/1/0:1 Qam1/1.2 8017 est P
<SNIP>
UBR10K_05_GE812 Mod8/1/1:0 Qam9/1.1 9101 est P
UBR10K_05_GE812 Mod8/1/1:1 Qam9/1.2 9102 est P
<SNIP>
UBR10K_05_GE812 Mod8/1/1:15 Qam9/1.16 9116 est P
EST (Established)
is the desired
state
show depi session endpoints
Not ACTIVE usually
indicates an issue
Reason will provide
troubleshooting direction
Poor Throughput Issues
CMs Reporting Poor Throughput
• 1. Look at overall interface numbers on CMTS• rf-channel bandwidth sharing, DBS Vs Static
• 2. Per CM DS and US throughput verification2.a - Identify the subscriber's CM having a problem on DS/US (or in both direction)
Perform throughput test
Look at the real numbers on CMTS
Install a test CM at headend on same US/DS interface, if possible
Perform FTP test from server behind CMTS
2.b – SID tracker verification for US throughput issues
Look at the real numbers on CMTS
• 3. Configure DS Bonding Resiliency• DS Bonding resiliency configuration and debugs
Best Way To Troubleshoot Is
1. CMTS Interface Numbers
• Configuration check for rf-channel bandwidth sharing
• ACFE – Adaptive CIR and Fair EIR Recommended
• show interfaces wideband-cable slot/subslot/port:instanceF241-38-05-uBR10K-01#show int wideband-Cable 8/1/1:0
Wideband-Cable8/1/1:0 is up, line protocol is up
Hardware is Wideband CMTS Cable interface, address is 1cdf.0f02.3e38 (bia 1cdf.0f02.3e38)
Description: 16 channel BG
MTU 1500 bytes, BW 600000 Kbit, DLY 1000 usec,
reliability 255/255, txload 12/255, rxload 1/255
Encapsulation MCNS, loopback not set
Keepalive set (10 sec)
ARP type: ARPA, ARP Timeout 04:00:00
Last input never, output 00:00:00, output hang never
Last clearing of "show interface" counters never
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
Interface Wideband-Cable5/0/0:0 queueing strategy: PXF Class-based
30 second input rate 0 bits/sec, 0 packets/sec
30 second output rate 30436000 bits/sec, 2728 packets/sec
0 packets input, 0 bytes, 0 no buffer
Received 0 broadcasts (0 multicasts)
0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort
2944540366 packets output, 4000910044396 bytes, 0 underruns
0 output errors, 0 collisions, 2 interface resets
0 output buffer failures, 0 output buffers swapped out
Total BW and current load of WB
interface
16 x 37.5 Mbps
No output drops
Current
output
rate
No Output Errors
1. Show rf-channel Counters on CMTS
• show controller modular-cable slot/subslot/port counters rf-channel
F241-38-05-uBR10K-01#sh controller modular-Cable 8/1/1 counters rf-channel
Contr RF Pkts Bytes Pkts DMPT DMPT* Sync MAP/UCD Med Pri
oller Chan Tx Tx Dropped Mbps pps Pkts Pkts Pkts
Tx Tx Tx
8/1/1 0 124553781 24379437801 36 25.12318 1100 11316742 226929966 169801
8/1/1 1 124571563 24385303280 36 27.12152 1100 11316820 226299619 169793
8/1/1 2 124578036 24380012584 6958 16.19951 1100 11317748 226317559 169816
8/1/1 3 124592826 24374980200 2 16.19050 1100 11317748 226317555 169815
<SNIP>
8/1/1 16 124570065 24535421656 0 24.32712 1100 11317746 226317511 169803
<SNIP>
Total: 1868968557 365703291136 7039 322.82472 16502 169765281 3394745165 2714168
* Does not include DEPI control plane or DLM packets. Current load on each
channel
MAP/UCD/SYNC Generated by
Cable line card (520,2020, or 3G60
show hw-module bay slot/subslot/port counters wideband-channel
channel
- Look for Tx packets increment / Look for Positive Mpeg Bps
Monitor channel utilization in EQAM (for M-CMTS deployment)
2.a - Per CM DS and US Throughput Verification
• Show cable modem {mac-address | ipaddress } wideband rcs-status
• Make sure CM is not in “Partial Service”
• MAC state will be “p-online(pt)” for DS partial service
Show Commands to be Used
F241-38-05-uBR10K-01#show cable modem 001d.d4d3.3122 service-flow
SUMMARY:
MAC Address IP Address Host MAC Prim Num Primary DS
Interface State Sid CPE Downstream RfId
001d.d4d3.3122 --- C8/1/1/UB w-online(pt) 3 1 Mo8/1/1:12 5220
Forwarding Interface: Wideband-Cable 8/1/1:0
Sfid Dir Curr Sid Sched Prio MaxSusRate MaxBrst MinRsvRate Throughput
State Type
371 US act 181 BE 0 5000000 8192 0 846
372 DS act N/A BE 0 101000000 3044 0 99000124
Show cable modem <mac/ip-add> service-flow
Current throughput
Wideband intf is for
forwarding.
Modular intf. Or Dynamic
WC Intf For WB CM in
partial service mode
US MAP grants and pxf QIDUPSTREAM SERVICE FLOW DETAIL:
SFID SID Requests Polls Grants Delayed Dropped Packets
Grants Grants
371 181 0 0 1323750 0 0 123933740
DOWNSTREAM SERVICE FLOW DETAIL:
SFID RP_SFID QID Flg Policer Scheduler FredIF
Xmits Drops Xmits Drops
372 33153 36355 134787 0 14131275 0 Wi8/1/1:0
No Dropped or
Delayed grants
Forwarding
Interface (Attribute
Mask / DS Bonding
Resil / Narrowband)
2.a - pxf cpu queue stats
F241-38-05-uBR10K-01#show cable modem 001d.d4d3.3122 service-flow verbose
<snip>
Bytes : 190345753
Rate Limit Delayed Packets : 0
Rate Limit Dropped Packets : 0
Current Throughput : 99000124 bits/sec
Application Priority : 0
LC_sfid: 372 rp_sfid: 36355 SF-ByteCount: 104678 SF-PacketCount: 539
SF-ConformXmitBytes: 104678
SF-ConformXmitPkts: 539
SF-ExceedXmitBytes: 0
SF-ExceedXmitPkts: 0
SF-ConformDropBytes: 0
SF-ConformDropPkts: 0
SF-ExceedDropBytes: 0
SF-ExceedDropPkts: 0
Queuing/Jib Tables:
ibus_channel: 49152 ds_key_index: 0 phs_rule: 0 tx_control: 0xD
jib_flag: 0 keyseq_mapcntrl: 0x0 ds_stat_index: 3 min_res_pkt_size: 0
jib_hdr_id: 0x0 docsis_hdr_len: 0 docsis_overhead_len: 12 ds_port: 0
qid: 134788 tx_pkts: 23539 qtail_drop_pkts: 0
LC and RP
sfid
Qtail_dropped packets
show cable modem mac-address service-flow verbose
If exceeding SLA
scheduler will
report drops
2.b – SID Tracker• On PRE or LC, Issue “debug cable interface Cable x/y/z sid <sid> track”
• On LC, Issue “show int Cable x/y/z up debug sid-tracking <sid> 0 40000”clc_8_1#sh int cable 8/1/1 up debug sid-track 3 0 40000
cmts_show_sid_track Cable8/1/1 sid 3
Count: 2490
[0 ]:BWREQ_3 100062974 0-usecs bytes:128 req_id:1 sid:3/ 1 psid 3/ 1 osid 3/ 1 cpu: 12 [1
]:GRANT 100064895 1921-usecs bytes:136 req_id:1 sid:3 chan:1 lo-queue: 0
<snip>
[3566]:BWREQ_3 495227019 14174-usecs bytes:7640 req_id:1342 sid:3/ 1 psid 3/ 1 osid 3/ 3 cpu: 5
[3567]:GRANT 495228817 1798-usecs bytes:1802 req_id:1342 sid:3 chan:1 lo-queue: 0
[3568]:PG 495228819 2-usecs bytes:5846 req_id:1342 sid:3 chan:1
[3569]:GRANT 495228828 9-usecs bytes:1802 req_id:1342 sid:3 chan:2 lo-queue: 0
[3570]:GRANT 495228837 9-usecs bytes:1802 req_id:1342 sid:3 chan:3 lo-queue: 0
[3571]:GRANT 495228845 8-usecs bytes:1802 req_id:1342 sid:3 chan:4 lo-queue: 0
[3572]:GRANT 495234817 5972-usecs bytes:484 req_id:1342 sid:3 chan:1 lo-queue: 0
[3573]:BWREQ_3P 495238448 3631-usecs bytes:1880 req_id:1343 sid:3/ 1 psid 3/ 1 osid 3/ 1 cpu: 4
[3574]:GRANT 495238818 370-usecs bytes:1698 req_id:1343 sid:3 chan:1 lo-queue: 0
[3575]:PG 495238820 2-usecs bytes:190 req_id:1343 sid:3 chan:1
[3576]:GRANT 495242818 3998-usecs bytes:200 req_id343 sid:3 chan:1 lo-queue: 0
[3577]:BWREQ_3 495252594 9776-usecs bytes:7640 req_id:1344 sid:3/ 2 psid 3/ 2 osid 3/ cpu: 5
[3578]:GRANT 495252821 227-usecs bytes:1802 req_id:1344 sid:3 chan:1 lo-queue: 0
[3579]:GRANT 495252832 11-usecs bytes:1802 req_id:1344 sid:3 chan:2 lo-queue: 0
[3580]:PG 495252834 2-usecs bytes:4052 req_id:1344 sid:3 chan:2
[3581]:GRANT 495252841 7-usecs bytes:1802 req_id:1344 sid:3 chan:3 lo-queue: 0
[3582]:GRANT 495252849 8-usecs bytes:1802 req_id:1344 sid:3 chan:4 lo-queue: 0
[3583]:GRANT 495258820 5971-usecs bytes:484 req_id:1344 sid:3 chan:1 lo-queue: 0
US BW request for128 bytes on ch 1
US BW Granted after 1921 usecs on ch 1
Pending
Grant of 5846
7640 bytes
requested
1802 bytes
issued on 1
1802 bytes
granted on ch
2,3 and 4
Piggyback
BW Request
2.b – Service Flow Segments
F241-38-05-uBR10K-01#show cable modem 001d.d4d3.3122 service-flow verbose
<snip>
UPSTREAM SERVICE FLOW DETAIL:
Sfid : 37
Mac Address : 001d.d4d3.3122
Type : Primary
Direction : Upstream
Current State : Active
Current QoS Indexes [Prov, Adm, Act] : [3, 3, 3]
Sid : 3
<SNIP>
US Bonded : YES
Upstream Bonding Group : UBG-811
Transmit Channel Set : 0xF
Sid Cluster : SC-0, Sid [ 3 3 3 3 ]
Sid Cluster : SC-1, Sid [ 4 4 4 4 ]
Segments Valid : 2945
Segments Discarded : 0
Segments Lost : 0
<SNIP>
USCB
modem
No Lost or
discarded
Segments
• Deep diver in to US service flow
For throughput >= 40M
2 sid clusters with 2 max request per sid
For fairly balanced utilization on US channels under one USBG per MD
Configure “cable upstream balance-scheduling” globally
2.b – Bonding Group Counters
F241-38-05-uBR10K-01#sh int c8/1/1 up bonding-group
Cable8/1/1: Upstream Bonding Group 811
4232142430 packets input, 961002075384 octets input
Segments: 2781840856 valid, 32519 discarded, 126663 lost
Reserved Bandwidth Max : 0 bits/sec
Reserved Bandwidth : 0 bits/sec
Available Bandwidth : 55946530 bits/sec
Total Service Flows On This Bonding Group: 542
Bonding Group
number
Bonding Group BW
Stats
Look for Available Bandwidth
F241-38-05-uBR10K-01#show int cable 8/1/1 mac-scheduler 1
DOCSIS 1.1 MAC scheduler for Cable8/1/1/U1: rate 15360000
wfq:None balance-scheduling: off
Queue[Rng Polls] 0/128, 0/0 drops, flows 0 max 0/2/0
Queue[CIR Grants] 0/256, 0/0 drops, flows 0 max 0/1/0 <SNIP>
Queue[BE(1) Grants] 0/96, 25/0 drops, flows 0 max 0/0/0
Queue[BE(0) Grants] 2/96, 0/0 drops, flows 2 max 0/11/0
Avg upstream channel utilization : 26%
UGS : 1 SIDs, Reservation-level in bps 172800
UGS-AD : 0 SIDs, Reservation-level in bps 0
RTPS : 0 SIDs, Reservation-level in bps 0
NRTPS : 74 SIDs, Reservation-level in bps 1049690
BE : 36071 SIDs, Reservation-level in b
Individual US
BW
Drops in BE
flow
Look for drops in scheduler queues
3. DS Bonding Resiliency
• DOCSIS 3.0 allows CMTS to transmit on Primary and NP RF channels
• If CM lose connectivity to Primary RF, CM goes offline
• If CM lose connectivity to NP RF, there will be data loss
• CM informs NP RF failure/recovery via CM-STATUS message
• RBG contains all RFs of original BG except the failed RFs
• Move primary DS Service Flow for CM to its dynamic RBG with 2 or more RFs
• Secondary SFs to dynamic RBG if configured with “Cable rf-change-trigger secondary” command
• CM remains in p-online state for tracking
Bonded CM operation without resiliency
CMTS/CM behavior with DS Bonding Resiliency
3. Config and Debugs for DS-Bonding Resiliency
cable rf-change-trigger percent 75 count 10
!
cable resiliency ds-bonding
interface Wideband-Cable8/1/1:23
cable ds-resiliency
!
interface Wideband-Cable8/1/1:24
cable ds-resiliency
!
interface Wideband-Cable8/1/1:25
cable ds-resiliency
DS Bonding resiliency
enabled under BG
Debugs Useddebug cable wbcmts resiliency
debug cable interface c8/1/1 mac-address 001d.d4d3.3122
Debugs for wideband resiliency
SLOT 8/1: Mar 26 16:40:06.183 EDT: CM 001d.d4d3.3122 n_rfch 15 CM_RFID 5215
SLOT 8/1: Mar 26 16:40:06.183 EDT: r 0 state UP[11] rfid 5208
SLOT 8/1: Mar 26 16:40:06.183 EDT: r 1 state UP[11] rfid 5209
SLOT 8/1: Mar 26 16:40:06.183 EDT: r 2 state UP[11] rfid 5210
<SNIP>
SLOT 8/1: Mar 26 16:40:06.183 EDT: r 14 state UP[11] rfid 5223
SLOT 8/1: Mar 26 16:40:06.183 EDT: r 15 state UP[11] rfid 5224
All channels are up in BG
Debug shows all RF-Channels are UP at the
moment
interface Wideband-Cable8/1/1:0
cable bundle 1
cable rf-channel 0 bandwidth-percent 1
cable rf-channel 1 bandwidth-percent 1
cable rf-channel 2 bandwidth-percent 1
cable rf-channel 3 bandwidth-percent 1
<SNIP>
cable rf-channel 15 bandwidth-percent 1
Static Bonding Group
(Not the DS Bonding
Resiliency BG)
Rf-change-trigger % and count of CM DS Resiliency Configuration
Global Configuration
Required
3. DS Bonding Resiliency Debugs
SLOT 8/1: Mar 26 16:40:13.203 EDT: handle_wb_rf_resil_event: 001d.d4d3.3122 n_rfch 15, event 2 n_ds_chid 1
SLOT 8/1: Mar 26 16:40:13.203 EDT: ds_chid 200 mc_info channel_id 200
SLOT 8/1: Mar 26 16:40:13.203 EDT: send_docsis_resil_event_trap: now sending docsis_resil event trap.
SLOT 8/1: Mar 26 16:40:13.203 EDT: no permit, bit=80, bitmap=0
SLOT 8/1: Mar 26 16:40:16.191 EDT: CM 001d.d4d3.3122 n_rfch 15 CM_RFID 5215
SLOT 8/1: Mar 26 16:40:16.191 EDT: r 0 state DOWN_PENDING[41] rfid 5208
<SNIP>
SLOT 8/1: Mar 26 16:40:16.191 EDT: r 14 state UP[11] rfid 5223
SLOT 8/1: Mar 26 16:40:16.191 EDT: r 15 state UP[11] rfid 5224
One DS Channel down
Channel went down for CM
because of impairments
018110: Mar 26 16:41:26.343 EDT: RESIL-IPC-RP: 001d.d4d3.3122, receiving 757 bytes
018111: Mar 26 16:41:26.343 EDT: RESIL-RP: message type 1
018112: Mar 26 16:41:26.343 EDT: RESIL-RP: tlv_len 740, RESIL-RP: bitmask down: 24
018113: Mar 26 16:41:26.343 EDT: RESIL-RP: current_interface 6952
018114: Mar 26 16:41:26.343 EDT: RP GOT REQUEST TO MOVE CM
<SNIP>
Original active RF members: 24-39
Needed RF members: 25-39
Down RF members: 24
Avail RF members: 25-39
018139: Mar 26 16:41:26.347 EDT: Find Best DBG: for 8/1/1:7 needed RF member: 25-39
018140: Mar 26 16:41:26.347 EDT: cmts_rf_resil_rp_dbg_get_unused(): WB Index checking match 8/1/1:8
018141: Mar 26 16:41:26.347 EDT: cmts_rf_resil_rp_dbg_get_unused(): WB Index was found to be free 8/1/1:8
Found free DBG to use,requesting create RF member: 25-39
018142: Mar 26 16:41:26.347 EDT: Creating Dyn WB interface 8/1/1:8 with bundle 1
Needed RF: 25-39
<SNIP>
018157: Mar 26 16:41:26.351 EDT: WB msg type 169 sent to LC 8/1
018158: Mar 26 16:41:26.351 EDT: %SNMP-5-LINK_UP: LinkUp:Interface Wideband-Cable8/1/1:8 changed state to up
RBG comes up with remaining channels
RP to look for RBG for
Wi 8/1/1:0
Dynamic WB intf. Created for
RBG
WB RBG with
remaining chans.
Comes up
Modem Resiliency
move necessary
3. DS Bonding Resiliency show commands
interface Wideband-Cable8/1/1:8
cable bundle 1
cable ds-resiliency
cable rf-channel 1 bandwidth-percent 1
cable rf-channel 2 bandwidth-percent 1
<SNIP>
cable rf-channel 15 bandwidth-percent 1
Resiliency WB running config
interface Wideband-Cable8/1/1:9
cable bundle 1
cable ds-resiliency
cable rf-channel 0 bandwidth-percent 1
cable rf-channel 2 bandwidth-percent 1
<SNIP>
cable rf-channel 15 banddith-percent 1
F241-38-05-uBR10K-01#show cable resiliency
BG Resil BG RF
Resil BG I/F ID State Count Time Ctrl Num
------------- ---- -------------- ----- --------------- ----------
Wi8/1/1:8 6953 Assigned 3 Mar 26 16:41:26 1 1
2
…
15
Wi8/1/1:9 6954 Assigned 1 Mar 26 17:11:32 1 0
…
15
Show cable resiliency
Current chans in a RBG, 15 channel (sans rf-ch 0)
Show cable modem resiliencyF241-38-05-uBR10K-01#show cable modem resiliency
Orig BG Curr BG
I/F MAC Address ID I/F RFs ID I/F RFs
------- -------------- ---------------------- ----------------------
C8/1/1 001d.d4d3.3122 6952 Wi8/1/1:7 16 6953 Wi8/1/1:8 15
C8/1/1 001d.d4d3.31d2 6952 Wi8/1/1:7 16 6954 Wi8/1/1:9 15
Channels in old and
new BG for a CM
F241-38-05-uBR10K-01#show cable modem partial-service
MAC Address IP Address I/F MAC DSxUS
Impaired Impaired State State DS US
001d.d4d3.3122 --- C8/1/1/UB p-online(pt)15x4
001d.d4d3.31d2 --- C8/1/1/UB p-online(pt)15x4
Show cable modem partial-service
Logical Suspend Suspend Flap Flap
RF Status Status Fails Count Time
-------- ------- ------- ----- ----- ----
8/1/1 0 DOWN N/A 0 22 Mar 24 19:15:57
1 UP N/A 0 3 Mar 24 19:15:57
2 UP N/A 0 0
<SNIP>
15 UP N/A 0 0
Show cable rf-status1st Channel went
down
Flap Fail and
Count
• CMTS Based Services Evolution
• Troubleshooting High Speed Data• DOCSIS 3.0 DS and US Channel Bonding Issues
• Troubleshooting DOCSIS Timing Issues
• Troubleshooting DOCSIS Load Balancing
• Troubleshooting Voice Services• Troubleshooting Voice Subscriber Issues
• Troubleshooting SNMP Services
• Troubleshooting DSG Services
• Summary
• Q & A
Agenda
Troubleshooting DOCSIS Timing Issues
• DOCSIS timing issues could be a HUGE reason for • Subscribers Issues
• Subscribers reporting sluggish throughput performance
• New CMs not coming up online
• CMs fall offline gradually on LCs or on CMTS
• Hardware issues
• N+1 Redundancy in RFGW-10 not working
• DS384 cards in RFGW-10 not booting up or stuck in boot cycle
• Possible Reasons
• DOCSIS clock issues on M-CMTS and RFGW-10
• Stable DOCSIS clock source not present on RFGW-10
M-CMTS DOCSIS Timing Architecture• DTI servers should be installed in root/slave mode
•Root server providing timing to DTI clients and Slave DTI server
•Slave server receiving clock from Root and providing clocking to DTI clients
• DTI clients (M-CMTS+RFGW-10) should have clock from Root and Slave server
F241-38-05-uBR10K-01#show cable clock dti status
Status of DTI component:
Active TCC card in slot 2/1
TCC Card 1/1 DTI status:
-------------------------------------------
Active Client port : 2
Active Client status : normal
Active Client Server status : freerun
Active Client frame error rate : < 2%
Active Client CRC error count : 0xAD
Standby Client Signal detected : yes
TCC Card 2/1 DTI status:
-------------------------------------------
Active Client port : 2
Active Client status : normal
Active Client Server status : freerun
Active Client frame error rate : < 2%
Active Client CRC error count : 0x15
Standby Client Signal detected : yes
Show cable clock status on M-CMTS
DOCSIS Clock Verification on M-CMTS
error rate has to be < 2%
DTI Port 2 is in Normal
Active Client status as warmup, fast, bridging, and holdover signals timing issue
CMTS getting clock from TCC in 2/1
DOCSIS Clock Verification on M-CMTS..Contd
Issue following commands on CMTS and RFGW-10
show log | include DTCC
show log | include TCC
show log | inc DTI
show log | inc DPLL (only on RFGW-10)
show log | inc UPX (only on RFGW-10)
Verify log/syslogs messages
Nov 12 20:28:43.121: %UBR10KTCC-1-DTIPORTCLIENT: DTCC slot 1/1 port 1 DTI client status changed to
Active normal
Nov 12 20:28:43.121: %UBR10KTCC-1-DTIPORTCLIENT: DTCC slot 1/1 port 2 DTI client status changed to
Standby
Nov 12 20:28:52.974: %SNMP-5-LINK_DOWN: LinkDown:Interface DTI1/1/0 changed state to down
Port 2 is identified as DTI/1/1/1 and Port1 as DTI1/1/0
Port DTI 1/1/0 went down
DTCC Port down on M-CMTS
Nov 12 20:31:41.711: %UBR10KTCC-1-DTIPORTPHYS: DTCC slot 1/1 port 2 DTI physical status changed:
Link status:Link detected, frame error rate: > 5%
Nov 12 20:31:42.711: %UBR10KTCC-1-DTIPORTPHYS: DTCC slot 1/1 port 2 DTI physical status changed:
Link status:Link detected, frame error rate: < 2%
Nov 12 20:31:42.984: %SNMP-5-LINK_UP: LinkUp:Interface DTI1/1/1 changed state to up
Port 2 on DTI 1/1 become UPDTCC Port come back up from Standby
Mar 13 19:41:37.672: %IPCOIR-3-TIMEOUT: Timeout waiting for a response from slot 2/1.
Mar 13 19:41:37.672: %IPCOIR-2-CARD_UP_DOWN: Card in slot 2/1 is down. Notifying 2cable-dtcc
driver.
Mar 13 19:41:37.672: %SNMP-5-LINK_DOWN: LinkDown:Interface DTI2/1/0 changed state to down
Mar 13 19:41:37.672: %SNMP-5-LINK_DOWN: LinkDown:Interface DTI2/1/1 changed state to down
DTI card Down on M-CMTS
Mar 13 19:41:38.384: %UBR10KTCC-2-ACTIVE_TCC: DTCC card 1/1 is active with DTI as clock reference
UC DTI: Notification sent to DTI module to set card 1/1 as active
UC Set role: card 1/1 role set to Active
Mar 13 19:41:38.384: %UBR10KTCC-4-CHG_CLK_REF: Clock reference source set to DTI for DTCC card
1/1 by process Card Config
Mar UC DTI: Notification sent to DTI module to set card 1/1 as active
UC Set role: card 1/1 role set to Active
Mar 13 19:41:38.680: %UBR10KTCC-4-CHG_CLK_REF: Clock reference source set to DTI for DTCC card
1/1
Standby card come back up
DOCSIS Clock Verification on M-CMTS..Contd
F241-38-03-RFGW10-02#sh cable clock
DTI Client status: TCC 13 - Active
-----------------
Client status : normal
Client clock type : ITU stratum 3
Client firmware version : 0x00000114
Client dti version : 1
Client transition t3 count : 0
Client transition t4 count : 1
Client transition t6 count : 0
Client transition t7 count : 0
<snip>
DTI Client Port 1 Status:
-------------------------
Port Status : Active
Signal detected : yes
CRC error count : 182
Frame error rate : < 2%
Cable advance : 0x0500
-- Connected server information ---Server
status : free-run
Root Server clock type : ITU type 1
Show cable clock status on RFGW-10
DOCSIS Clock verification on RFGW-10
Transition counters T3-T7 indicates clocking instability to client
DTI Client status: TCC 14 - Standby
-----------------
Client status : normal
Client clock type : ITU stratum 3
Client firmware version : 0x00000114
Client dti version : 1
Client transition t3 count : 0
Client transition t4 count : 1
Client transition t6 count : 0
Client transition t7 count : 0
Client port switch count : 1
Client Integral Frequency Term : -15704
Client EFC Value : -31333
DTI Client Port 1 Status:
-------------------------
Port Status : Active
Signal detected : yes
CRC error count : 3
Frame error rate : < 2%
Cable advance : 0x0F00
-- Connected server information ---
Server status : normal
Root Server clock type : ITU type 1
F241-38-03-RFGW10-02#show Inventory
Supervisor Led Color : Green
Module 1 Status Led Color : Green
Module 2 Status Led Color : Green
Module 3 Status Led Color : Red
Module 4 Status Led Color : Green
Module 5 Status Led Color : Green
Module 6 Status Led Color : Red
Module 7 Status Led Color : Red
Module 8 Status Led Color : Red
Module 9 Status Led Color : Green
Module 12 Status Led Color : Red
Module 13 Status Led Color : Off
Module 14 Status Led Color : Yellow
DS384 cards down on RFGW-10Hardware issues on RFGW-10 and CMTS
Mar 5 22:47:30.447 Eastern: %RFGW-3-LINECARD_ERRMSG_ERR: SLOT
12:BB_ERRMSG_DS_PHY_UPX_BOOTUP_OK_ERR_REASON:FWReady:failed to
detect a FW Ready signal from the Dogfish FPGA.
Mar 5 22:47:30.447 Eastern: %RFGW-3-LINECARD_ERRMSG_ERR: SLOT
12:BB_ERRMSG_HW_CTRL_INIT_FAILED: bb_hw_control: fpga setup
failed
Nov 5 22:47:30.449 Eastern: %RFGW-3-LINECARD_ERRMSG_ERR: SLOT
12:BB_ERRMSG_HW_CTRL_INIT_FAILED: bb_hw_control: HW
Mar 5 22:47:30.989 Eastern: %RFGW-2-LINECARD_ERRMSG_CRIT: SLOT
12:ERROR, process bb_hw_control launch too many times (1)
DS384 card failed to boot up in slot 12
Mar 5 22:58:59.494 UTC: %UBR10KTCC-6-DTIPORTCLIENT: DTCC slot 1/1 port 1 DTI client status changed to
Active bridging
Mar 5 22:58:59.494 UTC: %UBR10KTCC-5-DTISLOT: DTCC slot 2/1: card role changed to Active
Mar 5 22:59:00.494 UTC: %UBR10KTCC-5-DTISLOT: DTCC slot 1/1: card role changed to Standby
Mar 5 23:00:02.894 UTC: %UBR10KTCC-6-DTIPORTCLIENT: DTCC slot 1/1 port 1 DTI client status changed to
Active normal
Mar 5 23:05:53.243 UTC: %UBR10KTCC-6-DTIPORTCLIENT: DTCC slot 1/1 port 1 DTI client status changed to
Active bridging
Mar 6 00:28:41.747 UTC: %UBR10KTCC-6-DTIPORTCLIENT: DTCC slot 1/1 port 1 DTI client status changed to
Active normal
DTCC continuously changing status
Mar 4 06:41:12.109 UTC: %RFGW-0-LINECARD_ERRMSG_EMERG: SLOT
12:LC_ERRMSG_FAILOVER_TRIGGER UPX Hi priority event (SET):
module:1 idx:26 desc:DF VCXO 32M DPLL not locked,
DS384 cards reporting UPX/DPLL errors
• CMTS Based Services Evolution
• Troubleshooting High Speed Data• DOCSIS 3.0 DS and US Channel Bonding Issues
• Troubleshooting DOCSIS Timing Issues
• Troubleshooting DOCSIS Load Balancing
• Troubleshooting Voice Services• Troubleshooting Voice Subscriber Issues
• Troubleshooting SNMP Services
• Troubleshooting DSG Services
• Summary
• Q & A
Agenda
Troubleshooting DOCSIS Load Balancing issues
1. DOCSIS 3.0 CM not placed in proper DOCSIS LB group
1.a – Improper DOCSIS LB configuration
1.b - Special RLBG consideration
Common Issue
1.a – Improper Configuration Checklist
cable load-balance docsis-enable
cable load-balance rule 1 disable-throughput-lowerbound us 100
cable load-balance rule 2 disable-throughput-lowerbound ds 1000
cable load-balance docsis-policy 1 rule 1
cable load-balance docsis-policy 1 rule 2
Docsis LB rule definition
Docsis LB config and rule
cable tag 110
name HSD_BIZ
service-class us_hsd_biz
!
cable tag 101
name HSD_RES
service-class us_hsd_res
!
service-class name us_hsd_res
cable service class 101 name us_hsd_res
cable service class 101 upstream
cable service class 110 name us_hsd_biz
cable service class 110 upstream
Service Class
for business
service
Tag and service class definition
Service Class
for residential
service
cable fiber-node 2
description SAMPLE_NODE
downstream Modular-Cable 8/0/1 rf-channel 0-3 20-23
upstream Cable 8/0 connector 18
Fiber-Node configurationDS and US
chans. In FN
cable load-balance docsis-group 26 index 87
restricted
downstream Modular-Cable 8/0/1 rf-channel 12-23
upstream Cable8/0/1 3-4
method utilization
threshold load 30
policy pure-ds-load
init-tech-list 3
docsis-policy 1
tag HSD_RES
RLBG with HSD_RES tag
cable load-balance docsis-group 25 index 86
restricted
downstream Modular-Cable 8/0/1 rf-channel 0-11
upstream Cable8/0/1 0-2
method utilization
threshold load 30
policy pure-ds-load
init-tech-list 3
docsis-policy 1
tag HSD_BIZ
RLBG configuration
RLBG with HSD_BIZ tag
DS RF channels 0-11
Show cable load-balanceDOCSIS 3.0 LB Enabled: Yes
DOCSIS Group Status Interval DCC mask Policy Method Threshold
Group Index /UCC DS/US M/E/U/P/S
1 81 RE 30 0x10(3)/N 1 u/u 10/30/70/70/50
25 86 RE 30 0x10(3)/N 1 u/u 10/30/70/70/50
61 82 GE 50 0x10(3)/N 0 u/u 5/10/70/70/50
1.a – Show commands and debugs1.a – Show commands and debugs
Target interface State Trans Fail Fail Total Max-
Cmple Pend Retry (NB) (WB) Attem Fail
Mo1/0/0:0 (615 MHz) up 13 1 0 2 - 13 2
Mo1/0/0:0/U0 up 0 0 0 0 - 0 0
Mo1/0/0:0/U1 up 0 0 0 0 - 0 0
Mo1/0/0:0/U2 up 0 0 0 0 - 0 0
Mo1/0/0:0/U3 up 0 0 0 0 - 0 0
Show cable load-balance statistics
Interface State Group Target
Index
Mo1/0/0:0 (615 MHz) up 4
Mo1/0/0:0 (615 MHz) up 33539 Mo1/0/0:4 (639 MHz) [enforce]
Mo1/0/0:0/U0 up 4
Mo1/0/0:0/U0 up 33539 Mo1/0/0:0/U1 [enforce]
Mo1/0/0:0/U1 up 4
Show cable load-balance target
Feb 14 20:13:37:250 lb: CM 0025.2eab.87ac stid not configured. Get LBGID 0
Feb 14 20:13:37:250 lb: CM 0025.2eab.87ac try to set LBG by tag HSD_BIZ
Feb 14 20:13:37:250 lb: D3.0 modem 0025.2eab.87ac is in FN 2
Feb 14 20:13:37:250 lb: CM 0025.2eab.87ac clear group 86 by RLBG FN
Feb 14 20:13:37:250 lb: Assign 3.0 GLBG 62535 to CM 0025.2eab.87ac md_cm_sg 7602433
SLOT 8/0: Feb 14 20:13:37.207 PST: Found Network Access TLV
SLOT 8/0: Feb 14 20:13:37.207 PST: Ntw Access Control : 1
SLOT 8/0: Feb 14 20:13:37.207 PST: Found Max CPEs TLV
SLOT 8/0: Feb 14 20:13:37.207 PST: Maximum Number Of CPEs : 17
SLOT 8/0: Feb 14 20:13:37.207 PST: Found Upstream Service Flow TLV
SLOT 8/0: Feb 14 20:13:37.207 PST: Service Flow Reference : 1
SLOT 8/0: Feb 14 20:13:37.207 PST: QoS Parameter Set Type : 0x7
SLOT 8/0: Feb 14 20:13:37.207 PST: Service Class Name : us_hsd_biz
SLOT 8/0: Feb 14 20:13:37.207 PST: Found Downstream Service Flow TLV
Debug cable mac-address <mac of CM>
verbose
Debug cable load-balance
Debug cable load-balance docsis-lb
1.a – Show commands and debugs
CM registering with us_hsd_biz
Service Class
CM assigned to GLBG ?
Debugs needed for DLB issues
1.a – Show commands and debugs
DOCSIS load-balancing load
Interface State Group Utilization Rsvd NBCM WB/UB Flows
Weight
Index Total Total
Mo1/0/0:0 (615 MHz) up 4 10%(10%/14%) 2% 93 135 136 36.0
Mo1/0/0:0 (615 MHz) up 33539 14%(10%/14%) 2% 93 135 136 36.0
Mo1/0/0:0/U0 up 4 14% 3% 96 144 127 15.3
Mo1/0/0:0/U0 up 33539 14% 3% 96 144 127 15.3
Mo1/0/0:0/U1 up 4 12% 2% 49 148 80 15.3
Mo1/0/0:0/U1 up 33539 12% 2% 49 148 80 15.3
Show cable load-balance load
Debugs during Registration for WB CM bonding on 8/0/1 rf chan 0-3
Modem assigned to GLBG instead of RLBG..Why?
SLOT 8/0: Feb 14 10:42:05.997 PST: lb: Assign CM 0025.2eab.87ac LBG ID 0
SLOT 8/0: Feb 14 10:42:05.997 PST: lb: CM 0025.2eab.87ac stid not configured. Get LBGID 0
SLOT 8/0: Feb 14 10:42:05.997 PST: lb: CM 0025.2eab.87ac try to set LBG by tag HSD_BIZ
SLOT 8/0: Feb 14 10:42:05.997 PST: lb: D3.0 modem 0025.2eab.87ac is in FN 2
SLOT 8/0: Feb 14 10:42:05.997 PST: lb: Assign CM 0025.2eab.87ac LBG ID 86
SLOT 8/0: Feb 14 10:42:05.997 PST: lb: CM 0025.2eab.87ac Grp 86 Policy ID 1 from LB Group
cable load-balance docsis-group 25 index 86
restricted
downstream Modular-Cable 8/0/1 rf-channel 0-11
tag HSD_BIZ
cable load-balance docsis-group 25 index 86
restricted
downstream Modular-Cable 8/0/1 rf-channel 0-3
tag HSD_BIZ
MAC Address : 0025.2eab.87ac
<snip>
Downstream Channel DCID RF Channel : 150 8/0/1:0
Downstream Channel DCID RF Channel : 149 8/0/1:1
Downstream Channel DCID RF Channel : 151 8/0/1:2
Downstream Channel DCID RF Channel : 152 8/0/1:3
LB group ID assigned (index) : 25 (86)
LB group ID in config file (index) : N/A (N/A)
LB policy ID : 1
LB policy ID in config file : 0
LB priority : 0
Tag : HSD_BIZ
1.b – Special RLBG consideration
CM assigned to proper RLBG
CM tagged properly
We need 0-3 DS chans. in RLBGInstead of 0-11 DS chans. in RLBG
Channels in RLBG has to be the Subset of FN, not Superset
Load Balance Debugs during registration
Show cable modem <mac> verbose
• CMTS Based Services Evolution
• Troubleshooting High Speed Data• DOCSIS 3.0 DS and US Channel Bonding Issues
• Troubleshooting DOCSIS Timing Issues
• Troubleshooting DOCSIS Load Balancing
• Troubleshooting Voice Services• Troubleshooting Voice Subscriber Issues
• Troubleshooting SNMP Services
• Troubleshooting DSG Services
• Summary
• Q & A
Agenda
PacketCable™
The PacketCable architecture defines a platform to deliver Voice-over-Internet Protocol (VoIP) telephone service over a DOCSIS cable network
PacketCable™ Reference Architecture
Backbone
HFC Plant
RKS
LNP
STP
PSTN
Signaling GW
CMS/SoftSwitch MGC
DOCSIS
CMTS
Voice Path
Billing Events
VMCONFSRV
ANNSRV
Media Servers
CALEA
NCS EMTA (NID)
NCS EMTA
CM
NCS MTA
Provisioning Server
Provisioning
BAC
CUST. DB
DNS/DHCPTFTP
TODKDC
Signaling
Voice GatewayCisco uBR10012
V
BTS10200
Common Voice Subscriber Issues
• 1 - No dial tone1.a - EMTA not registered 1.b - Signaling flow not present1.c - High upstream utilization
• 2 - No audio and One way audio2.a - Gate establishment issues2.b - Dynamic service flow establishment issues2.c - Other possible issues
Not enough BW for DSHigh US utilizationLow SNR in HFC
• 3 - Choppy audio3.a – Service Flow Bandwidth Issues3.b – PXF drops
PacketCable™ DQoS
HFC PlantCMTSNCS EMTA
Initiate Call
Signaling
(NCS)
Allocate/Set
Gate, Start
Gate Timers
Continue Call
Signalig, w/
GateID
DSA, DSC with GateID
CMTS verifies DSX against
Gate, stops Gate Timers
DOCSIS 1.1
UGS Active !
CMS
1 – No Dial Tone
• 1.a - Check if the CM is online and MTA has IP add.
•
»nRTPS recommended for Signaling Flow
EMTA Not Registered and Signaling Flow Issues
CMTS#show cable modem 0000.cadd.a3ef
MAC Address IP Address I/F MAC Prim RxPwr Timing Num BPI
State Sid (dB) Offset CPE Enb
0000.cadd.a3ef 10.1.1.1 C8/0/2/U1 online 4 0.50 1515 1 N
show cable modem 0000.cadd.a3ef cpe
IP address MAC address
14.80.82.141 0000.cadd.a3f0 IP of MTA
CM is online
CMTS#show cable modem 0000.cadd.a3ef qos
Sfid Dir Curr Sid Sched Prio MaxSusRate MaxBrst MinRsvRate Throughput
State Type
108 US act 4 BE 1 2000000 3044 0 0
131 US act 63 NRTPS 4 64000 3044 32000 197
109 DS act N/A BE 1 6600000 12000 0 5
132 DS act N/A BE 4 64000 96000 32000 110
nRTPS signaling flow with
priority 4
• 1.b - Signaling flow not presentIssue “show cable modem mac-address service-flow”
1 – No Dial Tone
1.c - Check for Avg. US utilization
‒ show int cable 8/0/2 mac-scheduler 1
DOCSIS 1.1 MACscheduler for Cable8/0/2/U1
<snip>
Avg upstream channel utilization : 96%
Many things needs to be checked on BTS/CA
‒ Outside the scope
High US Utilization
2 – No Audio and One Way Audio
• show packetcable gate summary
• Other possible states : ALLOC, AUTH, RSVD,INVLD and UNKWN
• Debugs needed to check gate establishment• debug packetcable subscriber mta-ip verbose
• debug packetcable gate database
• debug packetcable gate control
• debug packetcable gate docsis-mapping
• debug packetcable gate commit
2.a – Gate Establishment Checking
CMTS#show packetcable gate su
GateID i/f SubscriberID GC-Addr State Type SFID(us) SFID(ds)
6576 Ca6/1/0 14.80.82.141 14.80.85.36 COMMIT DQoS 91 92
22940 Ca6/1/0 14.80.82.144 14.80.85.36 COMMIT DQoS 93 94
Total number of gates = 2
Total Gates committed(since bootup or clear counter) = 17
Gate has to be
in COMMIT
state
2.b – Service Flow Establishment
• DSX messages for dynamic service flow establishment
• Debugs for DSX messages and verification
debug cable mac-address cm-mac verbose
debug cable tlv
debug cable qos
debug cable dynsrv
DSX Messages
2.b – Dynamic Service Flow – Sample DebugsMar 9 19:28:49.792: DSA-REQ-RECD: OrgMac->0013.1050.3801 OrgId->89
Mar 9 19:28:49.792: DSA-STATE-CREATED: OrgMac->0013.1050.3801 OrgId->89
Mar 9 19:28:49.796: Found Upstream Service Flow TLV
Mar 9 19:28:49.796: Service Flow Reference : 1
Mar 9 19:28:49.796: QoS Parameter Set Type : 0x2
Mar 9 19:28:49.796: Scheduling Type : 6
Mar 9 19:28:49.796: Request/Transmission Policy : 0x17F
Mar 9 19:28:49.796: Unsolicited Grant Size : 232
Mar 9 19:28:49.796: Nominal Grant Interval : 20000
Mar 9 19:28:49.796: Tolerated Grant Jitter : 800
Mar 9 19:28:49.796: Grants Per Interval : 1
Mar 9 19:28:49.796: Found Upstream Packet Classifier TLV
Mar 9 19:28:49.796: Classifier Reference : 1
Mar 9 19:28:49.796: Service-Flow Reference : 1
Mar 9 19:28:49.796: Rule Priority : 128
Mar 9 19:28:49.796: Activation State : 0
Mar 9 19:28:49.796: Found IP Packet Classifier Sub-TLV
Mar 9 19:28:49.796: Protocol : 17
Mar 9 19:28:49.796: Source Address : 24.34.240.235
Mar 9 19:28:49.796: Destination Address : 24.34.240.247
Mar 9 19:28:49.796: Source Port Start : 53456
Mar 9 19:28:49.796: Source Port End : 53456
Mar 9 19:28:49.796: Destination Port Start : 53456
Mar 9 19:28:49.796: Destination Port End : 53456
DSA REQ Received
Mac-add of CM
Classifier not active yet
Admit Service Flow only
US Scheduling type UGS
Std. UGS size for G.711/20ms
160+40(ip)+18(eth)+14(docsis)
20 ms grant interval
RTP port numbers
2.b – Sample Debugs..contd..Mar 9 19:28:49.796: Found Downstream Service Flow TLV
Mar 9 19:28:49.796: Service Flow Reference : 2
Mar 9 19:28:49.796: QoS Parameter Set Type : 0x2
Mar 9 19:28:49.796: Traffic Priority : 5
Mar 9 19:28:49.796: Maximum Sustained Traffic Rate : 87200
Mar 9 19:28:49.796: Maximum Traffic Burst : 1522
Mar 9 19:28:49.796: Minimum Reserved Traffic Rate : 87200
Mar 9 19:28:49.796: Minimum Reserved Rate Packet Size : 218
Mar 9 19:28:49.796: Found Downstream Packet Classifier TLV
Mar 9 19:28:49.796: Classifier Reference : 2
Mar 9 19:28:49.796: Service-Flow Reference : 2
Mar 9 19:28:49.796: Rule Priority : 128
Mar 9 19:28:49.796: Activation State : 0
Mar 9 19:28:49.796: Found IP Packet Classifier Sub-TLV
Mar 9 19:28:49.796: Protocol : 17
Mar 9 19:28:49.796: Source Address : 24.34.240.247
Mar 9 19:28:49.796: Destination Address : 24.34.240.235
Mar 9 19:28:49.796: Destination Port Start : 53456
Mar 9 19:28:49.796: Destination Port End : 53456
Mar 9 19:28:49.796: Auth Block:
Mar 9 19:28:49.796: 0x0000: 01 06 01 04 00 00 14 3E
Mar 9 19:28:49.796: Sfref = 1, SFID = 103 <- Service Flow IDs assigned by CMTS
Mar 9 19:28:49.796: Sfref = 2, SFID = 104
Mar 9 19:28:49.796: Cfr-ref = 1, CFID = 33, SF-ref 1, SFID 103
Mar 9 19:28:49.796: Cfr-ref = 2, CFID = 34, SF-ref 2, SFID 104
Mar 9 19:28:49.796: Added Auth Block(633A2828) len=14
Mar 9 19:28:49.796: DSA-RSP-SENT: CM->0013.1050.3801 TranscId->89 ConfCode->0
Mar 9 19:28:49.896: DSA-ACK-RECD: OrgMac->0013.1050.3801 OrgId->89 ConfCode->0
Mar 9 19:28:50.196: DSA-REQ End : Transaction over-T8 timer expired. OrgMac->0013.1050.3801 OrgId->89
Mar 9 19:28:50.196: DYN-SRV-STATE-DESTROYED : OrgMac->0013.1050.3801 OrgId->89
Admit Service Flow only
DS service flow with high
priority
DQOS Gate ID contained here
SFID assigned for US and DS
DSA Response sent and ACK
received
2.b – DSX Message Stats
• Use “show controller cable slot/subslot/port”
Cable8/1/1 Downstream is up
Frequency 615.0000 MHz, Channel Width 6 MHz, 256-QAM, Symbol Rate 5.360537 Msps
FEC ITU-T J.83 Annex B, R/S Interleave I=32, J=4
Downstream channel ID: 119
<snip>
Dynamic Services Stats:
DSA: 0 REQs 21368 RSPs 0 ACKs
0 Successful DSAs 41 DSA Failures
DSC: 0 REQs 24761 RSPs 0 ACKs
0 Successful DSCs 0 DSC Failures
DSD: 38 REQs 21335 RSPs
0 Successful DSDs 37 DSD Failures
DBC: 0 REQs 0 RSPs(Rcvd) 0 ACKs
0 Successful DBCs 0 DBC Failures 0 DBC Partial
0 DBC Protocol Violations
DSA failures has to be 0
DSC failures has to be 0
DSD failures has to be 0
Only RSPs captured under DS
Use “show controller cable slot/subslot/port upstream us-number” for DSA, DSC and DSD Request stats
2.c – Other Possible Issues• Not enough bandwidth available for DS
• Issue “show interface cable x/y/z downstream”• Look for “Total downstream reserved/reservable bandwidth”
• High CPU for PRE and/or LC• Issue “show proc cpu” on PRE and on affected LC• Excessive SNMP polling• Excessive DS Bonding Resiliency activity
• Higher Upstream channel utilization• Issue “show interface cable x/y/z mac-schedular j”
Look for “Avg upstream channel utilization” and other flow related BW reservation
• Uncorrectable FEC and CRC errors should be between 1-5%
• Low SNR in HFC network• Issue “show controller cable x/y/z”
• Look for “US phy MER(SNR)_estimate for good packets” around 25db+
3 – Choppy Audio Issues
• Typically happens because of dropped packets
• Check for dynamic service flows for eMTA and packet count
• Use “show cable modem <mac/ip-add> service-flow” Or
show interfaces c5/0/0 service-flow | inc dyn | inc eeb6
Sfid Sid Mac Address QoS Param Index Type Dir Curr Active
Prov Adm Act State Time
142 77 0000.cad6.eeb6 0 10 10 dyn(S) US act 22:11
143 N/A 0000.cad6.eeb6 0 11 11 dyn(S) DS act 22:11
3.a – Service Flow Bandwidth Issues
3.a – Choppy Voice – Show Service-FlowF241-38-05-uBR10K-01#show cable modem 0000.cad6.eeb6 service-flow verbose
Sfid : 142
Mac Address : 0000.cad6.eeb6
Type : Secondary(Dynamic)
Direction : Upstream
Current State : Active
Current QoS Indexes [Prov, Adm, Act] : [0, 10, 10]
Active Time : 23:16
Sid : 77
Admitted QoS Timeout : 200 seconds
Current Throughput : 87200 bits/sec, 50 packets/sec
Application Priority : 3
Classifiers:
Classifier Id : 78
Service Flow Id : 142
CM Mac Address : 0000.cad6.eeb6
Direction : upstream
Activation State : active
Classifier Matching Priority : 128
PHSI : 0
Number of matches : -
IP Classification Parameters:
IP Source Address : 14.80.82.141
Source IP Address Mask : 255.255.255.255
Destination IP Address : 14.80.82.7
Destination IP Address Mask : 255.255.255.255
US dynamic service flow
US Service Flow
218 bytes@50 PPS=87200 bps
160(voice)+40(ip)+18(eth)x8(bits)x
50
Source IP of US flow
Destination IP of US flow
3.a – Choppy Voice – Show Service-FlowF241-38-05-uBR10K-01#show cable modem 0000.cad6.eeb6 service-flow verbose
Sfid : 143
Mac Address : 0000.cad6.eeb6
Type : Secondary(Dynamic)
Direction : Downstream
Current State : Active
Current QoS Indexes [Prov, Adm, Act] : [0, 11, 11]
Active Time : 24:02
Sid : N/A
Traffic Priority : 5
Minimum Reserved Rate : 87200 bits/sec
Admitted QoS Timeout : 200 seconds
Current Throughput : 87254 bits/sec, 50 packets/sec
Application Priority : 3
Classifiers:
Classifier Id : 79
Service Flow Id : 143
CM Mac Address : 0000.cad6.eeb6
Direction : downstream
Activation State : active
Classifier Matching Priority : 128
PHSI : 0
Number of matches : 72112
IP Classification Parameters:
IP Source Address : 14.80.82.7
Source IP Address Mask : 255.255.255.255
Destination IP Address : 14.80.82.141
Destination IP Address Mask : 255.255.255.255
DS dynamic service flow
DS Service Flow
Source IP of DS flow
Destination IP of DS flow
Min Reserve rate
Current throughput
High Priority for DS flow
3.b – Choppy Voice – PXF Drops
• Get the DS pxf_qid using command
• Show cable modem <mac/ip address> service-flow (verbose)
DOWNSTREAM SERVICE FLOW DETAIL:
SFID RP_SFID QID Flg Policer Scheduler FrwdIF
Xmits Drops Xmits Drops
143 37756 136278 14926 0 14926 0 Mo1/1/0:0
• Use “Show pxf cpu queue <qid>” and look for
Drops (tail/random/max_threshold) : 0/0/0
Drops (no_pkt_handle/buffer_low) : 0/0
WRED (weight/avg_smaller) : 0/0
Check for DS pxf Drops
• CMTS Based Services Evolution
• Troubleshooting High Speed Data• DOCSIS 3.0 DS and US Channel Bonding Issues
• Troubleshooting DOCSIS Timing Issues
• Troubleshooting DOCSIS Load Balancing
• Troubleshooting Voice Services• Troubleshooting Voice Subscriber Issues
• Troubleshooting SNMP Services
• Troubleshooting DSG Services
• Summary
• Q & A
Agenda
Troubleshooting SNMP Services
High CPU from SNMP Process
• Input Q Full - %SNMP-3-INPUT_QFULL_ERR: Packet dropped due to input queue full
Slow or timeout SNMP queries to CMTS
• Empty graphs
• Operations engineers complaining about not getting SNMP data
Common Issue
SNMP Engine UBR10K/IOS
• SNMP Object data on Cable Line Cards
Overview
UBR10K
Route Processor
(PRE5)
IPC
Linecard 5/0Linecard 5/1
Linecard 6/0
CPU
Impact to these components cause
Input Q Full, High CPU, Slow SNMP
Response
SNMP Response
SNMP Request docsIfDownstreamChannelEntry
docsIfUpstreamChannelEntry
docsIfSignalQualityEntry
docsIfCmtsMacEntry
docsIfCmtsStatusEntry
docsIfCmtsCmStatusEntry
docsIfCmtsServiceEntry
docsIfCmtsChannelUtilizationEntry
docsIfCmtsDownChannelCounterEntry
docsIfCmtsUpChannelCounterEntry
cdxCmtsMtcCmEntry
cdxCmtsUscbSflowEntry
docsSubMgtCpeIpEntry
IF_MIB
ENTITY
…
Troubleshooting SNMP Services
High CPU from SNMP Process
• DDTS/Bug
• Over-polling
• IPC Issues – BPE not in use
• BGSync and Caching
Slow or timeout SNMP queries to CMTS
• Audit / Analyze / Manage
• Best practices – Community View / ACL / Rate limit
Common IssueF241-38-05-uBR10K-01#sh proc cpu sort | i SNMP
328 8 2587 3 0.00% 0.00% 0.00% 0 uBR10K SNMP Proc
348 0 1 0 0.00% 0.00% 0.00% 0 CMTS SNMP SFLOG
372 0 4111 0 0.00% 0.00% 0.00% 0 EEM ED SNMP
388 0 2 0 0.00% 0.00% 0.00% 0 SNMP Timers
389 188 41698 4 0.00% 0.00% 0.00% 0 IP SNMP
391 3880 1661791 2 0.00% 0.00% 0.00% 0 SNMP ENGINE
392 0 2 0 0.00% 0.00% 0.00% 0 IP SNMPV6
393 0 1 0 0.00% 0.00% 0.00% 0 SNMP ConfCopyPro
394 724 10752 6 0.00% 0.00% 0.00% 0 SNMP Traps
401 0 1096 0 0.00% 0.00% 0.00% 0 CMTS SNMP CHAN U
424 0 1 0 0.00% 0.00% 0.00% 0 SNMP Manager
425 0 1 0 0.00% 0.00% 0.00% 0 SNMP Proxy Forwa
SNMP Slow Response Troubleshooting Strategy
Holistic approach to NMS and SNMP
Audit NMS SNMP Pollers and the CMTS • debug snmp packet
• SNMP CPU-Stat
Analyze each NMS Poller• Eliminate unnecessary, duplicate, or expensive objects
Manage SNMP polling to and on the CMTS• Offload services
• Community Views and ACLs
• Enhancements
Framework
SNMP Slow Response Troubleshooting
• Use SNMP CPU-Stat Collections for Object, Invoked, and total Time used per poll
SNMP CPU-Stat Feature
1. Requires configuration of service internal
2. term len 0
3. test snmp cpu-stat start
4. show snmp cpu-stat
5. test snmp cpu-stat stop
UBR10K-05#show snmp cpu-stat
Load for five secs: 51%/4%; one minute: 55%; five minutes: 47%
Time source is NTP, 01:20:41.588 EDT Wed Jun 18 2014
ADDR 5sec% 1min% 5min% Running(ms) Time(usecs) Invoked OID
41330A00 9.48 4.02 3.03 0 18 135 docsIfDownstreamChannelEntry.1
41330A00 9.47 4.00 3.03 0 11 133 docsIfDownstreamChannelEntry.2
41330A00 9.51 4.00 3.03 0 11 133 docsIfDownstreamChannelEntry.3
41330A00 9.52 4.00 3.03 0 11 132 docsIfDownstreamChannelEntry.4
41330A00 9.56 3.99 3.02 0 10 131 docsIfDownstreamChannelEntry.5
<SNIP>
413CAEC8 9.23 4.44 3.14 0 4619 192 docsIf3CmtsSignalQualityExtEntry.1
413E3368 9.50 4.11 3.06 0 3207 133 docsQos3DynamicServiceStatsEntry.10
SNMP Slow Response Troubleshooting
• Script or Automate interval collections on show snmp cpu-stat output
• Recommend parsing data into CSVs
• Case Study Example
SNMP CPU-Stat Feature
1. 30 second SNMP Cpu-stat collection
2. 24 hour total interval
3. Parse data into CSV format
4. CSV format allows us to sort, manipulate, analyze, the collected data
5. Allows analysis of SNMP Polling in deep detail
Sum of Time-per-Invoke Column Labels
Row Labels CMTS1 CMTS2 CMTS3 CMTS4 CMTS5 CMTS6 (blank) Grand Total
docsQosDynamicServiceStatsEntry.11 708207 957031 191688 1856926
docsQos3DynamicServiceStatsEntry.11 528262 438265 228324 471259 1666110
docsIf3CmtsCmRegStatusEntry.7 384282 335816 498340 1218438
docsIf3SignalQualityExtEntry.2 513677 275966 199443 989086
docsIfUpstreamChannelEntry.2 307042 216514 523556
docsQos3ServiceFlowEntry.7 491856 491856
docsIf3CmtsSignalQualityExtEntry.2 237637 218881 456518
docsIfUpstreamChannelEntry.4 255599 255599
docsIfCmtsUpChannelCounterEntry.6 225106 225106
docsIf3SignalQualityExtEntry.1 207745 207745
cdxCmtsServiceExtEntry.2 188181.5 188181.5
(blank)
Grand Total 897959 1236469 982944 2181465.5 2389153 391131 8079121.5
Node Date/Time SNMP Object Invoke Time 5min 1min 5sec Time-per-Invoke
CMTS1 11/4/14 14:00 docsIfCmtsMacEntry.1 9 8643 5.51 0.82 1.9 960.3333333
CMTS1 11/4/14 14:00 cdxIfUpstreamChannelExtEntry.10 761 8293 6.26 2.02 2.35 10.89750329
CMTS1 11/4/14 14:00 cdxIfUpstreamChannelExtEntry.12 154 6275 7.18 2.69 2.26 40.74675325
CMTS1 11/4/14 14:00 cdxIfUpstreamChannelExtEntry.7 761 4496 4.15 1.74 2.29 5.908015769
CMTS1 11/4/14 14:00 cdxIfUpstreamChannelExtEntry.4 771 4273 4.69 1.67 2.28 5.542153048
CMTS1 11/4/14 14:00 cdxIfUpstreamChannelExtEntry.14 756 4163 6.37 2.71 2.86 5.506613757
CMTS1 11/4/14 14:00 cdxIfUpstreamChannelExtEntry.6 154 3708 7.29 2.6 2.24 24.07792208
CMTS1 11/4/14 14:00 cdxIfUpstreamChannelExtEntry.5 455 3504 7.61 1.81 2.08 7.701098901
CMTS1 11/4/14 14:00 docsIfSignalQualityEntry.2 746 3376 4.43 2.56 2.84 4.525469169
SNMP Slow Response Troubleshooting
SNMP View – Allows ability to restrict SNMP objects polled
Access-list – Allows only permitted IP to access the SNMP Community
Rate-Limit – Service policy for NMS Pollers
SNMP Community View and ACL
Snmp-server view name object {include|exclude}
Snmp-server community community-name view [view-name] {RO|RW} [Access-list-number]
UBR10K-05#show run | b snmp
snmp-server view ENGView docsIfCmtsServiceEntry.4 excluded
snmp-server view ENGView docsIfCmtsServiceEntry.6 excluded
snmp-server view ENGView docsIfCmtsServiceEntry.8 excluded
snmp-server view ENGView cdxCmtsServiceExtEntry.1 excluded
snmp-server view OPSView iso.* excluded
snmp-server view OPSView ccsFlapEntry included
snmp-server community OPERATIONS view OPSView RO 98
snmp-server community ENGINEERING view ENGView RW 99
• CMTS Based Services Evolution
• Troubleshooting High Speed Data• DOCSIS 3.0 DS and US Channel Bonding Issues
• Troubleshooting DOCSIS Timing Issues
• Troubleshooting DOCSIS Load Balancing
• Troubleshooting Voice Services• Troubleshooting Voice Subscriber Issues
• Troubleshooting SNMP Services
• Troubleshooting DSG Services
• Summary
• Q & A
Agenda
DSG STB in one-way mode OR not working at all
• 1. Improper configuration
1.a – Verify Multicast routing, client-list, DSG Tunnel, DSG classifiers and Interface allocation configuations
• 2. DCD and US issues on the cable interface
2.a – Cable modem not online
2.b – DCD not incrementing on cable interface
• 3. Presence of TLV-13 (DSG DA-to-DSID) in MDD
• 4. Older firmware on DSG STB
• 5. Multicast DB 5000 Limit
Possible Reasons
Troubleshooting DSG Services
DSG Operation at a glanceDSG-CC
IP
BackboneDSG Client
CANH
DSG Forwarder
DSG Servers
eCM
HFCIP Flow
DSG Agent (CMTS)
Set-top Box
Classifier Rule
DSG Filter ListCMTS Provisioning
DSG-CC Processes
DCD
-- ---- --- ---- --- --- --- -- --- --- ---- --- ---
DCD
CMTS encodes in to
DCD
MDF Enabled
Classifier Rule
DSG Filter List
CMTS ProvisioningDSG-CC Processes
DCD
-- ---- --- ---- --- --- --- -- --- --- ---- --- ---
DCD
CMTS encodes in to
DCD
-- ---- --- ---- --- --- --- -- --- --- ---- --- ---
MDD
Unique DSID to Tunnel
Address Mapping
eCM uses MDD to find
DSID values
eCM
DSG Tunnel Processing
MDF Disabled
1.a DSG multicast routing and client configuration
cable dsg client-list 6 id-index 1 ca-system-id E00
cable dsg client-list 6 id-index 2 mac-addr 0001.a6ff.0006
cable dsg client-list 107 id-index 1 ca-system-id E00
cable dsg client-list 107 id-index 2 mac-addr 0001.a6ff.006b
cable dsg client-list 108 id-index 1 ca-system-id E00
cable dsg client-list 108 id-index 2 mac-addr 0001.a6ff.006c
cable dsg client-list 111 id-index 1 ca-system-id E00
cable dsg client-list 111 id-index 2 mac-addr 0001.a6ff.006f
cable dsg client-list 1000 id-index 1 ca-system-id E00
cable dsg client-list 1000 id-index 2 mac-addr 0001.a6fe.0000
cable dsg client-list 2000 id-index 1 application-id 2
ADSG Client-list configuration
ip multicast-routing
!
ip access-list standard SSM-ALLOW
permit 232.0.0.0 0.255.255.255
!
ip pim ssm range SSM-ALLOW
!
interface Bundle1
ip pim sparse-mode
ip mroute-cache
no cable match address 180
!
interface TenGigabitEthernet1/0/0
ip pim sparse-mode
ip mroute-cache
Multicast Routing and ACL configurationEnable Multicast routing
ACL to
allow 232/8
PIM and route-cache
enabled on interfaces
access-list 150 permit udp any eq rip any eq rip
access-list 150 deny igmp any any
access-list 150 deny icmp any 224.0.0.0 15.255.255.255
access-list 150 deny tcp any 224.0.0.0 15.255.255.255
access-list 150 deny udp any 224.0.0.0 15.255.255.255
access-list 150 deny pim any any
access-list 150 deny udp any 10.0.0.0 0.255.255.255 eq
snmp
access-list 150 permit ip any any
!
interface Bundle 1
ip access-group 150 in
Cable side ACL configuration
ACL to protect
rouge multicast
source from HSD
cable dsg tunnel 1000 mac-addr 0100.e80a.0a00 tg 1 clients 1000
cable dsg tunnel 1006 mac-addr 0100.e80a.0a06 tg 1 clients 6
cable dsg tunnel 1107 mac-addr 0100.e80a.0a6b tg 1 clients 107
cable dsg tunnel 1108 mac-addr 0100.e80a.0a6c tg 1 clients 108
cable dsg tunnel 1111 mac-addr 0100.e80a.0a6f tg 1 clients 111
cable dsg tunnel 2000 mac-addr 0100.e80a.0b00 tg 1 clients 2000
cable dsg tunnel 2006 mac-addr 0100.e80a.0b06 tg 1 clients 6
cable dsg tunnel 2107 mac-addr 0100.e80a.0b6b tg 1 clients 107
cable dsg tunnel 2108 mac-addr 0100.e80a.0b6c tg 1 clients 108
cable dsg tunnel 2111 mac-addr 0100.e80a.0b6f tg 1 clients 111
1.a DSG tunnel and classifiers configurationDSG tunnel configuration with multicast mac add ADSG Interface configuration
interface Cable5/0/0
cable downstream dsg tg 1 channel 500
!
interface Cable5/0/1
cable downstream dsg tg 1 channel 501
!
interface Cable5/0/2
cable downstream dsg tg 1 channel 502
!
<snip>
DSG
config
applied
per intf.
cable dsg cfr 1000 dest-ip 232.10.10.0 tunnel 1000 dest-port 2000 13821 priority 1 src-ip 67.244.183.99 in-dcd yes
cable dsg cfr 1006 dest-ip 232.10.10.6 tunnel 1006 dest-port 2000 13821 priority 1 src-ip 67.244.183.99 in-dcd yes
cable dsg cfr 1107 dest-ip 232.10.10.107 tunnel 1107 dest-port 2000 13821 priority 1 src-ip 67.244.183.99 in-dcd yes
cable dsg cfr 1108 dest-ip 232.10.10.108 tunnel 1108 dest-port 2000 13821 priority 1 src-ip 67.244.183.99 in-dcd yes
cable dsg cfr 1111 dest-ip 232.10.10.111 tunnel 1111 dest-port 2000 13821 priority 1 src-ip 67.244.183.99 in-dcd yes
cable dsg cfr 2000 dest-ip 232.10.11.0 tunnel 2000 dest-port 13821 13821 priority 1 src-ip 67.244.183.99 in-dcd yes
cable dsg cfr 2006 dest-ip 232.10.11.6 tunnel 2006 dest-port 2000 13821 priority 1 src-ip 67.244.183.99 in-dcd yes
cable dsg cfr 2107 dest-ip 232.10.11.107 tunnel 2107 dest-port 2000 13821 priority 1 src-ip 67.244.183.99 in-dcd yes
cable dsg cfr 2108 dest-ip 232.10.11.108 tunnel 2108 dest-port 2000 13821 priority 1 src-ip 67.244.183.99 in-dcd yes
cable dsg cfr 2111 dest-ip 232.10.11.111 tunnel 2111 dest-port 2000 13821 priority 1 src-ip 67.244.183.99 in-dcd yes
ADSG classifiers configurationClassifiers add IP layer info to
DCD
CMTS to
include
cfr in DCD
1.a DSG multicast routing configuration verification
Address Interfacee Ver/ Nbr Query DR DR
Mode Count Intvl Prior
10.15.128.1 Bundle1 v2/S 0 30 1 10.15.128.1
24.164.210.213 TenGigabitEthernet1/0/0 v2/S 1 30 1 24.164.210.213
24.164.210.215 TenGigabitEthernet3/0/0 v2/S 1 30 1 24.164.210.215
Show ip pim interfacePIM working fine on WAN
PIM Neighbor Table
Mode: B - Bidir Capable, DR - Designated Router, N - Default DR Priority,
P - Proxy Capable, S - State Refresh Capable
Neighbor Add Interface Uptime/Expires Ver DR Prio/Mode
24.164.210.212 TenGigabitEthernet1/0/0 6d22h/00:01:43 v2 1 / S P
24.164.210.214 TenGigabitEthernet3/0/0 6d22h/00:01:22 v2 1 / S P
Show ip pim neighbor PIM Adj. established
(67.244.183.99, 232.10.10.0), 6d07h/00:02:50, flags: sTI
Incoming interface: GigabitEthernet3/0/0, RPF nbr 24.24.16.14
Outgoing interface list:
Bundle1, Forward/Sparse, 6d07h/00:02:02, H
(67.244.183.99, 232.10.10.6), 6d07h/00:02:50, flags: sTI
Incoming interface: GigabitEthernet3/0/0, RPF nbr 24.24.16.14
Outgoing interface list:
Bundle1, Forward/Sparse, 6d07h/00:02:02, H
Show ip mrouteS,G entries in
multicast
routing table
Group: 232.10.10.0, Source count: 1, Packets forwarded:
59538411, Packets received: 59538432
Source: 67.244.183.99/32, Forwarding:
59538411/108/912/773, Other: 59538432/21/0
Group: 232.10.10.6, Source count: 1, Packets forwarded:
1346076, Packets received: 1346076
Source: 67.244.183.99/32, Forwarding:
1346076/19/721/113, Other: 1346076/0/0
Group: 232.10.10.108, Source count: 1, Packets
forwarded: 1345534, Packets received: 1345534
Source: 67.244.183.99/32, Forwarding:
1345534/18/711/107, Other: 1345534/0/0
Group: 232.10.10.111, Source count: 1, Packets
forwarded: 1345491, Packets received: 1345491
Source: 67.244.183.99/32, Forwarding:
1345491/18/713/110, Other: 1345491/0/0
Group: 232.10.10.107, Source count: 1, Packets
forwarded: 1345558, Packets received: 1345558
Source: 67.244.183.99/32, Forwarding:
1345558/18/707/103, Other: 1345558/0/0 <snip>
Show ip mroute count
Packets received
and forwarded in
multicast group
1.a DSG tunnel configuration verification
uBR10012#show cable dsg tunnel 1000 cfrs
Tunnel cfr cfr cfr destination ip source ip srcPre d_port d_port
Id id state pri address address length start end
-------- ------- ----- ---- ------------------- --------------- -------- --------- ---------
1000 1000 en 1 232.10.10.0 67.244.183.99 32 2000 13821
Check tunnel cfrs configuration for all tunnels
Source-Dest IP
add for tunnel
uBR10012#show cable dsg tunnel 1000
interfaces
tunnel downstream rule
id interface id
-------- ------------ ----
1000 Cable5/0/1 1
Cable5/0/2 1
<snip>
Cable6/0/1 1
Cable6/0/2 1
<snip>
Cable6/1/0 1
Cable6/1/1 1
<snip>
Cable7/0/0 1
Cable7/0/1 1
<snip>
Cable7/1/0 1
Cable7/1/1 1
<snip>
Check cable intf. Tunnel association
Tunnel-id to DS
interface
association
uBR10012#show cable dsg tunnel 1000 clients
Tunnel client client client client vendor
Id listId id id type address group
-------- ------- ------- ------------------ ------------------ ---------
1000 1000 1 CA System ID 0x0E00
2 MAC Addr 0001.a6ff.0000
Check Client-id and tunnel associationClient-id to
tunnel-id
association
uBR10012#show cable dsg tunnel 1000 statistics
tunnel cfr cfr destination ip source ip total total
id id state address address forwarded received
-------- ------ ----- ----------------- ------------------ -------------- --------------
1000 1000 en 232.10.10.0 67.244.183.99 339941423 339941444
uBR10012#show cable dsg tunnel 1006 statistics
tunnel cfr cfr destination ip source ip total total
id id state address address forwarded received
-------- ------ ----- ----------------- ------------------ -------------- --------------
1006 1006 en 232.10.10.6 67.244.183.99 27362242 27362242
Check DSG tunnel counters for all tunnels
Packets
forwarded and
received per
tunnel
2.a/2.b Cable modem online and DCD verification
uBR10012#scm 3c62.00dc.3644
MAC Address IP Address I/F MAC Prim RxPwr Timing Num I
State Sid (dBmv) Offset CPE P
3c62.00dc.3644 10.100.47.229 C7/0/2/U0 online(pt) 4605 0.00 1466 1 N
IP address MAC address Dual IP Device Class
10.151.27.89 3c62.00dc.3646 N Host
Cable modem online with cpe CM online and CPE has IP
uBR10012#sh int c6/1/5 dsg downstream dcd
IF dcd dcd dsg num of dcd num of dcd num of dcd num of
Name state Tx fwd sent fail change cnt fragment
---------- ---- ---- ---- --------------- -------------- ---------- --------
Mo6/1/1:16 en on en 123269 0 10 1
Mo6/1/1:17 en on en 123269 0 10 1
Mo6/1/1:20 en on en 123269 0 10 1
Check DCD for cable interface incrementing DCDs sent incrementing, 0
failed DCD
clc_6_1#
SLOT 6/1: Jan 25 13:20:06.583 EST: Cable6/1/0 123030 DCD msg sent, 9 change count 0 fails
SLOT 6/1: Jan 25 13:20:06.583 EST: Cable6/1/0 DCD Message Dump:
SLOT 6/1: Jan 25 13:20:06.583 EST: 0x0000: C2 00 01 17 00 00 01 E0 2F 00 00 01 60 73 5C 71
SLOT 6/1: Jan 25 13:20:06.583 EST: 0x0010: F7 74 01 05 00 00 03 03 20 00 09 01 01 32 20 01
SLOT 6/1: Jan 25 13:20:06.583 EST: 0x0020: 01 01 02 01 01 04 0C 03 02 0E 00 02 06 00 01 A6
SLOT 6/1: Jan 25 13:20:06.583 EST: 0x0030: FE 00 00 05 06 E8 0A 0A 00 00 00 06 02 03 E8 17
SLOT 6/1: Jan 25 13:20:06.583 EST: 0x0040: 1D 02 02 03 E8 05 01 01 09 14 05 04 E8 0A 0A 00
“Debug cable dsg dcd” on LC
DCDs are being
sent on an
interface
Show cable modem ip/mac verbose
uBR10012#scm 3c62.00dc.3644 verb
MAC Address : 3c62.00dc.3644
IP Address : 10.100.47.229
IPv6 Address : ---
<snip>
Number of Multicast DSIDs Support : 0
MDF Capability Mode : 0
IGMP/MLD Version : IGMPv2
CM do not
support MDF
3/4 TLV-13 and DSG Firmware issues
• Presence of TLV-13 (DSG DA-to-DSID) in MDD• Disable MDF
• cable multicast mdf-disable dsg
• cable multicast mdf-disable WB-Incapable-CM
• Upgrade DSG firmware that supports MDF
5. Multicast DB 5000
Show cable multicast db summary
Multicast DB and PXF commands
Show pxf cpu mroute multicast-ipdetail | include MC
Show pxf cpu queue cable slot/subslot/port
UBR10012# show cable multicast db summary
Interface Session Count CM Count
Bundle1 3295 864
UBR10012#show pxf cpu mroute 239.196.255.90 detail | inc
5/1/0
oif_next= 0x000523, IDB I/F name=Modular-Cable5/1/0:0
oif_next= 0x000528, IDB I/F name=Modular-Cable5/1/0:1
oif_next= 0x000529, IDB I/F name=Modular-Cable5/1/0:2
oif_next= 0x00052A, IDB I/F name=Modular-Cable5/1/0:3
oif_next= 0x00052B, IDB I/F name=Modular-Cable5/1/0:4
oif_next= 0x00052C, IDB I/F name=Modular-Cable5/1/0:5
oif_next= 0x00052D, IDB I/F name=Modular-Cable5/1/0:6
oif_next= 0x00052E, IDB I/F name=Modular-Cable5/1/0:7
oif_next= 0x000533, IDB I/F name=Modular-Cable5/1/0:8
oif_next= 0x000538, IDB I/F name=Modular-Cable5/1/0:9
oif_next= 0x00053D, IDB I/F name=Modular-Cable5/1/0:10
oif_next= 0x000542, IDB I/F name=Modular-Cable5/1/0:11
UBR10012#show pxf cpu queue cable 5/1/0
VCCI 2327:
Forwarding Interface: Modular-Cable5/1/0:0
Cable Interface Queues:
QID Len/Max Dequeues TailDrops MinRt Wt/Quantum ShapeRt
FlowId
(Kbps) (Kbps)
135325 0/255 38310354 0 0 1/240 0 2327
135326 0/255 117843 0 0 1/10000 0 34864
UBR10012#show pxf cpu queue cable 5/1/0
VCCI 2327:
Forwarding Interface: Modular-Cable5/1/0:0
Cable Interface Queues:
QID Len/Max Dequeues TailDrops MinRt Wt/Quantum ShapeRt
FlowId
(Kbps) (Kbps)
135325 0/255 38310457 0 0 1/240 0 2327
135326 0/255 117845 0 0 1/10000 0 34864
Session count less than 5000
Are the DSG-Enabled Intf MCs
all present?
Did your interface Q increment
at all?
Summary• Evolution of CMTS Based Services
• Common Data Service Issues and Troubleshooting
• DOCSIS Timing Issues
• DOCSIS Load Balancing
• Common VoIP Service issues
• SNMP Troubleshooting
• DSG troubleshooting
• Troubleshooting BSoD (Appendix A)
• Troubleshooting DDOS (Appendix B)
“Effective Troubleshooting Will Decrease Downtime and Increase Customer Satisfaction”
Q&A
Participate in the “My Favorite Speaker” Contest
• Promote your favorite speaker through Twitter and you could win $200 of Cisco Press products (@CiscoPress)
• Send a tweet and include
• Your favorite speaker’s Twitter handle #tejalpatel
• Two hashtags: #CLUS #MyFavoriteSpeaker
• You can submit an entry for more than one of your “favorite” speakers
• Don’t forget to follow @CiscoLive and @CiscoPress
• View the official rules at http://bit.ly/CLUSwin
Promote Your Favorite Speaker and You Could Be a Winner
Complete Your Online Session Evaluation
Don’t forget: Cisco Live sessions will be available for viewing on-demand after the event at CiscoLive.com/Online
• Give us your feedback to be entered into a Daily Survey Drawing. A daily winner will receive a $750 Amazon gift card.
• Complete your session surveys though the Cisco Live mobile app or your computer on Cisco Live Connect.
Continue Your Education
• Demos in the Cisco campus
• Walk-in Self-Paced Labs
• Table Topics
• Meet the Engineer 1:1 meetings
• Related sessions
Thank you
Troubleshooting BSoD (Appendix A)
Business Services over DOCSIS (BSoD)
• CMTS – Traditional High Speed Data vs newer BSoD usage
• HFC plant typically under-utilized in business hours
• Business services over HFC can maximize Return of Investment (ROI)
• CableLabs standardized Layer2 VPN Business Services over DOCSIS (BSoD)
• Multiple BSoD flavors
• Transparent LAN Services over DOCSIS – Cisco Proprietary
• Dot1Q based L2VPN BSoD – CableLabs Standard
• MPLS based L2VPN BSoD – CableLabs Compliant
Cable Multi Service Networks
Aggregation NetworkAccess/Hub
7600/ASR9K
Backbone Network
MPLS PE
HR
Corporate
Business
Residential
VoD
Content Network
TV SIP
CRS-x Core
VoD
Content Network
TV SIP
Core Network
IP / MPLS
Regional Network
CR
Regional IP /
MPLS Network
CR
IPoDWDM Optical Network
Deep Packet
Inspection
CMTS
EQAM
HFC
ME3400
FTTP
Corporate
Business
ME3400
STB
CM/
eMTA
Corporate
Business
ISR
Corporate
Business
L2VPN CM
L3 (IP/MPLS)L1/L2L2/L3
Video Complex
SDV VoD
PEG DS/DB
SDC (System/Div)
Voice /Video Complex
Network Mgt Complex
IP Security Complex
SDC (Regional) SDC (National)
Dot1Q Based L2VPN BSoD
MPLS Based L2VPN BSoD
Troubleshooting L2VPN BSoD
1. Verify Cable Modem is Online(pt)
2. Verify whether the CM is registered as L2VPN
3. Verify whether the MPLS pseudowire (PW) is UP
1. Verify CM Is Online
• Possible reasons for CM to be offline
• 1a. Basic DOCSIS issues• DHCP, TFTP, ToD and DOCSIS layer issues
• 1b. Feature not supported on CMTS Software Version• TLS: 12.2(15)BC or later crypto releases
• Dot1Q: 12.2(33)SCA or later crypto releases
• MPLS: 12.2(33)SCC or later crypto releases
CMTS-uBR10k#sh cable modem 0022.3a61.7bcf
Load for five secs: 0%/0%; one minute: 0%; five minutes: 0%
Time source is NTP, 19:29:47.278 EDT Wed Mar 24 2010
D
MAC Address IP Address I/F MAC Prim RxPwr Timing Num I
State Sid (dBmv) Offset CPE P
0022.3a61.7bcf 17.101.75.100 C5/1/0/U0 offline 9 0.75 1183 0 N
TLS
DOt1Q BSOD
MPLS BSoD
1c. Feature not supported on CM software version
1d. CMTS not configured for L2VPN BSoD
• 1e. Incorrect L2VPN encoding in the CM configuration file
• Cable Modem is online(pt) CMTS-uBR10k#sh cable modem 0022.3a61.7bcf
MAC Address IP Address I/F MAC Prim RxPwr Timing Num I
State Sid (dBmv) Offset CPE P
0022.3a61.7bcf 17.101.75.100 C5/1/0/U0 online(pt) 9 17.75 1183 0 N
Debug message:CMTS L2 VPN debugging is on
002134: Mar 24 19:54:59.973 EDT: cmts_docsis_l2vpn_add sanity failed: mac 0022.3a61.7bcf,
sid 16,vlanid 0,vpnid 2020
CMTS-uBR10k#sh cable logging layer2events
Cable logging: LAYER2EVENTS Enabled
002073: Mar 24 19:52:16.714 EDT: %UBR10000-6-CM_OFFLINE_WITH_MPLS_L2VPN_NOT_ENABLE: DOCSIS
MPLS L2VPN not enabled, sending CM 0022.3a61.7bcf offline
CMTS-uBR10k#sh cable modem 0022.3a61.7bcf verbose [Snippet]
sysDescr : Cisco DOCSIS Cable Modem<<HW_REV: 2.1; VENDOR: Cisco; BOOTR:
2.1.6d; SW_REV: v2.0.2r1256-100222; MODEL: DPC2100R2>>
Modem Status : {Modem= online(pt), Security=assign(tek)}
Security Capabilities : {Priv=BPI+, EAE=N, Key_len=56}
L2VPN Capabilities : {L2VPN=Y, eSAFE=N}
Sid/Said Limit : {Max US Sids=16, Max DS Saids=15}
Optional Filtering Support : {802.1P=Y, 802.1Q=Y, DUT=Y}
1. Verify CM Is Online
2. Verify Whether CM is Registered as L2VPN
CMTS-PE#sh cable l2-vpn xconnect mpls-vc-map
0022.3a61.7bcf verbose
MAC Address : 0022.3a61.7bcf
Prim Sid : 16
Cable Interface : Cable5/1/0
L2VPNs provisioned : 1
DUT Control/CMIM : Enable/0x8000FFFF
VPN ID : 2020
L2VPN SAID : 12296
SAII : 000007D1
TAII : 000007D1
Upstream SFID Summary : 27
Upstream SFID [27 ] : SID 16 MPLS-EXP 4
Downstream CFRID[SFID] Summary: Primary SF
CMIM : 0x60
MPLS PEER IpAddress : 99.1.1.22
MPLS PW VCID : 2001
MPLS PW TYPE : Ethernet
MPLS PW Circuit ID : Bu254:2001
MPLS PW Remote State : Down
MPLS PW Local State : UP
Total US pkts : 0
Total US bytes : 0
Total US pkt Discards : 0
Total US byte Discards: 0
Total DS pkts : 0
Total DS bytes : 0
Total DS pkt Discards : 0
Total DS byte Discards: 0
CMTS-uBR10k#sh cable l2-vpn xconnect dot1q-vc-
map 0022.3a61.7bcf verbose
MAC Address : 0022.3a61.7bcf
Prim Sid : 17
Cable Interface : Cable5/1/0
L2VPNs provisioned : 1
DUT Control/CMIM : Enable/0x8000FFFF
VPN ID : 0234560002
L2VPN SAID : 12302
Upstream SFID Summary : 29
Upstream SFID [29 ] : SID 17 UserPrio 3
Downstream CFRID[SFID]: Primary SF
CMIM : 0x60
Ethernet Interface : GigabitEthernet3/1/0
DOT1Q VLAN ID : 5
Total US pkts : 0
Total US bytes : 0
Total US pkt Discards : 0
Total US byte Discards: 0
Total DS pkts : 0
Total DS bytes : 0
Total DS pkt Discards : 0
Total DS byte Discards: 0
Dot1Q BSoD MPLS BSoD
2b. Verify L2VPN Parallel eXpress Forwarding (PXF)
CMTS-PE#sh pxf cable l2-vpn atom mac 0022.3a61.7bcf
ATOM L2VPN Information for CM 0022.3a61.7bcf Start:
DOCSIS ATOM L2VPN Upstream Information:
ATOM Enable SID : 16
vcci maptable flags : 0x4000
atom_us_index : 30
upstream index : 30
us_sid : 16
CM MAC addr : 0022.3a61.7bcf
vcci_out : 0x0
mac_rew_index : 0x0
rew_extension : 0x2F
ingress_flags : 0x40
upstream punt : Disabled
CM MAC table flags : 0x8000
esmac : 0000.0000.0000
cmim : 0x0
Forward packets : 0
Forward bytes : 0
Discard packets : 0
Discard bytes : 0
DOCSIS ATOM L2VPN Downstream Information:
Downstream index : 2001
flags : 0x1
l2vpn_said : 16
ds_classif_result : 0x8809D
Forward packets : 0
Forward bytes : 0
CMTS-uBR10k#sh pxf cable l2-vpn 0022.3a61.7bcf
Load for five secs: 0%/0%; one minute: 0%; five
minutes: 0%
Time source is NTP, 12:02:28.927 EDT Thu Mar 25
2010
MAC Address 0022.3a61.7bcf
upstream index 13, us_sid 17, vlan_hdr
0x6005,CM MAC addr 0022.3a61.7bcf,
vcci 142, flags 0x0, esmac 0000.0000.0000,
cmim 0x0
packets 0, bytes 0, discard packets 0,
discard bytes 0
downstream index 13, vcci 145, l2vpn_said 17
ds_classif_result 0x8809E
packets 0, bytes 0 discard packets 0,
discard bytes 0
Dot1Q BSoD
MPLS BSoD
3. Verify Whether the MPLS PW Is UP
Possible reasons for MPLS PW down• No Label Switch Path (LSP) established
• No targeted LDP session
• Interface MTU mismatch
Dot1Q-PE#sh mpls l2transport vc 2001
Load for five secs: 2%/1%; one minute: 2%; five minutes: 2%
Time source is NTP, 14:14:06.941 EDT Thu Mar 25 2010
Local intf Local circuit Dest address VC ID Status
------------- -------------------------- --------------- ---------- ----------
Gi4/23.1 Eth VLAN 11 99.1.1.22 2001 DOWN
DOt1Q BSOD
MPLS BSoD
CMTS-PE#sh mpls l2transport vc 2001
Load for five secs: 0%/0%; one minute: 0%; five minutes: 0%
Time source is NTP, 14:17:52.398 EDT Thu Mar 25 2010
Local intf Local circuit Dest address VC ID Status
------------- -------------------------- --------------- ---------- ----------
Bu254 DOCSIS 2001 99.1.1.22 2001 DOWN
3a. Verify Whether the MPLS PW Is UP
Dot1Q-PE#sh mpls l2transport vc 2001 detail
Local interface: Gi4/23.1 up, line protocol up, Eth VLAN 11 up
Interworking type is Ethernet
Destination address: 99.1.1.22, VC ID: 2001, VC status: up
Output interface: Te1/1, imposed label stack {51 64}
Preferred path: not configured
Default path: active
Next hop: 11.1.0.5
Create time: 4w0d, last status change time: 00:00:57
Signaling protocol: LDP, peer 99.1.1.22:0 up
Targeted Hello: 99.1.1.12(LDP Id) -> 99.1.1.22, LDP is UP
Status TLV support (local/remote) : enabled/supported
LDP route watch : enabled
Label/status state machine : established, LruRru
Last local dataplane status rcvd: No fault
Last local SSS circuit status rcvd: No fault
Last local SSS circuit status sent: No fault
Last local LDP TLV status sent: No fault
Last remote LDP TLV status rcvd: No fault
Last remote LDP ADJ status rcvd: No fault
MPLS VC labels: local 89, remote 64
Group ID: local 0, remote 0
MTU: local 4470, remote 4470
Remote interface description:
Sequencing: receive disabled, send disabled
Control Word: Off (configured: autosense)
SSO Descriptor: 99.1.1.22/2001, local label: 89
SSM segment/switch IDs: 16465/8252 (used), PWID: 8204
VC statistics:
transit packet totals: receive 34063315, send 11784915
transit byte totals: receive 10849943060, send 1205412300
transit packet drops: receive 0, seq error 0, send 0
Dot1Q BSOD
3a. Verify Whether the MPLS PW Is UPCMTS-PE#sh mpls l2transport vc 2001 detail
Load for five secs: 0%/0%; one minute: 0%; five minutes: 1%
Time source is NTP, 20:29:41.852 EDT Wed Mar 24 2010
Local interface: Bu254 up, line protocol up, DOCSIS 2001 up
Destination address: 99.1.1.22, VC ID: 2001, VC status: up
Output interface: Gi3/1/0, imposed label stack {101 31}
Preferred path: not configured
Default path: active
Next hop: 11.11.0.101
Create time: 00:33:28, last status change time: 00:07:40
Signaling protocol: LDP, peer 99.1.1.22:0 up
Targeted Hello: 99.1.1.13(LDP Id) -> 99.1.1.22
Status TLV support (local/remote) : enabled/supported
Label/status state machine : established, LruRru
Last local dataplane status rcvd: no fault
Last local SSS circuit status rcvd: no fault
Last local SSS circuit status sent: no fault
Last local LDP TLV status sent: no fault
Last remote LDP TLV status rcvd: no fault
MPLS VC labels: local 94, remote 31
Group ID: local 0, remote 0
MTU: local 1500, remote 1500
Remote interface description:
Sequencing: receive disabled, send disabled
SSO Descriptor: 99.1.1.22/2001, local label: 94
SSM segment/switch IDs: 8198/4100 (used), PWID: 4100
VC statistics:
packet totals: receive 23951, send 23951
byte totals: receive 2730414, send 2395100
packet drops: receive 0, seq error 0, send 0
MPLS BSoD
Troubleshooting MPLS L2VPN BSoD (Cont..)
• Debug cable l2-vpn
• Debug cable tlvs
• Debug cable mac-address
• Debug mpls l2transport vc status event
Note: Use caution when enabling above commands in production routers
Useful debug commands
Troubleshooting DDOS Attacks (Appendix B)
CMTS
RP
Common (D)DOS Attacks
Internet DDOS - Traffic gets punted (rp_dest) to the RP, increasing CPU usage
High volume traffic is being sent to the CPE, degrades service, induces manual reset or re-registration
Attack Vectors
PC1
Attacker
PC2
PC3
PC4
PC5
CPEInternet CM
24.24.24.24
CM offline
Slide Author: Lloyd D’Souza
Troubleshooting (D)DOSHow to detect (D)DOSCheck CPU usage.
UBR10k#show processes cpu sorted 5min
CPU utilization for five seconds: 99%/80%; one minute: 99%; five minutes: 99%
PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
176 3559252 944103 6462 16.75% 18.32% 16.40% 0 IP Input
7 923960 89379 10337 0.71% 0.31% 0.20% 0 Check heaps
...
Show to-RP queues twice to find rapidly incrementing counters and/or tail drops.
UBR10k#show pxf cpu queue
VCCI/ClassID ClassName QID Length/Avg Max Dequeues Drops(Tail/Random)
1/0 default 131072 0/0 1024 92781 0/0
1/1 src_vrfy 131073 0/0 1024 0 0/0
1/2 l2_ctrl 131074 0/0 1024 0 0/0
1/3 l3_ctrl 131075 0/0 1024 581496 0/0
1/4 ipc 131076 0/0 1024 0 0/0
1/5 l3 131077 0/3 1024 5995503 2307/0
1/6 netflow 131078 0/0 1024 0 0/0
1/7 acl 131079 0/0 1024 0 0/0
UBR10k#show pxf cpu queue
VCCI/ClassID ClassName QID Length/Avg Max Dequeues Drops(Tail/Random)
1/0 default 131072 0/0 1024 92782 0/0
1/1 src_vrfy 131073 0/0 1024 0 0/0
1/2 l2_ctrl 131074 0/0 1024 0 0/0
1/3 l3_ctrl 131075 0/0 1024 581505 0/0
1/4 ipc 131076 0/0 1024 0 0/0
1/5 l3 131077 0/3 1024 6026036 5812/0
1/6 netflow 131078 0/0 1024 0 0/0
1/7 acl 131079 0/0 1024 0 0/0
Look for other
RP related
processes.
e.g. Arp Input
Show diversion stats twice to find rapidly incrementing counters.
UBR10K#show pxf cpu stat diversion | exc = 0
Diversion Count
arp = 520981
fib bcast = 83685
fib dest = 1559489
cm nol = 16
ipv6_rp_dest = 2856003
...
UBR10K#show pxf cpu stat diversion | exc = 0
Diversion Count
arp = 520992
fib bcast = 83686
fib dest = 1997715
cm nol = 16
ipv6_rp_dest = 3347124
...
Same command to find the interface. This may require several steps.
ubr#show pxf cpu stat diversion | inc GigabitEthernet
GigabitEthernet4/0/0 174347 packets 34484268 bytes RP Rx
GigabitEthernet4/0/0 174347 packets 34484268 bytes Proc Enq
GigabitEthernet4/1/0 1544292 packets 191492208 bytes RP Rx
GigabitEthernet4/1/0 1548898 packets 192063352 bytes Proc Enq
ubr#show pxf cpu stat diversion | inc GigabitEthernet
GigabitEthernet4/0/0 174349 packets 34484670 bytes RP Rx
GigabitEthernet4/0/0 174350 packets 34484730 bytes Proc Enq
GigabitEthernet4/1/0 2584294 packets 196452456 bytes RP Rx
GigabitEthernet4/1/0 2587124 packets 199803376 bytes Proc Enq
Troubleshooting (D)DOSHow to detect (D)DOS
We are looking
for large
increments
• Configure Divert Rate limit on the Cable Sidecable divert-rate-limit rate 10 limit 10
• Configure Trusted Sites on the WAN side for common sources service divert-rate-limit trusted-site 0.0.0.0 0.0.0.0
• Configure the DRL global optionsservice divert-rate-limit ip fib_rp_glean rate 4 limit 4
service divert-rate-limit ip fib_rp_dest rate 4 limit 4
service divert-rate-limit ip fib_rp_punt rate 4 limit 4
service divert-rate-limit ipv6 ipv6_rp_dest rate 4 limit 4
service divert-rate-limit ipv6 ipv6_rp_punt rate 4 limit 4
service divert-rate-limit ipv6 ipv6_rp_glean rate 4 limit 4
service divert-rate-limit ipv6 icmpv6 rate 4 limit 4
• Configure the DRL Max-Rate options service divert-rate-limit max-rate wan fib_rp_glean rate 5000 limit 1000
service divert-rate-limit max-rate wan fib_rp_punt rate 5000 limit 1000
service divert-rate-limit max-rate wan fib_rp_dest rate 40000 limit 4000
service divert-rate-limit max-rate wan ipv6_fib_glean rate 5000 limit 1000
service divert-rate-limit max-rate wan ipv6_fib_punt rate 5000 limit 1000
service divert-rate-limit max-rate wan ipv6_fib_dest rate 40000 limit 4000
Troubleshooting (D)DOSMitigation
These
configurations
are general
Best Practices
Cable Side AttackStep 1 – Check CPU usage.
ubr#show processes cpu sorted 5min
CPU utilization for five seconds: 98%/80%; one minute: 82%; five minutes: 77%
PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
156 5559152 746133 6462 67.75% 38.32% 18.40% 0 IP Input <<<<<<<<<<
7 967780 56779 12347 0.61% 0.21% 0.10% 0 Check heaps
...
Step 2 – Show to-RP queues twice to find rapidly incrementing counters and/or tail drops.
ubr#show pxf cpu queue
VCCI/ClassID ClassName QID Length/Avg Max Dequeues Drops(Tail/Random)
1/0 default 131072 0/9 1024 719329 341/0 <<<<<<<<<<
1/1 src_vrfy 131073 0/0 1024 0 0/0
1/2 l2_ctrl 131074 0/0 1024 0 0/0
1/3 l3_ctrl 131075 0/0 1024 582838 0/0
1/4 ipc 131076 0/0 1024 0 0/0
1/5 l3 131077 0/0 1024 7783421 0/0
1/6 netflow 131078 0/0 1024 0 0/0
1/7 acl 131079 0/0 1024 0 0/0
ubr#
ubr#show pxf cpu queue
VCCI/ClassID ClassName QID Length/Avg Max Dequeues Drops(Tail/Random)
1/0 default 131072 0/2 1024 1055328 52719/0 <<<<<<<<<<
1/1 src_vrfy 131073 0/0 1024 0 0/0
1/2 l2_ctrl 131074 0/0 1024 0 0/0
1/3 l3_ctrl 131075 0/0 1024 582858 0/0
1/4 ipc 131076 0/0 1024 0 0/0
1/5 l3 131077 0/0 1024 7783421 0/0
1/6 netflow 131078 0/0 1024 0 0/0
1/7 acl 131079 0/0 1024 0 0/0
Cable Side AttackStep 3 – Show diversion stats twice to find rapidly incrementing counters.
ubr#show pxf cpu stat diversion | exc = 0
Diversion Count
arp = 584810
option = 1814448 <<<<<<<<<<
fib bcast = 93312
cm nol = 16
...
ubr#show pxf cpu stat diversion | exc = 0
Diversion Count
arp = 584818
option = 2029088 <<<<<<<<<<
fib bcast = 93312
cm nol = 16
Step 4 – Same command to find the interface. This may require several steps.
ubr#show pxf cpu stat diversion | inc Cable6/0
Cable6/0/0 4731382 packets 302815960 bytes RP Rx <<<<<<<<<<
Cable6/0/0 4731308 packets 283885992 bytes Proc Enq
Cable6/0/1 486853 packets 31160470 bytes RP Rx
Cable6/0/1 486841 packets 29212338 bytes Proc Enq
ubr#show pxf cpu stat diversion | inc Cable6/0
Cable6/0/0 4845974 packets 303749848 bytes RP Rx <<<<<<<<<<
Cable6/0/0 4845900 packets 284761512 bytes Proc Enq
Cable6/0/1 486861 packets 31160982 bytes RP Rx
Cable6/0/1 486849 packets 29212818 bytes Proc Enq
Step 5 – Attack is coming from a cable interface. Configure DRL appropriately.
ubr#conf t
ubr(config)#interf Cable6/0/0
ubr(config-if)#cable divert-rate-limit rate 4 limit 30
Cable Side AttackStep 3 – Show diversion stats twice to find rapidly incrementing counters.
ubr#show pxf cpu stat diversion | exc = 0
Diversion Count
arp = 584810
option = 1814448 <<<<<<<<<<
fib bcast = 93312
cm nol = 16
...
ubr#show pxf cpu stat diversion | exc = 0
Diversion Count
arp = 584818
option = 2029088 <<<<<<<<<<
fib bcast = 93312
cm nol = 16
Step 4 – Same command to find the interface. This may require several steps.
ubr#show pxf cpu stat diversion | inc Cable6/0
Cable6/0/0 4731382 packets 302815960 bytes RP Rx <<<<<<<<<<
Cable6/0/0 4731308 packets 283885992 bytes Proc Enq
Cable6/0/1 486853 packets 31160470 bytes RP Rx
Cable6/0/1 486841 packets 29212338 bytes Proc Enq
ubr#show pxf cpu stat diversion | inc Cable6/0
Cable6/0/0 4845974 packets 303749848 bytes RP Rx <<<<<<<<<<
Cable6/0/0 4845900 packets 284761512 bytes Proc Enq
Cable6/0/1 486861 packets 31160982 bytes RP Rx
Cable6/0/1 486849 packets 29212818 bytes Proc Enq
Step 5 – Attack is coming from a cable interface. Configure DRL appropriately.
ubr#conf t
ubr(config)#interf Cable6/0/0
ubr(config-if)#cable divert-rate-limit rate 4 limit 30
Cable Side AttackStep 6 – Show the DRL US-cable stats twice, checking for incrementing stats.
ubr#show pxf cpu stat drl us-cable threshold 500
Divert-Rate-Limit US-cable statistics :: threshold = 500
dropped identifier
753246 interface: Cable6/0/0 SID: 3
ubr#
ubr#show pxf cpu stat drl us-cable threshold 500
Divert-Rate-Limit US-cable statistics :: threshold = 500
dropped identifier
878734 interface: Cable6/0/0 SID: 3
Step 7 – Find the attacking source.
ubr#show cable modem Cable6/0/0
B D
MAC Address IP Address I/F MAC Prim RxPwr Timing Num P I
State Sid (dBmv) Offset CPE I P
0013.10bb.22e5 50.0.0.2 C6/0/0/U0 online 1 -0.75 1203 0 N N
0007.0e03.6b9d 50.0.0.3 C6/0/0/U0 online 2 0.00 2748 0 N N
0014.e84f.b440 50.0.0.4 C6/0/0/U0 online 3 0.00 1694 1 N N <<<<<<<<<<
Step 8 – Use customer-specific methods to cut off the attacking source. For example, configure ‘cable hotlist’ or reset the CM with a different registration file.
Step 9 – Confirm CPU usage.
ubr#show processes cpu sorted 5min
CPU utilization for five seconds: 0%/0%; one minute: 0%; five minutes: 7%
PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
156 224762 6372158 50 0.00% 0.00% 2.05% 0 IP Input
7 927216 89570 11349 0.00% 0.13% 0.11% 0 Check heaps
...
3 33 3
2 2
default
Src_verify
4 44 4
16 1616 16
L2_ctrl
L3_ctrll
Priority
6 66 6 netflow
Per Source/punt
rate limit
Trusted
Site
Per Punt cause
rate limit
PXF
Selected
Punt causes
• Add per-punt-cause (aggregated) based rate-limit for certain divert causes.
• New Priority queue for Network Control traffic.
Enhancements added via CSCun22070
To_RP
to_RP queues
1 1