Troubleshooting Forwarding Plane

Troubleshooting Forwarding Plane Telefónica España

Support Advanced Services EMEA [email protected] V1.1 20110527

2 Copyright © 2009 Juniper Networks, Inc. www.juniper.net

INTERFACE AND BYPASS-ROUTING


PING TO REMOTE ADDRESS – DEFAULT

M320 M120 so-1/0/0 so-4/1/0 10.2.2/30 .1

RE

CB

FPC #1

SIBs

PIC

so-5/1/0 .2

so-1/1/0 10.3.3/30

RE

CB

FEB #4 PIC 1/0 4/1

user@m320> ping 10.2.2.2

  Assuming default configuration (without default-address-selection)

(10.2.2.1, 10.2.2.2) (SA, DA) =

(10.2.2.2, 10.2.2.1)

FPC #4


LINK KEEPALIVES & ROUTING PROTOCOL PACKETS

M320 M120 so-1/0/0 so-4/1/0 10.2.2/30 .1

RE

CB

FPC #1

SIBs

PIC

so-5/1/0 .2

so-1/1/0 10.3.3/30

RE

CB

FEB #4 PIC 1/0 4/1

  HDLC/OAM/LMI keepalives follow the same hardware path as ping

FPC #4


PING TO REMOTE ADDRESS – INTERFACE OPTION

M320 M120 so-1/0/0 so-4/1/0 10.2.2/30 .1

RE

CB

FPC

#1

so-5/1/0 .2

so-1/1/0 10.3.3/30

RE

CB

(10.3.3.1, 10.2.2.2) (SA, DA) =

(10.2.2.2, 10.3.3.1)

user@m320> ping 10.2.2.2 interface so-1/1/0

  The interface option only alters the source IP address by default

PIC

1/0

FEB

#4

FPC #4

PIC 4/1

FEB

#5

FPC #5

PIC 5/1 PIC

1/1

  Similar to source option (monitor traffic interface displays packets)


PING TO REMOTE ADDRESS – BYPASS-ROUTING

M320 M120 so-1/0/0 so-4/1/0 10.2.2/30 .1

RE

CB

FPC #1

SIBs

PIC

so-5/1/0 .2

so-1/1/0 10.3.3/30

RE

CB

FEB #5 PIC 1/1 5/1

user@m320> ping 10.2.2.2 interface so-1/1/0 bypass-routing

  bypass-routing allows to force the packet to go out a given interface

(10.3.3.1, 10.2.2.2) (SA, DA) =

(10.2.2.2, 10.3.3.1)

FPC #5

  Only works properly at SONET/SDH interfaces


PING TO LOCAL ADDRESS – DEFAULT

M320 M120 so-1/0/0 so-4/1/0 10.2.2/30 .1

RE

CB

FPC #1 PIC

.2

RE

CB

FEB #4 PIC 1/0 4/1

user@m320> ping 10.2.2.1

  By default, ping to local address does not leave the RE

FPC #4

  Checked with show chassis ethernet-switch statistics

(10.2.2.1, 10.2.2.1)

(10.2.2.1, 10.2.2.1)


PING TO LOCAL ADDRESS – INTERFACE

M320 M120 so-1/0/0 so-4/1/0 10.2.2/30 .1

RE

CB

FPC #1 PIC

.2

RE

CB

FEB #4 PIC 1/0 4/1

user@m320> ping 10.2.2.1 interface so-1/0/0

  The interface option only alters the source IP address by default

FPC #4

  Still packet does not leave the Routing Engine

(10.2.2.1, 10.2.2.1)

(10.2.2.1, 10.2.2.1)


PING TO LOCAL ADDRESS – BYPASS-ROUTING

M320 M120 so-1/0/0 so-4/1/0 10.2.2/30 .1

RE

CB

FPC #1 PIC

.2

RE

CB

FEB #4 PIC 1/0 4/1


FPC #4 (10.2.2.1, 10.2.2.1)

(10.2.2.1, 10.2.2.1)

  bypass-routing allows to force the packet to go out a given interface   Only works properly at SONET/SDH interfaces


LOCAL AND REMOTE LOOPBACK


PING TO REMOTE ADDRESS – LOOPBACK REMOTE

RE

CB

FPC #1 PIC

RE

CB

FEB #4 PIC 1/0 4/1

user@m320> ping 10.2.2.2

  Packet loops until TTL expires

FPC #4

  The RE originating ICMP echo packets receive ICMP time exceeded   On the right: packet copies sent to the PFE hit firewall filters (counting)

(10.2.2.1, 10.2.2.2)

(10.2.2.1, 10.2.2.2)

user@m320> ping 10.2.2.2 PING 10.2.2.2 (10.2.2.2): 56 data bytes 36 bytes from 10.2.2.1: Time to live exceeded Vr HL TOS Len ID Flg off TTL Pro cks Src Dst 4 5 00 0054 8212 0 0000 01 01 1f91 10.2.2.1 10.2.2.2

M320 M120


PING TO REMOTE ADDRESS – LOOPBACK LOCAL (I)

RE

CB

FPC #1 PIC

RE

CB

FEB #4 PIC 1/0 4/1

user@m320> ping 10.2.2.2

  Packet loops until TTL expires

FPC #4

  The RE originating ICMP echo packets receive ICMP time exceeded

(10.2.2.2, 10.2.2.1) (10.2.2.1, 10.2.2.2)

M320 M120

user@m120> ping 10.2.2.1

user@M320# edit interfaces so-1/0/0 [ no-keepalives ; sonet-options loopback local; ]


(*) May be necessary to remove “family iso” and “family mpls” for the test


PING TO REMOTE ADDRESS – LOOPBACK LOCAL (II)

RE

CB

FPC #1 PIC

RE

CB

FEB #4 PIC 1/0 4/1

user@m320> ping 10.2.2.2

  Output firewall filters require double lookup and fabric pass

FPC #4

  The RE originating ICMP echo packets receive ICMP time exceeded

(10.2.2.2, 10.2.2.1) (10.2.2.1, 10.2.2.2)

[edit firewall family inet filter prueba-loopback] term unico then { count paquetes; accept; } [edit interfaces so-1/0/0 unit 0 family inet] filter output prueba-loopback;

user@m120> ping 10.2.2.1

SIBs


PING TO LOCAL ADDRESS – LOOPBACK REMOTE

RE

CB

FPC #1 PIC

RE

CB

FEB #4 PIC 1/0 4/1


FPC #4 (10.2.2.1, 10.2.2.1)

(10.2.2.1, 10.2.2.1)

user@M320# set interfaces so-1/0/0 no-keepalives

user@M120# set interfaces so-4/1/0 no-keepalives

user@M120# set interfaces so-4/1/0 sonet-options loopback remote

M320 M120

  Two simultaneous troubleshooting paths   Original packet looped by the remote PIC and sent back to originator   On the right: packet copies sent to the PFE hit firewall filters (counting)


PING TO LOCAL ADDRESS – LOOPBACK LOCAL

RE

CB

FPC #1 PIC

RE

CB

FEB #4 PIC 1/0 4/1


FPC #4

(10.2.2.2, 10.2.2.1) (10.2.2.1, 10.2.2.2)

M320 M120




  bypass-routing allows to force the packet to go out a given interface   Only works properly at SONET/SDH interfaces


IMPLEMENTATION DETAILS


IMPLEMENTATION DETAILS – BYPASS-ROUTING

RE

CB

FPC #1 PIC

RE

CB

FEB #4 PIC 1/0 4/1

  With a logical loop, the packet traverse both PIC framers

FPC #4

  This would spot interoperability issues between the framers   Problem can be isolated to be caused by the line or by the endpoints

•  Not necessarily by which of the endpoints

(10.2.2.1, 10.2.2.1)

M320 M120


SONET FRAMERS


IMPLEMENTATION DETAILS – LOOPBACK

RE

CB

FPC #1 PIC

RE

CB

FEB #4 PIC 1/0 4/1

  The PIC just loops the SONET frame

FPC #4

  The PIC framers do not modify the SONET frame at all   There is no way with loopbacks to traverse both PIC framers

(10.2.2.1, 10.2.2.2)

M320 M120

user@m320> show interfaces so-1/0/0 extensive | match trace

Received path trace: m320 so-1/0/0

Transmitted path trace: m320 so-1/0/0

SONET FRAMERS

loopback local at M120 so-4/1/0 loopback remote at M120 so-4/1/0


IMPLEMENTATION DETAILS – TRANSIT PING

RE

CB (control)

  The record-route option is useful to spot fabric failures   Different hardware path followed for each type of packet

FEB #4

PIC 4/0 FPC #4

FEB

#5 PIC 5/1 FPC #5

CB (fabric)

transit ping with record-route option transit ping with no special option


Interface Type loopback mode ping options

local remote interface interface & bypass-routing

SONET/SDH Yes Yes Yes Yes GE/100GE Yes No Yes No

ATM Yes No Yes No FR (E3 IQ) Yes No Yes No

NON-SONET INTERFACE – CAPABILITIES The bypass-routing option can be used, but it does not work

The remote loopback option is not available either

How to use loops? ping the remote link address, and count TTL expired packets

user@m320> show system statistics icmp | match exceed

time exceeded: 177


TOS OPTION


IP Precedence DSCP 3 bit 6 bit 8 bit

bin dec bin dec bin dec hex 000 0 000000 0 00000000 0 0x00 001 1 001000 8 00100000 32 0x20 010 2 010000 16 01000000 64 0x40 011 3 011000 24 01100000 96 0x60 100 4 100000 32 10000000 128 0x80 101 5 101000 40 10100000 160 0xa0 110 6 110000 48 11000000 192 0xc0 111 7 111000 56 11100000 224 0xe0

TOS VALUES – DIFFERENT ENCODINGS The table below displays the formats used for:

  3-bit & 6-bit bin: inet-precedence & dscp – classifiers & rewrite-rules   3-bit & 6-bit dec: from precedence & dscp | traffic-class – firewall filters   8-bit dec: ping command tos option, both for IPv4&IPv6   8-bit hex: dscp or traffic class field displayed in tcpdump decoding


ICMP AS CONTROL TRAFFIC Control traffic COS is determined by the RE

The Routing Engine sets the DSCP/IP Precedence as well as the internal FC+PLP values of a packet before sending it to the Egress PFE

By default, locally originated ICMP goes to queue 0   Regardless of the ping “tos” value

The ping “tos” option can change the DSCP/IP Precedence but not the queue the packet goes to

The ICMP echo reply mirrors the DSCP/IP Precedence from the original ICMP echo request

In Junos OS 10.4 output lo0 firewall filters support actions to rewrite FC,PLP (queue number) and DSCP/IP Precedence independently before sending packet to PFE

Egress control packets are never processed by rewrite rules


LAB DIAGRAMS


M320 M120 so-1/0/0 so-4/1/0 10.2.2/30 .1

so-5/1/0 .2

so-1/1/0 10.3.3/30

M7i

lo0.0 10.100.3.3

NETWORK DIAGRAM

lo0.0 10.100.1.1

lo0.0 10.100.2.2


FAILURE SCENARIO – COMPLETE TRAFFIC LOSS IN A LINK

RE

CB

FPC #1 PIC

RE

CB

FEB #4 PIC 1/0 4/1 FPC #4

M320 M120

test failure test success


FAILURE SCENARIO – TRAFFIC DEGRADATION IN A SINGLE LINK

RE

CB

FPC #1 PIC

RE

CB

FEB #5

PIC 1/1 5/1 FPC #5

M320 M120


PIC 1/1

FEB #4

PIC 4/1

PIC 4/0

FPC #4


FAILURE SCENARIO – TRAFFIC DEGRADATION IN A DOUBLE LINK

RE

CB

FPC #1 PIC

RE

CB

FEB #5

PIC 1/1 5/1

FPC

#5

M320 M120


PIC 1/0

FEB #4

PIC 4/1

PIC 4/0

FPC #4

M7i

CHANGE-LOG:

When Who Rev What

20110526 [email protected] v1.0 Presented to customer 20110527 [email protected] v1.1 Added lab slides, sending to customer

Documents

Troubleshooting Forwarding Plane