Troubleshooting BGPd2zmdbbm9feqrf.cloudfront.net/2010/usa/pdf/BRKRST-3320.pdf · Troubleshooting BGP Neighbor Establishment. ... Cisco Confidential 18 BGP Speakers Won’t Peer •

Session ID-BRKRST-3320

Troubleshooting BGP

© 2010 Cisco and/or its affiliates. All rights reserved. Cisco ConfidentialPresentation_ID 2

Agenda

• Housekeeping Items

• Knowing Your Crime Scene and Your Suspects

• Troubleshooting References

• Troubleshooting Peers

• Best Path

• BGP Convergence

• High Utilization

• BGP Routing Problems


Housekeeping

Cell Phones

Who are you?

“Traditional” Service Provider

Enterprise/State/School Offering Service Provider like services

Enterprise with a BGP Core

Enterprise Doing BGP Cause your SP told you to

CCIE

“Advanced” Class

Assume BGP Operational Experience

Basic config

Show commands

Understand BGP attributes


Knowing Your Crime Scene and Your Suspects



The Usual Suspects Configs

Layer 1

IGP

Memory

CPU

MTU

Design

Input Queues

Bug

………. BGP Table

BGP Best Path

RIB


AS 1BGPTCP

IP

AS 2

Layer 2 Cloud

eBGP

R2R1

BGPTCP

IP


Your Usual Suspects Not true point-to-point

Down detection

Presentation doesn’t match provisioning

………

+ the “usual” suspects


AS 65500

Intranet/IGP

iBGP

R2R1


Your Usual Suspects Network stability

IGP churn

........


BGPTCP

IP

BGPTCP

IP


Troubleshooting References


References

• BGP Support Pagehttp://www.cisco.com/en/US/customer/tech/tk365/tk80/tsd_technology_support_sub-protocol_home.html

(much more )

http://www.cisco.com/en/US/customer/tech/tk365/tk80/tsd_technology_support_sub-protocol_home.html�


Troubleshooting BGP

http://www.cisco.com/en/US/tech/tk365/technologies_tech_note09186a008009478a.shtml#main

BGP neighbor is flapping or unable toEstablish BGP peering

BGP Routes not in the Routing Table

BGP Route Advertisement

Multihoming/Loadsharing

Malloc Failures & BGP Memory Consumption

BGP and Firewalls

Troubleshooting Flapping BGP Routes

Issues with BGP Conditional Advertisement

References

http://www.cisco.com/en/US/tech/tk365/technologies_tech_note09186a008009478a.shtml�


ReferencesTroubleshooting BGP Neighbor Establishment

http://www.cisco.com/en/US/tech/tk365/technologies_tech_note09186a008009478a.shtml#bgp_trouble_neighbor

AS Number

IP Address

multihop

update-source

Extended Ping: peering address to

peering address

Access-lists

Extended Ping: With Max size Packet

(MTU)



ReferencesTroubleshooting Routes Missing from the Routing Table

http://www.cisco.com/en/US/tech/tk365/technologies_tech_note09186a008009478a.shtml#bgp_trouble_route_missing

Synchronization

Valid Next Hop?

Inbound filters

Does the path exist in the BGP Table?

Inbound filters

BGP NeighborNot Advertising



ReferencesTroubleshooting BGP Route Advertisement

Network command

Redistributing

Route-map or filter list

Aggregate-address

Classful network

Auto-summary Access-list or route-maps


ReferencesTroubleshooting Multihoming Inbound

http://www.cisco.com/en/US/tech/tk365/technologies_tech_note09186a008009478a.shtml#bgp_trouble_multi_in

Different ISPs

Single ISP andsingle router

Single ISP andmultiple routers



Troubleshooting Peers


router bgp {R1_AS#}neighbor {R2_BGP_Peer_Add} remote-as {R2_AS#}

R2router bgp {R2_AS#}neighbor {R1_BGP_Peer_Add} remote-as {R1_AS#}

R1

R1_BGP_Peer_Add

R2_BGP_Peer_Add

router bgp {R1_AS#}neighbor {R2_BGP_Peer_Add} remote-as {R2_AS#}neighbor {R2_BGP_Peer_Add} update-source {R1_BGP_Peer_Add}

R2router bgp {R2_AS#}neighbor {R1_BGP_Peer_Add} remote-as {R1_AS#}neighbor {R1_BGP_Peer_Add} update-source {R2_BGP_Peer_Add}

R1

Configs

BGP Speakers Won’t Peer


interface Loop0ip address 1.1.1.1/32

router bgp 100neighbor 2.2.2.2 remote-as {R2_AS#}neighbor 2.2.2.2 update-source Loop0

R2

interface Loop0ip address 2.2.2.2/32

router bgp 100neighbor 1.1.1.1 remote-as {R1_AS#}neighbor 1.1.1.1 update-source Loop0

R1

Extended ping - between peering addresses

R1 BGP session to 2.2.2.2 not successful

BGP Debugs on R1 showBGP: 2.2.2.2 open active, local address 1.1.1.1BGP: 2.2.2.2 open failed: Connection timed out; remote host not responding

Extended Ping Between Configured BGP Peering AddressesR1#ping Protocol [ip]: Target IP address: 2.2.2.2Repeat count [5]: Datagram size [100]: Timeout in seconds [2]: Extended commands [n]: ySource address or interface: 1.1.1.1..Sending 5, 100-byte ICMP Echos to 2.2.2.2.....Success rate is 0 percent (0/5)R1#




• Remember that BGP runs on top of IP, and can be affected by:

IP reachability problems (the underlying routing isn’t working)Access ListsTCP problemsMTU Issues – extended ping and sweep address rangesRate limitingTraffic shapingTunneling problemsEtc.


BGP Speakers Won’t PeerUseful Peer Troubleshooting Commands

show tcp brief all TCB Local Address Foreign Address (state)

64316F14 1.1.1.1.12345 2.2.2.2.179 ESTAB

6431BA8C *.179 2.2.2.2.* LISTEN

62FFDEF4 *.* *.* LISTEN

show tcp statistics Rcvd: 7005 Total, 10 no port

0 checksum error, 0 bad offset, 0 too short

....

0 out-of-order packets (0 bytes)

....

4186 ack packets (73521 bytes)

....

Sent: 9150 Total, 0 urgent packets

4810 control packets (including 127 retransmitted)

2172 data packets (71504 bytes)

....


BGP Speakers Won’t PeerUseful Peer Troubleshooting Commands

debug ip tcp transactions R1#sh log | i TCP0:

TCP0: state was ESTAB -> FINWAIT1 [12345 -> 2.2.2.2(179)]

TCP0: sending FIN

TCP0: state was FINWAIT1 -> FINWAIT2 [12345 -> 2.2.2.2(179)]

TCP0: FIN processed

TCP0: state was FINWAIT2 -> TIMEWAIT [12345 -> 2.2.2.2(179)]

TCP0: Connection to 2.2.2.2:179, advertising MSS 1460

TCP0: state was CLOSED -> SYNSENT [12346 -> 2.2.2.2(179)]

TCP0: state was SYNSENT -> ESTAB [12346 -> 2.2.2.2(179)]

TCP0: tcb 6430DCDC connection to 2.2.2.2:179, received MSS 1460, MSS is 1460

This can be very chatty, so be careful with this debug!



• If the connectivity is good, the next step is to check BGP itself

• debug ip bgpUse with cautionConfigure so the output goes to the log, rather than the console

logging buffered <size>

no logging console

It’s easier to find the problem points this wayrouter#show log | i NOTIFICATION



• show ip bgp neighbor 1.1.1.1 | include last reset

This should give you the resets for a peer

The same information as is shown through debug ip bgp

• bgp log-neighbor changesProvides much of the same information as debug ip bgp, as well


R2

R1

View from R2 Only - Wrong AS

10.1.2.1

10.1.2.2

R2# show log | include NOTI%BGP-3-NOTIFICATION: sent to neighbor 10.1.2.1 2/2 (peer in wrong AS) 2 bytes 0064 FFFF FFFF FFFF FFFF FFFF FFFF FFFF FFFF 002D 0104 0064 00B4 0101 0101 1002 0601 0400 0100 0102 0280 0002 0202 00

Sniff of BGP Notification Sent from R1 to R2

router bgp 200no synchronizationbgp log-neighbor-changesneighbor 10.1.2.1 remote-as 10no auto-summary

X ‘0064’ = decimal 100



R2

R1


10.1.2.1

10.1.2.2

R2# debug ip bgpBGP: 10.1.2.1 bad OPEN, remote AS is 100, expected 10BGP: 10.1.2.1 went from OpenSent to Closing%BGP-3-NOTIFICATION: sent to neighbor 10.1.2.1 2/2 (peer in wrong AS) 2 bytes 0064 FFFF FFFF FFFF FFFF FFFF FFFF FFFF FFFF 002D 0104 0064 00B4 0101 0101 1002 0601 0400 0100 0102 0280 0002 0202 00BGP: 10.1.2.1 send message type 3, length (incl. header) 23BGP: 10.1.2.1 local error close after sending NOTIFICATION





R2

R1


10.1.2.1

10.1.2.2 router bgp 200no synchronizationbgp log-neighbor-changesneighbor 10.1.2.1 remote-as 10no auto-summary

Question: What did R1 see

R1#sh log | include NOTI%BGP-3-NOTIFICATION: received from neighbor 10.1.2.2 2/2 (peer in wrong AS) 2 bytes 0064





BGP uses a TTL of 1 for eBGP peers

For eBGP peers that are more than 1 hop away a larger TTL must be used

neighbor x.x.x.x ebgp-multihop[2-255] R1

AS65000

R2AS65001

Default TTL

Configured TTL

eBGP Peers and Time to Live


BGP Speakers Won’t PeerDirectly Connected eBGP Peers

router bgp 100no synchronizationbgp log-neighbor-changesneighbor 2.2.2.2 remote-as 200neighbor 2.2.2.2 disable-connected-checkneighbor 2.2.2.2 update-source Loopback0no auto-summary

router bgp 100no synchronizationbgp log-neighbor-changesneighbor 2.2.2.2 remote-as 200neighbor 2.2.2.2 ebgp-multihop 2neighbor 2.2.2.2 update-source Loopback0no auto-summary

R2

R1

10.1.2.1

10.1.2.2


BGP Speakers Won’t PeerDirectly Connected eBGP Peers

router bgp 100no synchronizationbgp log-neighbor-changesneighbor 2.2.2.2 remote-as 200neighbor 2.2.2.2 disable-connected-checkneighbor 2.2.2.2 update-source Loopback0no auto-summary

R2

R1

10.1.2.1

10.1.2.2

“neighbor disable-connected-check

To disable connection verification to establish an eBGP peering session with a single-hop peer that uses a loopback interface, use the neighbor disable-connected-check command in Address Family or Router configuration mode. To enable connection verification for eBGP peering sessions, use the no form of this command.”

http://www.cisco.com/en/US/docs/ios/12_3t/ip_route/command/reference/ip2_n1gt.html#wp1109875

http://www.cisco.com/en/US/docs/ios/12_3t/ip_route/command/reference/ip2_n1gt.html�



unknown subcode The peer open notification subcode isn’t known

incompatible BGP version The version of BGP the peer is running isn’t compatible with the local version of BGP

peer in wrong AS The AS this peer is locally configured for doesn’t match the AS the peer is advertising

BGP identifier wrong The BGP router ID is the same as the local BGP router ID

unsupported optional parameter

There is an option in the packet which the local BGP speaker doesn’t recognize

authentication failure The MD5 hash on the received packet does not match the correct MD5 hash

unacceptable hold time The remove BGP peer has requested a BGP hold time which is not allowed (too low)

unsupported/disjoint capability The peer has asked for support for a feature which the local router does not support

%BGP-3-NOTIFICATION: sent to neighbor 2.2.2.2 2/2 (peer in wrong AS) 2 bytes 00C8 FFFF FFFF FFFF FFFF FFFF FFFF FFFF FFFF 002D 0104 00C8 00B4 0202 0202 1002 0601 0400 0100 0102 0280 0002 0202 00

Bad Messages


BGP Speaker Flap

• Here we see a message from bgp log-neighbor-changes telling us the hold timer expired

• We can double check this by looking at show ip bgp neighbor x.x.x.x | include last reset

R1

R2

%BGP-5-ADJCHANGE: neighbor 1.1.1.1 Down BGP Notification sent%BGP-3-NOTIFICATION: sent to neighbor 1.1.1.1 4/0 (hold time expired)

R2#show ip bgp neighbor 1.1.1.1 | include last resetLast reset 00:01:02, due to BGP Notification sent,hold time expired

Case Study


BGP Speaker Flap

There are lots of possibilities here

R1 has a problem sending keepalives?

The keepalives are lost in the cloud?

R2 has a problem receiving the keepalive?

R1

R2

Case Study

%BGP-5-ADJCHANGE: neighbor 1.1.1.1 Down BGP Notification sent%BGP-3-NOTIFICATION: sent to neighbor 1.1.1.1 4/0 (hold time expired)

R2#show ip bgp neighbor 1.1.1.1 | include last resetLast reset 00:01:02, due to BGP Notification sent,hold time expired


BGP Speaker Flap

• Did R1 build and transmit a keepalive for R2?debug ip bgp keepaliveshow ip bgp neighbor

• When did we last send or receive data with the peer?

R2#show ip bgp neighbors 1.1.1.1BGP neighbor is 1.1.1.1, remote AS 100, external linkBGP version 4, remote router ID 1.1.1.1BGP state = Established, up for 00:12:49Last read 00:00:45, last write 00:00:44, hold time is 180,

keepalive interval is 60 seconds

• If R1 did not build and transmit a KAHow is R1 on memory?What is the R1’s CPU load?Is R2’s TCP window open?

Case Study


BGP Speaker Flap

R1#show ip bgp sum | begin NeighborNeighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd2.2.2.2 4 2 53 284 10167 0 97 00:02:15 0

Case Study

R1#show ip bgp summary | begin NeighborNeighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd2.2.2.2 4 2 53 284 10167 0 98 00:03:04 0

At least one BGP keepalive interval apart

The number of packets generated is increasing

But the number of packets transmitted is not increasing

The keepalives aren’t leaving R1!


BGP Speaker Flap

• Go back to square one and check the IP connectivityThis is a layer 2 or 3 transport issue, etc.

R1#ping 2.2.2.2Type escape sequence to abort.Sending 5, 100-byte ICMP Echos to 2.2.2.2, timeout is 2 seconds:!!!!!Success rate is 100 percent (5/5), round-trip min/avg/max = 16/21/24 m

R2#ping 2.2.2.2 size 1500 df-bit

Type escape sequence to abort.Sending 5, 1500-byte ICMP Echos to 2.2.2.2, timeout is 2 seconds:Packet sent with the DF bit set…..Success rate is 100 percent (5/5), round-trip min/avg/max = 1/1/1 msR2#

Case Study


Best Path


Best Path

http://www.cisco.com/en/US/tech/tk365/technologies_tech_note09186a0080094431.shtml#bestpath

Algorithm 1 Weight Highest wins Scope is router only

2 Local Preference Highest wins Scope is AS only

3 Locally Originated

4 AS_Path Shortest wins Skipped if bgp bestpath as-path ignore configured

5 Origin Type Lowest IGP, EGP, Incomplete

6 MED Lowest MEDs are compared only if the first AS in the AS_SEQUENCE is the same for multiple paths.

7 eBGP over iBGP If bestpath is selected, go to #9 (multipath)

8 Metric to Next Hop Lowest Continue, even if bestpath is already selected

9 Multiple Paths in RIB

Continue, if bestpath not yet selected

10 Oldest Wins Unless BGP best path compare router-id configured

11 BGP Router ID Lowest

12 Cluster List Smallest Only present in BGP RR

13 Neighbor Address Lowest

http://www.cisco.com/en/US/tech/tk365/technologies_tech_note09186a0080094431.shtml�

http://www.cisco.com/en/US/tech/tk365/technologies_tech_note09186a0080094431.shtml�


Best Path

Code

Real Term Who Wins

Numerous N Next-hop

1 Wise W Weight Highest

2 LiP LP Local Preference

Highest

3 LOvers LO Locally Originated

4 APply AP AS PATH Shortest

5 ORal OR Origin

6 MEDication MED MED Lowest (see note 1 below)

Note 1: The default behavior for MED comparison is only done if the first AS in the AS_SEQUENCE is the same for multiple paths.

Policy Tools


BGP Convergence


BGP Slow Convergence

• Hey—Who are you calling slow?Slow is a relative term....

BGP probably won’t ever converge as fast as any of the IGPs

• Two general convergence situationsInitial startup between peers

Route changes between existing peers



• Initial convergence is limited byThe number of packets required to transfer the entire BGP database

The number of routes

The ability of BGP to pack routes into a small number of packets

The number of peer specific policies

TCP transport issues

How often does TCP go into slow start?

How much can TCP put into one packet?

Initial Convergence



BGP starts a packet by building an attribute set

It then packs as many destinations (NLRIs) as it can into the packet

Only destinations with the same attribute set can be placed in the packetDestinations can only be put into the packet until it’s full

Initial Convergence

Attr

ibut

eN

LRI

Attr

ibut

eN

LRI

Attr

ibut

eN

LRI

NLR

I

Less

Effi

cien

t

Mor

e E

ffici

ent

First rule of thumb: to increase convergence speed, decrease unique sets of attributes



The larger the packet BGP can build, the more destinations it can put in the packet

The more you can put in a single packet, the less often you have to repeat the same attributes

Initial Convergence

Attr

ibut

eN

LRI

Attr

ibut

eN

LRI

Attr

ibut

eN

LRI

NLR

I

Less

Effi

cien

t

Mor

e E

ffici

ent

Attr

ibut

eN

LRI

NLR

IA

ttrib

ute

NLR

IN

LRI

Attr

ibut

eN

LRI

NLR

IN

LRI

NLR

I

Less

Effi

cien

t

Mor

e E

ffici

ent

Second rule of thumb: allow BGP to use the largest packets possible



• BGP must create packets based the policies towards each peer

Initial Convergence

Attr

ibut

eN

LRI

NLR

I

Less

Effi

cien

t

Attr

ibut

eN

LRI

NLR

I

Attr

ibut

eN

LRI

NLR

IM

ore

Effi

cien

t

Third rule of thumb: Minimize the number of unique policies towards eBGP peers



• TCP Interactions

Each time a TCP packet is dropped, the session goes into slow start

It takes a good deal of time for a TCP session to come out of slow start

Initial Convergence

Fourth rule of Thumb: Try and reduce the circumstances under which a TCP segment will be dropped during initial convergence



• Bottom Line:Hold down the number of unique attributes per route

Don’t send communities if you don’t need to, etc

Hold down the number of policies towards eBGP peersTry to find a small set of common policies, rather than individualizing policies per peer

Stop TCP segment dropsIncrease input queues

Increase SPD thresholds

Make certain links are clean

Initial Convergence



There are two elements to route change convergence for BGP

How long does it take to see the failure?How long does it take to propagate information about the failure?

For faster peer down detection, there are several tools you can use

Fast layer two down detectionFast external fallover for directly connected eBGP peers

Faster keepalive and dead interval timers

Down to 3 and 9 are commonly used today

Route Change Convergence



Fast Session Deactivation• The address of each peer is registered with

the Address Tracking Filter (ATF) system

• When the state of the route changes, ATF notifies BGP

• BGP tears down the peer impacted

• BGP does not wait on the hold timer to expire


BGP/RIB ATF Interface

eBGP Multihop SessionLink fails; IGP converges

ATF Notifies BGP

BGP Tears Down eBGP Session



• Very dangerous for iBGP peersIGP may not have a route to a peer for a split second

FSD would tear down the BGP session

Imagine if you lose your IGP route to your RR (Route Reflector) for just 100ms

• Off by defaultneighbor x.x.x.x fall-over




• ATF can also be used to track changes in next hopsiBGP recurses onto an IGP next hop to find a path through the local AS

Changes in the IGP cost or reachability are normally seen only by the BGP scanner

Since the scanner runs every 60 seconds, by default, this means iBGP convergence can take up to 60 seconds on an IGP change....



• BGP Next Hop TrackingEnabled by default

[no] bgp nexthop trigger enable

• BGP registers all nexthops with ATFHidden command will let you see a list of nexthops

show ip bgp attr nexthop

• ATF will let BGP know when a route change occurs for a nexthop

• ATF notification will trigger a lightweight “BGP Scanner” runBestpaths will be calculated

None of the other “Full Scan” work will happen

BGP Slow ConvergenceRoute Change Convergence


• Once an ATF notification is received BGP waits 5 seconds before triggering NHT scan

bgp nexthop trigger delay <0-100>

May lower default value as we gain experience

• Event driven model allows BGP to react quickly to IGP changes

No longer need to wait as long as 60 seconds for BGP to scan the table and recalculate bestpaths

Tuning your IGP for fast convergence is recommended



• Dampening is used to reduce frequency of triggered scans

• show ip bgp internalDisplays data on when the last NHT scan occurred

Time until the next NHT may occur (dampening information)

• New commandsbgp nexthop trigger enable


show ip bgp attr next-hop ribfilter

debug ip bgp events nexthop

debug ip bgp rib-filter

• Full BGP scan still happens every 60 secondsFull scanner will no longer recalculate bestpaths if NHT is enabled



• How is the timer enforced for peer X?Timer starts when all routes have been advertised to X

For the next MRAI (seconds) we will not propagate any bestpath changes to peer X

Once X’s MRAI timer expires, send him updates and withdraws

Restart the timer and the process repeats…

• User may see a wave of updates and withdraws to peer X every MRAI

• User will NOT see a delay of MRAI between each individual update and/or withdraw

BGP would probably never converge if this was the case



• MRAI timeline for iBGP peer

• Bestpath Change #1 at t7 is TXed immediately

• MRAI timer starts at t7, will expire at t12

• Bestpath Change #2 at t10 must wait until t12 for MRAI to expire

• Bestpath Change #2 is TXed at t12

• MRAI timer starts at t12, will expire at t17

• MRAI expires at t17…no updates are pending

t0 t5 t10 t15 t20 t25

•TX update #1•Start MRAI

BestpathChange #2

BestpathChange #1

•MRAI Expires•TX update #2•Start MRAI

•MRAI Expires



• BGP is not a link state protocol

• May take several “rounds/cycles” of exchanging updates and withdraws for the network to converge

• MRAI must expire between each round!

• The more fully meshed the network and the more tiers of ASes, the more rounds required for convergence

• Think aboutHow many tiers of ASes there are in the Internet

How meshy peering can be in the Internet



• Internet churn means we are constantly setting and waiting on MRAI timers

One flapping prefix slows convergence for all prefixes

Internet table sees roughly 6 bestpath changes per second

• For iBGP and PE-CE eBGP peersneighbor x.x.x.x advertisement-interval 0

Will be the default in 12.0(32)S

• For regular eBGP peersLowering to 0 may get you dampened

OK to lower for eBGP peers if they are not using dampening



High Utilization


High Utilization

• High Processor Utilization

• Next Hop Tracking

• High Memory Utilization


High Processor Utilization

• Why?This could be for several reasons

High route churn is the most likely

Router#show process cpuCPU utilization for five seconds: 100%/0%; one minute: 99%; five minutes:

81%....139 6795740 1020252 6660 88.34% 91.63% 74.01% 0 BGP Router



• Check how busy the peers areThe Table Version

You have 150k routes and see the table version increase by 150k every minute … something is wrong

You have 150k routes and see the table version increase by 300 every minute … sounds like normal network churnThe InQ

Flood of incoming updates or build up of unprocessed updates

The OutQFlood of outgoing updates or build up of untransmitted updates

Router#show ip bgp summaryNeighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd10.1.1.1 4 64512 309453 157389 19981 0 253 22:06:44 111633172.16.1.1 4 65101 188934 1047 40081 41 0 00:07:51 58430



• If the Table Version is Changing QuicklyAre you in initial convergence with this peer?Is the peer flapping for some reason?

Examine the table entries from this peer: why are they changing?

If there is a group of routes which are constantly changing, consider route flap dampening

• If the InQ is highYou should see the table version changing quickly

If it’s not, the peer isn’t acting correctlyConsider shutting it down until the peer can be fixed

• If the OutQ is highLots of updates being generatedCheck table versions of other peers

Check for underlying transport problems



• If the Table Version is Changing Quickly

RR#sh ip bgp vpnv4 rd 81:4BGP table version is 8216218, local router ID is 128.109.210.148Status codes: s suppressed, d damped, h history, * valid, > best, i - internal,

r RIB-failure, S StaleOrigin codes: i - IGP, e - EGP, ? - incomplete

Network Next Hop Metric LocPrf Weight PathRoute Distinguisher: 81:4*>i24.181.203.0/24 128.109.210.217 0 100 0 65502 i*>i128.109.5.44/30 128.109.210.217 0 100 0 ?*>i152.1.99.0/24 128.109.210.197 0 100 0 11442 i*>i152.1.254.0/24 128.109.210.197 0 100 0 11442 I*>i152.1.255.115/32 128.109.210.197 0 100 0 11442 ?*>i152.1.255.205/32 128.109.210.197 0 100 0 11442 I*>i172.26.38.128/27 128.109.210.217 0 100 0 65502 ?*>i192.168.1.0/30 128.109.210.197 0 100 0 ?




RR#sh ip bgp vpnv4 rd 81:4 24.181.203.0BGP routing table entry for 81:4:24.181.203.0/24, version 8216205Paths: (1 available, best #1, no table)Advertised to update-groups:

165502, (Received from a RR-client)128.109.210.217 (metric 131584) from 128.109.210.217 (128.109.210.217)

Origin IGP, metric 0, localpref 100, valid, internal, bestExtended Community: RT:81:4Connector Attribute: count=1type 1 len 12 value 81:4:128.109.210.217mpls labels in/out nolabel/30




RR#sh ip bgp vpnv4 all version ? <1-4294967295> Version number and aboverecent Offset from the current table version

RR#sh ip bgp vpnv4 all version recent 1BGP table version is 8216322, local router ID is 128.109.210.148Status codes: s suppressed, d damped, h history, * valid, > best, i - internal,

r RIB-failure, S StaleOrigin codes: i - IGP, e - EGP, ? - incomplete

Network Next Hop Metric LocPrf Weight PathRoute Distinguisher: 81:2*>i128.109.70.104/30

128.109.210.248 0 100 0 ?Route Distinguisher: 81:3*>i1.3.1.0/24 128.109.210.248 0 7500 0 19401 i



• Check on the BGP ScannerWalks the table looking for changed next hops

Checks conditional advertisement

Imports from and exports to VPNv4 VRFs

Router#show processes | include BGP Scanner172 Lsi 407A1BFC 29144 29130 1000 8384/9000 0 BGP Scanner



To relieve pressure on the BGP ScannerUpgrade to newer code

Most of the work of the BGP Scanner has been moved to an event driven modelThis has reduced the impact of BGP Scanner significantly

Reduce route and view count

Reduce or eliminate other processes which walk the RIBSNMP routing table walks, for instance

Deploy BGP Next Hop Tracking (NHT)


Next Hop Tracking

• ATF is a middle man between the RIB and RIB clients BGP, OSPF, EIGRP, etc are all clients of the RIB

• A client tells ATF what prefixes he is interested in

• ATF tracks each prefixNotify the client when the route to a registered prefix changes

Client is responsible for taking action based on ATF notification

Provides a scalable event driven model for dealing with RIB changes


Next Hop Tracking

• BGP tells ATF to let us know about any changes to 10.1.1.3 and 10.1.1.5

• ATF filters out any changes for 10.1.1.1/32, 10.1.1.2/32, and 10.1.1.4/32

• Changes to 10.1.1.3/32 and 10.1.1.5/32 are passed along to BGP

RIB10.1.1.1/3210.1.1.2/3210.1.1.3/3210.1.1.4/3210.1.1.5/32

ATF

BGP BGP Nexthops10.1.1.310.1.1.5


Next Hop Tracking

• BGP Next Hop TrackingEnabled by default[no] bgp nexthop trigger enable

• BGP registers all nexthops with ATFHidden command will let you see a list of nexthopsshow ip bgp attr nexthop

• ATF will let BGP know when a route change occurs for a nexthop

• ATF notification will trigger a lightweight “BGP Scanner” runBestpaths will be calculatedNone of the other “Full Scan” work will happen


Next Hop Tracking

• Once an ATF notification is received BGP waits 5 seconds before triggering NHT scan


May lower default value as we gain experience

• Event driven model allows BGP to react quickly to IGP changes

No longer need to wait as long as 60 seconds for BGP to scan the table and recalculate bestpathsTuning your IGP for fast convergence is recommended


Next Hop Tracking

• Dampening is used to reduce frequency of triggered scans

• show ip bgp internalDisplays data on when the last NHT scan occurred

Time until the next NHT may occur (dampening information)

• New commandsbgp nexthop trigger enable


show ip bgp attr next-hop ribfilter

debug ip bgp events nexthop

debug ip bgp rib-filter

• Full BGP scan still happens every 60 secondsFull scanner will no longer recalculate bestpaths if NHT is enabled


High Memory Utilization

Why is BGP taking up so much memory?

A BGP speaker generally receives a number of copies of the same route or set of routes

Each of these copies of the same route or routes is called a view

A has two views of 10.1.1.0/24

10.1.1.0/24

A

B

C

D

First ViewSecond View

Views and Routes



Multiple views can come from:iBGP peers peering with the same remote AS

iBGP peers peering with remote AS’ with (generally) the same tableThis is common in the case of the global Internet

eBGP peers peering with the same remote AS

eBGP peers peering with remote AS’ with (generally) the same tableThis is common in the case of the global Internet

Views and Routes



• Multiple views exist in IGPs, as wellBut not on the same scale

Neighbor adjacencies in IGPs are generally on a lower scaleIn the hundreds, not the thousands

Neighbor adjacencies in IGPs normally pick up different routes, rather than the same route multiple times

• Each view takes up some amount of space250,000 routes x 100 views == a lot of memory usage

Views and Routes



• To reduce memory consumption:

Reduce the number of routesThis is particularly true in providers supporting L3VPN servicesThe route and view count can escalate quickly when supporting many customer’s L3VPNs

Filter aggressively

Accept partial routing tables, rather than full routing tables

Reduce the number of viewsUse route reflectors rather than full mesh iBGP peeringPeer only when needed

Views and Routes



• BGP implementations build their memory structures around minimizing storage

• Attributes are stored onceRather than once per routeEach route references an

attribute set, rather than storing the attribute set

• This is similar to the way BGP updates are formed

Attributes

10.1.1.0/24

10.1.2.0/24

10.1.3.0/24

10.1.4.0/24

AS Path 1

AS Path 2

Community Set 1

Community Set 2



• The more unique attribute sets you’re receiving, the more unique attribute sets you need to store

You might have the same number of routes and views over time, but memory utilization can increase

Attributes

10.1.1.0/24

10.1.2.0/24

10.1.3.0/24

10.1.4.0/24

AS Path 1

AS Path 2

Community Set 1

Community Set 2

AS Path 3

Community Set 3

Community Set 4



• To Conserve MemoryStrip unneeded attributes on the inbound side of eBGP peering sessions

Verify you don’t really need them, or they aren’t useful after the route has transited your AS

Communities are the biggest/only target

Use Communities wisely within your networkA large mishmash of communities can consume memory

Attributes



• B advertises 10.1.1.0/24 to A

• A filters the route locally

• The filters on A are changed to permit 10.1.1.0/24

• But how does A relearn 10.1.1.0/24?

Soft Reconfiguration

10.1.1.0/24

Advertise 10.1.1.0/24

Blocked by filterA

B



• With soft reconfiguration, A saves all the routes it receives from B

• Applies any inbound filters between this saved copy of B’s updates and the local BGP table

• If the local filters change, they can be reapplied by simply pulling all the updates from the saved table into the local BGP table


10.1.1.0/24

Blocked by filterA

B Advertise 10.1.1.0/24

Local Copy

Local Filter



• Keeping this local copy uses a lot of memory

• In general, don’t use soft-reconfiguration

• BGP now uses the route refresh capability to rebuild the local table if the local filters change


10.1.1.0/24

Blocked by filterA

B Advertise 10.1.1.0/24

Local Copy

Local Filter

Consumes large amounts of memory


Routing Problems


BGP Routing Problems

• Route Reflector Loops

• Route Reflector Suboptimal Routes

• BGP Multihoming


Route Reflector Loops

• Router BBGP Next-Hop: Router ALocal Next-Hop: Router ASet: Next-Hop-Self

• Router CBGP Next-Hop: Router BLocal Next-Hop: Router D

• Router DBGP Next-Hop: Router ELocal Next-Hop: Router C

• Router EBGP Next-Hop: Router ALocal Next-Hop: Router ASet: Next-Hop-Self

B

C

D

E

A

RR

C

RR

C


Route Reflector Loops

This results in a permanent routing loop

Route reflectors must always follow the topology

Never peer through a route reflector client to reach a route reflector

B

C

D

E

A

RR

C

RR

C


Route Reflector Suboptimal Routing

Route reflectors can also cause routing to be different (or suboptimal) compared to full mesh iBGP

E advertises 10.1.1.0/24 through eBGP to both B and C

The local preference, MED, AS Path length, and all other attributes are the same for 10.1.1.0/24 at both B and C

10 5

10

5

10

5

IGP costsA

B DC

E

eBGP

eBGP

10.1.1.0/24



• Assume A, B, C, and D are configured for full mesh iBGP

• A chooses B as its exit point because of the IGP cost

• D chooses C as its exit point, because of the IGP cost

10 5

10

5

10

5

IGP costsA

B DC

E

eBGP

eBGP

10.1.1.0/24

Best Path

Best Path



• Assume B, C and, D are configured as route reflector clients of A

• A chooses B as its best path because of the IGP cost

• A reflects this choice to C, but C chooses its locally learned eBGP route over the internal through B

• A reflects this choice to D, and D chooses the path through B, even though the path through C is shorter

10 5

10

5

10

5

IGP costsA

B DC

E

eBGP

eBGP

10.1.1.0/24

Best Path

Best Path



• There is little you can do about this

• Whenever you remove routing information, you risk suboptimal routing

• Keeping the route reflector topology in line with the layer 3 topology helps

• iBGP multipath can resolve some of these problemsAt the cost of additional memory

• Otherwise, use policy to choose the best exit point


BGP Multihoming

You must announce assigned address block to Internet

You may also announce subprefixes—reachability is not guaranteed

Current RIR* minimum allocation is /21Several ISPs filter RIR blocks on this boundary

Several ISPs filter the rest of address space according to the IANA assignments

This activity is called “net police” by some

*RIR: Regional Internet Registryhttp://en.wikipedia.org/wiki/Regional_Internet_Registry

http://en.wikipedia.org/wiki/Regional_Internet_Registry�


Assigned Netblock Filters

The RIRs publish their minimum allocation sizes at:AfriNIC: www.afrinic.net/docs/policies/afpol-v4200407-000.htm

APNIC: www.apnic.net/db/min-alloc.html

ARIN: www.arin.net/reference/ip_blocks.html

LACNIC: lacnic.net/en/registro/index.html

RIPE NCC: www.ripe.net/ripe/docs/ripe-ncc-managed-address-space.html

(Above picture from http://www.iana.org/numbers)


Assigned Netblock Filters

IANA* publishes the address space it has assigned to end-sites and allocated to the RIRs:

www.iana.org/assignments/ipv4-address-space

Several ISPs use this published information to filter prefixes on:

What should be routed (from IANA)

The minimum allocation size from the RIRs

Example….“Sprint reserves the right to aggregate any announcement for a network smaller than /19 when advertising to external peers such as AT&T, UUnet etc.”https://www.sprint.net/index.php?p=policy_bgp


One Path as Backup

This does not direct all the traffic through the one link, however

The AS path length doesn’t impact forwarding decisions within provider 2

Virtually all providers set the local preference to prefer routes learned from customers over routes learned from peers

Problem #1: We wanted the link between RtrB and RtrD to be used only as a backup but that is not happening

Provider 1 Provider 2

Local Pref Set to Prefer Routes Learned from Customers

10.1.1.0/20{AS#}

B

DC

A

10.1.1.0/20{AS#, AS#}

Upstream Customer 2


One Path as Backup

If customer 2 prefers the path through provider 2

They could have a default only to provider 2

They could be accepting a partial routing table

Etc. …

Then provider 2 will prefer the B->D link

Rather than taking the path through the Upstream->provider 1->A->C

All the traffic coming from provider 2’s customers will follow the B->D link

AS Path Prepend

Provider 1 Provider 2

Local Pref Set to Prefer Routes Learned from Customers

B

DC

A

Upstream Customer 2

10.1.1.0/20{AS#}

10.1.1.0/20{AS#, AS#}


10.1.1.0/24

Inbound Load Sharing

65600

65300 65200

65100

Why can’t we just AS path prepend for this?

Traffic from AS65500 will still flow through 65200

Traffic from AS65400 will still flow through AS65300

Only the traffic sourced from AS65600 will be impacted by AS path prepend by itself

This might work, or it might not

6550065400

Problem #2: We wanted to inbound load share between these 2 connections



What are our other options?

Longer match prefixes are still your friend…*

Advertise 10.1.0.0/22 through one connection

Advertise 10.1.0.0/22 and 10.1.0.0/23 through the other connection

Customer 2

Provider 2Provider 1

Upstream

A

10.1.0.0/2210.1.0.0/2310.1.0.0/22

DC

B

* If your service provider will support this



You can extend this concept by

Adding various longer prefix matches on both linksCombining advertisements of

longer prefixes out both links with AS path prepending

For instance, here we are Prepending the /22 to influence

traffic towards CAdvertising a longer prefix

to influence other traffic towards D

10.1.0.0/2210.1.0.0/2310.1.0.0/22

Customer 2


Upstream

A

DC

B



This example is more commonplace

Shows how ISPs and end-sites subdivide address space frugally, as well as use the AS-PATH prepend concept to optimise the load sharing between different ISPs

Notice that the /22 aggregate block is always announced

10.1.0.0/2210.1.0.0/2310.1.0.0/22

Customer 2


Upstream

A

DC

B


Show BGP Neighbors


XR – Show BGP NeighborRP/0/RP1/CPU0:CRS#sh bgp neighbor 128.109.210.148 detail

BGP neighbor is 128.109.210.148Remote AS 81, local AS 81, internal linkRemote router ID 128.109.210.148BGP state = Established, up for 02:46:51Last read 00:00:41, hold time is 180, keepalive interval is 60 secondsLast write 00:00:51, Last write before reset 02:47:50Last write pulse rcvd Jul 1 15:18:00.690 last full Jul 1 12:15:23.200 pulse count 25209Socket not armed for io, armed for read, armed for writePrecedence: internetNeighbor capabilities: Adv Rcvd

Route refresh: Yes Yes4-byte AS: Yes YesAddress family VPNv4 Unicast: Yes YesAddress family IPv4 MDT: Yes Yes

Message stats:InQ depth: 0, OutQ depth: 0

Last Sent Sent Last_Rcvd RcvdOpen: Jul 1 15:18:00.689 39 Jul 1 15:18:00.688 39Notification: Jul 1 15:17:40.486 33 --- 0Update: Jul 1 15:28:24.021 81254 Jul 1 16:31:32.984 60310Keepalive: Jul 1 18:04:01.011 12867 Jul 1 18:04:10.549 12597Route Refresh: Jun 26 22:47:30.307 4 --- 0Total: 94197 72946

Minimum time between advertisement runs is 0 seconds


XR – Show BGP Neighbor..For Address Family: VPNv4 UnicastBGP neighbor version 39025483Update group: 0.2 (Update Generation Throttled)

Route refresh request: received 0, sent 2Policy for incoming advertisements is pass-allPolicy for outgoing advertisements is pass-all1521 accepted prefixes, 1521 are bestpathsPrefix advertised 1503, suppressed 0, withdrawn 1500, maximum limit 524288Threshold for warning message 75%An EoR was received during read-only modeLast ack version 39025483, Last synced ack version 0Outstanding version objects: current 1, max 5

For Address Family: VPNv6 UnicastBGP neighbor version 82192Update group: 0.1Route refresh request: received 0, sent 2Policy for incoming advertisements is pass-allPolicy for outgoing advertisements is pass-all6 accepted prefixes, 6 are bestpathsPrefix advertised 1, suppressed 0, withdrawn 0, maximum limit 524288Threshold for warning message 75%An EoR was received during read-only modeLast ack version 82192, Last synced ack version 0Outstanding version objects: current 0, max 2


XR – Show BGP NeighborConnections established 39; dropped 38Last reset 03:19:07, due to BGP Notification sent: illegal networkTime since last notification sent to neighbor: 03:19:07Error Code: illegal networkNotification data sent:

FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF002A0200 00001380 0F100001 4260000000510000 0004806D D2D9

RP/0/RP1/CPU0:CRS#


7600 – Show BGP Neighbor7600#sh ip bgp all neighbors 128.109.210.148 .For address family: VPNv4 UnicastBGP neighbor is 128.109.210.148, remote AS 81, internal linkMember of peer-group RRs for session parametersBGP version 4, remote router ID 128.109.210.148BGP state = Established, up for 03:12:13Last read 00:00:31, last write 00:00:04, hold time is 180, keepalive interval is 60 secondsNeighbor sessions:

1 active, is not multisession capableNeighbor capabilities:

Route refresh: advertised and received(new)Address family IPv4 Unicast: advertisedAddress family IPv4 MDT: advertised and receivedAddress family VPNv4 Unicast: advertised and receivedAddress family VPNv6 Unicast: advertised and receivedMultisession Capability: advertised

Message statistics:InQ depth is 0OutQ depth is 0

Sent RcvdOpens: 1 1Notifications: 0 0Updates: 46 98Keepalives: 213 186Route Refresh: 0 0Total: 260 285

Default minimum time between advertisement runs is 0 seconds


7600 – Show BGP Neighbor..Address tracking is enabled, the RIB does have a route to 128.109.210.148Connections established 1; dropped 0Last reset neverTransport(tcp) path-mtu-discovery is enabledGraceful-Restart is disabled

Connection state is ESTAB, I/O status: 1, unread input bytes: 0 Connection is ECN DisabledMininum incoming TTL 0, Outgoing TTL 255Local host: 128.109.210.197, Local port: 18367Foreign host: 128.109.210.148, Foreign port: 179Connection tableid (VRF): 0

Enqueued packets for retransmit: 0, input: 0 mis-ordered: 0 (0 bytes)

Event Timers (current time is 0xB248B0):Timer Starts Wakeups NextRetrans 222 0 0x0TimeWait 0 0 0x0AckHold 215 185 0x0SendWnd 0 0 0x0KeepAlive 0 0 0x0GiveUp 0 0 0x0PmtuAger 1 1 0x0DeadWait 0 0 0x0Linger 0 0 0x0


7600 – Show BGP Neighbor..iss: 4216790207 snduna: 4216799431 sndnxt: 4216799431 sndwnd: 15890irs: 839066505 rcvnxt: 839118621 rcvwnd: 16308 delrcvwnd: 76

SRTT: 300 ms, RTTO: 303 ms, RTV: 3 ms, KRTT: 0 msminRTT: 0 ms, maxRTT: 300 ms, ACK hold: 200 msStatus Flags: noneOption Flags: higher precendence, nagle, path mtu capable

Datagrams (max data segment is 536 bytes):Rcvd: 527 (out of order: 0), with data: 306, total data bytes: 52115Sent: 507 (retransmit: 0 fastretransmit: 0),with data: 230, total data bytes: 9223


Summary



The Usual Suspects Configs

Layer 1

IGP

Memory

CPU

MTU

Design

Input Queues

Bug

………. BGP Table

BGP Best Path

RIB


AS 1BGPTCP

IP

AS 2

Layer 2 Cloud

eBGP

R2R1

BGPTCP

IP


Your Usual Suspects Not true point-to-point

Down detection

Presentation doesn’t match provisioning

………



AS 65500

Intranet/IGP

iBGP

R2R1


Your Usual Suspects Network stability

IGP churn

........


BGPTCP

IP

BGPTCP

IP


Questions?


Complete Your Online Session Evaluation

Give us your feedback and you could win fabulous prizes. Winners announced daily.

Receive 20 Cisco Preferred Access points for each session evaluation you complete.

Complete your session evaluation online now (open a browser through our wireless network to access our portal) or visit one of the Internet stations throughout the Convention Center.

Don’t forget to activate your Cisco Live and Networkers Virtual account for access to all session materials, communities, and on-demand and live activities throughout the year. Activate your account at any internet station or visit www.ciscolivevirtual.com.

http://www.ciscolivevirtual.com/�

Documents

Troubleshooting BGPd2zmdbbm9feqrf.cloudfront.net/2010/usa/pdf/BRKRST-3320.pdf · Troubleshooting BGP Neighbor Establishment. ... Cisco Confidential 18 BGP Speakers Won’t Peer •