Internet Routing (COS Internet Routing (COS 598A)598A)

Today: Interdomain Traffic Today: Interdomain Traffic EngineeringEngineering

Jennifer RexfordJennifer Rexford

http://www.cs.princeton.edu/~jrex/teaching/http://www.cs.princeton.edu/~jrex/teaching/spring2005spring2005

Tuesdays/Thursdays 11:00am-12:20pmTuesdays/Thursdays 11:00am-12:20pm

Outline

• Border Gateway Protocol– BGP protocol– BGP policies– BGP decision process

• BGP traffic engineering for outbound traffic– Predicting effects of policy changes– Identifying good policy changes

• What about inbound traffic?– AS prepending and MEDs– Inter-AS negotiation

Interdomain Routing: Border Gateway Protocol

• ASes exchange info about who they can reach– IP prefix: block of destination IP addresses– AS path: sequence of ASes along the path

• Policies configured by the AS’s operator– Path selection: which of the paths to use?– Path export: which neighbors to tell?

32 1

12.34.158.5

“12.34.158.0/24: path (2,1)” “12.34.158.0/24: path (1)”

data traffic data traffic

Components of BGP

• BGP protocol– Definition of how two BGP neighbors communicate– Message formats, state machine, route attributes, etc.– Standardized by the IETF

• Policy specification– Flexible language for filtering and manipulating routes– Indirectly affects the selection of the best route– Varies across vendors, though constructs are similar

• BGP decision process– Complex sequence of rules for selecting the best route– De facto standard applied by router vendors– Being codified in a new RFC for BGP coming soon

04/18/23

BGP Protocol: BGP Sessions

Establish session on TCP port 179

Exchange all active routes

Exchange incremental updates

AS1

AS2

While connection is ALIVE exchangeroute UPDATE messages

BGP session

BGP Protocol: Update Messages

• Update messages– Advertisement

• New route for the prefix (e.g., 12.34.158.0/24)• Attributes such as the AS path (e.g., “2 1”)

– Withdrawal• Announcing that the route is no longer available

• Numerous BGP attributes– AS path– Next-hop IP address– Local preference– Multiple-Exit Discriminator– …

BGP Policy: Applying Policy to Routes

• Import policy– Filter unwanted routes from neighbor

• E.g. prefix that your customer doesn’t own

– Manipulate attributes to influence path selection• E.g., assign local preference to favored routes

• Export policy– Filter routes you don’t want to tell your

neighbor• E.g., don’t tell a peer a route learned from other

peer

– Manipulate attributes to control what they see• E.g., make a path look artificially longer than it is

04/18/23

BGP Policy: Influencing Decisions

Best Route Selection

Apply Import Policies

Best Route Table

Apply Export Policies

Install forwardingEntries for bestRoutes.

ReceiveBGPUpdates

BestRoutes

TransmitBGP Updates

Apply Policy =filter routes & tweak attributes

Based onAttributeValues

IP Forwarding Table

Apply Policy =filter routes & tweak attributes

Open ended programming.Constrained only by vendor configuration language

BGP Decision Process: Path Selection on a Router

• Routing Information Base– Store all BGP routes for each destination prefix– Withdrawal message: remove the route entry– Advertisement message: update the route

entry

• Selecting the best route– Consider all BGP routes for the prefix– Apply rules for comparing the routes– Select the one best route

• Use this route in the forwarding table• Send this route to neighbors

BGP Decision Process: Multiple Steps

• Highest local preference – Set by import policies upon receiving advertisement

• Shortest AS path– Included in the route advertisement

• Lowest origin type– Included in advertisement or reset by import policy

• Smallest multiple exit discriminator– Included in the advertisement or reset by import

policy

• Smallest internal path cost to the next hop– Based on intradomain routing protocol (e.g., OSPF)

• Smallest next-hop router id– Final tie-break

BGP Decision Process in Action

“(2, 1)” “(3, 4, 1)”“(2, 1)”

But, what if the path “(3,4,1)” would be better?

Manipulating Policy to Move the Traffic

• Assign local preference to…– Prefer one neighbor over another for a prefix– Prefer certain AS paths over others

• Router configuration languages– Specifying rules for setting local-pref attribute– “if path(3, *, 1), then local-pref=110”– “else, local-pref=100”

• Allow policy to over-ride shortest AS path– Indirect way of making one path look better

or worse than another– Main way to do BGP traffic engineering today

BGP Traffic Engineering

BGP Traffic Engineering

• Limitations of intradomain traffic engineering– Alleviating congestion on edge links

– Making use of new or upgraded edge links

– Influencing choice of end-to-end path

• Extra flexibility by allowing changes to BGP policies– Direct traffic toward/from certain edge links

– Change the set of egress links for a destination

1

2

3

4

BGP Modeling for Traffic Engineering

• Predict effects of changes to import policies– Inputs: routing, traffic, and configuration data

– Outputs: flow of traffic through the network

TopologyBGP policy

configuration

eBGP routes

Offered traffic

BGP routingmodel

Flow of traffic through the network

Steps in the Model

• Effects of import policy on the routes– Inputs: BGP routes, and import policies– Output: BGP routes modified by policies

• Set of best routes for the network– Inputs: BGP routes modified by policies– Output: set of best routes (max local-pref, min AS path,

…)

• Best route per router– Inputs: set of BGP routes, and intradomain costs– Outputs: best route (closest egress) per prefix per

router

• Link-level paths– Inputs: best route per router, and intradomain costs– Outputs: link-level shortest path(s) from entry to egress

But, Hard to Select Good Import Policy Changes

• Can’t enumerate all choices– Many eBGP sessions– Around 100K prefixes– Highly programmable policies

• Risk of creating side-effects– Overhead of BGP routing changes– Unpredictable behavior of others

• Vulnerability to changes– New eBGP routes from neighbors– Shifts in the traffic volume per prefix

Traffic Engineering Guidelines

• Predictability– Ensure the BGP decision process is deterministic– Assume that BGP updates are (relatively) stable

• Outbound traffic (import policy, local preference)– Easier to control how traffic leaves the network– Cooperate with neighbor ASes for inbound traffic

• Limit overhead introduced by routing changes– Minimize frequency of changes to routing policies– Limit number of prefixes affected by changes

• Limit impact on how traffic enters the network– Avoid new routes that might change neighbor’s mind– Select route with same attributes, or at least path

length

Measurement Data

• Analysis of peering links– Links connecting AT&T to other large

providers– Relatively small # of high-end routers

• BGP routing tables– Log daily output of “show ip bgp” command– Extract BGP routes for each prefix

• Cisco Netflow– Collect continuous flow-level measurement– Aggregate the traffic by prefix– Focus on outbound traffic to peers

Controlling the Scale

• Destination prefixes– More than 90,000 destination prefixes

• Don’t want to have per-prefix routing policies

– Small fraction of prefixes contribute most of the traffic• Focus on the small number of heavy hitters

– Define routing policies for selected prefixes

• Routing choices– About 27,000 unique “routing choices”

• Help in reducing the scale of the problem

– Small fraction of “routing choices” contribute most traffic

• Focus on the very small number of “routing choices”

– Define routing policies on common attributes

Avoid Impacting Downstream Neighbors

Will traffic volume change???

Predictable Routing Change

• Predictability– Do not change the route sent to downstream neighbor– Focus on prefixes where all “best” routes are identical– Neighbors do not even receive a new BGP

advertisement!

• Example application– Multiple links to same peer, with one congested link– Assign lower local-pref at that link for some prefixes

• Empirical results – 83.5% of prefixes have shortest AS paths that are all

identical (same next-hop AS, same AS path, etc.)– These prefixes are responsible for 45% of the traffic– Plenty of scope to move traffic in a predictable fashion

Semi-Predictable Routing Change

• Semi-predictability– Do not change the length of the AS path sent to

neighbor– Neighbors receive new advertisement with same length– (Hopefully) they still make the same routing decision

• Example application– Need to move some traffic from one peer to another– Find prefixes with “best” paths via both neighbors– Assign lower local-pref at some links for some prefixes

• Empirical results – 10-15% of prefixes have shortest paths with 2 next

hops

– These prefixes contribute 35-40% of the traffic– Potential to move traffic in a semi-predictable fashion

Influence of AS Path Length

• AS path length– Plays a significant role in the BGP decision process– All “best” routes must have the same AS path length– 10% of prefixes have choices with different path

lengths

• An idea: a more flexible approach– Possible to disable consideration of AS path length,

and incorporate AS path length in local-pref assignment

– E.g., treat paths of length 3 and 4 as equally good

• AS prepending by other ASes– Inflating AS path length by adding fake hops– E.g., “701 80 80 80” instead of “701 80”– 18% of routes had some form of AS prepending

Inbound Traffic

Why Inbound Traffic is Hard to Manage

• Other ASes decide how to send to you– Destination-based routing– Other ASes decide which path to take– Based on their own policies

1

2

3

4

AS 2 doesn’t know how AS 1 will send traffic toward p

p

AS Prepending

• Artificial inflate AS path length– Prepend your own AS in the path– E.g., turn “3 4 5” into “3 3 3 4 5”– Hope to make the path less attractive

1

3

“3 3 3 4 5”

“3 4 5”

Multiple Exit Discriminator (MED)

• Tell your neighbor what you want– MED attribute to indicate receiver preference– Decision process picks route with smallest MED – Can use MED for “cold potato” routing– But, have to get your neighbor to accept MEDs

1

3

“3 4 5” with MED=2

“3 4 5” with MED=1

Inter-AS Negotiation

• Better to cooperate?– Negotiate where to

send– Inbound and outbound– Mutual benefits

• But, how to do it?– What info to exchange?– How to prioritize the

many choices?– How prevent cheating?

• Open research territory

Customer A

Customer B

multiplepeeringpoints

Provider A

Provider B

Early-exit routing

Next Time: Multi-Homed Customers

• Two papers– “A Measurement-Based Analysis of Multi-

homing” (SIGCOMM’03)– “Optimizing Cost and Performance for

Multihoming” (SIGCOMM’04)

• Reviews of both papers– Brief summary of the paper– Reasons to accept the paper– Reasons to reject the paper– Suggestions for future research directions