Embed Size (px)
Citation preview
D R A F T 1.9 W. B. Norton
Internet Service Providers and Peering
William B. Norton <[email protected]>Abstract
Internet Service Provider (ISP) peering hasemerged as one of the most important and effectiveways for ISPs to improve the efficiency of operation.Peering is defined as “an interconnection businessrelationship whereby ISPs provide connectivity toeach others’ transit customers.” ISPs seek peeringrelationships primarily for two reasons. First, peeringdecreases the cost and reliance on purchased Internettransit. As the single greatest operating expense, ISPsseek to minimize these telecommunications costs.Second, peering lowers inter-Autonomous System (AS)traffic latency. By avoiding a transit provider hop inbetween ISPs traffic between peering ISPs has lowerlatency. So how is peering done?
This paper details the ISP peering decision-makingprocess. Interviews with Internet Service Providers1
have highlighted three distinct decision phases of thepeering process : Identification (Traffic EngineeringData Collection and Analysis), Contact &Qualification (Initial Peering Negotiation), andImplementation Discussion (Peering Methodology).The first phases identifies the who and the why, whilethe last phase focuses on the how.
The appendix includes the description of a PeeringSimulation Game that has been used in workshops toplay out peering negotiations.
Introduction and DefinitionsInternet Service Providers (ISPs) connect end-usersand businesses to the public Internet. They competewith each other on price, performance, reliability, etc.but they also must cooperate with each other toprovide global connectivity to all other attachments onthe Internet. The cooperation is explicitly stated atdemarcation points where they interconnect, calledimprecisely “peering points”. This is an imperfectname since interconnection typically takes one of twoforms: a peering relationship or a transit relationship,and both use the Border Gateway Protocol (BGP) forrouting announcement exchange. Problems arise whenthe term “peering” is used interchangeably with a“transit” relationship. To lay the groundwork for thispaper, we introduce the following working definitions.
1 Interviews with 100 ISPs over the course of two yearsalong with presentations of the findings to ISPs atNANOG, RIPE and IEPG substantially validate thefindings.
Definition: Peering is the business relationshipwhereby ISPs reciprocally provide to each otherconnectivity to each others’ transit customers.
To illustrate peering, consider figure 1 below showinga much simplified Internet: the Internet with only threeISPs: WestNet, USNet, and EastNet. WestNet hascustomers shown as green circles. USNet hascustomers of its own (beige circles) and EastNet hasits customers shown as yellow circles.
Figure 1 - Peering and Transit relationships
In this example, WestNet has a peering relationshipwith USNet in which USNet announces reachability ofits beige customers to WestNet, and WestNetannounces reachability to its green customers toUSNet. This is the essence of the peering relationship;each ISP reciprocally provides access to each otherscustomers. EastNet also peers with USNet,announcing its yellow customers to USNet whileUSNet announces its blue customers to EastNet.
It is important to note that WestNet and EastNet cannot access each others customers in this configuration.(The boxes below the ISPs show their respectiverouting tables.) WestNet only knows how to get toblue and green customers, and EastNet knows how toreach only blue and yellow customers. The fact thatthey both peer with USNet is inconsequential; peeringis a non-transitive relationship.
Since peering is a reciprocal non-transitiverelationship, EastNet and WestNet must peer withevery other ISP or find another way of accessing everyother ISP.
Definition: Transit is the business relationshipwhereby one ISP provides (usually sells) access to
Peering
USNet EastNetWestNet
Peering
RoutingTables
Peering
USNet EastNetWestNet
Peering
RoutingTables
DRAFT Peering Decision Tree W. B. Norton
2 Comments to the Author Welcome
all destinations in its routing table2.
Consider a simplistic Internet model below infigure 2. In this picture, EastNet purchases transitfrom USNet, and in return gets connectivity to allnetworks in USNet’s routing table. As a customer ofUSNet, USNet announces EastNet routes across all ofits peering and transit interconnections.
Transit
USNetWestNet
Peering
By EastNet purchasing transit,EastNet is announced by USNet toUSNet Peering and Transit interconnections alike.
EastNet
…for a (transit) fee of course.
Figure 2 - Transit Relationship - selling access toentire routing table
To better illustrate transit, note that there are over8800 ISPs in the US3 alone; imagine the complexityand cost of trying to peer with all of them! (In realityyou wouldn’t have to peer with all ISPs, but youwould need to peer with their upstream (transit)providers to avoid transit entirely.) A more effectivedemonstration of transit is shown below, showing theopaque conglomeration of International networksaccessible via a transit relationship with a large transitprovider. Compared against the relatively smallnumber of routes received from a single peeringrelationship, one can see that transit is a valuable anddifferent service from peering.
2 Note that increasingly in Europe ISPs are offering andobtaining hybrids. For example, they may purchase“Regional Transit” from global players in a regionwithout adequate coverage. In a few rare cases ISPs havearranged “paid peering” to eliminate the cost of peeringto one or the other. INSNet and GXNet for example.
3 See the Boardwatch list: http://www.thelist.com for a listof ISPs.
Transit
WestNet
Peering
EastNet
Thousands ofother Int’l ISPs
USNet
The entire Internetas known by USNet
Figure 3 - Transit Providers with large network
As a side note, some service providers4 prefer a transit(customer) relationship with ISPs for business reasons,arguing that the threat of lost revenue is greater thanthe threat of terminating a peering arrangement ifperformance of the interconnection agreement isinadequate.
Now that we have introduced the notion of peering andtransit relationships and the difference between them,we will examine the role of the Peering Coordinator.
I. Phase 1: Identification ofPotential Peer: TrafficEngineering Data Collection andAnalysis
We spoke with over 100 peering coordinators inthis study to document how peering works, and howthey approach peering from a practical perspective.Peering Coordinators are typically charged withestablishing and managing the interconnectionsbetween their network and others. This multidisciplinejob crosses the boundaries of network architecture,technical (routing logistics), business, and legal. Thisjob therefore requires a mix of skill sets to be executedeffectively.
We’ll first examine the peering coordinatormotivations for peering and selection of peers.
Motivations: Why Peer?
4 Conversation with Allan Leinwand, Founder of DigitalIsland. Allan indicated that the financial “teeth” aremuch stronger with transit ISPs than with “peers”, andthe threat of lost revenue provides better quality andreliability in a transit relationship.
DRAFT Peering Decision Tree W. B. Norton
3 Comments to the Author Welcome
Lower Transit Costs. Choices made by InternetService Providers (ISP) are often dominated bytelecommunications cost issues. Highest among thesecosts is Internet transit service that provides the ISPwith connectivity to the global Internet. Transit Pricesfor DS-3 transit for example can be as high as$50,000/month5, and OC-3 transit can cost up to$150,000/month6. To reduce these costs, ISPs seekpeering (zero or reduced cost) relationships with otherISPs that provide more direct traffic exchange andreduce the load on these expensive transit services (asshown below).
Upstream Transit Provider
Transit$$$
Transit$$$
Peering $
ISP A ISP B
Figure 4 – Migrating traffic from Transit toPeering Interconnection
Lower Latency. As a side effect ofinterconnecting directly with peers, ISP customersexperience lower latency to the other ISP’s customers.Traffic destined for a local competitor’s customersmay need to traverse a couple of transit providers andpotentially across great distances (with high latency)before reaching the other customer. The worstexample highlighted traffic between the United ArabEmirates and Saudi Arabia traversing an overloadedexchange point in Washington DC7. Through directinterconnections (using direct circuits or regionalexchange points) ISP customers realize betterperformance.
Usage-based traffic billing. Some ISPs chargecustomers based upon metered traffic. Since packet
5 Varies by transit provider, backhaul costs vary by circuitmiles, carrier competition, etc.
6 Dave Rand interview with the Cook Report, and Authorinterview with Pat Binford-Walsh (UUNet) in 1998.Note that the bandwidth and transit costs have dropped.
7 Consulting work with the United Arab Emerites PTT.
loss and latency slows traffic consumption, theybenefit from a lower latency, lower packet lossInternet. It is in their best interest therefore to assurethat customers use as much bandwidth as possible byminimizing loss and latency through effective trafficengineering8.
Conversations with European ISPs showedincreasing adoption of the usage-based billing model.This has motivated them to compete for traffic andtherefore revenue.
Why not peer?
On the surface peering appears to be a good ideafrom a financial and technical perspective. However,the topic has generated more heat than light due to thefollowing conflicts of interest between ISPs.
• Traffic Asymmetry and Investmentasymmetry means that one party bearsmore of the cost as a result of peering. Forexample, consider the figure below whereExodus peers with GTEI. Web traffic (thedominant traffic flow on the Internet) isinherently asymmetric. Exodus is a netsource of content, therefore more GTEIresources (bandwidth) than Exodusbandwidth are consumed as a result ofpeering. GTEI could say that Exodus was“dumping traffic” onto GTEI’s backboneand GTEI was forced to “carry” the trafficacross the great distances. Meanwhile,Exodus and Exodus customers are able tosell advertising on their web pages andyield great revenues off of GTEI’scustomers. In some cases ISPs will peerwithout settlement up to a certain trafficratio (for example 4:1 traffic out to trafficin) and then on a usage basis beyond thaton a Mbps basis9. In the Exodus-GTEInegotiations, rumor has it that the solutionwas to peer at more locations and toengineer cold-potato10 routing to reducethe distance the traffic had to spend on theGTEI backbone11.
8 Interview with Avi Freedman, AboveNet.
9 Conversations with Frontier Global Center engineers atRIPE 35 in Amsterdam.
10 Cold-potato routing is a routing discipline whereby oneISP carries traffic as far as possible before handing it offto another ISP.
11 “GTEI, Exodus Make Peace On Peering”, Randy Barrett,
DRAFT Peering Decision Tree W. B. Norton
4 Comments to the Author Welcome
Figure 5 - ISP Traffic Asymmetry
• There may be the potential for transit salesif they don’t peer12. Some ISPs will notpeer if there is any existing or pendingcustomer-provider relationship betweenthe parties, even if the sale is completelyunrelated to interconnection (i.e. fiber saleor colo sale)13.
• Peering consumes resources (routerinterface slots, circuits, staff time, etc.)that could otherwise be applied to revenuegeneration. Router slots, cards,interconnection costs of circuits orInternet Exchange environments, staffinstall time are incrementalexpenditures14. Further, operating costs,particularly for peering sessions with ISPswithout the necessary on-call engineeringtalent, can require increased processingpower for filters and absorb time bettersuited to paying customers.
• Motivation not to commoditize IP transit.Tier 1 ISPs15 compete on the basis of
Inter@ctive Week, September 16, 1998 andanonymous interviews with parties involved with thenegotiations.
12 A case recited to the author: Level 3 refused to peer withGST Networks in the Washington DC area since GSTwas a Level 3 transit customer in New England. Whiletechnically the request could be accommodated, Level 3rationally preferred the transit revenue.
13 Apricot 2000 presentation by Lauren Nowlin, Onyx,March 2, 2000 in Seoul, Korea on “Peering andInterconnection Panel” with the author.
15 Tier 1 ISPs are defined as ISPs with global coverage anda full Internet routing table without acquiring transit
better performance. They accomplishbetter performance due to the largecustomer base of direct attachments andhigh-speed interconnections with otherTier 1 ISPs. Since peering with other ISPsimproves the performance of the “peer” iteffectively makes them a more powerfulcompetitor. Therefore, there is a strongdisincentive to peer and increase thenumber of top tier competitors.
• As a “peer” there are no Service LevelAgreements (SLAs) to guarantee rapidrepair of problems. Both parties benefitfrom the reparation of outages, and mayeven have clauses in the peeringagreements to work diligently to repair theproblem. However, a customerrelationship (with or without SLAs)generally has more contractual teeth(financial repercussion for failure toperform).
Traffic Ratio-based Peering. As a result ofthese forces, a traffic ratio-based paid peeringmodel is emerging. In this approach, peeringis free until traffic asymmetry reaches a certainratio (4:1 is common). At this point, the netsource of traffic will pay the net sink of traffica fee based upon traffic flow above this ratio.
With Whom to Peer?
If peering makes sense from a technical andfinancial perspective, the next question is, “Withwhom should we peer?” To identify potential peers,ISPs use a variety of criteria.
Quantities of traffic distributed between networksoften sets the pace of the negotiation; to quantify this,ISPs may systematically sample inbound andoutbound traffic flows. Flows then are mapped tooriginating AS, and calculations are made to determinewhere peering (direct interconnections) would mostreduce the load on the expensive transit paths. There issubstantial work involved here, as this traffic samplingresults in a large number of data. Alternativemeasurement methods include measuring portstatistics16.
Many peering coordinators indicated that peeringselection is accomplished by intuition17. Their sense
from any ISP.
16 Avi Freedman, AboveNet citing ATM and other switchmeasurement methods in use.
17 NANOG 17, Montreal, Peering Birds Of a Feather (BOF)
Traffic Asymetry
EXODUS
GTEI
WebFarm
SmallWeb
Request
Generates LargeWeb Response
Whose resources are consumedas a result of peering?
Peering
DRAFT Peering Decision Tree W. B. Norton
5 Comments to the Author Welcome
was that they knew where traffic was and would beheaded.
In either case, the end result of this first phase islist of the top 10 ISP candidates for peering.Interviews with Peering Coordinators highlighted afew other considerations.
Broader business arrangements between ISPsmay circumvent the peering negotiation phase andexpedite discussions directly to Phase III, the peeringmethodology negotiation phase.
Peering policies range across a wide spectrumfrom “open peering policy” meaning “we will peerwith anyone”, to “if you have to ask, we won’t peerwith you.18” Peering policies are often exposed onlyunder Non-disclosure agreements, and these policiesreduce the number and type of ISPs that are peeringcandidates.
In many cases peering requires interconnections atmultiple peering points, explicit specifications forrouting, migration from public (shared switch) peeringto private (non-shared switch) peering after a certaintraffic volume is reached, etc. It is beyond the scope ofthis document to fully explore the technical andpolitical motivation for peering policies; it is sufficientto be aware that these discussions can be cumbersomeand require a combination of technical and financialnegotiation.
The greatly simplified peer qualification decisiontree looks something like this:
meeting held by author when about two-thirds of theaudience indicated that they use ad-hoc, predictive, orintuition for selecting peering candidates.
18 Sentiment articulated by Sean Doran, shortly after leavingSprintLink. Peering policies are a politically sensitivesubject, and peering policies are often not explicitlyarticulated.
Partof a larger
business Transaction?
WillPeering
Have a positiveaffect on my
network?
LargeNew Customer Impact
expected?
Traffictransiting expensive
transit service?
Dominant TrafficFlow?
Phase 1
Proceed to Phase 2
Yes
Yes
Yes
YesYes
Figure 6- Phase 1 of Peering Selection DecisionTree
Once the measurements have been made andanalyzed, and it appears to be beneficial to peer, theISP enters into Phase 2, Contact & Qualification,Initial Peering Negotiation.
Emerging Migration Path from Transit toPeering. Interviews with tier 2 ISPs highlighted anemerging peering transition strategy:
1) Access the Internet via transit from a globalprovider,
2) Pursue peering arrangements on publicswitches at exchange points to reduce load ontransit links and improve performance
3) Migrate high traffic public peeringinterconnections to private interconnections(via fiber or direct circuits).
4) Ultimately migrate traffic away from transitpurchase and negotiate (free or for-fee)peering with former transit provider.
To illustrate this path, consider Telia, a global ISPbased in Sweden. Telia analyzed their transitcosts and recognized that approximately 85% oftheir traffic at MAE-East was to their transitprovider and the remaining 15% was throughpeering relationships. By focusing on establishingpeering relationships with the top 25 destinationASes they shifted the mix to 70% through privatepeering at an exchange with the remaining 30% oftraffic heading toward their transit provider19.
19 Interview with Anne Gibbens (Telia)
DRAFT Peering Decision Tree W. B. Norton
6 Comments to the Author Welcome
The result was increased traffic efficiency and areduction in the cost of transit20.
It should be stated that phase four of the migrationstrategy listed above may be overly optimisticand/or challenging for several reasons. First,transit providers prefer paying customers to peers.Second, transit providers typically have muchmore ubiquitous network infrastructure than theircustomers, and therefore will not see theircustomers as equal contributors. Finally, thetransit providers have an incentive to reduce thenumber of their own competitors.
To illustrate this migration path, Raza Rizvi(REDNET) said “We had to leave our upstreamprovider for 16 months with alternative access totheir route before they considered us not as acustomer lost but as a potential peering partner.“
After the top 10 potential peers are identified,peering coordinators proceed to Phase 2: Contact& Qualification, Initial Peering Negotiation.
II. Phase 2: Contact & Qualification,Initial Peering Negotiation
Internet Service Providers typically have a personor group specifically tasked with peering and trafficengineering issues. For example, UUNet has a“Peering Steering Committee” to evaluate peeringrequests21. Some variations of the following stepslead to the parties either leaving the negotiation orproceeding to peering methodology discussions.
Interviews have highlighted a key challenge forISPs. Finding the right person to speak with at thetarget ISP is a difficult and time intensive process.Peering Coordinators change jobs and there is nostandard way to find out who handles this task.Mergers and acquisitions cloud lines ofcommunication. Even if the name is known, PeeringCoordinators are often traveling, way behind in e-mail,and prioritizing e-mail based on the subject or thesender. This is where “people networking” helps agreat deal, and hiring expertise for their contactsspeeds this initial contact process up quite a bit. Insome cases, peering is expedited between ISPs simplybecause the decision makers have a previous
20 As compared with growing the transit connection.
21 Point made by Paul McNulty at the 1999 Apricot Sessiontitled “Next Generation Internet Infrastructure”.
relationship22. This was the dominant mode ofoperation in the early days of the Internet.
In any case, peering contacts are initiated in one ofthe following ways:
a) via electronic mail, using the pseudostandard peering@<ispdomain>.net or apersonal contact,
b) from contacts listed on an exchange pointparticipant list,
c) with tech-c or admin-c from DNS or ASNregistries,
d) informal meeting in an engineering forumlike NANOG, IETF, RIPE, etc.,
e) at trade shows from introductions amongspeakers, or with booth staff,
f) from the target ISP sales force,
g) from the target ISP NOC,
h) as part of a larger business transaction.
Second, mutual non-disclosures agreements(NDAs) may be negotiated and signed, and adiscussion of peering policy and prerequisites follow.Note that NDAs are an optional step, and many ISPsdo not require signed NDAs prior to discussions23.Traffic engineering discussions and data disclosuremay be used to justify the peering relationship. EachISP typically has a set of requirements for peering thatinclude peering at some number of geographicallydistributed locations, sometimes at public exchangepoints.
Traffic volume is usually a key determining factor.The decision rule hinges upon whether or not there issufficient savings from peering to justify spendingcapital on a port on a router and/or a portion of theinterconnection costs or augmenting existing capacityinto an exchange point. A Bilateral PeeringAgreement24(BLPA) is the legal form that details eachparties understanding of acceptable behavior, anddefines the arms length interactions that each wouldagreed to.
22 Discussion with Vab Goel, former VP of Engineering atQwest.
23 NANOG Peering BOF, NANOG 17 in Montreal, about70 Peering Coordinators of 125 indicated they do notrequire NDAs.
24 See http://www.linx.net/joininfo/peering-template/agreement-v4.html for sample BLPA
DRAFT Peering Decision Tree W. B. Norton
7 Comments to the Author Welcome
Another motivation for peering to factor inincludes lower latency and/or more regionaldistribution of traffic than existing connections allow.
This process is diagrammed below.
Contactswithin the broader business
Transaction?
Do bothparties find motivation
to continue peeringdiscussion?
Contact atOperationsForums?
Contact withPeering@ or personal
Contact?
Phase 2
Proceed to Phase 3
Yes
Yes
Contact withExchange Point contact
List?
ContactUsing DNS/ASN
Registry?Contact at Trade
Shows?
Contact viaSales
Force?
(optionally) Sign NDA
Share traffic statistics, Policies, BLPA,I.e. justification whythey should both peer
YesYes
Yes
Yes
Yes
End Peering DiscussionNo
Yes
Figure 7 – Phase 2: Contact & QualificationDecision Tree
After this initial discussion, either party maydecide to walk away from the peering discussions untilcertain criteria are met25. If both parties agree thattheir requirements are sufficiently met to discussmethodology (they both benefit from the peeringrelationship), they move onto Phase 3: ImplementationDiscussions.
III. Phase 3: Implementation Discussions:Peering Methodology
Since peering is seen as being of mutual benefit,both parties now explore the interconnectionmethod(s) that will most effectively exchange traffic.The primary goal is to establish point(s) ofinterconnection, and secondarily detail optimal trafficexchange behavior (using Multi-Exit Discriminators(MEDs) or other traffic weighting techniques).
To interconnect, ISPs face two distinct options:Direct Circuit Interconnection or Exchange-BasedInterconnection (or some global combination thereof).
25 According to participants at the NANOG17 Peering BOFled by the author, government agencies in Israel andAustralia forced ISP peering!
The “Interconnection Strategies for ISPs” whitepaper26 quantifies the economics and technicaltradeoffs between the first two options. To summarizethis report, the preferred methodology depends on thenumber of peers participating in the region andbandwidth required for its regional interconnections.ISPs that expect to interconnect at high or rapidlyincreasing bandwidth within the region, or expectinterconnections with more than five parties in theregion prefer the exchange-based solution. Those thatdo not anticipate a large number of regionalinterconnects prefer direct-circuits and typically decideto split the costs of interconnection with the peer byregion. On occasion the costs are covered in whole byone peer27.
Interconnection Method
Direct-CircuitInterconnection
Exchange-BasedInterconnection
Figure 8 – ISP Physical InterconnectionMethods
For direct-circuit interconnects, key issues centerupon interconnection location(s) and who pays for andmanages the interconnection. This becomes a materialcost issue as traffic grows and circuits increase in sizeand cost.
In either case, ISPs generally have the followinggoals for establishing peering:
1. get peering set up as soon as possible,
2. minimize the cost of the interconnection andtransit costs,
3. maximize the benefits of a systematic approach topeering,
26 Interconnection Strategies for ISPs, W. B. Norton,June99,presentation: http://www.nanog.org/mtg-9905/norton.html. A copy of this report can be requestedvia e-mail to [email protected].
27 Interviews found a pattern in which PSINet would peerwith ISPs provided that peer covered all interconnectioncosts.
DRAFT Peering Decision Tree W. B. Norton
8 Comments to the Author Welcome
4. execute the regional operations plan as strategydictates (may be architecture/networkdevelopment group goal), and
5. fulfill obligations of larger business agreement.
Exchange Environment Selection Criteria
This section details the selection criteria an ISPtypically uses when selecting an exchange. Note thatthese issues are listed in no particular order. Theseissues are shown graphically as flowcharts anddiscussed in detail in the paragraphs below.
TelecomAccessIssues
DeploymentIssues
ISPCurrent
PresencesIssue
OperationsIssues
BusinessIssues
CostIssues
ExchangeCredibility
Issues
DoesThis exchange
Have the credibilityTo bring in attractive
ISP Peers?
DoesThis exchange Havethe financial backing
To survive?
ExchangePopulation
Issues
Existing v.EmergingExchange
Issues
Weigh Credibility
AreWe inthere?
Aretheyin?
Add 1 to score
Add 1 to score
Weigh CurrentPresences
DoesExchange
existtoday?
Weigh Existingv. Emerging
Add 1 to score
FiberVendors inexchange?
MultipleCarriers?
Weigh TelecomAccess
Deploy+Install gear
easy?
Weigh Deployment
Are thereOthers there
That I’dLike to
Peer with?
Weigh ExchangePopulationWeigh Operations
Issues
Weigh BusinessIssues
Weigh CostIssues
PrivatePeering
Allowed?
Is thetotal cost In line
With theBenefits?
DoesParticipation
SupportCompetitor?
ScalesWell?
SecurityMeetsReq’s?
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
No
No
Figure 9 - Exchange Environment Selection
Telecommunications Access Issues
These issues have to do with gettingtelecommunications services into the exchange. Howfast can circuits be brought into the interconnectionenvironment? How many carriers compete forbusiness for circuits back to my local Point ofPresence (POP)? For facilities-based ISPs, what is thecost of trenching into the exchange (how far away andwhat obstacles present themselves)? Are there nearbyfiber providers that lease strands? These questions willhelp answer the most important question to ISPs: Howfast can my peer and I get connectivity into theexchange? Multiple carriers lead to speed and costefficiencies. Some ISPs have volume deals withcertain carriers or otherwise prefer carriers andtherefore prefer exchanges where these carriers canquickly provision circuits. These answers stronglyimpact the desirability of the exchange environment.
Deployment Issues
These issues have to do with getting equipmentinto the exchange. How do I get my equipment into theexchange (assuming it supports collocation)? Do I ship
equipment in or do I have to bring it with me as I flyin? Will someone act as remote hands and eyes to getthe equipment into the racks or do I do the installationmyself? Comparing exchange environments in thiscontext, what are the costs associated with deployment(travel, staff time, etc.) into this exchange? Does theexchange have sufficient space, power, airconditioning, etc. The answers to these questionsimpact the deployment schedule for the ISP(s)engineers and the costs of the interconnection method.
ISP Current Presences Issues
This issue is based on the following observation bythe peering coordinators: The most inexpensive andexpedient peering arrangements are the ones madebetween ISPs that are already located in the sameexchange. There is a hidden assumption here thatthere is sufficient capacity to interconnect at theexchange. Cross-connects or switching fabrics caneasily establish peering within a few hours or at mostdays. ISPs will prefer to interact where one or both ISPalready has a presence.
Operations Issues
These issues focus on the ongoing operationsactivities allowed within the exchange after initialinstallation. Does the exchange allow private networkinterconnections? Are there requirements to connect toa central switch? How is access and security handledat the facility28? Is there sufficient power, HVAC,capacity at the switch, space for additional racks, realtime staff support29? Is it easy to upgrade mypresence over time? Upgrading in this context meansthe ability to increase the speed of circuits into theexchange, the ability to purchase dark fiber, the abilityto increase the number of racks and cross connects inthe exchange, the ease of increasing the speed ofinterconnection. ISPs will prefer bandwidth-rich, ISP-friendly exchanges over those with restrictions overfuture operations.
Business Issues
“Bandwidth, strategic partner alliances, andcorporate ties often override the technical
28 For example, the NSPIXP is a major exchange in Japanyet has no staff on-site so engineers need to be called infor support. Escorted access could take hours to becoordinated.
29 MAE-East has been widely criticized for being a majorinterconnection point in the US without sufficientinfrastructure (power, A/C) to support expansion.
DRAFT Peering Decision Tree W. B. Norton
9 Comments to the Author Welcome
justification30.” – Lauren Nowlin, Peering Coordinatorfor Onyx Networks.
Perhaps the most far-reaching issue is strategic: dowe want to support this exchange operator, and dotheir interests enhance or conflict with ours?
Will using this exchange support a competitor(contribute to their net income, their credibility, theirpositioning)? A neutrally operated exchange (definedas one that is not owned or aligned with any carrier,fiber provider, or ISP) provides an open distortion-freemarketplace for carrier and ISP services.
Market distortions often result when an exchange isowned by one of its participants. This often manifestsitself in requirements (required use of their carrier orISP services) that constrain the market for serviceswithin the exchange31. Since it is difficult anddisruptive to move equipment out of an exchange,ISPs will prefer a neutrally operated exchangeenvironment that will not suffer from marketdistortions and limitations due to business conflicts ofinterest.
Cost Issues
This broad issue crosses all other issues. What isthe cost of using this exchange? What are the rackfees, cross connect fees, port fees, installation fees?What are the future operating fees going to be? Whatare the motivations and parameters surrounding thesefees? Cost issues shadow most of the other issueslisted in this paper. All else being equal, ISPs will seekto minimize the costs, particularly upfront costs,associated with the interconnection for peering.
Credibility Issue
The credibility issue is twofold.
First, credibility goes to the financial support of theexchange. Does the exchange exist today and will itexist tomorrow? During the early stages of theexchange, ISPs are asked to make a leap of faith whencommitting, and therefore prefer an exchange withstrong backing and the credibility to survive.
30 Discussions at NANOG 17 in Montreal.
31 MAE-East is owned and operated by MCI Worldcom andrequires use of MCI circuits to access MAE-Eastservices. Exodus requires use of its network and at onepoint restricted direct access between ISPs and Carriers.
Value of the Internet ExchangeVExchange
NParticipants
CostOf Coming In(Circuits+Routers+StaffTime)
VCapacity
The Exchange Startup Hump
First Carrier(s)First ISP(s)First CP(s) Critical Mass Point (Vexchange=CostExchange)
Larg
eFa
cilit
ySc
alin
g
Figure 10 - Value of exchange varies overpopulation
Second, does the exchange operator have thebacking and credibility to attract the more valuablepeering candidates? Since the value of the exchange(shown in the graph below) is proportional to thenumber and type of participants. Does this exchangehave the backing to attract my peers? Who ismanaging the exchange and what technology is in use?These answer signal the credibility and survivability ofthe exchange. ISPs will prefer an exchange withcredibility – one that is financially and technically wellbacked and likely to attract the most desirably peeringcandidates.
Exchange Population Issues
These issues focus on the side benefits to using thisexchange. Are there other ISPs at this exchange thatare peering candidates? Are there transit sales possibleat the exchange? In the context of the credibility issuediscussed above, who will likely be at the exchange inthe future, and when will the cost of participationequal the value of the interconnection (also known asthe Critical Mass Point)? ISPs will prefer anestablished and well-populated exchange, particularlyone with potential customers that can generaterevenue.
Existing Exchange vs. New Exchange?
There are many operational exchange points ineach region of the U.S. There are also emerging (soonto exist) exchanges that may be considered as peeringpoints. However, given the pace of ISP expansion, it isunlikely that emerging exchange offerings aredifferentiated or compelling enough to be preferredover existing exchanges. Chronic traffic congestioncan influence the decision to plan to peer in an existingmalfunctioning exchange or wait until a betterexchange opens. Customers with heavy flows of
DRAFT Peering Decision Tree W. B. Norton
10 Comments to the Author Welcome
regional traffic can also influence the decision. Longterm benefits (scalability) may lead to preferring anext generation exchange. However, all elseconsidered equal, ISPs generally prefer an existingexchange to an emerging one.
One Final Note on Exchange Criteria:Weighting
The ISPs we spoke with shared with us variedweightings of the importance of each of these issues.To some, the most important issues were the businessissues, and others weighted more heavily theoperations issues. Each ISP places higher or lowerimportance on different issues and not surprisinglyselect their operations environment based on theirspecific criteria.
IV. Summary
This paper provides a rough description of thedecision processes ISPs follow to identify andestablish peering relationships. It explores theimplementation phase and the criteria for exchangepoint selection.
The results of the interviews with ISP PeeringCoordinators can be summarized with the followingobservations:
1) ISPs seek peering primarily to reduce transitcosts and improve performance (lower latency).
2) Peering goals for ISPs include a) get peeringset up as soon as possible, b) minimize the costof the interconnection and their transit costs, c)maximize the benefits of a systematic approachto peering, d) execute the regional operationsplan as strategy dictates (may bearchitecture/network development group goal),and e) fulfill obligations of larger businessagreement.
3) The selection of an exchange environment ismade relatively late in the peering process.
4) ISPs highlighted 9 selection criteria forselection of exchange environment:telecommunications access issues, deploymentissues, current presences, operations issues,exchange population, cost issues, andcredibility of the exchange environmentoperator. ISPs weight these issues differentlyand will prefer and exchange environment thatbest suit these needs.
5) One major challenge facing PeeringCoordinators is the identification of potentialpeers and initiating discussions.
Acknowledgements
This paper was the result of interviews with keyInternet peering coordinators, interactions with ISPsupport staff, and side conversations at variousmeetings and forums. Special thanks to a few folkswho contributed their time and ideas to help create thisearly draft report: Ren Nowlin (Onyx networks), JoePayne (IXC), Dave Diaz (Netrail), Jake Khuon andAlan Hannan (Global Crossing), Dan Gisi and JeffRizzo (Equinix), Patricia Taylor-Dolan (Level 3Communications), Sean Donelan (AT&T Labs), AviFreedman (AboveNet), Patrick Gilmore (Onyx), GeoffHuston (Telstra), Steve Meuse (GTE), Aaron Dudek(Sprint), Raza Rizvi (REDNET), Wouter van Hulten(InterXion). Special thanks to John Odenwelder(Equinix) for editing the document.
References
[1] “Maturation in a Free Market: The ChangingDynamics of Peering in the ISP Market“ by JenniferDePalma
[2] “NAPs, Exchange Points, and Interconnectionof Internet Service Providers: Recent Trends, Part I:1997 Survey of Worldwide NAPs and ExchangePoints”, Mark Knopper
[3] Peering Contact Database, Restricted Access;for Peering Coordinators only. To received send e-mail to [email protected] with AS#, company name,peering e-mail address, mailing address, phonenumbers, and contact names.
[4] List of ISPs and NOC contact info:http://puck.nether.net/netops/nocs.cgi
[5] ISP Survival Guide, Geoff Huston, WileyPublishers
[6] Data Communications “Old Boys’ Network”,Robin Gareisshttp://www.data.com/issue/991007/peering.html
[7] Australian’s take on Interconnectioncompetition issues, http://www.accc.gov.au/
[8] IWS2000 Presentations on Internet Exchangefutures: http://hsn-crl.koganei.wide.ad.jp/ipsj-hqi/prev_workshops/5.html
DRAFT Peering Decision Tree W. B. Norton
11 Comments to the Author Welcome
Appendix A – Calculating the FinancialBenefits of Peering
Since the cost of transit is substantially higher32 thanthe typically zero cost of peering, ISPs try to reducethis cost with peering relationships. Before we discussthe tactics used to establish peering, we will take abrief diversion to quantify the financial value of thepeering.
The Financial Value of Peering
Peering is seen as so valuable that companies areacquired because of pre-existing peeringrelationships33. But how does one assess the valuepeering relationships? A rough approximation canbe34 made by measuring the traffic flow across thepeering connections and comparing the cost of thattraffic if sent across a transit interconnection. One canthen use a perpetual annuity function to roughly valuethe peering sessions. To be complete one would haveto factor in (at least modest) growth in traffictraversing that peering session, the chances of peeringtermination, improved performance and effect oncustomer retention, and a variety of other factors.
Upstream Transit Provider
Transit$$$
Transit$$$
Peering $
ISP A ISP B
Example: In a simple example, assume that ISP A andISP B both pay for transit at a transit fee of$100,00035 per month for an OC-3 (155 Mbps) worth
32 We’ll show some actual transit price quotes from aprevious research study. You can also look at theBoardwatch annual survey for this data also.
33 Anonymous source claims GeoNet was acquired by Level3 primarily because of its rich peering relationships, andNextLink purchased Concentric for the same reason.
34 Conversation with Nigel Titley (Level3) where theycalculated the peering cost savings to Level 3 in themillions.
35 Price quotes from interview with Dave Rand early 1999.
of transit. Assume further that one-tenth (15.5 Mbps)of that is traffic between ISP A and ISP B. If ISP Aand ISP B agree to peer over a large point-to-point(OC-3) at $11,400/month36, each pays half of thatcost37, and each pays for an interface card to supportprivate peering over that circuit, both ISP A and ISP Breduce their transit load by 10%38.
Assume a peering coordinator establishes onepeering session this year. One can approximate thefinancial value of a peering coordinator by the costsavings to the ISP:
10% savings on the transit OC-3: $10,000
-monthly cost of peering: $11,400/2=$ 5,570
= $5,430/month * 12months = $66,840/year
Note 1: These cost figures are highly variable andare based on early 1999 quotes. Since then the priceshave dropped dramatically, and ISPs have startedoffering tiered pricing. Fundamentally this doesn’tchange the dynamic; peering over point to point telcocircuits (or less expensively over private crossconnects within an exchange) has a profound effect onthe cost of telecommunications for ISPs.
Note 2: Since only 10% of the interconnect is usedbetween these two ISPs a smaller lower cost circuitcould suffice. This approach allows theinterconnection to scale and helps ensure that thisinterconnection is not a congestion point for sometime.
Note 3: This is a financial benefit and ignores thedecreased latency and perhaps decreased packet lossthat leads to increased traffic. This benefit ultimatelyresults in more bandwidth usage and therefore morerevenue for usage-based ISPs. This additional revenueeffect is ignored in this calculation.
36 The circuit prices we used were taken directly from acarrier in the Ashburn, VA area. We were quoted anOC-3 for $11,400/month and OC-12 for $23,000/month.
37 See the Interconnection Strategies for ISPs white paperdescribing
38 Lots of assumptions here: each ISP will save 10% iftraffic volume is symmetric.
DRAFT Peering Decision Tree W. B. Norton
12 Comments to the Author Welcome
Appendix B – European PeeringDifferences
During research for this paper we found severaldistinct differences between the US and Europe interms of peering in practice. For example, rather thanonly peering or transit, the Europeans more commonlybought or sold partial transit.
Interestingly enough, the peering wasn’t basedupon Europeans gaining partial transit access to theUS, but rather European ISPs getting partial transitaccess to other European cities! It was seen as farcheaper to buy transit to European cities from a singleprovider already expanded into those areas than tobuild across international boundaries themselves. Longhaul circuits and fiber to the US and establishingpeering on the East coast with other US players wasseen as relatively easy.
Peering Full TransitPaid Peering Partial Transit
$0 $ $$ $$$
CustomerRoutes Only
CustomerRoutes Only
Customer& RegionalRoutes All Routes
DRAFT Peering Decision Tree W. B. Norton
13 Comments to the Author Welcome
Appendix B: Peering Decision Tree Partof a larger
business Transaction?
WillPeering
Have a positiveaffect on my
network?
LargeNew Customer Impact
expected?
Traffictransiting expensive
transit service?
Dominant TrafficFlow?
Phase 1
Proceed to Phase 2
Yes
Yes
Yes
YesYes
Contactswithin the broader business
Transaction?
Do bothparties find motivation
to continue peeringdiscussion?
Contact atOperationsForums?
Contact withPeering@ or personal
Contact?
Phase 2
Proceed to Phase 3
Yes
Yes
Contact withExchange Contact
List?
ContactUsing DNS/ASN
Registry?Contact at Trade
Shows?
Contact viaSales
Force?
(optionally) Sign NDA
Share traffic statistics, Policies, BLPA,I.e. justification whythey should both peer
YesYes
Yes
Yes
Yes
End Peering DiscussionNo
Yes
TelecomAccessIssues
DeploymentIssues
ISPCurrent
PresencesIssue
OperationsIssues
BusinessIssues
CostIssues
ExchangeCredibility
Issues
DoesThis exchange
Have the credibilityTo bring in attractive
ISP Peers?
DoesThis exchange Havethe financial backing
To survive?
ExchangePopulation
Issues
Existing v.EmergingExchange
Issues
Weigh Credibility
AreWe inthere?
Aretheyin?
Add 1 to score
Add 1 to score
Weigh CurrentPresences
DoesExchange
existtoday?
Weigh Existingv. Emerging
Add 1 to score
FiberVendors inexchange?
MultipleCarriers?
Weigh TelecomAccess
Deploy+Install gear
easy?
Weigh Deployment
Are thereOthers there
That I’dLike to
Peer with?
Weigh ExchangePopulationWeigh Operations
Issues
Weigh BusinessIssues
Weigh CostIssues
PrivatePeering
Allowed?
Is thetotal cost In line
With theBenefits?
DoesParticipation
SupportCompetitor?
ScalesWell?
SecurityMeetsReq’s?
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
No
No
DRAFT Peering Decision Tree W. B. Norton
14 Comments to the Author Welcome
Appendix C – Peering Simulation Game
Setting
In order to illustrate the strategic and financial rolepeering plays in an ISPs strategy, we created a peeringsimulation game39. In this game, four ISPs (A, B, C,D) seek to maximize their revenues and minimizetheir costs. The revenues are determined by thenumber of regions or “squares” they occupyrepresenting their market coverage and a quantum ofcontent traffic (revenue) that market generates. Thecosts are determined by the number of squares thatothers occupy, representing the transit expense toaccess the rest of the Internet. The game board isshown below.
The Board
A
C
B
D
Transit Provider X
Transit Provider Y
IXW IXE
IXN
IXS
Y Y
XX
Figure 11 - The Peering Simulation Game
Play. Each ISP rolls the die and selects the numberof squares indicated by the die, building into theexchange points desired. For each square occupied,
39 First played at the Interconnect Billing and Accounting
write your name in the square and collect $2,000transit revenue. The ISP must then pay its upstreamtransit provider (shown as Transit Provider X orTransit Provider Y around the border of the board)$1,000 for each square the other ISPs own. The ISPfills in the score card and (if at an exchange point) canproceed to the peering negotiation stage.
Peering Negotiation. ISPs can reduce their transitcosts by building into an exchange point and peeringwith the other ISPs there. If both ISPs agree to peer,the transit costs to the other peer’s squares areeliminated. (Both ISPs’ transit costs are reduced by thenumber of squares the other ISP occupies). However,the ISPs must collectively cover the cost of peering($2,000 per round and two lost turns), split howeverthey see fit. (This is the peering negotiation.)
Objective
Generate as much revenue as possible by growingyour network and establish peering or transitrelationships to reduce network costs. Play is endedwhen any player can no longer execute a play (whenthe board is filled up). The winner is the player withthe most money at the end of the game.
Variations
Transit Sales Negotiation (Optional). ISPs canbuy/sell transit to each other at a reduced rate of$500/square. In the transit sale, the transit providergets the $500/square transit revenue and the transitpurchaser saves $500/square (compared with buyingtransit at $1000/square). The cost of transit ($2,000per round and two lost turns) is identical to the cost ofpeering and is split however the ISPs negotiate.
Merger and Acquisition (Optional). ISPs canagree to pool their interests and merge into a singleISP. There is functionally no difference in play exceptmoney can flow between the players and the new
merged company gets two turns. Transit fees must be
Workshop, London March 16, 2000
RO
UN
D
PLA
YE
R
Ro
ll(C
op
yd
ow
n)
#S
quar
esO
wne
d
Rev
enue
(Squ
ares
*$2
000)
#Oth
ersS
quar
es
Tra
nsit
Cos
t(*$
1000
)
Pee
ring
Cos
ts
Net
Run
ning
Xpe
erY
(cle
arco
nten
ts)
PLA
YE
R
Pay
for
Tra
nsit
toA
?
Pay
for
Tra
nsit
toB
?
Pay
for
Tra
nsit
toC
?
Pay
for
Tra
nsit
toD
?
Sum
ofT
rans
it$$
$$pa
idto
X
Sum
ofT
rans
it$$
$P
aid
toY
0 A FALSE 1 $0 3 $0 $0 $0 $0 A 1 1 1 -$
0 B FALSE 1 $0 3 $0 $0 $0 $0 B 1 1 1 -$
0 C FALSE 1 $0 3 $0 $0 $0 $0 C 1 1 1 -$
0 D FALSE 1 $0 3 $0 $0 $0 $0 D 1 1 1 -$copy A FALSE 1 $2,000 3 ($3,000) $0 ($1,000) $0 A 1 1 1 -$copy B FALSE 1 $2,000 3 ($3,000) $0 ($1,000) $0 B 1 1 1 -$copy C FALSE 1 $2,000 3 ($3,000) $0 ($1,000) $0 C 1 1 1 -$copy D FALSE 1 $2,000 3 ($3,000) $0 ($1,000) $0 D 1 1 1 -$
1 A FALSE 1 $2,000 3 ($3,000) $0 ($1,000) ($1,000) A 1 1 1 (3,000)$1 B FALSE 1 $2,000 3 ($3,000) $0 ($1,000) ($1,000) B 1 1 1 (6,000)$1 C FALSE 1 $2,000 3 ($3,000) $0 ($1,000) ($1,000) C 1 1 1 (3,000)$1 D FALSE 1 $2,000 3 ($3,000) $0 ($1,000) ($1,000) D 1 1 1 (6,000)$
DRAFT Peering Decision Tree W. B. Norton
15 Comments to the Author Welcome
paid until the two ISPs peer.
How this simulation is different from peeringreality:
1. The board is veiled allowing for gaming andbluffing during peering negotiations
2. ISPs move serially in the game, while in thereal world action is parallel.
3. The meaning of the board squares is severelyoverloaded to mean regional coverage andcorresponding revenue, a quantum of trafficgenerated, and a quantum of traffic transittedto all others. All customers are not equal inrevenue, traffic.
4. Customer transit revenue gained does notcause any additional financial load for the ISPin the game.
5. Traffic quantum is a vague notion thatignores the asymmetric nature of traffictoday.
6. Shared squares should cause revenue andcosts to be divided
7. Everyone starts with the same number ofsquares.
8. Everyone is financially backed to supportinfinite periods of financial loss. Well, thatmay reflect reality for some period of Internettime.
9. If ISPs fail to peer they must pay transit to getaccess to these squares. In reality, contentmulti-homes allowing alternative paths to thesame content.
10. Business motivations to sell transit instead ofpeer are an ignored dynamic in the game.
Summary
The basic peering game does a good job ofhighlighting the issues ISPs face when peering.Several comments from ISPs offeringenhancements add reality to the game at the costof complexity. For example, ISPs capture marketshare in order to be an attractive acquisitiontarget. Adding merger rules adds a realcomplexity, somewhat tangentially related topeering and transit. Adding rules for ISPs tobuy/sell transit to each other similarly addscomplexity but adds a negotiating dynamic thatISPs face today. Balancing the desire to explainand explore against the desire for the simulation tomatch reality has proven to be a challenge.
