Interconnection, Peering, and Settlements

Geoff Huston

Overview• Introduction

• Interconnection: Peer or client

• Interconnection architecture

• Interconnection financials

• Settlement models

• Settlement structures

• QOS & financial settlement

• Conclusions

Introduction• Tens of thousands of ISPs operating in

deregulated business space

• Complex environment– Transaction between two ISPs involves multiple

providers

• Basic financial cost distribution is based on bilateral relationships of customer/provider and mutual peering

• Internet industry use small set of physical interconnection mechanisms (LAN switches)

Interconnection: retailing, reselling, wholesale

• Internet is an outcome of business and technology interaction

• ISPs do not have clear precise roles:– Retail, resel, and wholesale– Retail ISP can easily become a wholesale provider– Many ISPs operate as client and provider– Hard to support stable segmentation into wholesale

and retail market sectors

Intrernet enviroment:

•There is no well ordered hierarchical model of a set of wholesale ISPs and retail ISPs

•Diverse ISPs operating as retailers and wholesale providers to other retailers

Peer or Client• Leads to a question of who is making subjective

decision and on what basis.

• Traditional public solution: pay a fee to a regulator that gives ISP a peer license

• Two problems:– Regulators are artificial in defining the market entities

(client vs. peers)– Discourages large-scale private investment, thus

putting funding burden on public sector

Things have changed

• Regulatory environment is changing to shift burden of comm. infrastructure – Public sector to private investment – Deregulated environment– No one can say whether two ISPs are client/provider or peers

• Who can say so in the industry?– Commercial Internet eXchange (CIX)– Based on description of infrastructure of each party– Peer: if you have a national transit infrastructure capability– Modified later: pay a fee to CIX assicaition– Not bilateral but multilateral relationship with other peers !!– Zero financial settlement (based on a fee)

• Other models use functional peer specification:– If ISP attaches to physical exchange entity, it is up to the ISP to open

bilateral negotiation with other ISPs attached to the exchange

• True peer relationship is based on the assumption that either party can terminate the interconnection– If one ISP relies on the interconnection more than the other

provider/client relationship– If there is a balance of mutual requirement between two ISPs peer

interconnection relationship– Problem: no metrics to quantify requirements (based on perception)

• There are various levels of peering in today’s internet due to business pressures– Local ISPs see competing local ISPs as peers– Local ISPs are clients of trunk ISPs

Interconnection Architecture

• Strict hierarchical structure– Worst case: traffic between two ISPs may traverse transit ISPs– Extended paths are inefficient and costly– To reduce costs, ISPs at different levels construct bilateral

interconnections

• Connect with all ISPs (full mesh, not scalable:N2 connections)• Local exchange model

– ISP connects to a single local exchange (scalable: N connections)– Exchange router is active component in peering policy

• Each ISP must have multilateral peering with all other ISPs

– Router must execute its own routing policy• When two ISPs advertise a route to same destination, router makes

decision on behalf of all other connected ISPs

– Router may not be completely neutral to all ISPs

How to connect with ISPs?

What do ISPs expect?

• Flexibilty of policy determination from exchange structure

• Bilateral interconnection at the exchange structure

• Make policy decisions when same destination is adverised by multiple providers

• Exchange must be neutral with respect to individual routing policies

• HOW?– Modify exchange model to use LAN switch as exchange element

• Each ISP:– has a dedicated router at the LAN exchange– has a bilateral peering agreement with another ISP by initiating

router peering session with the other’s router– If multiple peers advertise a path to same dest, ISP can use its

own policy-based preference to choose route

• Exchange environment must offer high degree of resilience and security (costs a lot)

Distributed exchange model

• Exchange comes to ISP location• Must have uniform access technology between every

exchange participant• Issues: switching speed (contention can be a problem)

Network Access Points NAPs

• Roles of NAP:– Exchange provider between ISPs– Transit purchase site to make agreements between

ISPs and trunk ISPs

Exchange Business Models• In the ISP industry, a common business model require the

internet exchange to be– Operated by neutral party that is not an ISP– Constructed in robust and secure fashion– Located in high density areas of internet market– Scalable– Operated in sound and stable fasion– Others (Performance of the exchange, QOS)

• Common business models use flat-fee structure– Based on the number of rack units used by an ISP at the exchange

• Other models are strcutured as cooperative entity between a number of ISPs– Problem: no ISP wants to financially take responsibilty for ensuring

quality of the exchange

Today’s Internet• Increasing ISPs will lead to increasing complexity of

interconnection structure

• Inability to reach stable cost distribution model makes each ISP optimize itself by making direct connections with peer ISPs (thru exchanges or direct 1:1 links)

• Curdity of inter-AS routing policy tools makes internet structure a source of considerable concern especially with– Absence of coherent policy (or even commonly accepted set of practices)

– Lack of administration of the AS space

Interaction Financials• Cost distribution:

– Compensation of all ISPs who participate in the delivery of a service to a customer of a single ISP

• Users want comprehensive end-to-end service with clients being parts of different ISPs

• How do all ISPs involve in a transaction?

• Who incur the cost in supporting the transaction?

• Who receive compensation?

• What is the cost distribtion model?

What is the currency of interaction?

• Routing advertisements– ISPs exchange routing entries to allow traffic flows – Traffic flows in opposite direction of route advertisement

Types of Routes

• Clients routes– Passed to ISP routing by contract with client, static configuration at edge of ISP,

learned by BGP, or part of DHCP addresses

• Internal ISP routes– DNS, SMTP, SNMP, POP..etc

• Upstream routes– Learned from making a transit service contract with upstream ISP

• Peer routes– Learned from exchange or private interconnection

• What is the route export policy ?

Internet settlement models

• Packet cost accounting (strawman model)– Everytime a packet crosses network boundry, it is sold to next ISP

– Ultimately, the packet is sold to the receiver client

• Pros:– Revenue gains from packets deliverd on egress from network

– Economic incentives not to drop packets in transit ISPs

• Cons:– Packet drop is inevitable

– Mechanism is open to abuse

TCP session accounting• Network boundry can

– Detect initial TCP handshake

– Count all subsequent packets with same TCP session

– Session initiator pays for entire traffic flow

• Such accounting allows for settlement based on dutration (TCP packets) or volume (TCP sessions)

• Problem: very hard to do– Router at the network boundry must do all work !!

• Real problem with any settlement models– Today’s internet have many retail pricing structures– Based on received volume, sent volume, mix, access capacity– There is no uniform retail pricing

Internet settlement structures

• Two structures of interactions between two ISPs– Customer/provider and peering with no form of financial settlement

• Sender Keep All (SKA)– Usually applies to domestic traffic between two ISPs

– Stable when both parties perceive equal benefit from interconnection

– Ex. Among local ISPs, regional ISPs, national ISPs

• How does it work? On each interconnection– each ISP ONLY presents/accepts to/from other ISP routes associated with

its customers

– Clients make contract with an ISP to present their routes to all other customers of the same ISP, to the upstream providers of the ISP, and all peer ISPs

How does all that look like ?

• Internet into two domains: transit ISPs, local ISPs• Transit ISPs: high capacity carriage infrastructure• Local ISPs: retail services

– Participate at exchanges with SKA peer interconnection with other ISPs

– Exchange does not have full connectivity ISPs purchase transit services

Negotianted financial settlement• Alternative to customer/provider and peer structures• Based on both parties selling services to each other

across the interconnection• Simple model:

– Measure volume of traffic in each direction

– Use single accounting rate for all traffic

– At the end of accounting period, two ISPs settle based on the agreed rate applied to the traffic

• Which way should the money flow in relation to traffic flow?– One model: Originating ISP should pay terminating ISP to deliver traffic

– Another model: when traffic is generated because of an action of a receiver (webpage, downloads), terminating ISP should pay

Settlement Debate

• Despite great ISP attention, today’s internet does not have sound models of financial settlements

• Why has the internet managed to pose hard challenge to the ISP industry?– Caused by adopted retail models of ISP services

– The internet as retailed to clients is not a comprehensive end-to-end service

– Internet works as a result of partial path paired services• Sender funds initial path component and receiver funds terminating path

component

• Natural outcome of today’s internet settlement environment is one of aggregation of ISPs

QOS and financial settlements• The shift towards end-to-end service model and support

of QOS are strong factors to change current ISP service model

• Meaningful inter-provider financial settlements depend highly on introducing end-to-end service retail models

• That in turn depends on universally shifting from best-effort regime to layered end-to-end service regimes

Conclusions

• $0 peering and customer/provider relationships are the only stable models within the internet

• As a consequence– deployment of end-to-end QOS is highly

unlikely in such an environment– Inability to support highly diverse ISP env– Aggregation within ISP industry