Upload
sunny-bhushan
View
220
Download
0
Embed Size (px)
Citation preview
8/2/2019 Modern Carrier Strategies for TE DrVishalSharma 2002-11-28
1/36
Modern Carrier Strategies for
Traffic Engineering
Dr. Vishal SharmaPrincipal ConsultantMetanoia, Inc.Voice: +1 408 394 6321Email: [email protected]: http://www.metanoia-inc.com
Metanoia, Inc.Critical Systems Thinking
Copyright 2002
All Rights Reserved
8/2/2019 Modern Carrier Strategies for TE DrVishalSharma 2002-11-28
2/36
Metanoia, Inc.Critical Systems Thinking
Copyright 2002,All Rights Reserved Modern Carrier Strategies for TE
2
Basic Service Provider Goals
The two fundamentaltasks before any service provider:
Deploy a physical topology that meets customers needs
Map customer traffic flows on to the physical topology
Earlier (1990s) the mapping task was uncontrolled!
By-product of shortest-path IGP routing
Often handled by over-provisioning
8/2/2019 Modern Carrier Strategies for TE DrVishalSharma 2002-11-28
3/36
Metanoia, Inc.Critical Systems Thinking
Copyright 2002,All Rights Reserved Modern Carrier Strategies for TE
3
The Early Years (< 1994-95): RoutedNetwork Topology
IP Router
Network Cloud
IP Router
IP RouterIP Router
RouterInterconnections
8/2/2019 Modern Carrier Strategies for TE DrVishalSharma 2002-11-28
4/36
Metanoia, Inc.Critical Systems Thinking
Copyright 2002,All Rights Reserved Modern Carrier Strategies for TE
4
The Early Years (< 1994-95): AStacked View
IP Routers
DCS/DXC
TDM overcopper (T1/T3)
FDDI rings(100 Mb/s)
Router
MUX
SDH/SONET
-- Service creation-- Pkt. switching-- Stat muxing
-- Connectivity
-- Speed match I/Fs
-- TDM transport-- Fault isolation-- Restoration
8/2/2019 Modern Carrier Strategies for TE DrVishalSharma 2002-11-28
5/36
Metanoia, Inc.Critical Systems Thinking
Copyright 2002,All Rights Reserved Modern Carrier Strategies for TE
5
TE in Carrier Networks: TraditionalRouted Core (pre 1994-95)
Prior to advent of ATM ...
... IP metricswere the only means available to control traffic
distribution through IP networks
Approach was ad-hoc
Observe traffic flow through network
Adjust weight of links with load lower/higher than desired
Overprovision network as, needed
8/2/2019 Modern Carrier Strategies for TE DrVishalSharma 2002-11-28
6/36
Metanoia, Inc.Critical Systems Thinking
Copyright 2002,All Rights Reserved Modern Carrier Strategies for TE
6
Two Paths to TE in IP Networks
With increase in traffic, emergence of ATM, and higher-
speed SONET, two approaches emerged
Use a Layer 2 (ATM) network
Build ATM backbone
Deploy complete PVC mesh,
bypass use of IP metrics
TE at ATM layer
With time, evolve ATM to MPLS-
based backbone
Use only Layer 3 (IP) network
Build SONET infrastructure
Rely on SONET for resilience
Run IP directly on SONET (POS)
Use metrics (systematically) to
control flow of traffic (more on
this later)
8/2/2019 Modern Carrier Strategies for TE DrVishalSharma 2002-11-28
7/36
Metanoia, Inc.Critical Systems Thinking
Copyright 2002,All Rights Reserved Modern Carrier Strategies for TE
7
Genesis of the ATM-Core
Growth in traffic needed faster backbones (> T3/45 Mb/s) Denser backbones metric manipulation impractical
IP routers lagged: offered only DS3 I/Fs & s/w forwarding
ATM emerged, was designed for WAN from start
In 1994-95 had OC-3, and later OC-12 I/Fs available
Allowed carriers to redesign their networks for high-speeds
As an evolutionary step, SPs moved to a switched ATM core
8/2/2019 Modern Carrier Strategies for TE DrVishalSharma 2002-11-28
8/36
Metanoia, Inc.Critical Systems Thinking
Copyright 2002,All Rights Reserved Modern Carrier Strategies for TE
8
ATM-based Cores (mid to late 1990s)
Router
ATM
MUX
SDH/SONET
-- Service creation-- Pkt. switching-- Stat muxing-- Connectivity
-- Traffic engg.-- Hardware fwding
-- Speed match I/Fs
-- TDM transport-- Fault isolation-- Restoration
8/2/2019 Modern Carrier Strategies for TE DrVishalSharma 2002-11-28
9/36
Metanoia, Inc.Critical Systems Thinking
Copyright 2002,All Rights Reserved Modern Carrier Strategies for TE
9
Physical Topology with an ATM Core
POP i+1
POP j
POP N
POP 1
POP 2
POP i
OC-12
OC-3
OC-3
Router
ATM Switch
ATM Core
OC-3 ATMSAR I/F
8/2/2019 Modern Carrier Strategies for TE DrVishalSharma 2002-11-28
10/36
Metanoia, Inc.Critical Systems Thinking
Copyright 2002,All Rights Reserved Modern Carrier Strategies for TE
10
Logical Topology with an ATM Core
ATM-core (usually) fully owned by SP
Dedicated (usually) to supporting IP backbone
Utilized ATM UBR or ATM VBR-rt/CBR, depending on classes of traffic
B
A C
D
Primary PVC Mesh
Secondary PVC Mesh
PVC 1'
PVC 2'
PVC 4'
PVC 3'
PVC 5'
PVC 1
PVC 2
PVC 3
PVC 4
PVC 5B
AC
D
ATM PVC Layout L3 Logical Topology
8/2/2019 Modern Carrier Strategies for TE DrVishalSharma 2002-11-28
11/36
Metanoia, Inc.Critical Systems Thinking
Copyright 2002,All Rights Reserved Modern Carrier Strategies for TE
11
Genesis of the IP-over-SONET/SDHApproach
Desire to minimize # of network layers Easier management
Simpler operation
Potentially scalable
Belief that high-speed SONET/SDH I/Fs would become
available with advances in components (vindicated with time)
Dictated partly by how (in time) a carriers network evolved
8/2/2019 Modern Carrier Strategies for TE DrVishalSharma 2002-11-28
12/36
Metanoia, Inc.Critical Systems Thinking
Copyright 2002,All Rights Reserved Modern Carrier Strategies for TE
12
SONET/SDH-based Cores (mid-to-late 1990s and beyond)
Router
MUX
SDH/SONET
-- Service creation-- Pkt. switching-- Stat muxing-- TE w/ metrics or
MPLS
-- Speed match I/Fs
-- Framing
-- Fault isolation-- Restoration(moving away)
DWDM
-- B/w on existingplant
8/2/2019 Modern Carrier Strategies for TE DrVishalSharma 2002-11-28
13/36
Metanoia, Inc.Critical Systems Thinking
Copyright 2002,All Rights Reserved Modern Carrier Strategies for TE
13
Physical Topology with SONET/SDH Core
POP i+1
POP j
POP N
POP 1
POP 2
POP i
OC-3/12 or
STM-1/4
Router
SONET/SDH Core
OC-3 SONET/STM-1 SDH I/F
OC-48 BLSR/MS-SPRing
OC-3/13 UPSR/SNCP Ring
Point-to-point SONET/SDH framed link
DXC
8/2/2019 Modern Carrier Strategies for TE DrVishalSharma 2002-11-28
14/36
Metanoia, Inc.Critical Systems Thinking
Copyright 2002,All Rights Reserved Modern Carrier Strategies for TE
14
Ckt. 1
Ckt. 2
Ckt. 3
Ckt. 4
Ckt. 5
Logical Topology with SONET/SDH Core
SONET/SDH infrastructure (usually) owned by SP
Logical links between POP routers realized over a physical SONET/SDH
circuit going over a fiber path
Parallel logical links (physically disjoint) provisioned b/w each router pair
A
B D
C
B
A C
D
Parallel Logial Links forLoad Sharing
Each logical link provisionedfor 2X the bandwidth
SONET/SDH Circuit LayoutL3 Logical Topology
8/2/2019 Modern Carrier Strategies for TE DrVishalSharma 2002-11-28
15/36
Metanoia, Inc.Critical Systems Thinking
Copyright 2002,All Rights Reserved Modern Carrier Strategies for TE
15
Global Crossing IP Backbone Network
100,000 route miles 27 countries 250 major cities5 continents200+ POPs
Courtesy: Thomas Telkamp, GBLX
M i I
8/2/2019 Modern Carrier Strategies for TE DrVishalSharma 2002-11-28
16/36
Metanoia, Inc.Critical Systems Thinking
Copyright 2002,All Rights Reserved Modern Carrier Strategies for TE
16
Global Crossing (GBLX): A Bit of History
First independent global fiber network Launched operations -- March 1997
First segment turned on -- May 1998
Expanded network & svcs. by acquisitions & JVs
Frontier Telecommunications, Sept 1999
Racal Telecom, Nov 1999
Hutchison Global Crossing, Jan 2000
IXNET/IPC, June 2000
International network, worldwide reach
100,000 route miles, 27 countries, 250 major cities
195 POPs (mid 2001)
M t i I
8/2/2019 Modern Carrier Strategies for TE DrVishalSharma 2002-11-28
17/36
Metanoia, Inc.Critical Systems Thinking
Copyright 2002,All Rights Reserved Modern Carrier Strategies for TE
17
Global Crossing IP Network
OC-48c/STM-16c (2.5Gbps) IP backbone Selected 10Gbps links operational (e.g. Atlantic)
Services offered Internet access & Transit services
IP VPNs -- Layer 3 and Layer 2
MPLS and DiffServ deployed globally
Edge Equipment
Core Equipment
Cisco GSR 12000/12400[12.0(17) SI]
Cisco 7500/7200 ESR, OSR
Juniper M10/20/40
M t i I
8/2/2019 Modern Carrier Strategies for TE DrVishalSharma 2002-11-28
18/36
Metanoia, Inc.Critical Systems Thinking
Copyright 2002,All Rights Reserved Modern Carrier Strategies for TE
18
Global Crossing:Network Design Philosophy
Ensure there are no bottlenecksin normal state
On handling congestion
Preventvia MPLS-TE
Managevia Diffserv
Over-provisioning
Well traffic engineerednetwork can handle all traffic
Can withstand failure of even the most critical link(s)
Avoid excessive complexity & features
Makes the network unreliable/unstable
M t i I
8/2/2019 Modern Carrier Strategies for TE DrVishalSharma 2002-11-28
19/36
Metanoia, Inc.Critical Systems Thinking
Copyright 2002,All Rights Reserved Modern Carrier Strategies for TE
19
Global Crossings Approach: Big Picture
WebServer
HR
DRBR
AR
CR
WR
DR
HR BR
AR
CR
WR
DR
HR BR
AR
CR
WR
EthernetSwitch
ModemBank
To other ISPs
To Customers
POP1
POP2
POP3
AR = Access RouterBR = Border Router
CR = Core Router
HR = Hosting Router
WR = WAN Router
DR = DSL Aggregation
OC-3/OC-12
OC-12/OC-48OC-48/OC-192
Metanoia Inc
8/2/2019 Modern Carrier Strategies for TE DrVishalSharma 2002-11-28
20/36
Metanoia, Inc.Critical Systems Thinking
Copyright 2002,All Rights Reserved Modern Carrier Strategies for TE
20
TE in the US IP Network:Deployment Strategy
Decision to adopt MPLS for traffic engineering & VPNs Y2000: 50+ POPs, 300 routers; Y2002: 200+ POPs
Initially, hierarchical MPLS system 2 levels of LSPs
Later, a flat MPLS LSP full mesh onlybetween core routers
Started w/ 9 regions -- 10-50 LSRs/region 100-2500 LSPs/region Within regions: Routers fully-meshed
Across regions: Core routers fully-meshed
Intra-region traffic ~Mb/s to Gb/s, Inter-region traffic ~ Gb/s
Source [Xiao00]
Metanoia Inc
8/2/2019 Modern Carrier Strategies for TE DrVishalSharma 2002-11-28
21/36
Metanoia, Inc.Critical Systems Thinking
Copyright 2002,All Rights Reserved Modern Carrier Strategies for TE
21
Design Principles: Statistics Collection
A
B
C
LSP1 = 15 Mb/s
LSP2 = 10 Mb/s
LSP3 = 10 Mb/s
Statistics on individual LSPs can
be used to build matrices
A
B
C
25 Mb/s
25 Mb/s
Using packets, we do not knowtraffic individually to B & C
Metanoia Inc
D i P i i l LSP C l &
8/2/2019 Modern Carrier Strategies for TE DrVishalSharma 2002-11-28
22/36
Metanoia, Inc.Critical Systems Thinking
Copyright 2002,All Rights Reserved Modern Carrier Strategies for TE
22
Design Principles: LSP Control &Management
B
A
D
D
B
OC-48
OC-192
10% in usebefore new req.
New Request
A to D = 2.2 Gb/s
New LSP takes
longer path
Links utilization ismore balancedManually move traffic away from
potential congestion via ERO
B
A
D
D
B
B
A
D
D
B
OC-192
2 LSPs of 1.2Gb/s each
LSPs split acrossalternate routes
Lowered load, greaterheadroom to grow
Load splittingratio = 0.5 each
OC-48
Adding new LSPs with aconfigured load splitting ratio
Metanoia Inc
Gl b l C i C t LSP
8/2/2019 Modern Carrier Strategies for TE DrVishalSharma 2002-11-28
23/36
Metanoia, Inc.Critical Systems Thinking
Copyright 2002,All Rights Reserved Modern Carrier Strategies for TE
23
Global Crossings Current LSPLayout and Traffic Routing
Region 1Region 2
Region 3
Region 4
POP1POP3
POP4
POP5POP2
Full LSP Meshin Core
Core LSP betweenWRs in POPs 1 & 5
Source
Destination
Metanoia Inc
8/2/2019 Modern Carrier Strategies for TE DrVishalSharma 2002-11-28
24/36
Metanoia, Inc.Critical Systems Thinking
Copyright 2002,All Rights Reserved Modern Carrier Strategies for TE
24
Sprint (FON): A Bit of History
A century of evolution ...
1899: Brown Telephone Co., Abilene, KS
1976: United Telephone, Kansas City, MS, 3.5M customers
1984: Began building first, all-digital, US-wide, fiber-opticnetwork
1986: United + GTE merge LD subsidiaries US Sprint1992: United buys GTEs stake, renaming co. to Sprint Corp.
2002: ~$23B revenue, 23M customers, 70 countries, 80,000 employees
110,000+ route miles in the long distance (LD) network
34,000+ in US, 78,000+ in rest of the world
Transport infrastructure common to voice, ATM, & IP network
Provides considerable leverage, as well see later
Metanoia Inc
8/2/2019 Modern Carrier Strategies for TE DrVishalSharma 2002-11-28
25/36
Metanoia, Inc.Critical Systems Thinking
Copyright 2002,All Rights Reserved Modern Carrier Strategies for TE
25
Sprint (FON): IP Network Timeline
'92 '93 '95 '96 '97 '98 '99 '01 '02
First IXC w/ Internet
svc. on T1 network
All DS3 IP network
DEC FDDI Gigaswitch POP
Work with Cisco for next
router for OC-3 backbone
Service via native IP bbone
OC-12 4F-BLSR deployed
Cisco GSR tested/deployed
GigaPOP bbone
GSR in POP
OC-3 WAN
Deploy OC-48
POS over DWDM
Deploy GSR12016
OC-192 TAT links
OC-192 in Europe
All bbone routers
GSR12416s
Expand to Asia,
South America
Metanoia Inc
8/2/2019 Modern Carrier Strategies for TE DrVishalSharma 2002-11-28
26/36
Metanoia, Inc.Critical Systems Thinking
Copyright 2002,All Rights Reserved Modern Carrier Strategies for TE
26
SprintLinkTM IP Backbone Network
19+ countries
30+ major intl. cities5 continents(reach S. America as well)
400+ POPs
Courtesy: Jeff ChaltasSprint Public Relations
Represents connectivityonly (not to scale)
110,000+ route miles(common with Sprint LD network)
Metanoia Inc
8/2/2019 Modern Carrier Strategies for TE DrVishalSharma 2002-11-28
27/36
Metanoia, Inc.Critical Systems Thinking
Copyright 2002,All Rights Reserved Modern Carrier Strategies for TE
27
SprintLinkTM IP Network
Tier-1 Internet backbone Customers: corporations, Tier-2 ISPs, univs., ...
Native IP-over-DWDM using SONET framing
4F-BLSR infrastructure (425 SONET rings in network)
Backbone US: OC-48/STM-16 (2.5 Gb/s) links
Europe: OC-192/STM-64 (10 Gb/s) links
DWDM with 8-40 s/fiber Equipment
Core: Cisco GSR 12000/12416 (bbone), 10720 metro edge router
Edge: Cisco 75xxx series
Optical: Ciena Sentry 4000, Ciena CoreDirector
Metanoia, Inc.
8/2/2019 Modern Carrier Strategies for TE DrVishalSharma 2002-11-28
28/36
Metanoia, Inc.Critical Systems Thinking
Copyright 2002,All Rights Reserved Modern Carrier Strategies for TE
28
SprintLinkTM IP Design Philosophy
Large networks exhibit arch., design & engg. (ADE) non-linearities not
seen at smaller scales
Even small things can & do cause huge effects (amplification)
More simultaneous events mean greater likelihood of interaction (coupling)
Simplicity Principle: simple n/wks are easier to operate & scale
Complexity prohibits efficient scaling, driving up CAPEX andOPEX!
Confine intelligenceat edges
No state in the network core/backbone
Fastest forwarding of packets in core
Ensure packets encounter minimal queueing
Metanoia, Inc.
8/2/2019 Modern Carrier Strategies for TE DrVishalSharma 2002-11-28
29/36
Metanoia, Inc.Critical Systems Thinking
Copyright 2002,All Rights Reserved Modern Carrier Strategies for TE
29
SprintLinkTM Deployment StrategyL2 failure detection triggers
switchover before L3 converges
ZA
Parallel links 50% utilizationunder normal state
1
2
3
4
SONET framing forerror detection
LineCard
LineCard
SONET
Overhead
IP Data
Metanoia, Inc.
8/2/2019 Modern Carrier Strategies for TE DrVishalSharma 2002-11-28
30/36
Metanoia, Inc.Critical Systems Thinking
Copyright 2002,All Rights Reserved Modern Carrier Strategies for TE
30
SprintLinkTM Design Principles
Great value on traffic measurement & monitoring
Use it for
Design, operations, management
Dimensioning, provisioning
SLAs, pricing
Minimizing the extent of complex TE & QoS in the core
Metanoia, Inc.
8/2/2019 Modern Carrier Strategies for TE DrVishalSharma 2002-11-28
31/36
eta o a, cCritical Systems Thinking
Copyright 2002,All Rights Reserved Modern Carrier Strategies for TE
31
Sprints Approach to Monitoring
AccessRouter
AccessRouter
AccessRouter
BackboneLinks
Peering LinksProbe
BackboneRouter
Customers Customers Customers
Adapted from [Diot99]
Analysis platform located atSprint ATL, Burlingame, CA
Metanoia, Inc.
8/2/2019 Modern Carrier Strategies for TE DrVishalSharma 2002-11-28
32/36
,Critical Systems Thinking
Copyright 2002,All Rights Reserved Modern Carrier Strategies for TE
32
Sprint Approach to TE
Aim: Thoroughly understand backbone traffic dynamics
Answer questions such as:
Composition of traffic? Origin of traffic?
Between any pair of POPs
What is the traffic demand?
Volume of traffic?
Traffic patterns? (In time? In space?)
How is this demand routed?
How does one design traffic matrics optimally?
8/2/2019 Modern Carrier Strategies for TE DrVishalSharma 2002-11-28
33/36
Metanoia, Inc.
8/2/2019 Modern Carrier Strategies for TE DrVishalSharma 2002-11-28
34/36
,Critical Systems Thinking
Copyright 2002,All Rights Reserved Modern Carrier Strategies for TE
34
A Peek at a Row of a Traffic Matrix
Summary of Data Collected
Adapted from [Bhattacharya02]
Distribution of aggregate accesstraffic across other POPs in the Sprintbackbone
Peer 1
Peer 2
Web 2
Web 1
ISP
To Backbone
Sprint POP under study
Metanoia, Inc.
8/2/2019 Modern Carrier Strategies for TE DrVishalSharma 2002-11-28
35/36
Critical Systems Thinking
Copyright 2002,All Rights Reserved Modern Carrier Strategies for TE
35
Applications of Traffic Matrices
Traffic engineering
Verify BGP peering
Intra-domain routing
SLA drafting
Customer reports
Metanoia, Inc.
8/2/2019 Modern Carrier Strategies for TE DrVishalSharma 2002-11-28
36/36
Critical Systems Thinking
Copyright 2002
Acknowledgements
Thomas Telkamp, Global Crossing
Robert J. Rockell, Jeff Chaltas, Ananth Nagarajan, Sprint
Steve Gordon, Cable and Wireless
Jennifer Rexford, Albert Greenberg, Carsten Lund, AT&T Research
Wai-Sum Lai, AT&T
Fang Wu, NTT America
Arman Maghbouleh, Alan Gous, Cariden Technologies
Yufei Wang, VPI Systems