Upload
trisetiawan
View
224
Download
0
Embed Size (px)
Citation preview
7/26/2019 01 ISP Network Design
1/110
ISP Network Design
ISP Workshops
1Last updated 27thFebruary 2015
7/26/2019 01 ISP Network Design
2/110
ISP Network Design
! PoP Topologies and Design
! Backbone Design
! Upstream Connectivity & Peering
! Addressing
! Routing Protocols
! Security
! Out of Band Management
! Operational Considerations
2
7/26/2019 01 ISP Network Design
3/110
Point of PresenceTopologies
3
7/26/2019 01 ISP Network Design
4/110
PoP Topologies
! Corerouters high speed trunkconnections
! Distributionrouters and Accessrouters
high port density! Borderrouters connections to other
providers
! Servicerouters hosting and servers
! Some functions might be handled by asingle router
4
7/26/2019 01 ISP Network Design
5/110
PoP Design
! Modular Design
! Aggregation Services separated accordingto
"
connection speed"
customer service
"
contention ratio
" security considerations
5
7/26/2019 01 ISP Network Design
6/110
Modular PoP Design
6
Backbone linkto another PoP
Backbone linkto another PoP
Business customer
aggregation layer
for leased line circuit deliveryChannelised circuits
NetworkOperations
Centre
Consumer
Dial Access
NetworkCore
Consumer cable,xDSL and
wireless Access
for MetroE circuit deliveryGigE fibre trunks
MetroE customer
aggregation layer
ISP Services(DNS, Mail, News,
FTP, WWW)
Hosted Services &Datacentre
Other ISPs
Web Cache
7/26/2019 01 ISP Network Design
7/110
Modular Routing Protocol Design
! Modular IGP implementation
"
IGP areaper PoP
" Core routers in backbone area (Area 0/L2)
" Aggregation/summarisation where possible
into the core
! Modular iBGP implementation
" BGP route reflector cluster
" Core routers are the route-reflectors
" Remaining routers are clients & peer withroute-reflectors only
7
7/26/2019 01 ISP Network Design
8/110
Point of Presence Design
8
7/26/2019 01 ISP Network Design
9/110
PoP Modules
! Low Speed customer connections
"
ADSL2, Cable, Public Wireless, PSTN access
" Low bandwidth needs
" Low revenue, large numbers
! Business customer connections
" 10Mbps to 100Mbps speed range
" Delivery over SDH or MetroEthernet links
"
Medium bandwidth needs
" Medium revenue, medium numbers
9
7/26/2019 01 ISP Network Design
10/110
PoP Modules
! Highband Business customer connections
"
From 100Mbps to over 1Gbps
" Ethernet, Fibre, or high end SDH
" High bandwidth needs
"
High revenue, low numbers
10
7/26/2019 01 ISP Network Design
11/110
PoP Modules
! PoP Core
"
Two dedicated routers
" High Speed interconnect
" Backbone Links ONLY
"
Do not touch them!
! Border Network
" Dedicated border router to other ISPs
"
The ISPs frontdoor
" Transparent web caching?
" Twoin backbone is minimum guarantee for
redundancy 11
7/26/2019 01 ISP Network Design
12/110
PoP Modules
! ISP Services
"
DNS (cache, secondary)
" News (still relevant?)
" Mail (POP3, Relay, Anti-virus/anti-spam)
"
WWW (server, proxy, cache)
! Hosted Services/DataCentres
" Virtual Web, WWW (server, proxy, cache)
"
Information/Content Services
" Electronic Commerce
12
7/26/2019 01 ISP Network Design
13/110
PoP Modules
! Network Operations Centre
"
Consider primary and backup locations
" Network monitoring
" Statistics and log gathering
"
Direct but secure access
! Out of Band Management Network
" The ISP Network Safety Belt
13
7/26/2019 01 ISP Network Design
14/110
PSTN & Broadband Access Module
14
To Core Routers
Telephone Network BRAS
SSG, DHCP, TACACS+or Radius Servers/Proxies,
DNS resolver, Content
Web Cache
Access Network
Gateway Routers
Cable RAS
DSLAM
IP, ATM
Primary Rate
T1/E1
Digital ModemAccess Servers
Cable System
7/26/2019 01 ISP Network Design
15/110
Business Customer Access Module
15
To Core Routers
Channelised T1/E1
Metro Ethernet
Direct Fibre Links
Aggregation Edge
7/26/2019 01 ISP Network Design
16/110
High Speed Access Module
16
To Core Routers
Metro Ethernet
Direct Fibre Links
Channelised OC3/OC12
Aggregation Edge
7/26/2019 01 ISP Network Design
17/110
ISP Services Module
17
To core routers
WWW
cache
Service Network
Gateway Routers
SMTPRelay(out)
DNSResolver
SMTPHost(in)
POP3sIMAPs
DNSSecondary
7/26/2019 01 ISP Network Design
18/110
Hosted Services Module
18
Customer 7Customer 3Customer 4Customer 5Customer 6
To core routers
Hosted Network
Gateway Routers
Customer 2Customer 1
vlan12vlan11 vlan13 vlan14 vlan15 vlan16 vlan17
7/26/2019 01 ISP Network Design
19/110
Border Module
19
To core routers
Network
Border Routers
To local IXP
NB: router has no default route +
local AS routing table onlyISP1 ISP2
7/26/2019 01 ISP Network Design
20/110
NOC Module
20
Primary DNS
To core routersHosted Network
Gateway Routers
SYSLOG
serverTACACS+serverNetwork Operations Centre Staff
Out of Band
Management Network
async terminal server
NetFlow
Analyser
Firewall
Billing, Database
and AccountingSystems
Corporate LAN
Critical ServiceModule
7/26/2019 01 ISP Network Design
21/110
Out of Band Network
21
Out of Band
Management Network
Terminal serverTo the NOC
Out of Band Ethernet
NetFlow
Collector
NetFlow
enabled
routers
Router
consoles
7/26/2019 01 ISP Network Design
22/110
Backbone NetworkDesign
22
7/26/2019 01 ISP Network Design
23/110
Backbone Design
! Routed Backbone
! Switched Backbone
" ATM/Frame Relay core network
"
Now obsolete! Point-to-point circuits
" nx64K, T1/E1, T3/E3, OC3, OC12, GigE, OC48,10GigE, OC192, OC768, 100GE
! ATM/Frame Relay service from telco
" T3, OC3, OC12, delivery
" Easily upgradeable bandwidth (CIR)
"
Almost vanished in availability now 23
7/26/2019 01 ISP Network Design
24/110
Distributed Network Design
! PoP design standardised
"
operational scalability and simplicity
! ISP essential services distributed aroundbackbone
! NOC and backupNOC
! Redundant backbone links
24
7/26/2019 01 ISP Network Design
25/110
Distributed Network Design
25
POP One
POP Two
POP Three
Customerconnections
Customerconnections
Customerconnections
Externalconnections
Externalconnections
Operations Centre
BackupOperations Centre
ISP Services
ISP Services
ISP Services
7/26/2019 01 ISP Network Design
26/110
Backbone Links
! ATM/Frame Relay
"
Virtually disappeared due to overhead, extraequipment, and shared with other customersof the telco
" MPLS has replaced ATM & FR as the telcofavourite
! Leased Line/Circuit
"
Most popular with backbone providers
" IP over Optics and Metro Ethernet verycommon in many parts of the world
26
7/26/2019 01 ISP Network Design
27/110
Long Distance Backbone Links
! These usually cost more
! Important to plan for the future
" This means at least two years ahead
" Stay in budget, stay realistic
" Unplanned emergencyupgrades will bedisruptive without redundancy in the networkinfrastructure
27
7/26/2019 01 ISP Network Design
28/110
Long Distance Backbone Links
! Allow sufficient capacity on alternativepaths for failure situations
" Sufficient can depend on the business strategy
" Sufficient can be as little as 20%
" Sufficient is usually over 50% as this offersbusiness continuity for customers in the caseof link failure
" Some businesses choose 0%
!
Very short sighted, meaning they have no sparecapacity at all!!
28
7/26/2019 01 ISP Network Design
29/110
Long Distance Links
29
POP One
POP Two
POP Three
Long distance link
Alternative/Backup Path
7/26/2019 01 ISP Network Design
30/110
Metropolitan Area Backbone Links
! Tend to be cheaper
"
Circuit concentration
" Choose from multiple suppliers
! Think big
" More redundancy
" Less impact of upgrades
" Less impact of failures
30
7/26/2019 01 ISP Network Design
31/110
Metropolitan Area Backbone Links
31
POP One
POP Two
POP Three
Metropolitan Links
Metropolitan Links
Traditional Point to Point Links
7/26/2019 01 ISP Network Design
32/110
Upstream Connectivityand Peering
32
7/26/2019 01 ISP Network Design
33/110
Transits
! Transit provider is another autonomous systemwhich is used to provide the local network withaccess to other networks
" Might be local or regional only
" But more usually the whole Internet
! Transit providers need to be chosen wisely:
" Only one
! no redundancy
" Too many
! more difficult to load balance
! no economy of scale (costs more per Mbps)
! hard to provide service quality
!
Recommendation: at least two, no morethan three
7/26/2019 01 ISP Network Design
34/110
Common Mistakes
! ISPs sign up with too many transit providers" Lots of small circuits (cost more per Mbps than larger
ones)
" Transit rates per Mbps reduce with increasing transitbandwidth purchased
" Hard to implement reliable traffic engineering thatdoesnt need daily fine tuning depending on customeractivities
! No diversity
" Chosen transit providers all reached over same satellite
or same submarine cable" Chosen transit providers have poor onward transit and
peering
7/26/2019 01 ISP Network Design
35/110
Peers
! A peer is another autonomous system with whichthe local network has agreed to exchange locallysourced routes and traffic
!
Private peer
" Private link between two providers for the purpose ofinterconnecting
!
Public peer
" Internet Exchange Point, where providers meet andfreely decide who they will interconnect with
!
Recommendation: peer as much as possible!
7/26/2019 01 ISP Network Design
36/110
Common Mistakes
! Mistaking a transit providers Exchangebusiness for a no-cost public peering point
! Not working hard to get as much peeringas possible
" Physically near a peering point (IXP) but notpresent at it
"
(Transit sometimes is cheaper than peering!!)
! Ignoring/avoiding competitors because
they are competition" Even though potentially valuable peering
partner to give customers a better experience
7/26/2019 01 ISP Network Design
37/110
Private Interconnection
! Two service providers agree tointerconnect their networks
" They exchange prefixes they originate into therouting system (usually their aggregated
address blocks)"
They share the cost of the infrastructure tointerconnect
! Typically each paying half the cost of the link (be itcircuit, satellite, microwave, fibre,)
! Connected to their respective peering routers
" Peering routers only carry domestic prefixes
37
7/26/2019 01 ISP Network Design
38/110
Private Interconnection
! PR = peering router" Runs iBGP (internal) and eBGP (with peer)
"
No default route" No full BGP table
" Domestic prefixes only
! Peering router used for all private interconnects
PR
PR
ISP1
ISP2
Upstream
Upstream
38
7/26/2019 01 ISP Network Design
39/110
Public Interconnection
! Service provider participates in anInternet Exchange Point
" It exchanges prefixes it originates into therouting system with the participants of the IXP
" It chooses who to peer with at the IXP
! Bi-lateral peering (like private interconnect)
! Multi-lateral peering (via IXPs route server)
" It provides the router at the IXP and provides
the connectivity from their PoP to the IXP
" The IXP router carries only domestic prefixes
39
7/26/2019 01 ISP Network Design
40/110
Public Interconnection
! ISP1-PR = peering router of our ISP" Runs iBGP (internal) and eBGP (with IXP peers)
"
No default route" No full BGP table
" Domestic prefixes only
! Physically located at the IXP
ISP1-PR ISP1
Upstream
40
IXP
ISP2-PRISP3-PR
ISP4-PR
ISP5-PR
ISP6-PR
7/26/2019 01 ISP Network Design
41/110
Public Interconnection
! The ISPs router IXP peering router needs carefulconfiguration:
" It is remote from the domestic backbone
" Should not originate any domestic prefixes
" (As well as no default route, no full BGP table)
" Filtering of BGP announcements from IXP peers (in andout)
!
Provision of a second link to the IXP:
" (for redundancy or extra capacity)
" Usually means installing a second router! Connected to a second switch (if the IXP has two more more
switches)
! Interconnected with the original router (and part of iBGP mesh)
41
7/26/2019 01 ISP Network Design
42/110
Public Interconnection
! Provision of a second link to the IXP meansconsidering redundancy in the SPs backbone
"
Two routers" Two independent links
" Separate switches (if IXP has two or more switches)
ISP1-PR1 ISP1
Upstream
42
IXP
ISP2-PRISP3-PR
ISP4-PR
ISP5-PR
ISP6-PR
ISP1-PR2
7/26/2019 01 ISP Network Design
43/110
Upstream/Transit Connection
! Two scenarios:
"
Transit provider is in the locality
! Which means bandwidth is cheap, plentiful, easy toprovision, and easily upgraded
" Transit provider is a long distance away
! Over undersea cable, satellite, long-haul crosscountry fibre, etc
! Each scenario has different considerationswhich need to be accounted for
43
7/26/2019 01 ISP Network Design
44/110
Local Transit Provider
! BR = ISPs Border Router
" Runs iBGP (internal) and eBGP (with transit)
" Either receives default route or the full BGP table from
upstream" BGP policies are implemented here (depending on
connectivity)
" Packet filtering is implemented here (as required)
AR
BR
Transit
ISP1
44
7/26/2019 01 ISP Network Design
45/110
Distant Transit Provider
! BR = ISPs Border Router" Co-located in a co-lo centre (typical) or in the upstream providers
premises
" Runs iBGP with rest of ISP1 backbone" Runs eBGP with transit provider router(s)
"
Implements BGP policies, packet filtering, etc
" Does not originate any domestic prefixes
AR1
TransitISP1
45
BR
AR2
7/26/2019 01 ISP Network Design
46/110
Distant Transit Provider
! Positioning a router close to the TransitProviders infrastructure is stronglyencouraged:
"
Long haul circuits are expensive, so the router
allows the ISP to implement appropriatefiltering first
" Moves the buffering problem away from the
Transit provider
" Remote co-lo allows the ISP to choose anothertransit provider and migrate connections withminimum downtime
46
7/26/2019 01 ISP Network Design
47/110
Distant Transit Provider
! Other points to consider:
"
Does require remote hands support
" (Remote hands would plug or unplug cables,power cycle equipment, replace equipment, etc
as instructed)" Appropriate support contract from equipment
vendor(s)
" Sensible to consider two routers and two long-
haul links for redundancy
47
7/26/2019 01 ISP Network Design
48/110
Distant Transit Provider
! Upgrade scenario:" Provision two routers
"
Two independent circuits" Consider second transit provider and/or turning up at
an IXP
AR1
TransitISP1
48
BR1
AR2BR2
7/26/2019 01 ISP Network Design
49/110
Summary
! Design considerations for:
"
Private interconnects
! Simple private peering
" Public interconnects
! Router co-lo at an IXP
" Local transit provider
! Simple upstream interconnect
" Long distance transit provider
! Router remote co-lo at datacentre or Transit
premises
49
7/26/2019 01 ISP Network Design
50/110
Upstream Connectivityand Peering Case Study
How Seacom chose theirinternational peering locations
and transit providers
50
7/26/2019 01 ISP Network Design
51/110
Objective
! Obtain high grade Internet connectivity forthe wholesale market in Africa to the restof the world
! Emphasis on:
" Reliability
" Interconnectivity density
"
Scalability
51
7/26/2019 01 ISP Network Design
52/110
Metrics Needed in DeterminingSolution (1)
! Focusing on operators that cover the destinationsmostly required by Africa
" i.e., English-speaking (Europe, North America)
!
Include providers with good connectivity into
South America and the Asia Pacific.! Little need for providers who are strong in the
Middle East, as demand from Africa for thoseregions is very, very low.
52
7/26/2019 01 ISP Network Design
53/110
Metrics Needed in DeterminingSolution (2)
! Split the operators between Marseille (where theSEACOM cable lands) and London (where there isgood Internet density)
" To avoid outages due to backhaul failure across Europe
" And still maintain good access to the Internet
! Look at providers who are of similar size so asnot to fidget too much (or at all) with BGP tuning.
!
The providers needed to support:
" 10Gbps ports
"
Bursting bandwidth/billing" Future support for 100Gbps or N x 10Gbps
53
7/26/2019 01 ISP Network Design
54/110
Metrics Needed in DeterminingSolution (3)
! Implement peering at major exchange points inEurope
" To off-set long term operating costs re: upstreamproviders.
54
7/26/2019 01 ISP Network Design
55/110
Implementing Solution
! Connected to Level(3) and GT-T (formerlyInteliquent, formerly Tinet) in Marseille
! Connected to NTT and TeliaSonera in London
!
Peered in London (LINX)
! Peered in Amsterdam (AMS-IX)
! BGP setup to prefer traffic being exchanged atLINX and AMS-IX
!
BGP setup to prefer traffic over the upstreamsthat we could not peer away
!
No additional tuning done on either peered ortransit traffic, i.e., no prepending, no de-aggregation, etc. All traffic setup to flow naturally55
7/26/2019 01 ISP Network Design
56/110
End Result
! 50% of traffic peered away in less than 2xmonths of peering at LINX and AMS-IX
! 50% of traffic handled by upstream providers
!
Equal traffic being handled by Level(3) and GT-T
in Marseille! Equal traffic being handled by TeliaSonera and
NTT in London
!
Traffic distribution ratios across all the transitproviders is some 1:1:0.9:0.9
! This has been steady state for the last 12xmonths
" No BGP tuning has been done at all56
7/26/2019 01 ISP Network Design
57/110
Addressing
57
7/26/2019 01 ISP Network Design
58/110
Where to get IP addresses and ASnumbers
! Your upstream ISP
! Africa
" AfriNIC http://www.afrinic.net
!
Asia and the Pacific
" APNIC http://www.apnic.net
! North America
" ARIN http://www.arin.net
!
Latin America and the Caribbean
" LACNIC http://www.lacnic.net
! Europe and Middle East" RIPE NCC http://www.ripe.net/info/ncc
58
7/26/2019 01 ISP Network Design
59/110
Internet Registry Regions
59
7/26/2019 01 ISP Network Design
60/110
Getting IP address space
! Take part of upstream ISPs PA spaceor
!
Become a member of your Regional InternetRegistry and get your own allocation
" Require a plan for a year ahead" General policies are outlined in RFC2050, more
specific details are on the individual RIR website
! There is no more IPv4 address space at IANA
" APNIC and RIPE NCC are now in their final /8IPv4
delegation policy phase" Limited IPv4 available
" IPv6 allocations are simple to get in most RIR regions60
7/26/2019 01 ISP Network Design
61/110
What about RFC1918 addressing?
! RFC1918 defines IPv4 addresses reserved forprivate Internets
" Not to be used on Internet backbones
" http://www.ietf.org/rfc/rfc1918.txt
! Commonly used within end-user networks" NAT used to translate from private internal to public
external addressing
" Allows the end-user network to migrate ISPs without amajor internal renumbering exercise
! ISPs must filter RFC1918 addressing at theirnetwork edge
" http://www.cymru.com/Documents/bogon-list.html 61
7/26/2019 01 ISP Network Design
62/110
What about RFC1918 addressing?
! There is a long list of well known problems:" http://www.rfc-editor.org/rfc/rfc6752.txt
! Including:
" False belief it conserves address space
" Adverse effects on Traceroute
" Effects on Path MTU Discovery
" Unexpected interactions with some NAT implementations
" Interactions with edge anti-spoofing techniques
" Peering using loopbacks
" Adverse DNS Interaction
" Serious Operational and Troubleshooting issues
" Security Issues
! false sense of security, defeating existing securitytechniques
62
b b
7/26/2019 01 ISP Network Design
63/110
Private versus Globally Routable IPAddressing
! Infrastructure Security: not improved by usingprivate addressing
" Still can be attacked from inside, or from customers, orby reflection techniques from the outside
! Troubleshooting: made an order of magnitudeharder
" No Internet view from routers
" Other ISPs cannot distinguish between down and broken
! Summary:
" ALWAYS use globally routable IP addressing for ISPInfrastructure
63
Add i l S
7/26/2019 01 ISP Network Design
64/110
Addressing Plans ISPInfrastructure
! Address block for router loop-back interfaces
! Address block for infrastructure
" Per PoP or whole backbone
" Summarise between sites if it makes sense
" Allocate according to genuine requirements, not historicclassful boundaries
!
Similar allocation policies should be used for IPv6as well
" ISPs just get a substantially larger block (relatively) soassignments within the backbone are easier to make
64
7/26/2019 01 ISP Network Design
65/110
Addressing Plans Customer
! Customers are assigned address spaceaccording to need
! Should not be reserved or assigned on aper PoP basis
" ISP iBGP carries customer nets
" Aggregation not required and usually not
desirable
65
7/26/2019 01 ISP Network Design
66/110
! Phase Two
Addressing Plans ISP Infrastructure
! Phase One
66
223.10.0.0/21
Customer assignments Infrastructure Loopbacks
/24223.10.6.255223.10.0.1
223.10.0.0/20
Original assignments New Assignments
/24/24223.10.0.1
223.10.5.255 223.10.15.255
7/26/2019 01 ISP Network Design
67/110
Addressing PlansPlanning
! Registries will usually allocate the nextblock to be contiguous with the firstallocation
"
Minimum allocation could be /21
" Very likely that subsequent allocation willmake this up to a /20
" So plan accordingly
67
7/26/2019 01 ISP Network Design
68/110
Addressing Plans (contd)
! Document infrastructure allocation
"
Eases operation, debugging and management
! Document customer allocation
" Contained in iBGP
" Eases operation, debugging and management
" Submit network object to RIR Database
68
7/26/2019 01 ISP Network Design
69/110
Routing Protocols
69
7/26/2019 01 ISP Network Design
70/110
Routing Protocols
! IGP Interior Gateway Protocol
"
Carries infrastructure addresses, point-to-pointlinks
" Examples are OSPF, ISIS,...
! EGP Exterior Gateway Protocol" Carries customer prefixes and Internet routes
" Current EGP is BGP version 4
! No connection between IGP and EGP
70
7/26/2019 01 ISP Network Design
71/110
Why Do We Need an IGP?
! ISP backbone scaling
"
Hierarchy
" Modular infrastructure construction
" Limiting scope of failure
" Healing of infrastructure faults using dynamicrouting with fast convergence
71
7/26/2019 01 ISP Network Design
72/110
Why Do We Need an EGP?
! Scaling to large network
"
Hierarchy
" Limit scope of failure
! Policy
" Control reachability to prefixes
" Merge separate organizations
" Connect multiple IGPs
72
I t i E t i R ti
7/26/2019 01 ISP Network Design
73/110
Interior versus Exterior RoutingProtocols
! Interior" Automatic neighbour
discovery
" Generally trust your IGProuters
" Prefixes go to all IGProuters
" Binds routers in one AStogether
! Exterior" Specifically configured
peers
" Connecting with outsidenetworks
" Set administrativeboundaries
" Binds ASs together
73
I t i E t i R ti
7/26/2019 01 ISP Network Design
74/110
Interior versus Exterior RoutingProtocols
! Interior" Carries ISP
infrastructure addressesonly
" ISPs aim to keep the
IGP small for efficiencyand scalability
! Exterior" Carries customer
prefixes
" Carries Internetprefixes
" EGPs are independentof ISP network topology
74
7/26/2019 01 ISP Network Design
75/110
Hierarchy of Routing Protocols
75
BGP4
BGP4and OSPF/ISIS
Other ISPs
CustomersIXP
Static/BGP4
BGP4
R ti P t l
7/26/2019 01 ISP Network Design
76/110
Routing Protocols:Choosing an IGP
! OSPF and ISIS have very similar properties" Review the ISIS vs OSPF presentation
! Which to choose?
" Choose which is appropriate for your operatorsexperience
" In most vendor releases, both OSPF and ISIS havesufficient nerd knobsto tweak the IGPs behaviour
" OSPF runs on IP
" ISIS runs on infrastructure, alongside IP
" ISIS supports both IPv4 and IPv6
" OSPFv2 (IPv4) plus OSPFv3 (IPv6)
76
R ti Pr t l
7/26/2019 01 ISP Network Design
77/110
Routing Protocols:IGP Recommendations
! Keep the IGP routing table as small as possible" If you can count the routers and the point-to-point links
in the backbone, that total is the number of IGP entriesyou should see
! IGP details:
" Should only have router loopbacks, backbone WANpoint-to-point link addresses, and network addresses ofany LANs having an IGP running on them
" Strongly recommended to use inter-routerauthentication
" Use inter-area summarisation if possible
77
Ro ting Protocols
7/26/2019 01 ISP Network Design
78/110
Routing Protocols:More IGP recommendations
! To fine tune IGP table size more, consider:
"
Using ip unnumberedon customer point-to-point links saves carrying that /30 in IGP! (If customer point-to-point /30 is required for
monitoring purposes, then put this in iBGP)
" Use contiguous addresses for backbone WAN
links in each area then summarise intobackbone area
" Dont summarise router loopback addresses
as iBGP needs those (for next-hop)"
Use iBGP for carrying anything which does notcontribute to the IGP Routing process
78
Routing Protocols:
7/26/2019 01 ISP Network Design
79/110
Routing Protocols:iBGP Recommendations
! iBGP should carry everything whichdoesnt contribute to the IGP routingprocess
"
Internet routing table
" Customer assigned addresses" Customer point-to-point links
"
Access network dynamic address pools,passive LANs, etc
79
Routing Protocols:
7/26/2019 01 ISP Network Design
80/110
Routing Protocols:More iBGP Recommendations
! Scalable iBGP features:
"
Use neighbour authentication
" Use peer-groups to speed update process andfor configuration efficiency
" Use communities for ease of filtering" Use route-reflector hierarchy
! Route reflector pair per PoP (overlaid clusters)
80
7/26/2019 01 ISP Network Design
81/110
Security
81
7/26/2019 01 ISP Network Design
82/110
Security
! ISP Infrastructure security
! ISP Network security
! Security is not optional!
! ISPs need to:
" Protect themselves" Help protect their customers from the Internet
" Protect the Internet from their customers
!
The following slides are general recommendations
" Do more research on security before deploying any
network
82
7/26/2019 01 ISP Network Design
83/110
ISP Infrastructure Security
! Router & Switch Security
"
Use Secure Shell (SSH) for device access &management! Do NOT use Telnet
" Device management access filters should onlyallow NOC and device-to-device access
! Do NOT allow external access
" Use TACACS+ for user authentication and
authorisation
! Do NOT create user accounts on routers/switches
83
7/26/2019 01 ISP Network Design
84/110
ISP Infrastructure Security
! Remote access
"
For Operations Engineers who need accesswhile not in the NOC
" Create an SSH server host (this is all it does)
! Or a Secure VPN access server
" Ops Engineers connect here, and then they canaccess the NOC and network devices
84
7/26/2019 01 ISP Network Design
85/110
ISP Infrastructure Security
! Other network devices?" These probably do not have sophisticated security
techniques like routers or switches do
" Protect them at the LAN or point-to-point ingress (onrouter)
! Servers and Services?" Protect servers on the LAN interface on the router
" Consider using iptables &c on the servers too
! SNMP
" Apply access-list to the SNMP ports
" Should only be accessible by management system, notthe world
85
7/26/2019 01 ISP Network Design
86/110
ISP Infrastructure Security
! General Advice:
"
Routers, Switches and other network devicesshould not be contactable from outside the AS
" Achieved by blocking typical management
access protocols for the infrastructure addressblock at the network perimeter
! E.g. ssh, telnet, http, snmp,
" Use the ICSI Netalyser to check access levels:
! http://netalyzr.icsi.berkeley.edu
" Dont block everything: BGP, tracerouteand ICMP still need to work!
86
7/26/2019 01 ISP Network Design
87/110
ISP Network Security
! Effective filtering
"
Protect network borders from traffic whichshould not be on the public Internet, forexample:
! LAN protocols (eg netbios)
! Well known exploit ports (used by worms andviruses)
! Drop traffic arriving and going to private and non-routable address space (IPv4 and IPv6)
"
Achieved by packet filters on border routers! Remote trigger blackhole filtering
87
7/26/2019 01 ISP Network Design
88/110
ISP Network Security RTBF
! Remote trigger blackhole filtering" ISP NOC injects prefixes which should not be accessible
across the AS into the iBGP
" Prefixes have next hop pointing to a blackhole address
" All iBGP speaking backbone routers configured to point
the blackhole address to the null interface" Traffic destined to these blackhole prefixes are dropped
by the first router they reach
! Application:
" Any prefixes (including RFC1918) which should not have
routability across the ISP backbone
88
7/26/2019 01 ISP Network Design
89/110
ISP Network Security RTBF
! Remote trigger blackhole filtering example:" Origin router:
" iBGP speaking backbone router:
89
router bgp 64509
redistribute static route-map black-hole-trigger
!
ip route 10.5.1.3 255.255.255.255 Null0 tag 66
!route-map black-hole-trigger permit 10
match tag 66
set local-preference 1000
set community no-export
set ip next-hop 192.0.2.1
!
ip route 192.0.2.1 255.255.255.255 null0
7/26/2019 01 ISP Network Design
90/110
ISP Network Security RTBF
! Resulting routing table entries:
90
gw1#sh ip bgp 10.5.1.3
BGP routing table entry for 10.5.1.3/32, version 64572219
Paths: (1 available, best #1, table Default-IP-Routing-Table)
Not advertised to any peer
Local
192.0.2.1 from 1.1.10.10 (1.1.10.10)
Origin IGP, metric 0, localpref 1000, valid, internal, best
Community: no-export
gw1#sh ip route 10.5.1.3
Routing entry for 10.5.1.3/32
Known via "bgp 64509", distance 200, metric 0, type internal
Last update from 192.0.2.1 00:04:52 ago
Routing Descriptor Blocks:
* 192.0.2.1, from 1.1.10.10, 00:04:52 agoRoute metric is 0, traffic share count is 1
AS Hops 0
7/26/2019 01 ISP Network Design
91/110
ISP Network Security uRPF
! Unicast Reverse Path Forwarding
! Strongly recommended to be used on all
customer facing staticinterfaces
" BCP 38 (tools.ietf.org/html/bcp38)
" Blocks all unroutable source addresses the
customer may be using
" Inexpensive way of filtering customers connection(when compared with packet filters)
! Can be used for multihomed connections too, butextreme care required
91
7/26/2019 01 ISP Network Design
92/110
What is uRPF?
! Router compares source address of incomingpacket with FIB entry
"
If FIB entry interface matches incoming interface, thepacket is forwarded
" If FIB entry interface does not match incoming interface,the packet is dropped 92
router
FIB:172.16.1.0/24 fa0/0
192.168.1.0/24 se0/1
fa0/0 se0/1src=172.16.1.1
src=192.168.1.1
7/26/2019 01 ISP Network Design
93/110
Security Summary
! Implement RTBF" Inside ISP backbone
" Make it available to BGP customers too
! They can send you the prefix you need to block with aspecial community attached
!
You match on that community, and set the next-hop to thenull address
! Implement uRPF
" For all static customers
! Use SSH for device access
! Use TACACS+ for authentication
93
7/26/2019 01 ISP Network Design
94/110
Out of Band Management
94
7/26/2019 01 ISP Network Design
95/110
Out of Band Management
! Not optional!
! Allows access to network equipment intimes of failure
! Ensures quality of service to customers
" Minimises downtime
" Minimises repair time
" Eases diagnostics and debugging
95
7/26/2019 01 ISP Network Design
96/110
Out of Band Management
! OoB Example Access server:"
modem attached to allow NOC dial in
" console ports of all network equipmentconnected to serial ports
" LAN and/or WAN link connects to networkcore, or via separate management link to NOC
! Full remote control access under all
circumstances
96
7/26/2019 01 ISP Network Design
97/110
Out of Band Network
97Ethernet
to the NOC
Router, switch
and ISP serverconsoles
(Optional) Out of band
WAN link to other PoPs
Modem access
to PSTN for out of
band dialin
Equipment RackEquipment Rack
7/26/2019 01 ISP Network Design
98/110
Out of Band Management
! OoB Example Statistics gathering:"
Routers are NetFlow and syslog enabled
" Management data is congestion/failuresensitive
" Ensures management data integrity in case offailure
! Full remote information under all
circumstances
98
7/26/2019 01 ISP Network Design
99/110
Test Laboratory
99
7/26/2019 01 ISP Network Design
100/110
Test Laboratory
! Designed to look like a typical PoP"
Operated like a typical PoP
! Used to trial new services or new softwareunder realistic conditions
! Allows discovery and fixing of potentialproblems before they are introduced tothe network
100
7/26/2019 01 ISP Network Design
101/110
Test Laboratory
! Some ISPs dedicate equipment to the lab
! Other ISPs purchase aheadso thattodays lab equipment becomestomorrows PoP equipment
! Other ISPs use lab equipment for hotsparesin the event of hardware failure
101
7/26/2019 01 ISP Network Design
102/110
Test Laboratory
! Cant afford a test lab?"
Set aside one spare router and server to trialnew services
" Never ever try out new hardware, software or
services on the live network! Every major ISP in the US and Europe has
a test lab
" Its a serious consideration
102
7/26/2019 01 ISP Network Design
103/110
OperationalConsiderations
103
7/26/2019 01 ISP Network Design
104/110
Operational Considerations
104
Why design the world
s bestnetwork when you have notthought about what operational
good practices should be
implemented?
Operational Considerations
7/26/2019 01 ISP Network Design
105/110
pMaintenance
! Never work on the live network, no matter howtrivial the modification may seem
" Establish maintenance periods which your customers areaware of
! e.g. Tuesday 4-7am, Thursday 4-7am
! Never do maintenance on the last working daybefore the weekend
" Unless you want to work all weekend cleaning up
! Never do maintenance on the first working day
after the weekend
" Unless you want to work all weekend preparing
105
Operational Considerations
7/26/2019 01 ISP Network Design
106/110
pSupport
! Differentiate between customer supportand the Network Operations Centre
" Customer support fixes customer problems
" NOC deals with and fixes backbone and
Internet related problems! Network Engineering team is last resort
" They design the next generation network,
improve the routing design, implement newservices, etc
"
They do not and should not be doing support!
106
Operational Considerations
7/26/2019 01 ISP Network Design
107/110
pNOC Communications
! NOC should know contact details forequivalent NOCs in upstream providersand peers
! Or consider joining the INOC-DBA system
" Voice over IP phone system using SIP" Runs over the Internet
"
www.pch.net/inoc-dbafor more information
107
7/26/2019 01 ISP Network Design
108/110
ISP Network Design
Summary
108
7/26/2019 01 ISP Network Design
109/110
ISP Design Summary
! KEEP IT SIMPLE & STUPID ! (KISS)
! Simple is elegant is scalable
! Use Redundancy, Security, and
Technology to make life easier for yourself
! Above all, ensure quality of service for
your customers
109
7/26/2019 01 ISP Network Design
110/110
ISP Network Design
ISP Workshops