BGP Peering Strategy & Data - 2017. · PDF file28/02/2017 BGP Traffic Engineering, Andy Davidson 3 Complexity Life starts out very simply, “send traffic to peers if possible, then

BGPPeeringStrategy&Data

[email protected]

CTO@AsteroidInternationalAPRICOT2017,HCMC,VietnamDirector@LONAP,IXLeeds,Euro-IX March2017

BGPTrafficEngineering– version2.0(2016)

28/02/2017 BGPTrafficEngineering,AndyDavidson 2

WhydoTrafficEngineering?

Manageyourcapacitydemands

Ensureservicequality

RecoverfromFailures

Manageservice/circuitcosts

HandletrafficgrowthJamesCridland http://www.flickr.com/photos/jamescridland/


Complexity

Lifestartsoutverysimply,“sendtraffictopeersifpossible,thentransitproviders”

Butwhataboutwhenyournetworkgrows?

Whataboutwhenyourtrafficgrows?

Whatifyouaddmorecities/POPs/exchanges?Caisey Hussain Bisson -http://www.flickr.com/people/maisonbisson/

Realexamples

• Circuitswithcostdifference>$100/Mbit• Regionalnetworks- poorlocalpeering• Circuitfailurecausingcongestion• Changingcustomerdemand/behaviour– Increasedqualityexpectation– Newhighbandwidthservicessuchasvideo


InternalnetworkTE

• Simple comparedwithInterdomain TE

• Youadministratebothsides– Youknowtheprice ofallpaths– TheIGPknowsthecapacity ofallpaths– IGPprotocolsletyoumapprice,capacitytoshaperoutingusingcost.


Inter-domainTE

• YoudoNOTcontrolbothsides– Pathvectorprotocolshidemetric,capacity,cost– SimplicityofBGPprotocolimposeslimitations– Volumeoftrafficmatters,not#ofroutes

• However,largevolumeoftrafficisusuallywithasmallnumberofotherASNs


Youneeddata


AS-StatsManuelKasperhttps://neon1.net/as-stats/

..Butmoreonthislater

Netflow

• Export informationaboutpacketsroutedthroughyournetwork

• Normallysampled• SenttoacollectoroverUDP• Avarietyofcommercialandopen-sourcetoolssortanddisplaytheseflowrecords.


DifferentFlowprotocols

• Netflow – DesignedbyCiscoin‘90s,publishedasastandard(v9isRFC3954andsupportsIPv6)

• IPFIX(RFC5101)BasedonNetflow 9,2008• sFlow – NiceprotocolbutincompatiblewithNetflow,typicallyimplementedonL2switch.

• Jflow – EssentiallyNetflow onJunipers


Otherwaystogetdata

• Logfileanalysis– Usefulbeforeyouhaveanetwork,forworkingoutthebenefitofbuildinganetwork/peering.

– Bestfor‘singleservice’networks• DNSprovidershaveDNSlogswithtime&IP• Webprovidershaveweblogswithtime&IP• Hostedemailprovidershavemaillogs…



IPAddress Timeanddate AmountofTraffic

Datatellsyou

• Yourtrafficdirection–Mainlyinbound–Mainlyoutbound– Balanced

• Yourtoptrafficoriginatorsordestinations


As-stats

• https://neon1.net/as-stats/

• Opensource• Quicktosetup,simpletouse• Resourceintensive



Whoaremykeypeers?

(orpotentialpeers)

Toporiginatorsoftraffictome,topconsumersofmycontent

Chartcolour relatestoaninterfaceontheedgeofmynetwork


Historicaldata

Newpeeringadded,trafficgrowth!=$$$$$formynetwork!

Bonusplease!


Footballstarts

Footballends

Videocdn

SocialNetwork

Inmon sflow-rt

• http://www.sflow-rt.com/download.php

• Producesrealtime trafficgraphs• “Whatisthesituationrightnow”• Usefultocheckpeeringconfig hastakeneffect• Alsousefulinabusemitigation


Realtime aggregatedata


Realtime transitanalysis


ExportfromSflow-RT

• RESTful exportintologstash/influxdb/grafanaforhistoricaldata

• http://www.slideshare.net/pphaal/network-visibility-and-control-using-industry-standard-sflow-telemetry


pmacctd

• http://www.pmacct.net

• OpenSource• Highperformance,highscale,powerful• Mostflexible,mostconfigurationrequired

• Collectorà ownreports


1)Configureacollectorsfacctd_port: 2100sfacctd_as: sflowsfacctd_renormalize: true!plugins: print[testprint]!aggregate[testprint]: in_iface, out_iface, proto, peer_src_ip, peer_dst_ip, peer_dst_as, peer_src_as, src_as, dst_as!print_output_file[testprint]: /path/to/spool/blabla-$peer_src_ip-%Y%m%d-%H%M.txtprint_output[testprint]: csvprint_output_separator[testprint]: ;print_refresh_time[testprint]: 60print_history[testprint]: 1mprint_history_roundoff[testprint]: m


UsetheASNdatafromtherouterifitexists,noneedtosetupBGPflowexport

2)GetareportSRC_AS;DST_AS;PEER_SRC_AS;PEER_DST_AS;PEER_SRC_IP;PEER_DST_IP;IN_IFACE;OUT_IFACE;PROTOCOL;PACKETS;BYTES41230;224;41230;2603;x.x.x.253;x.x.x.246;3;4;tcp;2048;15155241230;15169;41230;15169;x.x.x.253;x.x.x.246;3;4;tcp;10240;88064041230;50247;41230;24724;x.x.x.253;x.x.x.246;3;4;tcp;2048;16793641230;9269;41230;1273;x.x.x.253;x.x.x.237;3;3;tcp;2048;13516841230;3356;41230;1273;x.x.x.253;x.x.x.237;3;3;tcp;32768;237568041230;209;41230;1273;x.x.x.253;x.x.x.237;3;3;udp;2048;294092820940;0;20940;0;x.x.x.253;x.x.x.246;3;4;tcp;43008;65458176


RedlinerepresentsasingleflowwithGoogleonbehalfofauser

Keepinghistoricaldataplugins: mysql[5mins], mysql[hourly]

sql_optimize_clauses: truesql_dont_try_update: truesql_multi_values: 1024000

sql_history_roundoff[5mins]: msql_history[5mins]: 5msql_refresh_time[5mins]: 300sql_table[5mins]: acct_bgp_5mins

sql_history_roundoff[hourly]: hsql_history[hourly]: 1hsql_refresh_time[hourly]: 3600sql_table[hourly]: acct_bgp_1hr

plugin_buffer_size: 10240plugin_pipe_size: 1024000aggregate: tag, src_as, dst_as, peer_src_as, peer_dst_as, peer_src_ip,

peer_dst_ip, local_pref, as_path



Examplebypmacct authorPaoloLucente

• Exportintoreports,webinterface,spreadsheet• Multiplebackendssupported,includingtimeseriesdatabases• Veryflexibleapproachbutneedsmoresetuptime

Kentik

• https://www.kentik.com

• Hostedsolution• Zeroconfiguration,zeroequipmentneeded• PointNetflow attheircollectorandreportsfollow


TrafficbySourceASN


Breakdownbyregion(US)


Topflowsperinterface


Outboundvs Inbound• Outboundheavynetworks– Somewhateasierlife

• Inboundheavynetworks– Youmusttrick theBestPathSelectionmethodsofnetworks sendingyoutraffic.

– Their config changewillmoveyour traffic.


OutboundHeavyTrafficEngineering

• BGPBestPathSelectionAlgorithm• Trafficengineeringisabout‘tricking’thisprocess• Affectstrafficinoutbounddirection– LocalPreference– ASPATHlength– LowestOriginType– LowestMED– PrefereBGP paths– LowestIGPMetric– Oldestroute


Mainlyoutbound,singlePOP

• Localpref– Ahammer – blunttool,inflexible..Butitisatool.– “Generally”prefertosendtraffictocustomers,thenpeers,thentransits.

–Managetop‘n’networks,sothatthereisapreferredpath,andafailurepath,withcapacityonbothcircuits.


Localpref – blunthammer


10.0.0.0/8Localpref 100via10012310.0.0.0/8Localpref 500via300200200200200123

Whichlinkwillyouprefer?AS123hereistryingtoshapeinboundtrafficviaAS100.Why?

Highercapacitylink?Morereliable?

Whatshouldyoudo?

Answer:Itdependsonthevolume oftraffic,cost ofcapacity,value oftraffic

Mainlyoutbound– ManyPOPs

• Usehotpotatoroutingtobesteffect– Nearestexitrouting– UnderstandwhoyourtoptrafficsinksareandpeeratallPOPs

– IgnoreMEDsfromothers– unlessyouwanttocarrythetrafficonyourbackbone


AS_PATHprepending(outbound)• MakeBGPprefixpaths”appear”longervialesspreferred

circuits

• “BGPpathfor10.0.0.0/8654_789iscongested”

• 123456789• 654789

• 100100100654789

• Willnotvaryinboundpacketroute(thoughthisroutemightnotbecongested!)


Inboundtrafficengineering

• Muchharder– Trickothers’BestPathcalculations– Youdonotadministrateoriginpartyrouter

• Butremember…– Largestflowscomefromasmallnumberofnetworks– Contentnetworkswanttodelivertraffictoyouaswellaspossible!


SelectiveAnnouncements


– Shortestprefix– LocalPreference– ASPATHlength– LowestOriginType– LowestMED– PrefereBGP paths– LowestIGPMetric– Oldestroute

PrefixlengthconsideredbeforeBGP.

10.0.0.0/16vs 10.0.0.0/17&10.128.0.0/17

ProblemofSelectiveAnnouncements

• Oftenfiltered• Consideredrude – mightleadtodepeering• Neverannounce‘globally’


…Butcanbeusedtogreateffect

• Tothesamepeerortransitprovider,announceaggregateandregionalpfx


Berlin Frankfurt

Transit

10.0.0.0/810.0.0.0/9 10.0.0.0/8

10.128.0.0/9

AddNO_EXPORTcommunity

Usewithpermission

AS_PATHprepending(inbound)

• SignalpreferredpathbygrowingAS_PATHonlesspreferredpaths

• Marginaleffectwhichdegradesquickly• SignalbackuplinktoasingleAS,butload-balancingcapacityismuchharder

• Maynotbeheardat‘distant’ASNs• Another‘blunt’tool,butcanmovesometraffic.



MEDs

• Lowest MEDwins.– OppositeofNearestExitrouting,“carrytraffictome”– Onlyworkstothesamepeerinmultipleregions– CopyIGPmetrictoMED– Normallysubjecttonegotiation

• Sometimeshonoured,oftenwhennetworktrafficislatencyorlosssensitive.


MEDsareoftenfiltered

• ManynetworkssetMEDto0whentheylearnprefixes,sothathotpotatoroutingwilloverride MED.

route-map peers-in permit 10set local-preference 200set metric 0


Originchanging

• IGP• EGP• Incomplete


route-map PEERS permit 10set origin igp

route-route-map TRANSIT permit 10set origin incomplete

Oftenpeerssetto‘igp’or‘egp’staticallyonroutersto nullify effectsofOriginchanging.

Highestpriority

Inbound– whatdoesworkwell?• Overprovisioning• Peerwithtopnetworkswidely (buyoptions!)– Failureofsinglelinkwillnotbreakadjacency– Failurescanbehandledinpredictableways

• Buildrelationships• Constantlymonitorandmanage• Ifyoucareaboutyourtraffic,letitgo.J

– Playinggameswithpeeringhurtsyourcustomers’traffic

• AffectingdistantASNsisveryhard– aregionmayonlyseeasinglenext-hopASN.


Deterministicrouting


– LocalPreference– ASPATHlength– LowestOriginType– LowestMED– PrefereBGP paths– LowestIGPMetric– Oldestroute

Topflowsshouldleaveyournetworkviadeterministicmeans,andnotlefttoBGPBestPathselection(ortochance).

Ifyouarerelyingonoldestroutetomakethedecision,yourisktraffictakingunpredictableroutes.

However,oldestroutesdobreakthe‘flappingsessions’problem.Youneedtomonitorandmanageyourtopflowsconstantly.

Whatdoes“managerelationships”mean?

• Gobacktoyourdata– Collectandshareinformationwithpeeringco-ordinatorsatforumslikethis

– Youwillstandoutifyouknowexactlyhowmuchtrafficyouwillexchangeatpeakwithapeer

– Protectyourpeer’sinterests• Discussmutualpointsofinterconnectionthatsuitboth• Respondtoabusecomplaintspromptly• Usecontactstoreachotherpeeringco-ordinators• RespondpromptlytoBGPsessiondown/flapping• ListyournetworkonPeeringDB!


PublishinIRRDB

• PublishyourcircuitandadjacencyinformationinIRRDB

• Makeitpossibleforothernetworkstofigureouthowyouareintending yourtraffictoberouted



RT1 RT2

EX2EX1

PP1

PP2 PP3

Transit

6Gbit

5Gbit

2Gbit

4Gbit 4Gbit

AS2isyourlargestflow- viaPP2- maybeneedsasecondprivatepeerbackuponRT2?

4Gbit 4Gbit

AS12345AS2

AS1

Manypeers Manypeers

AS3


RT1 RT2

EX2EX1

PP1

PP2 PP3

Transit

6Gbit

5Gbit

2Gbit

4Gbit 4Gbit

AS2isyourlargestflow- viaPP2- maybeneedsasecondprivatepeerbackuponRT2?AS1viaPP1,configureabackupoverEX1orEX2fordeterministicrouting?

4Gbit 4Gbit

AS12345AS2

AS1

Manypeers Manypeers

AS3


RT1 RT2

EX2EX1

PP1

PP2 PP3

Transit

6Gbit

5Gbit

2Gbit

4Gbit 4Gbit


CanyoumovelargerpeersbehindEX1andEX2ontoprivatepeering?

4Gbit 4Gbit

AS12345AS2

AS1

Manypeers Manypeers

AS3


RT1 RT2

EX2EX1

PP1

PP2 PP3

Transit

6Gbit

5Gbit

2Gbit

4Gbit 4Gbit


CanyoumovelargerpeersbehindEX1andEX2ontoprivatepeering?Ifthereisanexchangefailure,wherewillthetrafficgo?Howbigaflowshouldyoucareabout?

4Gbit 4Gbit

AS12345AS2

AS1

Manypeers Manypeers

AS3


RT1 RT2

EX2EX1

PP1

PP2 PP3

Transit

6Gbit

5Gbit

2Gbit

4Gbit 4Gbit


CanyoumovelargerpeersbehindEX1andEX2ontoprivatepeering?Ifthereisanexchangefailure,wherewillthetrafficgo?Howbigaflowshouldyoucareabout?

IfyouloseRT2,howwilltraffictoPP3andtrafficvolumeviaEX2bedelivered?

4Gbit 4Gbit

AS12345AS2

AS1

Manypeers Manypeers

AS3

Buyingtransitinasmartway

• Buyingfromawellpeeredtransitprovider:– Canimprovequalityforthereasonsdiscussed– Hidescapacityproblemsfromyouautomatically

• Buyingfromyourtoptrafficdestination– Ifyourbusinessreliesonthetrafficquality,itmaymakesensetopay

– Data mayhelpyounegotiategoodterms


Dealingwitha“no”topeering

• Paidpeeringisoneoption– OftenmoreexpensivethanfullIPtransit– “Onceacustomer,neverapeer”

• Payforotherservicesinreturnforpeering– Transportforexample

• Peeraroundtheproblem– Trytopeerdirectlywithdownstreamcustomers– Trytoselldirectlytodownstreamcustomers• Ifyouarebetterpeered,youcansellbasedonquality


Constantlymanage

• PeeringontheInternetchanges everyday.• CapacityontheInternetgrows everyday.• Smallnetworksbecomelarge.• Largenetworksbecomelarger(consolidation)• A“bad”pathmightbecomegoodovernight


Questions?______________________________

[email protected]

Emailmetorequestacopyofthispresentation!

Feedbackandintroductiontopeeringco-ordinators welcome

Twitter:@andyd

Documents

BGP Peering Strategy & Data - 2017. · PDF file28/02/2017 BGP Traffic Engineering, Andy Davidson 3 Complexity Life starts out very simply, “send traffic to peers if possible, then