Virtualized PE for BGP/MPLS L3-VPN using Open …...Virtualized PE for BGP/MPLS L3-VPN using Open-Source Software NANOG 74 (October 2018) Bilal Anwer, Robert Bays, Vijay Gopalakrishnan,

VirtualizedPEforBGP/MPLSL3-VPNusingOpen-SourceSoftwareNANOG74(October2018)

BilalAnwer,RobertBays,VijayGopalakrishnan,BoHan,DewiMorgan,PatrickRuddy,AmanShaikh,SusheelaVaidya,ChengweiWangandGeorgeWilkie

2

IntroductionObjective•  DemonstratefeasibilityofcreatingaBGP/MPLSL3-VPNvPEusingopen-sourcesoftware

Motivation•  Use-caseforAT&T’sDANOS(DisaggregatedNetworkOS)•  WhyL3-VPNvPEfromopen-sourcesoftware?–  L3-VPN•  Allowscreationofmultiplelayer-3virtualnetworksontopofasharedservice-providernetwork

•  Widelyusedservicebyenterprises

–  vPE•  EnablerVNFwhichactsastheingressandegressforL3-VPNtrafficintheservice-providernetwork

–  Opensourcesoftware•  Allowsincreasedagilityinprovidingnewfeatureswhilereducingthecost

Challenges•  Requiredfunctionalandintegration-relatedextensionstoopen-sourcecomponents

NANOG74:OpenSourcevPEforBGP/MPLSL3-VPN

3

SoftwareComponentsofOpenSourcevPE Control-plane•  FRR(5.1-dev,snapshote8f9540)forOSPF,LDPandZebra•  GoBGP(version1.31.1=version1.31+ourenhancements)

Data-plane•  AT&T-Vyatta’s(DPDK-based)data-plane

• Wealsoverifiedfeasibilitywith…–  Linuxdata-plane(kernel4.14.4-mpls)

–  VPPdata-plane(release1801+routerplug-inwithourenhancementswhichhavebeenup-streamed)


DANOSUse-Case

4

SoftwareArchitectureofDANOSOpenSourcevPE


Control-plane

FRROSPF

FRRLDPGoBGP

AT&T-VyattaDPDKbasedData-plane

FRRZebrazserv.api

vPE

LinuxForwardingTable

RIB

NIC

KernelFIB

NetLinkLinuxNetworkingStack

TAPInterface

LinuxUserSpace

LinuxKernelSpace

FIB

AT&T-VyattaRouteBrokerReliable&highscaleroutedownload

Existingclientsofnetlink

vCEred_east

Videoserver

172.16.0.2

RedEastOffice

5

VerifyingFeasibility

•  Demonstratedfeasibilitybyconcurrentlyrunningtwovideostreams–  KeepvideotrafficseparatedespitesameIPaddressesbeingusedbytwocustomers

–  Eachclient/serverhasastaticroutepointingtoitsupstreamCE

–  EachCEadvertisesappropriateprefixtothePENANOG74:OpenSourcevPEforBGP/MPLSL3-VPN

Corerouter+

VPNv4RR

vPEwest vPEeast

OSPFLDP

VPNv4BGP

OSPFLDP

VPNv4BGP

VRFred VRFred

VRFblue VRFblue

ServiceProvider

SameIPaddress SameIPaddress

vCEblue_west

Videoclient

172.16.0.1

BlueWestOffice

vCEred_west

Videoclient

172.16.0.1

RedWestOffice

Videoserver

172.16.0.2

BlueEastOfficevCE

blue_east

Videostream

Videostream

6

PacketCaptureatCoreRouterduringVideoStreaming


`CorerouterClients vCEs

vPEeast

vPEwest ServersvCEs

redserver→redclient

redclient→redserver

blueclient→blueserver

blueserver→blueclient

7

ConfigurationDetailsEstablishLSP(LabelSwitchedPaths)betweenPEs•  EnableIPandMPLSforwarding

•  ConfigureOSPFandLDPonserviceproviderrouters

EnableL3-VPNservice•  ConfigureVRFs•  ConfigureeBGPsessionsbetweenPEsandCEs•  ConfigureiBGPsessionsbetweenPEsandroutereflector


Note:WeusedAT&T/DANOSYangModulesforconfiguringvPEswherepossible,butshowequivalentLinux,FRRandGoBGPcommandsinsubsequentslides

8

TaleofTwoLoopbacksConfiguredtwoloopbackaddressesonvPEsandcorerouter•  Loopback1–  UsedforIPtrafficincludingcontrol-planetraffic•  Loopback2–  UsedforMPLStraffic•  HencealltrafficfromVPNcustomers


9

vPEEast:ConfiguringMPLSForwarding



vPEeast

vPEwest ServersvCEs

# Enable IPv4 forwarding $ sudo sysctl –w net.ipv4.ip_forward=1 # Load MPLS modules $ sudo modprobe mpls_router $ sudo modprobe mpls_iptunnel # Enable MPLS forwarding on the interface facing the core router $ sudo sysctl -w net.mpls.conf.ens4.input=1 # Allocate entries in MPLS label table $ sudo sysctl -w net.mpls.platform_labels=1048575

10

vPEEast:FRROSPFandLDPConfigurations



vPEeast

vPEwest ServersvCEs

OSPFConfiguration LDPConfiguration

11

vPEEast:VRFConfiguration



vPEeast

vPEwest ServersvCEs

# Create VRF blue and bring it up $ sudo ip link add blue type vrf table 1 $ sudo ip route add table 1 unreachable default metric 4278198272 $ sudo ip link set dev blue up # Add interface to vCE blue-east to VRF blue $ sudo ip link set dev ens6 master blue # Allow BGP to listen on port 179 over the VRF-bound interface $ sudo sysctl –w net.ipv4.tcp_l3mdev_accept=1 $ sudo sysctl –w net.ipv4.udp_l3mdev_accept=1

12

vCEBlueEast:FRRBGPConfiguration



vPEeast

vPEwest ServersvCEs

AllowsconfigurationofeBGPsessiontovPEEastloopback

13

vPEEast:GoBGPConfiguration



vPEeast

vPEwest ServersvCEs

Useofpolicyforsettingnext-hop-selfrequirespolicy

CommunicatetoLinuxkernelandVyattadata-planeviaFRRZebra

iBGPsessionwithRoute-Reflector

14

vPEEast:Defining“SetNext-HopSelf”policy



vPEeast

vPEwest ServersvCEs

Matchontheroutereflectorasneighbor

Setloopback2ofthisPEasthenext-hop

15

vPEEast:AddingVRFandeBGPNeighborviaGoBGPCLI



vPEeast

vPEwest ServersvCEs

CommandsforaddingblueVRFandeBGPsessiontovCEeast-blue

$ ip link show blue

11: blue: <NOARP,MASTER,UP,LOWER_UP> mtu 65536 qdisc noqueue state UP

mode DEFAULT group default qlen 1000

link/ether ae:07:ef:a3:f3:f7 brd ff:ff:ff:ff:ff:ff

$ gobgp add vrf blue id 11 rd 100:1 rt both 100:1

$ gobgp nei add 10.31.4.10 as 65101 vrf blue

UseofifIndexvalueassignedbyLinuxasVRFid

16

GoBGPConfigurationforRouterReflector



vPEeast

vPEwest ServersvCEs

NoneedtocommunicatewithFRRZebrasinceVPNv4routesarenotinstalledinforwardingtable

17

Implementation:L3-VPNSupportinGoBGPKeybuildingblocks•  InternetroutingwithBGP–  Messagehandling,routecomputation,andpolicies

•  PartitionofroutingtableintoglobalandVRF–  AssignBGPsessionstoappropriatepartition•  VPNv(4|6)BGPaddressfamily–  IPprefix,RouteDistinguisher(RD)andMPLSlabel

•  Routetargets(RTs)–  ToassociaterouteswithVRF(s)

Interactionwith“outsideworld”•  AllowconfigurationofVRF(s)–  AssociateaneBGPsessionwithCEtoaVRF•  HandlearoutereceivedfromaCE

•  HandlearoutereceivedfromRR(orremotePEs)

•  CommunicatewithZebraNANOG74:OpenSourcevPEforBGP/MPLSL3-VPN

Existingsupportwasadequate

Neededsomeenhancements

18

GoBGP:HandlingaRouteReceivedfromanAdjacentCE•  InstalltherouteinVRFforwardingtableviaZebra–  AllowsvPEtosendtrafficcomingfromotherPEstotheCE

•  AttachVRFlabeltotheroutebeforesendingtoRR


GoBGPpullrequest1587

vCEVRF VRF Global

Control-plane

FRRZebra

ZebraAPI

vPE

GoBGP

Data-plane

VRF VRF GlobalRIB

VRF VRF GlobalFIB

VRF VRF GlobalBGPtable

RR

IPv4BGProuteforprefixpp

Ourenhancement;pullrequest1572acceptedintoGoBGPupstreamrepository

VPNv4BGProuteforprefixpwithblueRDandbluelabel

p

p

RouteBroker

p

p

p

19

GoBGP:HandlingaRouteReceivedfromanRRoraPE•  PreventtheroutefrombeinginstalledinglobalFIB

•  ImporttherouteintoappropriateVRFbasedonroutetarget

•  InstalltheroutewithlabelinLinuxVRFforwardingtableviaZebra•  SendtheroutetoadjacentCE(s)belongingtotheVRF


vCEVRF VRF Global

Control-plane

FRRZebra

ZebraAPI

vPE

GoBGP

Data-plane

VRF VRF GlobalRIB

VRF VRF GlobalFIB


RR

IPv4BGProuteforprefixpp

VPNv4BGProuteforprefixpwithblueRDandbluelabel

p

p

p IPv4BGProuteforprefixpwithbluelabel

p

OurenhancementtoGoBGP

Wefixedthis

RouteBroker

p

p

p

20

SummaryDemonstratedfeasibilityofcreatinganL3-VPNvPEusingOpen

SourceSoftware•  Control-plane:–  GoBGP,FRR(OSPF,LDPandZebra)•  Data-plane:–  AT&T-VyattaDPDKbaseddata-plane–  AlsoverifiedfeasibilitywithVPPandLinuxdata-planes

RequiredustomakesomeenhancementstoGoBGP1.31•  ProperinstallationofroutesintoFIB•  AssignMPLSlabelstoVPNv4routes

• ModificationsavailableonGithubat:https://github.com/amanshaikh75/gobgp/tree/zapi_version_5

DANOSURL:https://www.danosproject.org/


21

AcknowledgementsAT&T•  BillBenson,RamanaChinnapa,KennethDuell,JenniferYates

CumulusNetworks•  DavidAhern(forexplaininghowLinuxVRFswork)FRR•  DonaldSharp,RenatoWestphal,RussWhite,https://github.com/paulzlabn

GoBGP•  IwaseYusukeVPP• MichaelBorokhovich,PierrePfister,JeffShaw


Backup

23

Open-SourceSoftwareacrosstheFeasibilityTest-bed

Control-plane•  GoBGP1.31.1=version1.31+ourenhancements

•  FRR5.1-dev=snapshote8f9540WhenLinuxisusedasdata-planeonvPE•  vPEOS:Ubuntu16.04.3LTS,Linuxkernel4.14.4-mpls

WhenVPPisusedasdata-planeonvPE•  vPEdata-plane:VPPrelease1801+ourenhancementstorouterplug-in

•  OS:sameaswhenLinuxisusedasdata-plane

NetworkFunction VNFOS Control-plane Data-plane

vCE Ubuntu16.04.2LTSLinuxKernel4.4.0-64generic

FRR5.1-devBGPandZebra Linux

vPE Debian4.14.62-0Vyatta1+9.1LinuxKernel4.14.0-trunk-vyatta-amd64…(DANOS)

GoBGP1.31.1FRR5.1-devOSPF,LDPandZebra(snapshote8f9540)

AT&T-VyattaDPDK

Corerouter Ubuntu16.04.3LTSLinuxkernel4.14.4-mpls(customconfiguration)

GoBGP1.31.1FRR5.1-devOSPF,LDPandZebra(snapshote8f9540)

Linux


24

GoBGP:ConfigurationofVRFandAssociatedCESessionsImportmatchingVPNv4routesintoVRF•  RoutesreceivedfromRRsand/orotherPEs

•  Outstandingissue;weuseaworkaroundObtainuniquelabelforVRFfromZebra•  Zebraactsasacentralagentforlabelassignment

–  PreventslabelcollisionbetweendifferentprotocolslikeBGPandLDP

•  Pullrequest1587toGoBGPrepositoryInstallanMPLSrouteforthelabelin

Linuxdefaultforwardingtable•  AllowsvPEtohandletrafficfromotherPEs

• WeenhancedGoBGPcode


VRF VRF Global

Control-plane

FRRZebra

ZebraAPI

vPE

GoBGP

Data-plane

VRF VRF GlobalRIB

VRF VRF GlobalFIB


ConfigureblueVRFandaddBGPsessiontoCE

ImportrelevantroutesfromglobaltableintoblueVRF

AssignMPLSlabel

# Example of MPLS route installed # in Linux kernel by GoBGP $ ip -f mpls route 144 dev blue proto bgp

RouteBroker

InstallMPLSlabelroute



25

GoBGP:InteractingwithZebraGoBGPbydefaultusesAPIversion4for

interactionwithZebra•  APIversion4doesnothaveallfeaturestosupportL3-VPN

–  Example:lackofsupportformulti-levelrecursivenext-hoplookup

•  RequiredustoupgradetoZebraAPIversion5AddedpartialsupportforAPIversion5

inGoBGP•  SupportforpartsrequiredforL3-VPN,noteverything


VRF VRF Global

Control-plane

FRRZebra

ZebraAPIVersion5

vPE

GoBGP

Data-plane

VRF VRF GlobalRIB

VRF VRF GlobalFIB


RouteBroker

Documents

Virtualized PE for BGP/MPLS L3-VPN using Open …...Virtualized PE for BGP/MPLS L3-VPN using Open-Source Software NANOG 74 (October 2018) Bilal Anwer, Robert Bays, Vijay Gopalakrishnan,