Upload
lamminh
View
213
Download
0
Embed Size (px)
Citation preview
ScalableControlPlaneSubstrate
SachinKa3,JohnOusterhout,GuruParulkar,MarcosAguilera,CurtKolovson
ONRC+ON.Lab+RAMCloud,VMWare
MoEvaEon
• SeparaEonofcontrolplaneisacommontrend:networks/systems– Challengingtodesign,hardrequirementsonthroughput,latency,consistency
• Wehavebeenbuildingcontrolplanesforourspecificsystems– ONOS,SoORAN,andRAMCloudCoordinator
• Canwedesignageneralizedscalablecontrolplanewith– AcommonfoundaEonthatcanbecustomizedfordifferentcontexts
ExamplesofControlPlanes
• SDNControllers– Datacenter,enterprise,serviceproviderwiredandcellularnetworks– IoT
• VMmanager(e.g.NovainOpenStack)– DataCenters
• Nameservers(“namenode”)– distributedfilesystem
• Scheduler– Manydistributedsystems
• SoOwaredefinedstorage• …
New“Control”Planes–Drones,Cars
• Controllingfleetsofdronesandcars– Collisionavoidance,sharingtheairspace– Airtrafficcontrolfordrones– LogisEcsanddelivery
App1 App2
ControlPlane
OpenInterface(s)
SeparaEonofControlPlane
SeparaEonofControlPlane
GlobalViewApp3
OpenInterface
Server
Server
Storage
SwitcheseNBsServersStorage
Server
LogicallyCentralizedControlPlane
• Providesglobalview
• Makesiteasytoprogramcontrol,management,configapps
• Enablesnewapps
KeyPerformanceRequirements
Control
Apps Apps
Global Network View / State Global View / State
HighVolumeofState:~500GB-2TBHighThroughput:~500K-20Mops/second~100Mstateops/secondLowLatencytoEvents:1-10sms
Difficultchallenge!
Highthroughput|Lowlatency|Consistency|Highavailability
Server
Server
Server
Storage
● Scalability,HighAvailabilityandPerformance● NorthboundandSouthboundAbstracEons● Modularity
ControlPlaneChallenges
ExampleScalableControlPlanes
ONOSSoORAN
RAMCloudCoordinator
Scale-OutDesignw/MulEpleONOSInstances
ONOS
Apps Apps
Global Network View / State Global Network View / State
Each instance is identical
One can add and remove instances seamlessly
Each instance is a master for a subset of switches
It works like a single system for apps and network devices
ONOS Instance 1 ONOS Instance 2 ONOS Instance 3
ONOSArchitectureTiers
Northbound - Application Intent Framework (policy enforcement, conflict resolution)
OpenFlow NetConf . . .
Apps Apps
Distributed Core (scalability, availability, performance, persistence)
Southbound (discover, observe, program, configure)
Northbound Abstraction: - network graph - application intents Core: - distributed - protocol independent Southbound Abstraction: - generalized OpenFlow - pluggable & extensible
DistributedCore
Southbound
“Provision 10G path from Datacenter 1 to Datacenter2 optimized for cost”
Intents translated and Compiled into specific instructions for network devices.
ApplicaEonIntentFramework:AbstracEons,APIs,PolicyEnforcement,ConflictresoluEon
DistributedCore
SouthboundCoreAPI
OpenFlow NETCONF SouthboundInterface
ONOSApplicaEonIntentFrameworkIwanttodefine“what”Ineedwithoutworryingabout“how”…
DistributedCore:StateManagement
● Topology● Flows● Intents● Switchtocontrollermapping● ResourceallocaEons● NetworkConfiguraEon● AndaplethoraofapplicaEon
generatedstate
DifferentstateshavedifferentsizeandR/WpakernsTheycanusedifferentconsistency,replicaEon,sharding
KeytoPerformance/Scalability:DistributedStateManagement
172.16.0.0 172.16.0.0
172.16.0.0 172.16.0.0
Application Intents - immutable - durable & replicated Global Network View - eventually consistent - fully replicated
Flow Table Entries - strongly consistent - partitioned
Optimistic Replication - gossip based - anti-entropy
Optimistic Replication - gossip based - anti-entropy - partial ordering
Master/Backup Replication
172.16.0.0 172.16.0.0
172.16.0.0 172.16.0.0
172.16.0.0 172.16.0.0
172.16.0.0 172.16.0.0
Switch è Master Switch è Master Switch è Master SwitchtoController-stronglyconsistent-replicatedfordurability
Consensusforstrongconsistency
Switch è Master Switch è Master Switch è Master ResourceAssignment-stronglyconsistent-parEEonedforscale
Consensusforstrongconsistency
ONOSPerformanceMetricsandResults● Device&linksensinglatency
o Lessthan100mso ONOSprocessinglessthan10ms
● FlowruleoperaEonsthroughputo 500Kto3Mops/sec
● IntentoperaEonsthroughputo 200kops/sec
● IntentoperaEonslatencyo Lessthan50ms
SoORAN
GlobalNetworkView:eNBs+smallcelleNBs+signalstrengthsfromdevicesto
neighboringeNBs
So#RAN:So#wareDefinedRadioAccessNetwork
Macrocell
Picocell
Microcell Macrocell
PicocellMicrocell
So#RANOS
LTE-A LTE-IoT MVNO
Upto100Kmobiledevices;fewflowsperdevice
Asdevicesmove,needtohandover
50MacroeNBs,600MicroeNBs
Signalstrengthsconstantlychange–withinterference
Goal:Maximizeefficiencyofpreciousspectrum
Createpersonalizeduserexperience
Minimizecapex/opex
withrapidly
increasingdevicesandtheirbandwidth
requirements
CriEcaldecisions:Handover
DevicetoeNBassignmentPowerassignment
GlobalNetworkVieweNBs+smallcelleNBs+signalstrengthsfromdevicesto
neighboringeNBs
ControlPlaneRequirementsforSo#RAN
Macrocell
Picocell
Microcell Macrocell
PicocellMicrocell
So#RANOS
LTE-A LTE-IoT MVNO
Upto100Kmobiledevices;fewflowsperdevice
50MacroeNBs,600MicroeNBs
AllocaEonof50resourceblockspereNBstomobiledeviceseveryms
SoORANKeyPerformanceRequirements
Control
Apps Apps
Global Network View / State Global View / State
HighVolumeofState:~500GB-2TBHighThroughput:~500K-5Mops/second~20Mstateops/secondLowLatencytoEvents:1-10sms
Difficultchallenge!Server
Server
Server
Storage
Orderofmagnitudehigherthroughputandlowerlatencyrequirement
Upto100Kmobiledevices;fewflowsperdevice
OpportunityforParallelismStatemanagement/processingcanbedoneindependently–
notasEghtlycoupled
GlobalNetworkViewMulEpleTBs
Updatedeveryfewms
ControlPlaneChallengesforSo#RAN:SimplificaConandOpCmizaCon
So#RANOS
LTE-A LTE-IoT MVNO
50MacroeNBs,600MicroeNBs
OnlydevicesintransiConrequireimmediateprocessing
RAMCloudCoordinator
RAMCloudControlPlane
RAMCloudArchitecture:ControlPlane
Appl.Library
DatacenterNetwork
…1000–100,000ApplicaConServers
Appl.Library
Appl.Library
Appl.Library
MasterBackup
MasterBackup
MasterBackup
MasterBackup…
Coordinator
CoordinatorStandby
ExternalStorage(ZooKeeper)
1000–10,000StorageServers● Meta-state:mappingofkeyspacetotablesonservers,listofacEveservers,
listofacEveclient-leases● AppseverscachethestateandsocoordinatornotinthecriEcalpath
RAMCloudCoordinator:Requirements
Apps Control Apps
Global Network View / State Global System View / State
High Volume of State: ~500GB Medium Throughput: ~M updates per second Low Latency to Response: usec to msec
MasterBackup
MasterBackup
MasterBackup…
10,000Servers
…
100,000AppServers
Appl.Library
Appl.Library
Appl.Library
GeneralizedandCustomizableScalableControlPlane
Whatisdomainspecificandwhatisnt?
• Domainindependent– Statemanagementwithconfigurableconsistency,availability– Streamingeventprocessing– GraphabstracEons
• Domainspecific– Northboundandsouthboundinterfaces– NoEficaEonAPIs
GeneralizedandCustomizableScalableControlPlane
Northbound Absrtractions/APIs (C/C++, Declective Programming, REST) Strongly consistent, trasaction semantics?
OpenFlow/NetConf RAN X2 Interface RPC
Apps
Distributed Core Cluster of Servers, 10-100Gbps, low latency RPC
Distributed State Management Primitives
Southbound Plug-in for different contexts
NorthboundAbstracEon:-ProvidedifferentAPIs-CustomizeforthecontextCore:-distributed-contextindependentSouthboundAbstracEon:-Plug-insfordifferentcontexts
Server
Server
Storage
SwitcheseNBsServersStorage
Server
TimeVaryingStreamingGraphAbstracEon
• CommonstateabstracEonformanycontrolplanesàGraph– E.g:Innetworks,graphsofnodes(routers,basestaEons,phones)with
statechangingfrequently(linkload,cellconnecEvity)– O(1million)nodesingraph,relaEvelysmallcomparedtowebgraphs
• Updatestographsinstreamingfashion&highlydynamic– E.g:Incellularnetworks,everyedgeupdatedevery3-5msà
O(2-5million)updateseverysecond
ComplexEventProcessing
• ComputeabstracEon:Updatestographs(events)triggercomplexprocessingthat’sdomainspecific– E.g:linkdowneventtriggeringarouteupdatetomaintainconnecEvity
• TriggeredeventprocessingcanhaveEghtlatencyconstraints– E.g:linkdowneventcancausedisconnecEon,needfastacEon
(~10-100ms)tokeepthenetworkconnected
• Ideally,oneshouldbeabletowritetheeventlogicinahighleveldeclaraEvefashion
Summary• SeparaEonofcontrolplaneisacommontrend:networks/systems
• Wehavebeenbuildingcontrolplanesforourspecificsystems– ONOS,SoORAN,andRAMCloudCoordinator
• Canwedesignageneralizedscalablecontrolplanewith– AcommonfoundaEonthatcanbecustomizedfordifferentcontexts
Backup
GeneralizedandCustomizableScalableControlPlane
• AcommonfoundaEonforallcontrolplanes– Aclusterofcommodityservers– Highperformancenetwork:10-100Gbps,speedoflightlatency– LowlatencyRPC:~5usecendtoendlatency– Customizeddistributedstatemanagement– Differentconsistency,replicaEon,andshardingmodelsfordifferentstate
• Customizableforspecificcontexts– Allowthespecificsystemtopickdistributedstatemanagements– AppprogrammingabstracEons/models/APIs
• C/C++,DeclaraEveProgramming,RESTAPIs
– Supportdifferentconsistencymodels• Canwegetawaywitheverythingbeingstronglyconsistent?
Outline• Examplecontrolplanes:requirementsanddesigns
– ONOSforserviceprovidernetworks– SoORANforcellularRANs– CoordinatorforRAMCloud– Otherexamples
• Opportunityforaunifiedscalablecontrolplane– Whatmakesusbelievethisispossible– Whatarethechallenges
• Highlevelarchitectureofaunifiedscalablecontrolplane– Commonbuildingblocks– CustomizaEon
• Nextsteps
So#RAN:DataPlaneAbstracCon
• Flow-graph:thepipeline/graphofpacketprocessingatomsthatneedtobeexecutedonalldata/packetsmatchingtheflow
• Flowisflexiblydefined(e.g.IPflow,UEbearer,QoSclass)
• AtomsaresimplepacketprocessingfuncEons(e.g.headercompression,segmentaEon,FFT)
• Flow-graphscanhavelatencyandcomputethroughputrequirementsassociatedwiththem
A1
A2
A3
A4 A5
An
Match:IfPktàFlow1AcEon:ExecuteFlow-GraphF
Flow
-Graph
Fwith
throughp
utand
latency
requ
iremen
t
So#RAN:ControlPlaneAbstracCon• Update/Reduceoncontrol-graphsthat
representnetwork/applicaEonstate– Control-graphisdefinedflexibly(e.g
networkgraphwitheNBs,UEsandedgesannotatedwithlinkstrength,flowrecords,CDNgraphwithcontentnodes)
– LocalupdatecomputaEonateachnodeinthegraph,messagepassingonedgesandglobalreducefuncEons
• SimpleabstracEonissufficienttoexpressalargerangeofcontrolfuncEons
N1 N2 N3
N4 N6 N5
R
Update(NodeState) Update(NodeState) Update(NodeState)
Reduce(GlobalState)
Update(NodeState) Update(NodeState) Update(NodeState)
DistributedMessagePassing
So#RAN:Expressingnetworkservicesusingflowgraphs&controlgraphs
A1
A2
A3
A4 A5
An
N1
N2
N3
N4
N6
N5
R
Update(NodeState) Update(NodeState) Update(NodeState)
Reduce(GlobalState)
Update(NodeState) Update(NodeState) Update(NodeState)
DistributedMessagePassing
SoORANOS(Compiler,scheduler&resourceorchestrator)
Macrocell
PicocellMicrocell Macrocell
PicocellMicrocell
So#RANOS:PlasormtoexecuteanycombinaEonofflowandcontrolgraphsexpressedinhighlevelprogramsonCOTSdistributedhardwarewhilemeeEnglatency&throughputrequirements
Data/ControlInterface
H1
H2
H3
H4
H5
H6
H7
H8
H9
H1H2H3H4H5H6 H7H8H9H1H2H3H4H5H6 H7H8H9
ControlPlaneforServiceProviderNetworks
ONOS• Networkstate
– 1TB– Changesmoreslowly
• Controlops– 1-3Mops/sec
• Latencyforprocessing– 10ms
• Highavailability
So#RAN• Networkstate
– MulEpleTBs– Changeseveryfewms
• Controlops– 20Mops/sec
• Latencyforprocessing– 1ms
• Highavailability
SoORANoffersopportuniEesforsimplificaEonandopEmizaEons
RAMCloudControlPlane
RAMCloudArchitecture
Appl.Library
DatacenterNetwork
…1000–100,000ApplicaConServers
Appl.Library
Appl.Library
Appl.Library
MasterBackup
MasterBackup
MasterBackup
MasterBackup…
Coordinator
CoordinatorStandby
ExternalStorage(ZooKeeper)
1000–10,000StorageServers
High-speednetworking:• 5µsround-trip• FullbisecEonbandwidth
CommodityServers
64-256GBperserver
ScalableControlPlane:BreakoutSessionSummary• HeardposiEvecommentsabouttheoverallgoalsandproject
– withareminderthatwearetakingonsomethingchallenging– generalandcustomizableandhighperformancereobviouslyatodds
witheachother
• Scalablecontrolplanesfornetworking—easiertoseefromIoTtoRANtoothernetworks– Aretheyinadifferentcategorycomparedtoscalablecontrolplanefor
distributedsystems
• Examplesofscalablecontrolplanesfordistributedsystems– Schedulers:clustersandmulEpleclusters– Schedulingofmicro-servicesorassociatedworkload
• Whatisdomainorcontextspecificandwhatisdomain/contextindependent– Distributedstatemanagementisdomainindependent– ApplicaEonabstracEonsandAPIsandinterfacestodevicesbeingcontrolledare
domainspecific
• Whatistherightmodelorparadigmforthedomainindependentpart– Distributedstatemanagementbutthereareothermodels– Emevaryingnetworkgraphasadatabaseandstandingqueries– streamingdatabasewithabilitytocustomize:consistency,replicaEon,sharding– Complexeventprocessingonstreamingdatabase
• ApplicaEonabstracEonsandAPIs– DeclaraEve/easytoprogram