Embed Size (px)
DESCRIPTION
Dinesh Dutt presentation about Morphology of Modern Data Center Networks at YaC 2013 in Moscow, Russia.
Citation preview
Data Center Topologies
Morphology of Modern Data Center Networks
Dinesh G Dutt | Chief Scientist
YAC 2013
Dawn of the Modern Data CenterNetwork 2.0Routing Taming the Configuration Beast
Agenda
04/07/2023 2
(Pictures courtesy of Wikimedia, where not stated)
YAC 2013
Dawn of the Modern Data CenterNetwork 2.0RoutingTaming the Configuration Beast
Agenda
04/07/2023 3
YAC 2013
Evolution of the Data Center Application
04/07/2023 4
Traditional Enterprise Applications
L2-centric Sensitive to network
failures Mostly static
VLANs
No Server Virtualization
Mostly North-South
Lower Capacity
100s-few thousand endpoints
Modern Data Center Applications
IP-centric Workaround network
failures Dynamic
Clouds
Server Virtualization
Mostly East-West
High Capacity
Thousands to millions of endpoints
YAC 2013
Challenges: Large failure domain
Agg box failure
Unscalability of agg boxes MAC/ARP
VLANs
Choke point for E-W
Complex
HA
Too many protocols
Many proprietary enhancements
Each vendor has their version of the same feature
04/07/2023 5
Traditional Enterprise DC Network Design
L3
L2
Access
Aggregation
Core
VRRP VRRP
STP/VTP/GVRP/UDLD
ECMP
STP/VTP/GVRP/UDLD
YAC 201304/07/2023 6
Network's Function is to Serve the Application Needs
YAC 2013
Dawn of the Modern Data CenterNetwork 2.0RoutingTaming the Configuration Beast
Agenda
04/07/2023 7
YAC 2013
Folded CLOS Network
04/07/2023 8
YAC 2013
ECMPIP fabric
ubiquitous
Better Failure Handling
Predictable Latency
Simple Feature Set
Scalable
L2/L3 Boundary
ToR vs. EoR design
Characteristics Of CLOS Network
04/07/2023 9
LEAF
SPINE
YAC 2013
Calculating Network Size
04/07/2023 10
TIER-1
TIER-2
TIER-3
2 Tier FabricFor smaller environments
3 Tier FabricFor large-sale environmentsPods can be of dissimilar size
LEAF
SPINE
Cluste
r or P
od
Cluste
r or P
od
Inte
r-Clus
ter
YAC 2013
Calculating Network Size
04/07/2023 11
2 Tier Fabric• #ports @ToR = (m*n)/2• Max #ports @ToR= 2K with
64px10GE at Tor/Spine• Max #ports @ToR= 4608 with
96px10GE at Tor/Spine
3 Tier Fabric• #ports @ToR = (m*n*o)/4• Max #ports @ToR= 65K with
64px10GE at Tor/Spine/Spine• Max #ports @ToR = 221K with
96x10GE at Tor/Spine/Spine
m
m
n
on
YAC 2013
Oversubscription & Such
04/07/2023 12
Number of servers: Number of uplinks
Non-blocking after this first layer
Using Trident and 40 servers per rack: Oversubscription is 2.5
Using Trident2 in same config:Oversubscription can be 1
04/07/2023 13YAC 2013
Paganini Variations
04/07/2023 14YAC 2013
Size Does MatterFine grained failure domainLarge boxes vs small boxesInterconnect link Scheduling DowntimeTrying on new clothesMulti-vendor
Failure Analysis
YAC 2013
Dawn of the Modern Data CenterNetwork 2.0Routing Taming the Configuration Beast
Agenda
04/07/2023 15
Picture courtesy Nanoer.com @flickr
04/07/2023 16YAC 2013
What ProtocolLink state (OSPF/ISIS) or BGP
Managing IPv4/v6Separate session/protocol or unified
Multi-Vendor SupportDeployment Experience
Questions That Affect Routing Protocol
YAC 2013
Commonly deployed protocol within enterprisesSimplify config:
Only 2 area IDs, backbone and non-BBUnnumbered interfaces
Run OSPFv3 also if you have IPv6Route summarization possible, not desired due to non-optimal routing
04/07/2023 17
OSPF
Backbone area
Area 0.0.0.1 Area 0.0.0.1
YAC 2013
Simple up-down routingUse Private AS numbersRoute summarization not possibleInterface addresses only Single BGP session for v4/v6 or separate sessions
04/07/2023 18
eBGP
ASx ASx1 ASx2 ASxn
ASy1ASy1ASy1ASy1ASy ASy ASy ASy
ASz ASz ASz ASz
ASx3 ASx ASx1 ASx2 ASxnASx3
YAC 2013
Simple up-down routingNo IGPEliminates AS number distractionUse of NH Self with RRSingle-hop BGP peer, use interface addressSingle BGP session for v4/v6 or separate sessions
04/07/2023 19
iBGP
RR RR RR RR RR RR RR RR
RR RR RR RR
YAC 2013
VMVMVMVM VMVM
Logical switch
Logical switch
Great fit for modern data center appsLayer complex applications such as clouds as an overlay
L2 as a service
04/07/2023 20
Network Virtualization
YAC 2013
Dawn of the Modern Data CenterNetwork 2.0Routing Taming the Configuration Beast
Agenda
04/07/2023 21
04/07/2023 22YAC 2013
To err is human, to automate divineBut traditional networking gear is a black box
OS functions more like an embedded OS
No programmable way to configure the box
Primitive network management tool chain
Vendor-specific
Automate Configuration
23
Turn Black box into White & use Linux as the network OSWhy Linux ?
Well established and open APIVibrant community fueling innovationSophisticated management tool chainExcellent networking support
Linux As The Network OS
04/07/2023 YAC 2013
24
Server management tools to manage networksPuppet, Chef, Ansible or in house
Common Toolset
04/07/2023 YAC 2013
04/07/2023 25YAC 2013
Verify connectivity is as per operator specified cabling planUser defined actions on topology check result
For example, routing adjacency is brought up only if physical connectivity check passes
Example: T1, port1 is connected to M1, port1 T1, port2 is connected to M2, port1 … M1, port 3 is connected to S1, port1 M1, port 4 is connected to S2, port1
…
Validating Physical Topology
S2
M2M1
T2T1
M4M3
T4T3
S1
04/07/2023 26YAC 2013
Graphviz: Network topology specified via DOT languageWell understood graph modeling languageWide range of supported toolsOpen source
Central management tool: Network topology is pushed out to all nodesEach node determines its relevant information
LLDP: Use the discovery protocol to verify connectivity
Graph G {S1:p1 – M1:p3;S1:p2 – M2:p3;S1:p3 – M3:p3;S1:p4 – M4:p3;S2:p1 – M1:p4;S2:p2 – M2:p4;S2:p3 – M3:p4;S2:p4 – M4:p4;M1:p1 – T1:p1;M1:p2 – T2:p2;…M4:p2 – T4:p2;
}
ptmd: Prescriptive Topology Manager
https://github.com/CumulusNetworks/ptm
04/07/2023 27YAC 2013
CLOS Fabric as the foundation for modern data center networksLayer Complex applications such as Clouds on top with overlaysAutomate Configuration & Simplify Networking
Linux as the network OS to use sophisticated management tools
Simplify networking further with tools such as ptmd
Conclusion
2804/07/2023 YAC 2013
www.cumulusnetworks.com
@cumulusnetworks
Spasibo!
Web:
Email:
Twitter:
