Embed Size (px)
Citation preview
© 2006 Cisco Systems, Inc. All rights reserved. Cisco Confidential
Design & Architecture of Data Center Network Platforms 1
Design and Architecture ofData Center Networking Platforms
Lincoln Dale <[email protected]>TME, Data Center Switching
© 2006 Cisco Systems, Inc. All rights reserved. Cisco Confidential 2Design & Architecture of Data Center Network Platforms
Agenda
Part I—Design Considerations
Part II—Architectures
Part III—Real World Implementation
Part IV—The Future
© 2006 Cisco Systems, Inc. All rights reserved. Cisco Confidential
Design & Architecture of Data Center Network Platforms 3
Part I – Design Considerations
© 2006 Cisco Systems, Inc. All rights reserved. Cisco Confidential 4Design & Architecture of Data Center Network Platforms
Part I—Design Considerations
Many Factors to WeighBufferingMultiprotocolOne Plus One Is Not Necessarily Equal to TwoFuture (Investment Protection)
© 2006 Cisco Systems, Inc. All rights reserved. Cisco Confidential 5Design & Architecture of Data Center Network Platforms
Many Factors to Weigh
Standards requirementsMarket requirementsDesignabilitySilicon technologyProcessor technology
Applicable to Any Switch/Router Design
Product is chassis and networkManufacturabilityTime to marketFlexibilityBudget
© 2006 Cisco Systems, Inc. All rights reserved. Cisco Confidential 6Design & Architecture of Data Center Network Platforms
Many Factors to Weigh
BufferingNo packet dropThroughputPort countModularNo single point of failureIn-order deliveryFuture protocol compatibility
Data PlaneModularRestartable (including active-active state handling)Non-disruptive code load & activationNo single point of failureScaleableUnit TestableFuture protocol compatibility
Control plane
Baseline Data Center Switch requirements
© 2006 Cisco Systems, Inc. All rights reserved. Cisco Confidential 7Design & Architecture of Data Center Network Platforms
Part I—Design Considerations
Many Factors to WeighBufferingMultiprotocolOne Plus One Is Not Necessarily Equal to TwoFuture (Investment Protection)
© 2006 Cisco Systems, Inc. All rights reserved. Cisco Confidential 8Design & Architecture of Data Center Network Platforms
In A
In B
Out C
Out D
SwitchSwitch
Buffering
A-C
Out
put
Out
put
Inpu
tIn
put
© 2006 Cisco Systems, Inc. All rights reserved. Cisco Confidential 9Design & Architecture of Data Center Network Platforms
Out C
Out D
SwitchSwitch
Out
put
Out
put
Inpu
tIn
put
In A
In B
SwitchSwitch
Buffering
A-C
© 2006 Cisco Systems, Inc. All rights reserved. Cisco Confidential 10Design & Architecture of Data Center Network Platforms
In A
In B
Out C
Out D
Buffering
A-C
SwitchSwitch
Out
put
Out
put
Inpu
tIn
put
© 2006 Cisco Systems, Inc. All rights reserved. Cisco Confidential 11Design & Architecture of Data Center Network Platforms
In A
In B
Out C
Out D
Buffering
A-C
B-C
SwitchSwitch
Out
put
Out
put
Inpu
tIn
put
© 2006 Cisco Systems, Inc. All rights reserved. Cisco Confidential 12Design & Architecture of Data Center Network Platforms
In A
In B
Out C
Out D
SwitchSwitch
Out
put
Out
put
Inpu
tIn
put
Buffering
XXA-C
B-C
© 2006 Cisco Systems, Inc. All rights reserved. Cisco Confidential 13Design & Architecture of Data Center Network Platforms
Buffering
A-C
B-C
In A
In B
Out C
Out D
SwitchSwitch
Out
put
Out
put
Inpu
tIn
put
© 2006 Cisco Systems, Inc. All rights reserved. Cisco Confidential 14Design & Architecture of Data Center Network Platforms
Out C
Out D
Out C
Out D
In A
In B
SwitchSwitch
Out
put
Out
put
Inpu
tIn
put
SwitchSwitch
Buffering
A-C
B-C
© 2006 Cisco Systems, Inc. All rights reserved. Cisco Confidential 15Design & Architecture of Data Center Network Platforms
Out C
Out D
SwitchSwitch
Out
put
Out
put
Inpu
tIn
put
In A
In B
Buffering
A-C
B-C
© 2006 Cisco Systems, Inc. All rights reserved. Cisco Confidential 16Design & Architecture of Data Center Network Platforms
Out C
Out D
SwitchSwitch
Out
put
Out
put
Inpu
tIn
put
In A
In B
Buffering
A-C
B-C
© 2006 Cisco Systems, Inc. All rights reserved. Cisco Confidential 17Design & Architecture of Data Center Network Platforms
Buffering
A-C
B-C
In A
In B
Out C
Out D
SwitchSwitch
Out
put
Out
put
Inpu
tIn
put
© 2006 Cisco Systems, Inc. All rights reserved. Cisco Confidential 18Design & Architecture of Data Center Network Platforms
Part I—Design Considerations
Many Factors to WeighBufferingMultiprotocolOne Plus One Is Not Necessarily Equal to TwoFuture (Investment Protection)
© 2006 Cisco Systems, Inc. All rights reserved. Cisco Confidential 19Design & Architecture of Data Center Network Platforms
Multi ProtocolProtocol AgnosticProtocol TranslationDrop BehaviorShared UplinksMultiple Topologies
IP/Ethernet802.11 “WiFi”802.16 “WiMax”FDDIVinesXNSAppleTalkIPX/SPXNetbeuiDECnetOSI/ISOArcNetL2TPv3
10Base{2,5,T} / 100Base{S,T,TX} /1000BaseT10GbEFibre ChannelFICONESCONToken Ring100BaseVG
IP & TCPIPv4/IPv6802.1q VLANsMPLS /MPLS {P,PE} /MPLE {CE,..}, ..VPLSq-in-q VLAN encapsulation802.2 LLC/SNAPATMFrame RelayDOCSISAToM
© 2006 Cisco Systems, Inc. All rights reserved. Cisco Confidential 20Design & Architecture of Data Center Network Platforms
Part I—Design Considerations
Many Factors to WeighBufferingMultiprotocolOne Plus One Is Not Necessarily Equal to TwoFuture (Investment Protection)
© 2006 Cisco Systems, Inc. All rights reserved. Cisco Confidential 21Design & Architecture of Data Center Network Platforms
One Plus One Does Not Equal TwoSw
itch
ASw
itch
A
Switc
h B
Switc
h B
10 Links @ 1Gbps EachBandwidth = 10Gbps
Flow Bandwidth = 1GbpsSerialization Delay = 20uS
1 Link @ 10Gbps EachBandwidth = 10Gbps
Flow Bandwidth = 10GbpsSerialization Delay = 2uS
Switc
h A
Switc
h A
Switc
h B
Switc
h B
© 2006 Cisco Systems, Inc. All rights reserved. Cisco Confidential
Design & Architecture of Data Center Network Platforms 22
Part II –Architectures
© 2006 Cisco Systems, Inc. All rights reserved. Cisco Confidential 23Design & Architecture of Data Center Network Platforms
Part II—Architectures
Centralized MemorySwitch on a Chip / “System on Chip” (SoC)
BusMesh
Centralized memory in a mesh
Two-Tier / Banyan / ClosCentralized memory in two tiers
Cross-Bar
© 2006 Cisco Systems, Inc. All rights reserved. Cisco Confidential 24Design & Architecture of Data Center Network Platforms
Part II—Architectures
Centralized MemorySwitch on a chip / “System on Chip” (SoC)
BusMesh
Centralized memory in a mesh
Two-Tier / Banyan / ClosCentralized memory in two tiers
Cross-Bar
© 2006 Cisco Systems, Inc. All rights reserved. Cisco Confidential 25Design & Architecture of Data Center Network Platforms
Centralized Memory
Inpu
t 3In
put 2
Inpu
t 1
Out
put 3
Out
put 2
Out
put 1
Scalability limited by memory bandwidth/sizeTypically optimized for fixed configuration—inflexiblePort count 8–32Cost effective with small port countsOften used as building blockAlmost required for Blade Server Applications
© 2006 Cisco Systems, Inc. All rights reserved. Cisco Confidential 26Design & Architecture of Data Center Network Platforms
Inpu
t 3In
put 2
Inpu
t 1
Out
put 3
Out
put 2
Out
put 1
Centralized Memory
Scalability limited by memory bandwidth/sizeTypically optimized for fixed configuration— inflexiblePort count 8–32Cost effective with small port countsOften used as building blockAlmost required for Blade Server Applications
© 2006 Cisco Systems, Inc. All rights reserved. Cisco Confidential 27Design & Architecture of Data Center Network Platforms
Inpu
t 3In
put 2
Inpu
t 1
Out
put 3
Out
put 2
Out
put 1
Centralized Memory
Scalability limited by memory bandwidth/sizeTypically optimized for fixed configuration— inflexiblePort count 8–32Cost effective with small port countsOften used as building blockAlmost required for Blade Server Applications
© 2006 Cisco Systems, Inc. All rights reserved. Cisco Confidential 28Design & Architecture of Data Center Network Platforms
Inpu
t 3In
put 2
Inpu
t 1
Out
put 3
Out
put 2
Out
put 1
Centralized Memory
Scalability limited by memory bandwidth/sizeTypically optimized for fixed configuration— inflexiblePort count 8–32Cost effective with small port countsOften used as building blockAlmost required for Blade Server Applications
© 2006 Cisco Systems, Inc. All rights reserved. Cisco Confidential 29Design & Architecture of Data Center Network Platforms
Inpu
t 3In
put 2
Inpu
t 1
Out
put 3
Out
put 2
Out
put 1
Centralized Memory
Scalability limited by memory bandwidth/sizeTypically optimized for fixed configuration— inflexiblePort count 8–32Cost effective with small port countsOften used as building blockAlmost required for Blade Server Applications
© 2006 Cisco Systems, Inc. All rights reserved. Cisco Confidential 30Design & Architecture of Data Center Network Platforms
Inpu
t 3In
put 2
Inpu
t 1
Out
put 3
Out
put 2
Out
put 1
Centralized Memory
3
2
1
Scalability limited by memory bandwidth/sizeTypically optimized for fixed configuration— inflexiblePort count 8–32Cost effective with small port countsOften used as building blockAlmost required for Blade Server Applications
© 2006 Cisco Systems, Inc. All rights reserved. Cisco Confidential 31Design & Architecture of Data Center Network Platforms
Inpu
t 3In
put 2
Inpu
t 1
Out
put 3
Out
put 2
Out
put 1
Centralized Memory
3
2
1
Scalability limited by memory bandwidth/sizeTypically optimized for fixed configuration— inflexiblePort count 8–32Cost effective with small port countsOften used as building blockAlmost required for Blade Server Applications
© 2006 Cisco Systems, Inc. All rights reserved. Cisco Confidential 32Design & Architecture of Data Center Network Platforms
Inpu
t 3In
put 2
Inpu
t 1
Out
put 3
Out
put 2
Out
put 1
Centralized Memory
3
2
1
Scalability limited by memory bandwidth/sizeTypically optimized for fixed configuration— inflexiblePort count 8–32Cost effective with small port countsOften used as building blockAlmost required for Blade Server Applications
© 2006 Cisco Systems, Inc. All rights reserved. Cisco Confidential 33Design & Architecture of Data Center Network Platforms
Inpu
t 3In
put 2
Inpu
t 1
Out
put 3
Out
put 2
Out
put 1
Centralized Memory
3
Scalability limited by memory bandwidth/sizeTypically optimized for fixed configuration— inflexiblePort count 8–32Cost effective with small port countsOften used as building blockAlmost required for Blade Server Applications
© 2006 Cisco Systems, Inc. All rights reserved. Cisco Confidential 34Design & Architecture of Data Center Network Platforms
Centralized Memory
Inpu
t 3In
put 2
Inpu
t 1
Out
put 3
Out
put 2
Out
put 1
Scalability limited by memory bandwidth/sizeTypically optimized for fixed configuration— inflexiblePort count 8–32Cost effective with small port countsOften used as building blockAlmost required for Blade Server Applications
© 2006 Cisco Systems, Inc. All rights reserved. Cisco Confidential 35Design & Architecture of Data Center Network Platforms
Part II—Architectures
Centralized Memory“Switch on a chip” / System on Chip (SoC)
BusMesh
Centralized memory in a mesh
Two-Tier / Banyan / ClosCentralized memory in two tiers
Cross-Bar
© 2006 Cisco Systems, Inc. All rights reserved. Cisco Confidential 36Design & Architecture of Data Center Network Platforms
BusIn
put 1
Inpu
t 2In
put 3
Out
put 1
Out
put 2
Out
put 3
3
2
1
© 2006 Cisco Systems, Inc. All rights reserved. Cisco Confidential 37Design & Architecture of Data Center Network Platforms
Inpu
t 1In
put 2
Inpu
t 3
Out
put 1
Out
put 2
Out
put 3
Bus
3
2
1
© 2006 Cisco Systems, Inc. All rights reserved. Cisco Confidential 38Design & Architecture of Data Center Network Platforms
Inpu
t 1In
put 2
Inpu
t 3
Out
put 1
Out
put 2
Out
put 3
Bus
XX
XX3
2
1
© 2006 Cisco Systems, Inc. All rights reserved. Cisco Confidential 39Design & Architecture of Data Center Network Platforms
Inpu
t 1In
put 2
Inpu
t 3
Out
put 1
Out
put 2
Out
put 3
XX
Bus
XX
3
2
1
© 2006 Cisco Systems, Inc. All rights reserved. Cisco Confidential 40Design & Architecture of Data Center Network Platforms
Inpu
t 1In
put 2
Inpu
t 3
Out
put 1
Out
put 2
Out
put 3
XX
Bus
XX3
2
1
© 2006 Cisco Systems, Inc. All rights reserved. Cisco Confidential 41Design & Architecture of Data Center Network Platforms
Inpu
t 1In
put 2
Inpu
t 3
Out
put 1
Out
put 2
Out
put 3
Bus
3
2
1
© 2006 Cisco Systems, Inc. All rights reserved. Cisco Confidential 42Design & Architecture of Data Center Network Platforms
Inpu
t 1In
put 2
Inpu
t 3
Out
put 1
Out
put 2
Out
put 3
Bus
3
2
1
© 2006 Cisco Systems, Inc. All rights reserved. Cisco Confidential 43Design & Architecture of Data Center Network Platforms
BusIn
put 1
Inpu
t 2In
put 3
Out
put 1
Out
put 2
Out
put 3
© 2006 Cisco Systems, Inc. All rights reserved. Cisco Confidential 44Design & Architecture of Data Center Network Platforms
Bus with centralized memory
In 1In 2In 3In 4In 5In 6In 7In 8
In 1In 2In 3In 4In 5In 6In 7In 8
In 1In 2In 3In 4In 5In 6In 7In 8
Out 1Out 2Out 3Out 4Out 5Out 6Out 7Out 8
Out 1Out 2Out 3Out 4Out 5Out 6Out 7Out 8
Out 1Out 2Out 3Out 4Out 5Out 6Out 7Out 8
© 2006 Cisco Systems, Inc. All rights reserved. Cisco Confidential 45Design & Architecture of Data Center Network Platforms
Part II—Architectures
Centralized MemorySwitch on Chip / “System on Chip” (SoC)
BusMesh
Centralized memory in a mesh
Two-Tier / Banyan / ClosCentralized memory in two tiers
Cross-Bar
© 2006 Cisco Systems, Inc. All rights reserved. Cisco Confidential 46Design & Architecture of Data Center Network Platforms
Mesh
1616--PortPortCentralCentralMemoryMemory
1616--PortPortCentralCentralMemoryMemory
1616--PortPortCentralCentralMemoryMemory
1616--PortPortCentralCentralMemoryMemory
13 ExternalPorts
Total 52 External Ports13:1 Blocking Factor
© 2006 Cisco Systems, Inc. All rights reserved. Cisco Confidential 47Design & Architecture of Data Center Network Platforms
1616--PortPortCentralCentralMemoryMemory
1616--PortPortCentralCentralMemoryMemory
1616--PortPortCentralCentralMemoryMemory
1616--PortPortCentralCentralMemoryMemory
Total 52 External Ports
Mesh
Input A Output B
© 2006 Cisco Systems, Inc. All rights reserved. Cisco Confidential 48Design & Architecture of Data Center Network Platforms
1616--PortPortCentralCentralMemoryMemory
1616--PortPortCentralCentralMemoryMemory
1616--PortPortCentralCentralMemoryMemory
1616--PortPortCentralCentralMemoryMemory
Total 52 External Ports
Input A1 Output B1’
Mesh
13:1 Over-Subscription
Input A2
Input A13
Output B2’
Output B13’
© 2006 Cisco Systems, Inc. All rights reserved. Cisco Confidential 49Design & Architecture of Data Center Network Platforms
1616--PortPortCentralCentralMemoryMemory
1616--PortPortCentralCentralMemoryMemory
1616--PortPortCentralCentralMemoryMemory
1616--PortPortCentralCentralMemoryMemory
10 ExternalPorts
Total 40 External Ports
Mesh
© 2006 Cisco Systems, Inc. All rights reserved. Cisco Confidential 50Design & Architecture of Data Center Network Platforms
1616--PortPortCentralCentralMemoryMemory
1616--PortPortCentralCentralMemoryMemory
1616--PortPortCentralCentralMemoryMemory
1616--PortPortCentralCentralMemoryMemory
Total 40 External Ports
Mesh
Input A1 Output B1’
10:2 Over-Subscription
Input A2
Input A10
Output B2’
Output B10’
© 2006 Cisco Systems, Inc. All rights reserved. Cisco Confidential 51Design & Architecture of Data Center Network Platforms
1616--PortPortCentralCentralMemoryMemory
1616--PortPortCentralCentralMemoryMemory
1616--PortPortCentralCentralMemoryMemory
1616--PortPortCentralCentralMemoryMemory
Total 16 External PortsNon-Blocking
Mesh
4 ExternalPorts
4 Links
© 2006 Cisco Systems, Inc. All rights reserved. Cisco Confidential 52Design & Architecture of Data Center Network Platforms
Mesh
SAME
Total 16 External PortsNon-Blocking
1616--PortPortCentralCentralMemoryMemory
1616--PortPortCentralCentralMemoryMemory
1616--PortPortCentralCentralMemoryMemory
1616--PortPortCentralCentralMemoryMemory 4 Links
1616--PortPortCentralCentralMemoryMemory
© 2006 Cisco Systems, Inc. All rights reserved. Cisco Confidential 53Design & Architecture of Data Center Network Platforms
Mesh-Based Switch
16 Port Switch
16 Port Switch
16 Port Switch
16 Port Switch
16 P
ort S
witc
h
16 P
ort S
witc
h
16 P
ort S
witc
h
16 P
ort S
witc
h
16 P
ort S
witc
h
16 P
ort S
witc
h
16 P
ort S
witc
h
16 P
ort S
witc
h
16 Port Switch
16 Port Switch
16 Port Switch
16 Port Switch
16 Port Switch
16 Port Switch
16 Port Switch
16 Port Switch
16 P
ort S
witc
h
16 P
ort S
witc
h
16 P
ort S
witc
h
16 P
ort S
witc
h
16 P
ort S
witc
h
16 P
ort S
witc
h
16 P
ort S
witc
h
16 P
ort S
witc
h
© 2006 Cisco Systems, Inc. All rights reserved. Cisco Confidential 54Design & Architecture of Data Center Network Platforms
16 Port Switch
16 Port Switch
16 Port Switch
16 Port Switch
16 P
ort S
witc
h
16 P
ort S
witc
h
16 P
ort S
witc
h
16 P
ort S
witc
h
16 P
ort S
witc
h
16 P
ort S
witc
h
16 P
ort S
witc
h
16 P
ort S
witc
h
16 Port Switch
16 Port Switch
16 Port Switch
16 Port Switch
16 Port Switch
16 Port Switch
16 Port Switch
16 Port Switch
16 P
ort S
witc
h
16 P
ort S
witc
h
16 P
ort S
witc
h
16 P
ort S
witc
h
16 P
ort S
witc
h
16 P
ort S
witc
h
16 P
ort S
witc
h
16 P
ort S
witc
h
Mesh-Based Switch
Linecard
© 2006 Cisco Systems, Inc. All rights reserved. Cisco Confidential 55Design & Architecture of Data Center Network Platforms
16 Port Switch
16 Port Switch
16 Port Switch
16 Port Switch
16 P
ort S
witc
h
16 P
ort S
witc
h
16 P
ort S
witc
h
16 P
ort S
witc
h
16 P
ort S
witc
h
16 P
ort S
witc
h
16 P
ort S
witc
h
16 P
ort S
witc
h
16 Port Switch
16 Port Switch
16 Port Switch
16 Port Switch
16 Port Switch
16 Port Switch
16 Port Switch
16 Port Switch
16 P
ort S
witc
h
16 P
ort S
witc
h
16 P
ort S
witc
h
16 P
ort S
witc
h
16 P
ort S
witc
h
16 P
ort S
witc
h
16 P
ort S
witc
h
16 P
ort S
witc
h
Mesh-Based Switch
© 2006 Cisco Systems, Inc. All rights reserved. Cisco Confidential 56Design & Architecture of Data Center Network Platforms
16 Port Switch
16 Port Switch
16 Port Switch
16 Port Switch
16 P
ort S
witc
h
16 P
ort S
witc
h
16 P
ort S
witc
h
16 P
ort S
witc
h
16 P
ort S
witc
h
16 P
ort S
witc
h
16 P
ort S
witc
h
16 P
ort S
witc
h
16 Port Switch
16 Port Switch
16 Port Switch
16 Port Switch
16 Port Switch
16 Port Switch
16 Port Switch
16 Port Switch
16 P
ort S
witc
h
16 P
ort S
witc
h
16 P
ort S
witc
h
16 P
ort S
witc
h
16 P
ort S
witc
h
16 P
ort S
witc
h
16 P
ort S
witc
h
16 P
ort S
witc
h
Mesh-Based Switch
A
B
© 2006 Cisco Systems, Inc. All rights reserved. Cisco Confidential 57Design & Architecture of Data Center Network Platforms
16 Port Switch
16 Port Switch
16 Port Switch
16 Port Switch
16 P
ort S
witc
h
16 P
ort S
witc
h
16 P
ort S
witc
h
16 P
ort S
witc
h
16 P
ort S
witc
h
16 P
ort S
witc
h
16 P
ort S
witc
h
16 P
ort S
witc
h
16 Port Switch
16 Port Switch
16 Port Switch
16 Port Switch
16 Port Switch
16 Port Switch
16 Port Switch
16 Port Switch
16 P
ort S
witc
h
16 P
ort S
witc
h
16 P
ort S
witc
h
16 P
ort S
witc
h
16 P
ort S
witc
h
16 P
ort S
witc
h
16 P
ort S
witc
h
16 P
ort S
witc
h
Mesh-Based Switch
A
B
C
D
© 2006 Cisco Systems, Inc. All rights reserved. Cisco Confidential 58Design & Architecture of Data Center Network Platforms
16 Port Switch
16 Port Switch
16 Port Switch
16 Port Switch
16 P
ort S
witc
h
16 P
ort S
witc
h
16 P
ort S
witc
h
16 P
ort S
witc
h
16 P
ort S
witc
h
16 P
ort S
witc
h
16 P
ort S
witc
h
16 P
ort S
witc
h
16 Port Switch
16 Port Switch
16 Port Switch
16 Port Switch
16 Port Switch
16 Port Switch
16 Port Switch
16 Port Switch
16 P
ort S
witc
h
16 P
ort S
witc
h
16 P
ort S
witc
h
16 P
ort S
witc
h
16 P
ort S
witc
h
16 P
ort S
witc
h
16 P
ort S
witc
h
16 P
ort S
witc
h
A
B
C
D
E
F
Mesh Based Switch
2:1 Over-Subscription2:1 Over-Subscription
G
H
© 2006 Cisco Systems, Inc. All rights reserved. Cisco Confidential 59Design & Architecture of Data Center Network Platforms
16 Port Switch
16 Port Switch
16 Port Switch
16 Port Switch
16 P
ort S
witc
h
16 P
ort S
witc
h
16 P
ort S
witc
h
16 P
ort S
witc
h
16 P
ort S
witc
h
16 P
ort S
witc
h
16 P
ort S
witc
h
16 P
ort S
witc
h
16 Port Switch
16 Port Switch
16 Port Switch
16 Port Switch
16 Port Switch
16 Port Switch
16 Port Switch
16 Port Switch
16 P
ort S
witc
h
16 P
ort S
witc
h
16 P
ort S
witc
h
16 P
ort S
witc
h
16 P
ort S
witc
h
16 P
ort S
witc
h
16 P
ort S
witc
h
16 P
ort S
witc
h
Mesh-Based Switch
A
B
© 2006 Cisco Systems, Inc. All rights reserved. Cisco Confidential 60Design & Architecture of Data Center Network Platforms
16 Port Switch
16 Port Switch
16 Port Switch
16 Port Switch
16 P
ort S
witc
h
16 P
ort S
witc
h
16 P
ort S
witc
h
16 P
ort S
witc
h
16 P
ort S
witc
h
16 P
ort S
witc
h
16 P
ort S
witc
h
16 P
ort S
witc
h
16 Port Switch
16 Port Switch
16 Port Switch
16 Port Switch
16 Port Switch
16 Port Switch
16 Port Switch
16 Port Switch
16 P
ort S
witc
h
16 P
ort S
witc
h
16 P
ort S
witc
h
16 P
ort S
witc
h
16 P
ort S
witc
h
16 P
ort S
witc
h
16 P
ort S
witc
h
16 P
ort S
witc
h
Mesh-Based Switch
AC
BD
2:1 Over-Subscription2:1 Over-Subscription
© 2006 Cisco Systems, Inc. All rights reserved. Cisco Confidential 61Design & Architecture of Data Center Network Platforms
16 Port Switch
16 Port Switch
16 Port Switch
16 Port Switch
16 P
ort S
witc
h
16 P
ort S
witc
h
16 P
ort S
witc
h
16 P
ort S
witc
h
16 P
ort S
witc
h
16 P
ort S
witc
h
16 P
ort S
witc
h
16 P
ort S
witc
h
16 Port Switch
16 Port Switch
16 Port Switch
16 Port Switch
16 Port Switch
16 Port Switch
16 Port Switch
16 Port Switch
16 P
ort S
witc
h
16 P
ort S
witc
h
16 P
ort S
witc
h
16 P
ort S
witc
h
16 P
ort S
witc
h
16 P
ort S
witc
h
16 P
ort S
witc
h
16 P
ort S
witc
h
A
B
C
D
Mesh-Based Switch
E
F
G
H
4:1 Over-Subscription4:1 Over-Subscription
© 2006 Cisco Systems, Inc. All rights reserved. Cisco Confidential 62Design & Architecture of Data Center Network Platforms
16 Port Switch
16 Port Switch
16 Port Switch
16 Port Switch
16 P
ort S
witc
h
16 P
ort S
witc
h
16 P
ort S
witc
h
16 P
ort S
witc
h
16 P
ort S
witc
h
16 P
ort S
witc
h
16 P
ort S
witc
h
16 P
ort S
witc
h
16 Port Switch
16 Port Switch
16 Port Switch
16 Port Switch
16 Port Switch
16 Port Switch
16 Port Switch
16 Port Switch
16 P
ort S
witc
h
16 P
ort S
witc
h
16 P
ort S
witc
h
16 P
ort S
witc
h
16 P
ort S
witc
h
16 P
ort S
witc
h
16 P
ort S
witc
h
16 P
ort S
witc
h
A
B
C
D
Mesh-Based Switch
E
F
G
H
4:1 Over-Subscription4:1 Over-SubscriptionA
B
C
D
E
F
G2:1 Over-Subscription2:1 Over-Subscription
H
ACBD
1:1 Non-Blocking1:1 Non-Blocking
© 2006 Cisco Systems, Inc. All rights reserved. Cisco Confidential 63Design & Architecture of Data Center Network Platforms
Part II—Architectures
Centralized MemorySwitch on Chip / “System on Chip” (SoC)
BusMesh
Centralized memory in a mesh
Two-Tier / Banyan / ClosCentralized memory in two tiers
Cross-Bar
© 2006 Cisco Systems, Inc. All rights reserved. Cisco Confidential 64Design & Architecture of Data Center Network Platforms
Two-Tier a.k.a. Banyan a.k.a. Clos
4 PortSW
4 PortSW
4 PortSW
4 PortSW
4 PortSW
4 PortSW
4 PortSW
4 PortSW
4 PortSW
4 PortSW
4 PortSW
4 PortSW
8 Port Switch
© 2006 Cisco Systems, Inc. All rights reserved. Cisco Confidential 65Design & Architecture of Data Center Network Platforms
Two-Tier
8 PortSW
8 PortSW
8 PortSW
8 PortSW
8 PortSW
8 PortSW
8 PortSW
8 PortSW
8 PortSW
8 PortSW
8 PortSW
8 PortSW
8 PortSW
8 PortSW
8 PortSW
8 PortSW
8 PortSW
8 PortSW
8 PortSW
8 PortSW
8 PortSW
8 PortSW
8 PortSW
8 PortSW
32 Port Switch
© 2006 Cisco Systems, Inc. All rights reserved. Cisco Confidential 66Design & Architecture of Data Center Network Platforms
8 Port Switch
Two-Tier
4 PortSW
4 PortSW
4 PortSW
4 PortSW
4 PortSW
4 PortSW
4 PortSW
4 PortSW
4 PortSW
4 PortSW
4 PortSW
4 PortSW
© 2006 Cisco Systems, Inc. All rights reserved. Cisco Confidential 67Design & Architecture of Data Center Network Platforms
Two-Tier
In 1
In 2
In 3
In 4
Out 1
Out 2
Out 3
Out 4
4 Po
rtSW4 Po
rtSW
4 Po
rtSW4 Po
rtSW
4 Po
rtSW4 Po
rtSW
4 Po
rtSW4 Po
rtSW
4 Po
rtSW4 Po
rtSW
4 Po
rtSW4 Po
rtSW
8 Port Switch
© 2006 Cisco Systems, Inc. All rights reserved. Cisco Confidential 68Design & Architecture of Data Center Network Platforms
In 1
In 2
In 3
In 4
Out 1
Out 2
Out 3
Out 4
4 Po
rtSW4 Po
rtSW
4 Po
rtSW4 Po
rtSW
4 Po
rtSW4 Po
rtSW
4 Po
rtSW4 Po
rtSW
4 Po
rtSW4 Po
rtSW
4 Po
rtSW4 Po
rtSW
8 Port Switch
Two-Tier
© 2006 Cisco Systems, Inc. All rights reserved. Cisco Confidential 69Design & Architecture of Data Center Network Platforms
In 1
In 2
In 3
In 4
Out 1
Out 2
Out 3
Out 4
4 Po
rtSW4 Po
rtSW
4 Po
rtSW4 Po
rtSW
4 Po
rtSW4 Po
rtSW
4 Po
rtSW4 Po
rtSW
4 Po
rtSW4 Po
rtSW
4 Po
rtSW4 Po
rtSW
8 Port Switch
Two-Tier
© 2006 Cisco Systems, Inc. All rights reserved. Cisco Confidential 70Design & Architecture of Data Center Network Platforms
In 1
In 2
In 3
In 4
Out 1
Out 2
Out 3
Out 4
4 Po
rtSW4 Po
rtSW
4 Po
rtSW4 Po
rtSW
4 Po
rtSW4 Po
rtSW
4 Po
rtSW4 Po
rtSW
4 Po
rtSW4 Po
rtSW
4 Po
rtSW4 Po
rtSW
8 Port Switch
Two-Tier
Blocking
Rearrangeably non-blockingRearrangeably non-blocking
© 2006 Cisco Systems, Inc. All rights reserved. Cisco Confidential 71Design & Architecture of Data Center Network Platforms
In 1
In 2
In 3
In 4
Out 1
Out 2
Out 3
Out 4
4 Po
rtSW4 Po
rtSW
4 Po
rtSW4 Po
rtSW
4 Po
rtSW4 Po
rtSW
4 Po
rtSW4 Po
rtSW
4 Po
rtSW4 Po
rtSW
4 Po
rtSW4 Po
rtSW
8 Port Switch
Two-Tier
Rearrangeably non-blockingRearrangeably non-blocking
© 2006 Cisco Systems, Inc. All rights reserved. Cisco Confidential 72Design & Architecture of Data Center Network Platforms
In 1
In 2
In 3
In 4
Out 1
Out 2
Out 3
Out 4
4 Po
rtSW4 Po
rtSW
4 Po
rtSW4 Po
rtSW
4 Po
rtSW4 Po
rtSW
4 Po
rtSW4 Po
rtSW
4 Po
rtSW4 Po
rtSW
4 Po
rtSW4 Po
rtSW
8 Port Switch
Two-Tier
© 2006 Cisco Systems, Inc. All rights reserved. Cisco Confidential 73Design & Architecture of Data Center Network Platforms
In 1
In 2
In 3
In 4
Out 1
Out 2
Out 3
Out 4
4 Po
rtSW4 Po
rtSW
4 Po
rtSW4 Po
rtSW
4 Po
rtSW4 Po
rtSW
4 Po
rtSW4 Po
rtSW
4 Po
rtSW4 Po
rtSW
4 Po
rtSW4 Po
rtSW
8 Port Switch
Two-Tier
2
© 2006 Cisco Systems, Inc. All rights reserved. Cisco Confidential 74Design & Architecture of Data Center Network Platforms
In 1
In 2
In 3
In 4
Out 1
Out 2
Out 3
Out 4
4 Po
rtSW4 Po
rtSW
4 Po
rtSW4 Po
rtSW
4 Po
rtSW4 Po
rtSW
4 Po
rtSW4 Po
rtSW
4 Po
rtSW4 Po
rtSW
4 Po
rtSW4 Po
rtSW
8 Port Switch
Two-Tier
2
© 2006 Cisco Systems, Inc. All rights reserved. Cisco Confidential 75Design & Architecture of Data Center Network Platforms
In 1
In 2
In 3
In 4
Out 1
Out 2
Out 3
Out 4
4 Po
rtSW4 Po
rtSW
4 Po
rtSW4 Po
rtSW
4 Po
rtSW4 Po
rtSW
4 Po
rtSW4 Po
rtSW
4 Po
rtSW4 Po
rtSW
4 Po
rtSW4 Po
rtSW
8 Port Switch
Two-Tier
2
© 2006 Cisco Systems, Inc. All rights reserved. Cisco Confidential 76Design & Architecture of Data Center Network Platforms
Two-Tier-Based Switch—64 Ports
© 2003, Cisco Systems, Inc. All rights reserved.
© 2006 Cisco Systems, Inc. All rights reserved. Cisco Confidential 77Design & Architecture of Data Center Network Platforms
Two-Tier-Based Switch—128 Ports
© 2006 Cisco Systems, Inc. All rights reserved. Cisco Confidential 78Design & Architecture of Data Center Network Platforms
Part II—Architectures
Centralized MemorySwitch on Chip / “System on Chip” (SoC)
BusMesh
Centralized memory in a mesh
Two-Tier / Banyan / ClosCentralized memory in two tiers
Cross-Bar
© 2006 Cisco Systems, Inc. All rights reserved. Cisco Confidential 79Design & Architecture of Data Center Network Platforms
Cross-Bar
3
2
1
Inpu
t 1
Out
put 1
Out
put 2
Out
put 3
Inpu
t 2In
put 3
Cross-Bar
© 2006 Cisco Systems, Inc. All rights reserved. Cisco Confidential 80Design & Architecture of Data Center Network Platforms
Inpu
t 1
Out
put 1
Out
put 2
Out
put 3
Inpu
t 2In
put 3
Cross-Bar
Cross-Bar
3
2
1
© 2006 Cisco Systems, Inc. All rights reserved. Cisco Confidential 81Design & Architecture of Data Center Network Platforms
3
2
1
Cross-BarIn
put 1
Out
put 1
Out
put 2
Out
put 3
Inpu
t 2In
put 3
Cross-Bar
© 2006 Cisco Systems, Inc. All rights reserved. Cisco Confidential 82Design & Architecture of Data Center Network Platforms
Cross-BarIn
put 1
Out
put 1
Out
put 2
Out
put 3
Inpu
t 2In
put 3
Cross-Bar
© 2006 Cisco Systems, Inc. All rights reserved. Cisco Confidential 83Design & Architecture of Data Center Network Platforms
Cross-Bar
In 1In 2In 3In 4In 5In 6In 7In 8
In 1In 2In 3In 4In 5In 6In 7In 8
In 1In 2In 3In 4In 5In 6In 7In 8
Out 1Out 2Out 3Out 4Out 5Out 6Out 7Out 8
Out 1Out 2Out 3Out 4Out 5Out 6Out 7Out 8
Out 1Out 2Out 3Out 4Out 5Out 6Out 7Out 8
Cross-Bar
© 2006 Cisco Systems, Inc. All rights reserved. Cisco Confidential 84Design & Architecture of Data Center Network Platforms
Cross-Bar
In 1In 2In 3In 4In 5In 6In 7In 8
In 1In 2In 3In 4In 5In 6In 7In 8
In 1In 2In 3In 4In 5In 6In 7In 8
Out 1Out 2Out 3Out 4Out 5Out 6Out 7Out 8
Out 1Out 2Out 3Out 4Out 5Out 6Out 7Out 8
Out 1Out 2Out 3Out 4Out 5Out 6Out 7Out 8
Cross-Bar
© 2006 Cisco Systems, Inc. All rights reserved. Cisco Confidential 85Design & Architecture of Data Center Network Platforms
Cross-Bar
In j
Out k
SwitchSwitch
© 2006 Cisco Systems, Inc. All rights reserved. Cisco Confidential 86Design & Architecture of Data Center Network Platforms
Cross-Bar
In 1
In 2
In 3
In n
In 4
Out 1 Out 2 Out 3 Out 4 Out n
© 2006 Cisco Systems, Inc. All rights reserved. Cisco Confidential 87Design & Architecture of Data Center Network Platforms
In 1
In 2
In 3
In n
In 4
Out 1 Out 2 Out 3 Out 4 Out n
Cross-Bar
3
2
1
4
n
3 21 4 n
© 2006 Cisco Systems, Inc. All rights reserved. Cisco Confidential 88Design & Architecture of Data Center Network Platforms
Advantages of Cross-Bar
ScalableMulti-speed capableNon-blockingEfficient
Cross-BarCross-Bar
Line CardLine Card Line CardLine Card Line CardLine Card Line CardLine Card Line CardLine Card
© 2006 Cisco Systems, Inc. All rights reserved. Cisco Confidential
Design & Architecture of Data Center Network Platforms 89
Part III – Real World Implementation
© 2006 Cisco Systems, Inc. All rights reserved. Cisco Confidential 90Design & Architecture of Data Center Network Platforms
Part III—Real World Implementation
Cross-bar PerformanceClassic crossbar vs. Buffered Crossbar
Virtual Output QueuingHead-of-Line Blocking
Completing the SwitchOther Features
© 2006 Cisco Systems, Inc. All rights reserved. Cisco Confidential 91Design & Architecture of Data Center Network Platforms
Part III—Real World Implementation
Cross-bar PerformanceClassic crossbar vs. Buffered Crossbar
Virtual Output QueuingHead-of-Line Blocking
Completing the SwitchOther Features
© 2006 Cisco Systems, Inc. All rights reserved. Cisco Confidential 92Design & Architecture of Data Center Network Platforms
Cross-Bar Model
© 2006 Cisco Systems, Inc. All rights reserved. Cisco Confidential 93Design & Architecture of Data Center Network Platforms
Performance Issue with Cross-Bars
Source: M. J. Karol, M.G. Hluchyj, S. P. Morgan, “Input Versus Output Queueing [sic] on a Space-Division Packet Switch”, IEEE Transactions on Communications, Vol COM-35, No 12, December 1987, page 1353
58.6%
© 2006 Cisco Systems, Inc. All rights reserved. Cisco Confidential 94Design & Architecture of Data Center Network Platforms
Cat6K & MDS Family Cross-Bar
ClassicCross-Bar2.2Tbps
ClassicCross-Bar2.2Tbps
FIFO 1FIFO 1
FIFO 2FIFO 2
FIFO 3FIFO 3
FIFO 18FIFO 18
20/23.07 GbpsEach
60/69.21 GbpsEach
FIFO 1FIFO 1
FIFO 2FIFO 2
FIFO 3FIFO 3
FIFO 18FIFO 18
60/69.21 GbpsEach
20/23.07 GbpsEach
Buffered Cross-Bar with 3X Over-SpeedSingleDevice
© 2006 Cisco Systems, Inc. All rights reserved. Cisco Confidential 95Design & Architecture of Data Center Network Platforms
Part III—Real World Implementation
Cross-bar PerformanceClassic crossbar vs. Buffered Crossbar
Virtual Output QueuingHead-of-Line Blocking
Completing the SwitchOther Features
© 2006 Cisco Systems, Inc. All rights reserved. Cisco Confidential 96Design & Architecture of Data Center Network Platforms
Buffering—Input Buffering
© 2006 Cisco Systems, Inc. All rights reserved. Cisco Confidential 97Design & Architecture of Data Center Network Platforms
Head of Line Blocking
Stadium
Beer/Soda/Chips
Kwiky Mart
© 2006 Cisco Systems, Inc. All rights reserved. Cisco Confidential 98Design & Architecture of Data Center Network Platforms
Head of Line Blocking
Stadium
Beer/Soda/Chips
Kwiky Mart
© 2006 Cisco Systems, Inc. All rights reserved. Cisco Confidential 99Design & Architecture of Data Center Network Platforms
Stadium
Output Queuing
Beer/Soda/Chips
Kwiky Mart
© 2006 Cisco Systems, Inc. All rights reserved. Cisco Confidential 100Design & Architecture of Data Center Network Platforms
Head of Line Blocking
A
B
C
B
© 2006 Cisco Systems, Inc. All rights reserved. Cisco Confidential 101Design & Architecture of Data Center Network Platforms
Head of Line Blocking
A
B
C
C
© 2006 Cisco Systems, Inc. All rights reserved. Cisco Confidential 102Design & Architecture of Data Center Network Platforms
Head of Line Blocking
A
B
C
A
© 2006 Cisco Systems, Inc. All rights reserved. Cisco Confidential 103Design & Architecture of Data Center Network Platforms
Head of Line Blocking
A
B
C
ABC?
© 2006 Cisco Systems, Inc. All rights reserved. Cisco Confidential 104Design & Architecture of Data Center Network Platforms
Virtual Output Queues
A
B
C
ARB
B F
© 2006 Cisco Systems, Inc. All rights reserved. Cisco Confidential 105Design & Architecture of Data Center Network Platforms
Virtual Output Queues
A
B
C
ARB
C F
© 2006 Cisco Systems, Inc. All rights reserved. Cisco Confidential 106Design & Architecture of Data Center Network Platforms
AACBA
Virtual Output Queues
A
B
C
ARB
F
© 2006 Cisco Systems, Inc. All rights reserved. Cisco Confidential 107Design & Architecture of Data Center Network Platforms
Virtual Output Queues
F
F
FInput Port 1
Input Port 2
Input Port 3
Output Port 1
Avoids Head of line Blocking
Output Port 2
Output Port 3
© 2006 Cisco Systems, Inc. All rights reserved. Cisco Confidential 108Design & Architecture of Data Center Network Platforms
Virtual Output Queues
F
F
F
Larger Effective Buffers
Input Port 1
Input Port 2
Input Port 3
Output Port 1
Output Port 2
Output Port 3
© 2006 Cisco Systems, Inc. All rights reserved. Cisco Confidential 109Design & Architecture of Data Center Network Platforms
Virtual Output Queues
F
F
F
Better Isolation of Congestion Sources
Input Port 1
Input Port 2
Input Port 3
Output Port 1
Output Port 2
Output Port 3
© 2006 Cisco Systems, Inc. All rights reserved. Cisco Confidential 110Design & Architecture of Data Center Network Platforms
Virtual Output Queues
F
F
F
Better Isolation of Congested Outputs
Input Port 1
Input Port 2
Input Port 3
Output Port 1
Output Port 2
Output Port 3
© 2006 Cisco Systems, Inc. All rights reserved. Cisco Confidential 111Design & Architecture of Data Center Network Platforms
Virtual Output Queues
F
F
F
Better Isolation of Congestion In General
Input Port 1
Input Port 2
Input Port 3
Output Port 1
Output Port 2
Output Port 3
© 2006 Cisco Systems, Inc. All rights reserved. Cisco Confidential 112Design & Architecture of Data Center Network Platforms
Virtual Output Queues
F
F
FInput Port 1
Input Port 2
Input Port 3
Output Port 1
Output Port 2
Output Port 3
Buffer Bandwidth Matched to Port Bandwidth
© 2006 Cisco Systems, Inc. All rights reserved. Cisco Confidential 113Design & Architecture of Data Center Network Platforms
Virtual Output Queuing - Summary
PROAvoids Head-of-line blockingMore efficient buffer usageBetter traffic isolationBuffer bandwidth matched to port bandwidthNecessary for lossless networks.(e.g. Fibre Channel, Infiniband, ..)
CONComplex to implementLocks in scalability
© 2006 Cisco Systems, Inc. All rights reserved. Cisco Confidential 114Design & Architecture of Data Center Network Platforms
Part III—Real World Implementation
Cross-bar PerformanceClassic crossbar vs. Buffered Crossbar
Virtual Output QueuingHead-of-Line Blocking
Completing the SwitchOther Features
© 2006 Cisco Systems, Inc. All rights reserved. Cisco Confidential 115Design & Architecture of Data Center Network Platforms
Completing the Swtich
FWD
FWD
FWD
ARB
© 2006 Cisco Systems, Inc. All rights reserved. Cisco Confidential 116Design & Architecture of Data Center Network Platforms
Completing the Swtich
ARB
FWDMACMAC
FWDMACMAC
FWDMACMAC
MACMAC
MACMAC
MACMAC
© 2006 Cisco Systems, Inc. All rights reserved. Cisco Confidential 117Design & Architecture of Data Center Network Platforms
Completing the Swtich
Port
ARB
FWDMACMAC
FWDMACMAC
FWDMACMAC
MACMAC
MACMAC
MACMAC
© 2006 Cisco Systems, Inc. All rights reserved. Cisco Confidential 118Design & Architecture of Data Center Network Platforms
Re-Draw
MACMAC MACMAC MACMAC
ARB
FWD FWD FWD
© 2006 Cisco Systems, Inc. All rights reserved. Cisco Confidential 119Design & Architecture of Data Center Network Platforms
Larger Switch
EthFlashCard Console
Cross-Bar
Supervisor/Fabric
Back Plane
Arb
I/F uP
P P P
M M M
F
Q
F
Q
F
Q
I/F uP
P P P
M M M
F
Q
F
Q
F
Q
I/F uP
P P P
M M M
F
Q
F
Q
F
Q
I/F uP
P P P
M M M
F
Q
F
Q
F
Q
uP
© 2006 Cisco Systems, Inc. All rights reserved. Cisco Confidential 120Design & Architecture of Data Center Network Platforms
Larger Switch
MACPort Speeds:• 1 / 2 / 4 / 10GbpsPort Type / Modes:• F_Port,• Loop port (FL_Port),• Private Loop (TL_Port),• ISL (E_Port),• EISL (TE_Port),• SPAN Port (SD_Port)
PHY / MAC ForwardingQueuing
PHYOptical to Electrical interface:• SFP (1 / 2 / 4 Gbps)• X2 (10Gbps) optics
• Error-Check ingress frame (CRC etc)• Ingress rate-limiting• Prepend switch-internal header
(Ingress port, Ingress VSAN, QoS bits)• Timestamp ingress frame arrival time
• Transmit, loopback or Expire framebased on ingress timestamp
• Remove switch-internal header from frame• Prepend EISL header onto frame where
egress port is a Trunked E_Port (TE_Port)
• Per-VSAN Ingress ACLs (Hard Zoning)• Per-VSAN Forwarding (Fibre Channel, MPLS)• Load-balancing decision (determine finalegress interface where destination is aPortChannel or equal-cost FSPF route)
• Inter VSAN Routing (Fibre Channel NAT)• Update src/dst / frame/byte statistics
• Virtual Output Queuing• Frame Buffering• Quality of Service (QoS)• Arbitration and Fairness• Congestion Control (FCC)
• Per-VSAN Egress ACLs (Hard Zoning)including LUN Zoning & read-only Zones
• Re-Validate frame CRC• Inter VSAN Routing (Fibre Channel NAT)• Quality of Service (egress QoS)
CentralArbiter
CrossbarSwitchFabric
CrossbarSwitchFabric
CentralArbiter
Dual Redundant Centralized Arbiters(capable of schedulingmore than 5 billionframes per second)
• Dual Redundant crossbar switch fabrics providehigh-speed non-blocking non-oversubscribed capacityfrom any slot to any slot (25 Gbps rawcapacity per channel, 4 channels per linecard)
• Crossbar operates 3X over-speed (75 Gbps) andutilizes superframing to improves crossbar efficiency
© 2006 Cisco Systems, Inc. All rights reserved. Cisco Confidential 121Design & Architecture of Data Center Network Platforms
Part III—Real World Implementation
Cross-bar PerformanceClassic crossbar vs. Buffered Crossbar
Virtual Output QueuingHead-of-Line Blocking
Completing the SwitchOther Features
© 2006 Cisco Systems, Inc. All rights reserved. Cisco Confidential 122Design & Architecture of Data Center Network Platforms
Other Features
Bridging and RoutingSVIsMulticastBroadcastACLs (+VACLs)SPAN + RSPANSecurity (e.g. 802.1x)RedundancyL4-7 Deep Packet Inspection
NATMPLS*PLSGREVirtualizationVirtual Router Instances Services / Service ModulesRoles / RBACDiagnostics(insert here…)
© 2006 Cisco Systems, Inc. All rights reserved. Cisco Confidential
Design & Architecture of Data Center Network Platforms 123
Part IV – The Future
© 2006 Cisco Systems, Inc. All rights reserved. Cisco Confidential 124Design & Architecture of Data Center Network Platforms
Future Data Center considerations
Power & Heat becoming the primary considerationDensity of servers rapidly shrinking;Server density + Server capacity typically growing >3x faster than associated Cooling/AC infrastructure
Moore’s LawToday’s Commodity CPUs were yesterday’s Supercomputers
VirtualizationServer, Network, Storage, …
Future Data Center NetworkAt most, dual server connectionsHigher Speeds, Lower Latency100 Mbps GbE 10GbE 100+GbE?Infrastructure Simplification
© 2006 Cisco Systems, Inc. All rights reserved. Cisco Confidential 125Design & Architecture of Data Center Network Platforms
Future Data Center considerations
Power & Heat becoming the primary considerationDensity of servers rapidly shrinking;Server density + Server capacity typically growing >3x faster than associated Cooling/AC infrastructure
Moore’s LawToday’s Commodity CPUs were yesterday’s Supercomputers
VirtualizationServer, Network, Storage, …
Future Data Center NetworkAt most, dual server connectionsHigher Speeds, Lower Latency100 Mbps GbE 10GbE 100+GbE?Infrastructure Simplification
© 2006 Cisco Systems, Inc. All rights reserved. Cisco Confidential 126Design & Architecture of Data Center Network Platforms
Data Center Physical InfrastructureDesire is to Last 10 to 15 Years
Useful life of a Data Center facility is expected to exceed the life of the gear it contains due to the tremendous cost of relocating or renovating data center facilities
Yr 5 Yr 10 Yr 15
2 to 3 Network Infrastructure Upgrades
5 to 7 Server Upgrade CyclesMainframes 1/2RU Servers Multi-Core CPU’s Blade Server
10/100 10/100/1000 with L4-7 Services 10GbE Attached hosts
© 2006 Cisco Systems, Inc. All rights reserved. Cisco Confidential 127Design & Architecture of Data Center Network Platforms
Planning and Coordination Between IT Staff and Facilities Managers is a Must
Obtaining 10-15 Years of Useful Life Requires Planning for Growth of Power and Cooling
Yr 5 Yr 10 Yr 15
2 to 3 Network Infrastructure Upgrades
5 to 7 Server Upgrade CyclesMainframes 1/2RU Servers Multi-Core CPUs Blade Server
10/100 10/100/1000 with L4-7 Services 10GbE Attached hostsConverged LAN/SAN/IPC
2kW/Rack 6kW/Rack 15+kW/Rack!
© 2006 Cisco Systems, Inc. All rights reserved. Cisco Confidential 128Design & Architecture of Data Center Network Platforms
Future Data Center considerations
Power & Heat becoming the primary considerationDensity of servers rapidly shrinking;Server density + Server capacity typically growing >3x faster than associated Cooling/AC infrastructure
Moore’s LawToday’s Commodity CPUs were yesterday’s Supercomputers
VirtualizationServer, Network, Storage, …
Future Data Center NetworkAt most, dual server connectionsHigher Speeds, Lower Latency100 Mbps GbE 10GbE 100+GbE?Infrastructure Simplification
© 2006 Cisco Systems, Inc. All rights reserved. Cisco Confidential 129Design & Architecture of Data Center Network Platforms
1978: 8088: 20K transistors1985: Intel® 386™: 275K transistors1989: Intel® 486™: 1.2M transistors1993: Intel® Pentium™: 3.1M transistors1999: Intel® Pentium 3™: 9.5M transistors2000: Intel® Pentium 4™: 42M transistors2006: Intel® Core™2 Duo: 291M transistors
Moore’s LawAn empirical observation“The transistor density of Integrated Circuits,with respect to minimum component cost,will double every 24 months.”― Gordon E Moore, co-founder, Intel, 1965
© 2006 Cisco Systems, Inc. All rights reserved. Cisco Confidential 130Design & Architecture of Data Center Network Platforms
Compute Capacity is IncreasingNetwork Challenges & Implications
Top 500 Super Computer (reported)
-1
0
1
2
3
4
5
6
1993 1994 1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006
Year
log
(gig
aflo
ps)
#1
#500
Growth is not slowing downMore effective way to scale layer 2 networks is neededReady access to compute AND storage resourcesPowering 10GbE over Copper PHYs on high density linecards (40+ ports)Impact of an outage/downtimewill be magnified- maximizeoperational uptime and cluster utilizationLeast disruptive upgrade path – hardware, software& services
© 2006 Cisco Systems, Inc. All rights reserved. Cisco Confidential 131Design & Architecture of Data Center Network Platforms
Future Data Center considerations
Power & Heat becoming the primary considerationSize of servers rapidly shrinking;Server density + Server capacity typically growing >3x faster than associated Cooling/AC infrastructure
Moore’s LawToday’s Commodity CPUs were yesterday’s Supercomputers
VirtualizationServer, Network, Storage, …
Future Data Center NetworkAt most, dual server connectionsHigher Speeds, Lower Latency100 Mbps GbE 10GbE 100+GbE?Infrastructure Simplification
© 2006 Cisco Systems, Inc. All rights reserved. Cisco Confidential
Design & Architecture of Data Center Network Platforms 132
VIRTUALIZATIONThe capability to decouple logical functions from physical devices.
Increase resource usability –Lower CAPEXLoosely couple re-usable functions –FlexibilityDynamic allocation of virtual instances –AutomationCentralized policy Management –Lower TCODistributed Capabilities –Broad use of functions
© 2006 Cisco Systems, Inc. All rights reserved. Cisco Confidential 133Design & Architecture of Data Center Network Platforms
Data Center VirtualizationExamples
Type Example Benefit
Pooling
802.3ad, Port ChannelTCP Connection PoolingSymmetric Multi-processingOracle Cache Fusion
Improved availabilityCost effective scalingSimplify topologies and configuration
PartitioningVLAN, VSANVirtual Firewall, ACEVMWare
Increased utilizationReduce number of devicesSegmentation of policy/ traffic
AbstractionHSRP/VRRP, GLBP, WWNSVIP/NATvFrame
Simplify provisioningProvision ACROSS technology silos
© 2006 Cisco Systems, Inc. All rights reserved. Cisco Confidential 134Design & Architecture of Data Center Network Platforms
Evolution in Data Center Computing
• Compute, Storage and Interconnect Resources PhysicallyIntegrated
• Few Applications
• Proprietary Architecture
• Minimal Connectivity Requirements
• Compute, Storage and Interconnect Resources PhysicallyIntegrated
• Few Applications
• Proprietary Architecture
• Minimal Connectivity Requirements
• Reduction Apps Development Cost
• 1st Gen CPU’s not very powerful
• Client/Server Model
• Reduction Apps Development Cost
• 1st Gen CPU’s not very powerful
• Client/Server Model
• DAS migration to NAS, SAN, FC-SAN, VSAN
• iSCSI
• DAS migration to NAS, SAN, FC-SAN, VSAN
• iSCSI
• PCI, PCIe, PCIx• Emergence of
Blade Servers• Compute nodes
increasingly more powerful
• PCI, PCIe, PCIx• Emergence of
Blade Servers• Compute nodes
increasingly more powerful
• Location of Servers Dynamic
• Multiple Apps per CPU
• Location of Servers Dynamic
• Multiple Apps per CPU
Mainframe and SMP
x86 uprocessor
Clusters/ multi-core
CPU’sVirtual
MachinesUtility
Computing
• Clustering Technologies
• TCP/SSL Offload• VLAN’s and L3• VTP and VLAN
pruning• ACL’s and
Security
• Clustering Technologies
• TCP/SSL Offload• VLAN’s and L3• VTP and VLAN
pruning• ACL’s and
Security
• Myrinet and IB as extension of PCI
• QoS• 10/100 ->
10/100/1000 -> 10GbE
• Myrinet and IB as extension of PCI
• QoS• 10/100 ->
10/100/1000 -> 10GbE
• Large, flat L2 domains
• L4-7 service chaining (ACE)
• Services Centric Provisioning Model (vFrame)
• Large, flat L2 domains
• L4-7 service chaining (ACE)
• Services Centric Provisioning Model (vFrame)
• 4G and 10G FC• Parallel File
Systems with SAN Backend
• iSLB
• 4G and 10G FC• Parallel File
Systems with SAN Backend
• iSLB
• VSANs with Hard-zoning
• Dynamic SAN environment
• VSANs with Hard-zoning
• Dynamic SAN environment
• Compute, Storage and Interconnect ResourcesLogicallyIntegrated
• Geographic Separation of Resources
• Large # of Applications
• On Demand Service Provisioning
• Compute, Storage and Interconnect ResourcesLogicallyIntegrated
• Geographic Separation of Resources
• Large # of Applications
• On Demand Service Provisioning
ComputeCompute
LANLAN
SANSAN
© 2006 Cisco Systems, Inc. All rights reserved. Cisco Confidential 135Design & Architecture of Data Center Network Platforms
Future Data Center considerations
Power & Heat becoming the primary considerationDensity of servers rapidly shrinking;Server density + Server capacity typically growing >3x faster than associated Cooling/AC infrastructure
Moore’s LawToday’s Commodity CPUs were yesterday’s Supercomputers
VirtualizationServer, Network, Storage, …
Future Data Center networkAt most, dual server connectionsHigher Speeds, Lower Latency100 Mbps GbE 10GbE 100+GbE?Infrastructure Simplification
© 2006 Cisco Systems, Inc. All rights reserved. Cisco Confidential 136Design & Architecture of Data Center Network Platforms
At most, dual server connections
Today’s infrastructure
ServerProcesssor + Memory
LAN
I/O
SAN
I/O
IPC
I/O
Tomorrow’s infrastructure
ServerProcesssor + Memory
Converged I/O
LAN + SAN + IPC
I/O Consolidation simplifies platform architectures,reducing overall cost
© 2006 Cisco Systems, Inc. All rights reserved. Cisco Confidential 137Design & Architecture of Data Center Network Platforms
1995: Catalyst 5000 introduced:16K MAC addresses192KB buffering / 8 ports (24KB/port) (10Mbps, 10/100Mbps)
1999: Catalyst 6000 introduced:128K MAC addresses (32K-80K usable)512KB/port buffering (GbE port)
2003: Catalyst 6500 Series 4 x 10GbE modules18MB/port buffering (10GbE ports)
2006: Catalyst 6500 Series 8 x 10GbE module200MB/port buffering (10GbE ports)Metro Ethernet environments demanding >128K MAC addresses
Growing demands of Ethernet
© 2006 Cisco Systems, Inc. All rights reserved. Cisco Confidential 138Design & Architecture of Data Center Network Platforms
CAM tables (L2), ACLs, QoS, CEF FIB tables (L3), Statistics all typically built using Tenary CAMs
Tenary CAM ≈ Static RAM (SRAM)–SRAM requires 6 transistors per bit–‘always ON’ (high power)
Growing demands of Ethernet
A six-transistor CMOS SRAM cell.source: Wikipedia
http://en.wikipedia.org/wiki/Static_random_access_memory
Transistor count for128K MAC addresses,CEF FIB, QoS, Statistics, ...
now typical to have over 200Mtransistors just in tables
comparison:Intel® Core™2 Duo “Extreme Edition”:291M transistors
© 2006 Cisco Systems, Inc. All rights reserved. Cisco Confidential 139Design & Architecture of Data Center Network Platforms
1978: 8088: 20K transistors1985: Intel® 386™: 275K transistors1989: Intel® 486™: 1.2M transistors1993: Intel® Pentium™: 3.1M transistors1999: Intel® Pentium 3™: 9.5M transistors2000: Intel® Pentium 4™: 42M transistors2006: Intel® Core™2 Duo: 291M transistors
Moore’s LawAn empirical observation“The transistor density of Integrated Circuits,with respect to minimum component cost,will double every 24 months.”― Gordon E Moore, co-founder, Intel, 1965
© 2006 Cisco Systems, Inc. All rights reserved. Cisco Confidential 140Design & Architecture of Data Center Network Platforms
Cisco CRS-1 Carrier Routing System
Heart of each linecard is the “Metro” ASIC
Programmable 40Gbps Network Processor192 Cores per chipFully programmablePowerPC-like instruction set
Programmability
© 2006 Cisco Systems, Inc. All rights reserved. Cisco Confidential 141Design & Architecture of Data Center Network Platforms
Present a Unified Fabric Comprising Multiple Technologies. Service vs Technology Virtualization
Increase Service/ Applications Deployment Velocity while Reducing Marginal Costs
Enable Layer 2 Networks to continue to scale beyond spanning tree limitations
Non-Intrusively Improve Applications Performance and Availability
Reduce the Number of Audit Points
Enable Per Business Unit and Per Application SLA’s
Support 24x7x365 operations
Cisco Data Center Network Design Goals
© 2006 Cisco Systems, Inc. All rights reserved. Cisco Confidential 142Design & Architecture of Data Center Network Platforms
Q and AQuestions?
© 2006 Cisco Systems, Inc. All rights reserved. Cisco Confidential 143Design & Architecture of Data Center Network Platforms
