View
3
Download
0
Category
Preview:
Citation preview
Advanced Software Engineering
Lecture 8: Data Centerby
Prof. Harold Liu
Agenda
IntroductionDesign and ConstructionManagement and MaintenanceHot Topics
Real‐time traffic analysisSDNWireless Data Center Networking
Core Elements of a Data Center
ApplicationsDatabases – Database Management System (DBMS) and the physical and logical storage of dataServers/Operating SystemsNetworksStorage Arrays
3
An Example
Consider an order processing system consisting of:Application for order entry.Database Management System (DBMS) to store customer and product information.Server/Operating System on which the Application and Database programs are run.Networks that provide
Connectivity between Clients and the Application/Database ServerConnectivity between the Server and the Storage system.
Storage Array
Local AreaNetwork
Storage Area Network
Storage Array
Client
Server
Application User
InterfaceDatabase
An Example ..Closer Look
A customer order is entered via the Application User Interface on a client.
Local AreaNetwork
Storage Area Network
ClientServer
Application User
Interface
Storage Array
An Example ..Closer Look
A customer order is entered via the Application User Interface on a clientThe client accesses the Server over a Local Area Network.
Storage Area Network
ClientServer
Storage Array
Local AreaNetwork
An Example ..Closer Look
A DBMS uses the operating system on the server to read and write this data to the physical location on a disk.
Storage Area Network
ClientServer
O/S and DBMS
Storage Array
Local AreaNetwork
An Example ..Closer Look
A DBMS uses the operating system on the server to read and write this data to the physical location on disk.A Network provides the communication link between the server and the storage array, and transports the read/write commands and data between the server and the storage array.
Storage Area Network
ClientServer
Storage Array
Local AreaNetwork
An Example ..Closer Look
A DBMS uses the operating system on the server to read and writethis data to the physical location on disk.A Network provides the communication link between the client andthe server, and transports the read/write commands and data between the server and the storage array.A storage array receives the read/write commands and data from the server and performs the necessary operations to store the data on the physical disks.
Storage Area Network
ClientServer
Storage Array
Local AreaNetwork
Database
Google Data Center
Google位于俄勒冈州的Dalles数据中心
at Lenoir, North Carolinahttp://www.google.com/about/datacenters/inside/streetview/
Galleryhttp://www.google.cn/about/datacenters/gallery/#/all
IBM Data Center
Equinix Data Center
Elements of a Data CenterThe SiteCommand CenterCable ManagementNetwork InfrastructureTerminal ServersEnvironmental ControlsPower
Standards ‐ 1
Standard ‐ 2
Structural Aspects
Structural Aspectsa raised floor ceilingBasement data center locations near water are not a good idea.Their must be a pathway for equipment to be moved in & out of the data center.Make sure the floor where the data center is to be located is rated for the estimated load.
Power and EnvironmentDual street power supply with UPSWhen necessary, a diesel generator as a second backup
Environmental Control FacilitiesTemperatureMoisturedust
HVAC (Heating, Ventilation, and Air Conditioning)
Hot‐Aisle and Cold‐Aisle Layout
Networking3‐layer architectureCore Layer Switch/RouterLayer‐2 SwitchInternal NIC Ethernet connection
DLink 48ports Gbps Layer-2 Switch
Cisco Core Layer Router
SecurityPhysical AccessLevels of AccessMonitoring
Past to Future
a single computer occupied the space of an entire Data Center. That same space can be occupied by thousands of servers today.
Let us Make it Online
HWSWRackDeployment and TestingMaintenance
ServersTower
Low cost, customizedSpace cost, not for parallel computing
Blade• save space, high computing ability• suitable for large‐size data center
Rack Server• small, easy to manage• need good cooling
Rack Unit (RU)Unit=44.45mm (height)Width=19‐inch or 23‐inch1U, 2U, 4U (half rack)
1U4U
Server Rack and Cluster
Data Center TopologyInternet
Servers
Layer-2 switchAccess
Data Center
Layer-2/3 switchAggregation
Layer-3 routerCore
Top of Rack Switch
Top of Rack (ToR)Switch
Data Center CostsJames Hamilton published basic 2008 breakdown
Servers: 45%CPU, memory, diskInfrastructure: 25%UPS, cooling, power distributionPower draw: 15%Electrical utility costsNetwork: 15%Switches, links, transit
Power CostApproximate distribution of peak power usage by hardware subsystem in one of Google’s datacenters (circa 2007).
Data Center ChallengesTraffic load balanceSupport for VM migrationAchieving bisection bandwidthPower savings / CoolingNetwork management (provisioning)Security (dealing with multiple tenants)
Non‐Virtualized Data CentersToo many servers for too little work
High costs and infrastructure needsMaintenanceNetworkingFloor spaceCoolingPowerDisaster Recovery
What is Virtualization?Run multiple OSes and user applications on the same hardware
e.g., run both Windows and Linux on the same laptopHow is it different from dual‐boot?
Both OSes run simultaneouslyOSes are completely isolated from each other
Reduce costs by consolidating services onto the fewest number of physical machines
Dynamic Data CenterVirtualization helps us break the “one service per server” modelConsolidate many services into a fewer number of machines when workload is low, reducing costsConversely, as demand for a particular service increases, we can shift more virtual machines to run that serviceWe can build a data center with fewer total resources, since resources are used as needed instead of being dedicated to single services
VM Workload Multiplexing
Multiplex VMs’ workload on same physical server
Separate VM sizing
VM multiplexing
s1s2
s3
We expect s3 < s1 + s2. Benefit of multiplexing !
So, it is just like Java VM, right?
Two Types of Hypervisors (or VMM)Hypervisor is a software layer that allows several VMs to runon a physical machineThe physical OS and hardware are called the HostVM OS and applications are called the Guest
VMware ESX, Microsoft Hyper‐V, Xen
Hardware
Hypervisor
VM1 VM2
Type 1 (bare‐metal)
Host
Guest
Hardware
OS
Process Hypervisor
VM1 VM2
Type 2 (hosted)
VMware Workstation, Microsoft Virtual PC, Sun VirtualBox, QEMU, KVM
Host
GuestProcess Process
Process Process
Bare‐metal or Hosted?Bare‐metal
Has complete control over hardwareDoesn’t have to “fight” an OS
HostedAvoid code duplication: need not code a process scheduler, memory management system – the OS already does thatCan run native processes alongside VMsFamiliar environment – how much CPU and memory does a VM take? Use top! How big is the virtual disk? ls –lEasy management – stop a VM? Sure, just kill it!
A combinationMostly hosted, but some parts are inside the OS kernel for performance reasons, e.g., KVM
5151
VM on Multi‐core CPUsEach core can be configured for multiple VMs
A Quad‐core CPU could be configured as a 32 node multi‐computerLimiting factor is often memory. Each guest OS has its own requirements (512 MB?)
Installing a Virtual machineBase OS is Windows 7
First install Sun VirtualBox as hypervisorThen, Guest OS will be Ubuntu 12.04.1
52
Installing Sun/Oracle VirtualBox
53
Installing VirtualBox
54
Installing VirtualBox
55
Installing VirtualBox
56
Installing Ubuntu VM
57
Installing Ubuntu VM
58
Installing Ubuntu VM
59
Installing Ubuntu VM
60
61
Installing Ubuntu VM
62
63
Installing Ubuntu VM
64
Data Center TopologyInternet
Servers
Layer-2 switchAccess
Data Center
Layer-2/3 switchAggregation
Layer-3 routerCore
Top of Rack Switch
Oversubscription
MotivationFlow‐based traffic monitoringVolume of processed data is reducedPopular flow statistics tools
Cisco NetFlowsFlow
What is sFlow?Accurate sampling is simple enough to be performed in hardware, at wire speed. Good accuracy under heavy loads Detailed complete packet header and switching/routing information for L2‐L7 traffic flows.Scalable capable of monitoring networks at 10Gbps, 100Gbps and beyond. Thousands of devices can be monitored by a single sFlow Collector.Low Cost sFlow Agent is very simple to implement and adds negligible cost to a switch or router.Timely an up to the minute view, for real‐time controls. QoS and DoS attack.
sFlow Architecture
SwitchingASIC
1 in N sampling
sFlow in Operation
packet header src/dst i/f sampling parms forwarding user ID URL i/f counterssFlow agent
forwarding tables
interface counters
sFlow Datagram
eg 128B ratepool
src 802.1p/Qdst 802.1p/Qnext hopsrc/dst maskAS pathcommunitieslocalPref
src/dstRadiusTACACS
sFlow Collector & Analyzer
Switch/Router
Statistical Model for Packet Sampling
NncNc⋅=
Total number of frames = NTotal number of samples = nNumber of samples in class = cNumber of frames in the class estimated by:
Relative Sampling Error
0%
25%
50%
75%
100%
1 10 100 1000 10000
Number of Samples in Class
% E
rror
c%error 1196 ⋅≤
Estimating Traffic per Protocol
Even so, Data Volume is Huge!
Short‐term period of flow dataMassive flow data from anomaly traffic data of Internet worm and DDoS
Cluster file system and cloud computing platformGoogle’s programming model, MapReduce, big tableOpen‐source system, Hadoop
Flow data in our campus network ( /16 prefix )# of Routers 1 Day 1 Month 1 Year
1 1.2 GB 13 GB 156 GB5 6 GB 65 GB 780 GB
10 12 GB 130 GB 1.5 TB200 240 GB 2.6 TB 30 TB
73
An Experiment: Travel Booking Website2 BNT switches
Traditional Flow‐based Traffic Monitoring
Run on a high performance central server
Flow Data
Routers
High Performance Server
Storage
75
Occupancy of Different Network Service Types
Heavy Hitters Workload
Offline Spike Analysis
Google System Health
Software Defined Networking (SDN)What is SDN?
BackgroundAn OS for networks
What is OpenFlow?How it helps SDN
The current status & the future of SDNConclusions
79
Limitations of Current Networks
80
Switches
Million of linesof source code
5400 RFCs Barrier to entry
500M gates10Gbytes RAM
Bloated Power Hungry
Many complex functions baked into the infrastructureOSPF, BGP, multicast, differentiated services,Traffic Engineering, NAT, firewalls, MPLS, redundant layers, …
An industry with a “mainframe‐mentality”
We have lost our way
Specialized Packet Forwarding Hardware
OperatingSystem
App App App
Routing, management, mobility management, access control, VPNs, …
Operating SystemOperating System
Reality
AppApp
App
Specialized Packet Forwarding HardwareSpecialized Packet
Forwarding HardwareSpecialized Packet
Forwarding Hardware
OperatingSystem
App App App
• Lack of competition means glacial innovation• Closed architecture means blurry, closed interfaces• Vertically integrated, complex, closed, proprietary• Not suitable for experimental ideas• Not good for network owners & users• Not good for researchers
Glacial process of innovation made worse by captive standards process
DeploymentIdea Standardize
Wait 10 years
• Driven by vendors• Consumers largely locked out• Lowest common denominator features• Glacial innovation
Old Ways to Configure a Network
Specialized Packet Forwarding Hardware
App App App
Specialized Packet Forwarding Hardware
App App App
Specialized Packet Forwarding Hardware
App App App
Specialized Packet Forwarding Hardware
App App App
Specialized Packet Forwarding Hardware
OperatingSystem
OperatingSystem
OperatingSystem
OperatingSystem
OperatingSystem
App App App
84
No control plane abstraction for the whole network! It’s like old times – when there was no OS…
Wilkes with the EDSAC, 1949
85
Idea: An OS for Networks!!!
Specialized Packet Forwarding Hardware
App App App
Specialized Packet Forwarding Hardware
App App App
Specialized Packet Forwarding Hardware
App App App
Specialized Packet Forwarding Hardware
App App App
Specialized Packet Forwarding Hardware
OperatingSystem
OperatingSystem
OperatingSystem
OperatingSystem
OperatingSystem
App App App
Closed
Idea: An OS for Networks
Specialized Packet Forwarding Hardware
App App App
Specialized Packet Forwarding Hardware
App App App
Specialized Packet Forwarding Hardware
App App App
Specialized Packet Forwarding Hardware
App App App
Specialized Packet Forwarding Hardware
OperatingSystem
OperatingSystem
OperatingSystem
OperatingSystem
OperatingSystem
App App App
Network Operating System
Control Programs
87
Idea: An OS for Networks
Simple Packet Forwarding Hardware
Simple Packet Forwarding Hardware
Simple Packet Forwarding Hardware
Simple Packet Forwarding Hardware Simple Packet
Forwarding Hardware
Network Operating System
Control Programs
88
Idea: An OS for Networks“NOX: Towards an Operating System for Networks”
Global Network View
Protocols Protocols
Control via forwarding interface
Network Operating System
Control Programs
Software‐Defined Networking (SDN)
89
Windows(OS)Windows(OS)
Linux MacOS
x86(Computer)
Windows(OS)
AppApp
LinuxLinuxMacOSMacOS
Virtualization layer
App
Controller 1
AppApp
Controller2
Virtualization or “Slicing”
App
OpenFlow
Controller 1NOX(Network OS)
Controller2Network OS
Trend
Computer Industry Network Industry
Outline
What is SDN?Limitations of current networksThe idea of Network OS
What is OpenFlow?How it helps SDN
The current status & the future of SDNConclusions 92
OpenFlow“OpenFlow: Enabling Innovation in Campus Networks”
Like hardware drivers – interface between switches and Network OS
93
OpenFlow
94
Data Path (Hardware)Data Path (Hardware)
Control Path (Software)Control Path (Software)
OpenFlow
95Data Path (Hardware)Data Path (Hardware)
Control PathControl Path OpenFlowOpenFlow
OpenFlowOpenFlow ControllerController
OpenFlow Protocol (SSL/TCP)
OpenFlow ProtocolSSL‐TCP
Network OS
Control Program A Control Program B
OpenFlow Basics
IBM 10 gigabit ethernet OpenFlow switch G8264, which has 48×10 GbE SFP+ ports and 4 × 40 GbE QSFP+ ports
OpenFlow Switching
97
Controller
PCHardwareLayer
SoftwareLayer
OpenFlow TableMACsrc
MACdst
IPSrc
IPDst
TCPsport
TCPdport Action
OpenFlow Client
**5.6.7.8*** port 1
port 4port 3port 2port 1
1.2.3.45.6.7.8
Control Program A Control Program B
Network OS
Flow Table
PacketForwarding Packet
Forwarding
PacketForwarding Packet
Forwarding
PacketForwarding Packet
Forwarding
FlowTable(s)Flow
Table(s)
“If header = p, send to port 4”
“If header = ?, send to me”
“If header = q, overwrite header with r, add header s, and send to ports 5,6”
Flow Table
Rule(exact & wildcard) Action Statistics
Rule(exact & wildcard) Action Statistics
Rule(exact & wildcard) Action Statistics
Rule(exact & wildcard) Default Action Statistics
Flow 1.
Flow 2.
Flow 3.
Flow N.
Flow EntryMatch fields
Match against packetsAction
Modify the action set or pipeline processingStats
Update the matching packets
Match Fields StatsAction
In Port SrcMAC
DstMAC
Eth Type Vlan Id IP Tos IP
Proto IP Src IP Dst TCP SrcPort
TCP DstPort
Layer 2 Layer 3 Layer 4
1. Forward packet to port(s)2. Encapsulate and forward to controller3. Drop packet 4. Send to normal processing pipeline
1. Forward packet to port(s)2. Encapsulate and forward to controller3. Drop packet 4. Send to normal processing pipeline
1. Packet2. Byte counters1. Packet2. Byte counters
ExamplesSwitching
*
SwitchPort
MACsrc
MACdst
Ethtype
VLANID
IPSrc
IPDst
IPProt
TCPsport
TCPdport Action
* 00:1f:.. * * * * * * * port6
Flow Switching
port3
SwitchPort
MACsrc
MACdst
Ethtype
VLANID
IPSrc
IPDst
IPProt
TCPsport
TCPdport Action
00:20.. 00:1f.. 0800 vlan1 1.2.3.4 5.6.7.8 4 17264 80 port6
Firewall
*
SwitchPort
MACsrc
MACdst
Ethtype
VLANID
IPSrc
IPDst
IPProt
TCPsport
TCPdport Action
* * * * * * * * 22 drop
ExamplesRouting
*
SwitchPort
MACsrc
MACdst
Ethtype
VLANID
IPSrc
IPDst
IPProt
TCPsport
TCPdport Action
* * * * * 5.6.7.8 * * * port6
VLAN Switching
*
SwitchPort
MACsrc
MACdst
Ethtype
VLANID
IPSrc
IPDst
IPProt
TCPsport
TCPdport Action
* * vlan1 * * * * *port6, port7,port9
00:1f..
OpenFlow UsageController
PC
OpenFlowSwitch
OpenFlowSwitch
OpenFlowSwitch
Alice’s codeAlice’s code
Decision?OpenFlowProtocol
Alice’s Rule
Alice’s Rule
Alice’s Rule
Alice’s Rule
Alice’s Rule
Alice’s Rule
103
OpenFlow UsageController
PC
Alice’s codeAlice’s code
104
» Alice’s code:˃ Simple learning switch ˃ Per Flow switching˃ Network access control/firewall
˃ Static “VLANs”˃ Her own new routing protocol: unicast, multicast, multipath
˃ Home network manager˃ Packet processor (in controller)˃ IPvAlice
OpenFlowStandard way to control flow‐tables in commercial switches and routers
Just need to update firmware
Essential to the implementation of SDN
105
Centralized/Distributed Control
Centralized Control
OpenFlowSwitch
OpenFlowSwitch
OpenFlowSwitch
Controller
Distributed Control
OpenFlowSwitch
OpenFlowSwitch
OpenFlowSwitch
Controller
Controller
“Onix: A Distributed Control Platform for Large‐scale Production Networks”, UNIX OSDI 2010.
Virtualizing OpenFlowNetwork operators “Delegate” control of subsets of network hardware and/or traffic to other network operators or usersMultiple controllers can talk to the same set of switchesImagine a hypervisor for network equipmentsAllow experiments to be run on the network in isolation of each other and production traffic
Switch Based Virtualization
Normal L2/L3 Processing
Flow Table
Production VLANs
Research VLAN 1
ControllerResearch VLAN 2
Flow Table
Controller
108
FlowVisorA network hypervisor developed by StanfordA software proxy between the forwarding and control planes of network devices
FlowVisor‐based Virtualization
OpenFlowSwitch
OpenFlowProtocolOpenFlowProtocol
OpenFlow FlowVisor& Policy Control
Craig’sController
Heidi’sControllerAaron’s
Controller
OpenFlowSwitch
OpenFlowSwitch
110
Topology discovery is per slice
OpenFlowProtocolOpenFlowProtocol
OutlineOutlineWhat is SDN?
Limitations of current networksThe idea of Network OS
What is OpenFlow?How it helps SDN
The current status & the future of SDN
Conclusions111
OpenFlow Building Blocks
ControllerNOXNOX
SlicingSoftwareFlowVisorFlowVisor
FlowVisorConsole
112
ApplicationsLAVILAVIENVI (GUI)ENVI (GUI) ExpedientExpedientn‐Castingn‐Casting
NetFPGANetFPGASoftware Ref. SwitchSoftware Ref. Switch
BroadcomRef. SwitchBroadcomRef. Switch
OpenWRTOpenWRT PCEngineWiFi APPCEngineWiFi AP
Commercial Switches Stanford Provided
OpenFlowSwitches
ONIXONIX
Stanford Provided
Monitoring/debugging toolsoflopsoflopsoftraceoftrace openseeropenseer
Open vSwitchOpen vSwitch
HP, NEC, Pronto, Juniper.. and many more
HP, NEC, Pronto, Juniper.. and many more
BeaconBeacon TremaTrema MaestroMaestro
Current status of SDNHardware support
113
Ciena Coredirector
NEC IP8800
More coming soon...
Juniper MX‐series
HP Procurve 5400
Pronto 3240/3290
WiMax (NEC)
PC EnginesNetgear 7324
Current status of SDNIndustry support
Google built hardware and software based on the OpenFlow protocolVMware purchased Nicira for $1.26 billion in 2012IBM, HP, NEC, Cisco and Juniper also are offering SDNs that may incorporate OpenFlow, but also have other elements that are specific to that vendor and their gear.
Future Focuses of SDNNew policies for securityProgrammable WLANsThe placement of controllers (amount; location; centralized/distributed)Debugger for SDN
115
ConclusionsWhat is SDN?
A system‐layered abstractionProgrammable, flexible, and extensible
What is OpenFlow?Interface between switches and controllersEnabling SDN
Future SDNEnabling innovation
116
Wireless Data Center Networking60 GHz spectrum 7 GHz (57–64 GHz) wavebanddata rate over 1 Gb/s10 metersLine‐of‐Sight (LoS)
Recommended