Upload
eliza-croen
View
642
Download
1
Tags:
Embed Size (px)
DESCRIPTION
Building highly efficient cloud infrastructure, and lessons learned from real deployments: The session will cover how to build converged cloud solution based on industry standard components and open source software, to deliver the best cost/performance, lowest $/GB storage, and lowest $/VM, and the right balance of compute, network, and storage resources. This is based on the speaker experience of working with multiple OpenStack based cloud providers, integrators, and internal implementation of OpenStack private cloud in Mellanox The session will also discuss various software defined storage (SDS) and commercial options, what’s the benefit of one vs the other, how to efficiently combine SSD & HDD, and expiriance with BigData and Hadoop applications, will cover latest innovations in the space of high-performance networking and storage (VXLAN in hardware, DPDK/NFV, Cinder acceleration, Ceph over RDMA, ..) , and will go over a concrete for high-density, high-perform
Citation preview
Agenda
• OpenStack Overview, Benefits, and Challenges
• Overall topology and setup
• OpenStack Networking
• Options, Overlays, Challenges
• Storage in OpenStack
• Options (Ceph, iSCSI, iSER, Swift, HDD/SSD), Pro and Cons
• Building Cost effective solutions with application in mind
• Summary
• Based on commodity
• Reduce software costs
• Automation
What Does OpenStack Buy You ?
Flexibility & FeaturesCost Effective Cloud
Integration & Interoperability
• Endless amount of services
• OpenSource and Extensible
• Many configuration and scripting options
• Compute, Storage, Networks, and Apps under one system
• Modular approach with Pluggable services
• Testing, integration, and certification by multiple vendors and communities
Deploy OpenStack, Practical View ?
• Require strong IT group with DevOps professionals, not suitable for small/non-technical organizations (yet)
• Must think cloud to take advantage, and not tie your self to Enterprise paradigms
• Start with solid distribution (RedHat RHOS, Ubuntu, Mirantis, ..) or use professional services to ramp up
• It’s a community, if you want to get help, you must also contribute
• Test on small scale, validate assumptions, before going to production setups
Typical OpenStack Deployment
Firewall, L3 Routing, NAT, DHCP services
(can have multiple)
CinderiSCSI or Ceph based storage
NovaCompute (VM) nodes
Management servicescan also be installed on
the same server
Source: stackops.org
OpenStack in Converged Network Environment
Native integration of
Mellanox products with
Neutron
Hardware-based support for
security, and isolation functions
Accelerating storage
access by up to 5x
DPDK DPDK
Source: Mellanox
Types of Networks in OpenStack
• Hypervisor Networks
• Console
• Storage
• Messaging
• VM (Tennant) Networks
• VLAN Based
• Overlay based (VXLAN, GRE)
• Connection/Port Types
• Para-virtualized, e.g. using OpenvSwitch
• Direct Attached, using SR-IOV, for highest performance & native NIC features (RDMA, OS Bypass, ..)
Logical View of an OpenStack Multi-Tenant Cloud
Implemented via software or hardware appliances
Using VLANs or Tunnels (VXLAN)
Mellanox Network Virtualization (Neutron) Plug-in
OSVM
Para-virtual
OSVM
OSVM
OSVM
tap tap SR-IOV to the VM
Provision VM & Fabric Policy in hardware, through standard APIsBenefits: Isolation, functionality, performance & offload, simpler SDN
Embedded Switch
Mellanox Nova
Plug-in
Create/delete, configure policy per VM vNIC
Neutron Plug-Ins
Servers
Manager
OpenStack Manager
Compare eSwitch vs OVS Qperf (TCP) Latency
Source: Mellanox
Typical VxLAN Overlay Network Deployment
OSVM
OSVM
OSVM
UDP
OSVM
BR0 BR1
VXLAN Overlay
IP
VNI100 VNI300Open vSwitch (OVS)
Underlay Network (Layer 2 or Layer 3)
Hypervisor
vTap vTap vTap vTap
OSVM
OSVM
OSVM
UDP
OSVM
BR0 BR1 BR2
VXLAN Overlay
IP
VNI100 VNI200 VNI300Open vSwitch (OVS)
Hypervisor
vTap vTap vTap vTap vTap
VxLAN Overlay (tenant) networks
UDP
BR0 BR1
VXLAN
IP
VNI200 VNI300
Router / NAT Software Gateway
vEth1vEth0
Internet
Gateway (Network) Nodes
OVS
SDN Manager e.g. OpenStack Neutron, ODL,
VMware NSX
What is VxLAN
• “Virtual Extensible LAN (VXLAN) is a network virtualization technology that attempts to ameliorate the scalability problems associated with large cloud computing deployments. It uses a VLAN-like encapsulation technique to encapsulate MAC-based OSI layer 2 Ethernet frames within layer 3 UDP packets.” Wikipedia
Performance & CPU Advantage Using VxLAN Offload
1 VM 2 VMs 3 VMs
NO VxLAN
11 19 21
VxLAN in software
2 3 3.5
VxLAN HW Off-load
10 19 21
2.512.522.5
Total VM Bandwidth when using VxLANB
and
wid
th [
Gb
it/s
ec]
1 VM 2 VMs 3 VMs
NO VxLAN
0.545454545454545
0.68421052631579
0.666666666666667
VxLAN in soft-ware
3.5 3.33333333333333
4.28571428571429
VxLAN HW Off-load
0.9 0.894736842105263
1.19047619047619
0.50
2.50
CPU Usage Per Gbit/sec with VxLAN
CP
U%
/ B
an
dw
idth
(G
bit
/sce
)
5x More Bandwidth with VxLAN Offload4x Less CPU
Overhead with VxLAN Offload
Test Details- Test command: netperf -t TCP_STREAM –H- 1-3 VMs talking to 1-3 VMs on a second server- OpenvSwitch (OVS) with VxLAN support- Servers:
- HP ProLiant DL380p Gen8- 2 x Intel(R) Xeon(R) CPU E5-2650 0 @ 2.00GHz- 32GB RAM
- Hypervisor OS: Linux Upstream Kernel 3.14-rc1 + KVM- Guest VM OS: RHEL 6.5 2.6.32-431.el6.x86_64 - NIC: ConnectX-3Pro , FW: 2.30.8000- CPU% and Bandwidth measures on the Hypervisor (aggregate
1-3 VMs)
Source: Mellanox
Large Scale Topology Diagram
Example: Redundant and shared storage connectivity, and non-redundant server connectivity (to save costs)
OpenStack Storage OptionsOption Pro Con Perf $/GB*
Ceph • Free • Multiple front-ends• Built in HA, Snaps,
DR, ..
• Slow & CPU intensive
• Complexity & Stability
• Replication (with apps like Hadoop)
Low-Med
0.40
iSCSI-LVM or iSER-LVM
• Free• Fast
• HA Features • Cinder only
Med-High
0.20
Commercial • Performance• Stability & Support• Features
• Cost• Flexibility
Med-High
0.50 - 2
Swift • Free• Low-cost
• Performance • Object only
Low 0.30
Reference: Amazon AWS
• No OpEx overhead • Slow• data transfer
costs
Low 0.30/yr
* - all inclusive, using 4TB HDDs, with redundancy
Ceph (Vision) – All Purpose Storage
Cinder iSCSI Deployment Example
iSCSI or iSER Storage Servers/Heads
JBODs or Mid-range RAID Enclosure
HA
SAS/SATA
OpenSource software (Cinder LVM) or commercial software/system
Initiators/Hosts accessing through iSCSI or iSER
(iSCSI RDMA)
Note: need to use the right ratio of disks per Head to optimize cost/performance
With 40GbE and RDMA offload, each head can support 5x more disks, lowering overall costs
• Using OpenStack Built-in components and management (Open-iSCSI, tgt target, Cinder), no additional software is required, RDMA is already inbox and used by our OpenStack customers !
RDMA Provide Fastest OpenStack Storage Access
Hypervisor (KVM)
OSVM
OSVM
OSVM
Adapter
Open-iSCSI w iSER
Compute Servers
Switching Fabric
iSCSI/iSER Target (tgt)
Adapter Local Disks
RDMA Cache
Storage Servers
OpenStack (Cinder)
Using RDMA to accelerate
iSCSI storage
1 10 100 10000
1000
2000
3000
4000
5000
6000
7000iSER 4 VMs WriteiSER 8 VMs WriteiSER 16 VMs WriteiSCSI Write 8 vmsiSCSI Write 16 VMs
I/O Size [KB]
Band
wid
th [M
B/s]
PCIe Limit
6X
RDMA enable 6x More Bandwidth, 5x lower I/O latency, and lower CPU%
Source: Mellanox
Storage TiersWorkload Comment $/GB
(Raw)
HDD Big Data workloads (e.g. Hadoop, Video)
Very slow IOPs, VM cross interference
0.04
HDD with SSD as Cache(e.g. LSI FlashCache, Bcache, commercial)
Localized disk access (e.g. VM Images)
0.20
SSD Storage IOPs/latency sensitive apps (e.g. Databases, VDI)
1-2
SSD/TLC Read mostly, Random Fast reads Low-endurance SSDs
0.40
Challenge: How to pre-allocate storage to different tiers and different usage models (Block, Object, Files, ..)
Food For Thought
Should I use 4TB or 2TB Disks ?
$/GB cost with 60 x 3.5” HDD JBOD(Disks + Enclosure costs / Disks #)
Disk $/GB
4TB 0.08
2TB 0.11
For 30% extra cost you can double the IOPs & BW
Highly Efficient Rack Example U36 27 2835 25 2634 23 2433 21 2232 19 2031 17 1830 15 1629 13 1428 11 1227 9 1026 7 825 5 624 3 423 1 2222120 Storage19
18 Storage17
16 Storage15
14 Storage1312 Storage1110 Storage98 Storage76 Storage54 Storage32 Storage1
Switch: 36 x 40/56Gb (IB/Eth)2nd Switch (Optional)
28 x 0.5U (twin) servers• Each with Mellanox ConnectX3-Pro 40/56Gb
Adapter• Optional up to 6 Disks per server
2 x High performance storage servers, each with • 2U 24 bay server + 2 x 45 bay JBODs • 24 x 2.5” 500-1000GB SSDs • 90 x 3.5” 2-4TB SAS/SATA HDDs• 1-2 x ConnectX3-Pro 40/56Gb Adapter• Run Cinder/LVM (iSCSI/iSER) and/or Ceph
1-2 x Switches (e.g. Mellanox SX1036/6036)• 28 x 40/56Gb copper links to servers • 4 x 56Gb Copper links to storage • 4 x 56Gb optical uplinks to core
Thank You !Yaron Haviv, [email protected]
Manila – File Shares as A Service
What is Manila?– Multitenant, secure file share as a service– “Cinder for shared file systems”– NFS & CIFS protocols supported today; more to come
Will be available Junos
OpenStack Netron Network Topology Examples