Thank you to our sponsor,files.meetup.com/2979972/OpenStackDC_2013-06-18.pdf · 2013-06-26 · and highest traffic sites ... • Ryu • NEC OpenFlow ... GFXBench 2.7 T-Rex HD Fixed

OpenStackDC Meetup

June 18th 2013

Thank you to our sponsor,

"Virtual Reality: Orchestrating Physical and Virtual Networks" by Richard Whitney, Arista Networks Inc.

"GPU-accelerated VDI on OpenStack“ by Brian Schott, CTO, Nimbis Services, Inc.

"Disk IO Performance Opportunities in Virtualized Environments" by Joel Sciandra, Senior Software Architect at Neustar, Inc.

Open floor discussion

PRESENTATIONS

Software Defined Cloud Networks

Extending Quantum for the Physical Network

Richard Whitney

Systems Engineer, Arista Networks

[email protected]

• Founded in 2005 – FCS 2008

• Focused on Data Centre L2/L3 Ethernet switching solutions

• Pre-IPO, Privately held and profitable

• >1900 customers globally

•Successful franchise in many of the world’s largest DCs

and highest traffic sites

• Award Winning Solutions

Arista - Who are we?

7048T

48-port Data

Center Class

Gigabit Ethernet

Switch

7500E

Lossless, High

Density, Modular

Switching System

supporting up to

1152 x 10G,

288 x 40G and

96 x 100G

wirespeed ports

7050 S/Q

16 Port 40Gb

64/52 Port 10Gb

Dense

Virtualization

10GbE/40GbE DC

7150S & 7124FX

Ultra Low Latency

24,52,64-port SFP+

1G-40Gb Switches

Intelligent

Application Switch

7050 T

Dense 36-64 Port

1/10GBASE-T

Data Center

Switches

Arista Networks: The Best Data Center Portfolio

NEW

Supported Quantum Plugins in Grizzly

• Open vSwitch

• Cisco

• Linux Bridge

• Nicira NVP

• Ryu

• NEC OpenFlow

• Big Switch, Floodlight REST Proxy

• PLUMgrid

• Hyper-V Plugin

• Brocade Plugin

• Midonet Plugin

http://docs.openstack.org/grizzly/openstack-network/admin/content/

OVS Virtual

Driver

Quantum

Nova

Problem

• Quantum is great at orchestrating virtual tenant

networks

• physical network orchestration left to the user

• result - suboptimal network design, manual

intervention required, or need an outside system to

automate

• Some physical network vendors have developed their

own quantum plugins

• the model is wrong - your choice of virtual switch

should not be slaved to your physical network

infrastructure

• All proprietary, single-vendor solution


Extending Quantum for the Physical Network

• Extend quantum for the physical

network

• Enable seamless provisioning of

both the physical and virtual network


OVS Virtual

Driver

Physical

Driver

Quantum

Nova

VM

Quantum

Network

Controller

OVS Plugin

VM

VLAN 1000

OpenStac

k

Dashboar

d

(Horizon)

What are we solving?

VLAN 1001

OVS

Bridge

OVS

Agent

Green Network = Vlan

1000

Red Network = Vlan

1001

Rabbit MQ

OVSPlugin configures

each vswitch on the

compute hypervisor.

VLANs are used for

multi-tenancy

isolation.

Comput

e

node

VM

Quantum

Network

Controller

OVS Plugin

VM

VLAN 1000

OpenStac

k

Dashboar

d

(Horizon)

Issues: •Manual hardware configuration

•Broadcast Traffic on all ports

•Security hole with multi-tenancy

•Friends don’t let friends build large

L2 networks

VLAN 1001

OVS

Bridge

OVS

Agent


1000

Red Network = Vlan

1001

VM VM

VLAN 1000 VLAN 1001

OVS

Bridge

OVS

Agent

Vendor

hardware switch

Vendor hardware

switch

Compute

Node

Compute

Node

eth3 eth4

V

M

Quantum

Network

Controller

OVS Plugin

Vendor HW

Driver

Vendor-specific

Management

Entity

eth2

Et4

Et8

Et9

Compute

Node

V

M V

M

V

M

VLAN 1000

VLAN 1001

Provisioning API

OpenStac

k

Dashboar

d

(Horizon)


1000

Red Network = Vlan

1001

Driver

API

Compute

Node

Compute

Node

eth3 eth4

VM

eth2

Compute

Node

VM VM VM

VLAN 1000

VLAN 1001

Vendor hardware

switch

Quantum

Network

Controller

OVS Plugin

Vendor HW

Driver

Vendor-specific

Management

Entity

Et4

Et8

Et9

Provisioning API

OpenStac

k

Dashboar

d

(Horizon)


1000

Red Network = Vlan

1001

Driver

API

Why introduce a driver layer?

• Allows multiple hardware vendor drivers to co-exist below the

OVSPlugin.

• Sits below OVSPlugin, so a completely new plugin is not

required for each hardware vendor.

• OVSPlugin is the most deployed open-source vswitch

solution.

• Driver layer sits in a common directory and can easily be

imported into other plugins (Can also easily be adapted into

the ML2 framework).

• Each hardware device vendor needs to implement a small

set of APIs to conform to the driver layer and integrate with

the OVSPlugin.

• OVSPlugin manages VLAN allocation, and the hardware

driver manages the vendor-specific hardware provisioning.

The Driver API

• This API is between the OVSPlugin and the

hardware driver:

o create_network( network_id )

o plug_host( network_id, segmentation_id, host_id)

o unplug_host( network_id, segmentation_id, host_id )

o delete_network( network_id )

• Minor enhancement in Nova to pass VM hypervisor

location to Quantum:

o allocate_for_instance()

Directory Structure

• etc/quantum/hardware_driver/vendor_driver.ini

• quantum/common/hardware_driver/driver_adapter.py

• quantum/common/hardware_driver/drivers/vendor_driver.py

• quantum/plugins/openvswitch/ovs_quantum_plugin.py

Blueprint

• Blueprint

o https://blueprints.launchpad.net/quantum/+spec/ovsplugin-

hardware-devices

• Code Review

o https://review.openstack.org/#/c/20754/

o https://review.openstack.org/#/c/21141/

https://blueprints.launchpad.net/quantum/+spec/ovsplugin-hardware-devices






https://review.openstack.org/




Thank You!

Page 18

GPU-Accelerated VDI on OpenStack

NIMBIS SERVICES, INC.


OpenStack DC MeetUp

June 18, 2013

Nimbis Services, Inc.

http://www.nimbisservices.com

Brian Schott, CTO

[email protected]

http://www.nimbisservices.com

mailto:[email protected]

Page 19



About Nimbis Services

Nimbis Services is lowering the barriers to high-

performance computing (HPC) and offering a secure

marketplace for technical computing in the cloud.

Nimbis partners with leading HPC & cloud providers, software

vendors, and domain experts to configure and broker secure

on-demand pay-as-you-go access to applications.

Page 20



Challenges for GPU-Accelerated Virtual Desktop Infrastructure (VDI)

on OpenStack

Making the GPU available to a virtual machine.

(3 basic approaches, ok, maybe 4).

Bare Metal / Containers (lxc)

PCI Pass-Through

Para-virtualization (proxy device drivers)

Remote Display Protocols (X, NX, VirtualGL, WebGL)

Remote desktop protocol that supports GPU acceleration.

Host and Guest Combinations: Linux or Windows

Remote Clients: Linux, Windows, OSX, Web Browser, Tablet

Ports: “The Joys of Firewalls”, or “Any Port as Long as it is 443”

Licenses: “This is too good to be free.”

Provisioning GPU resources from within OpenStack.

Nova compute virtualization driver (plugin).

GPU instance types, resource discovery, and scheduling.

This is still under development in the community.

Full disclosure: Nimbis hand-deployed the GPU instances for testing.

Page 21



Nimbis Testbed Hardware

“Production” Cisco System

(1) UCS C220 M3 – dual quad-core 3.3GHz

Intel Xeon E5-2643, 192GB SDRAM, 2TB

SAS-7K, dual channel 10GbE interface.



SAS-7K, dual channel 10GbE interface.



SAS-7K, dual-channel 10GbE.

• (1) Nvidia GRID K2 Board

(2) Nexus 3064 10GbE Switches

(2) ASA 5520 Adaptive Security Appliances

(2) Cisco Catalyst 2960S 1GbE switches

“Development” Cisco System

(3) Cisco UCS C200 M2

(1) Nexus 3064 10GbE Switch

(1) ASA 5520 Adaptive Security Appliance

(1) Cisco Catalyst 2960S 1GbE switch

Cores: 72

Memory: 1440 GB

Disk: 36 TB

Servers: 9*

Switches: 9*

* What do you

expect with a

Cisco testbed?

Page 22



Nvidia GRID GPU Virtualization (Kepler Architecture)

Source: http://www.nvidia.com/object/grid-boards.html

Source: 1http://www.nvidia.com/content/PDF/kepler/NV_DS_Tesla_KCompute_Arch_May_2012_LR.pdf

GPU Virtualization¹

GRID boards feature

the NVIDIA Kepler

architecture that, for the

first time, allows

hardware virtualization

of the GPU. This

means multiple users

can share a single GPU,

improving user density

while providing true

PC performance and

compatibility.

Page 23



GPU-Accelerated VDI Approaches Considered

Customer Requirements

Combination of Windows and Linux guest images.

Applications using OpenGL on both Windows and Linux.

Variety of Windows (95, 7, 8) clients, OSX clients, and web browsers

(IE7, IE8, Firefox, Chrome, Safari).

Variety of firewall and IT/FSO software challenges.

Technologies Tested

Microsoft Hyper-V and RemoteFX

XenServer with VNC / RDP / Guacamole

Open Source Xen (libvirt) on Ubuntu Linux with VNC

Possible Alternatives

Linux Containers (lxc)

• USC/ISI team has the blueprint and code targeted for Havana.

• No Windows support.

XenDesktop with HDX 3D Pro (not tested)

• Requires a custom client, Citrix Receiver, OpenGL support unclear.

• Does not integrate with OpenStack.

Page 24



Microsoft Hyper-V and RemoteFX

Technology

Windows Host GPU device with para-virtualized driver for windows guests.

Host / Hypervisor

Microsoft Windows Server 2012 Datacenter Edition (RemoteFX license).

Latest Nvidia driver for Windows.

Guests

No GPU pass-through available or driver for Linux guests.

Accelerates Windows DirectX calls, does software OpenGL rendering.

Clients

Good performance with standard Windows RDC > 8.0.

Good performance from FreeRDP* OSX client, not Microsoft client.

iOS and Android? Not tested, but FreeRDP clients exist on both.

OpenStack Status

Integrates with OpenStack Hyper-V nova compute drivers.

Nova scheduler support for Hyper-V extra specs not on blueprint radar.

Possible to use today with custom nova scheduler plugin.

* http://www.freerdp.com

Page 25



Hyper-V and RemoteFX Benchmarks with Nvidia GRID K2

GFXBenchTest MetricRDPWithoutRemoteFX

NVidiaQuadro600(~$150)*

RDPWithRemoteFX

NVidiaQuadroK5000(~$1800)*

SparkLines

Fillrate-Offscreen(1080p) MTexels/s 76 3291 34507 39776

Fillrate-Onscreen MTexels/s 60 2324 8760 7949

GFXBench2.5EgyptHDC24Z16-Offscreen(1080p) Frames 124 4274 6610 70733

GFXBench2.5EgyptHDC24Z16-Offscreen(1080p) Frames/s 1.1 76.3 58.5 626

GFXBench2.5EgyptHDC24Z16-Onscreen Frames 74 2622 4682 6776

GFXBench2.5EgyptHDC24Z16-Onscreen Frames/s 0.7 46.8 41.4 60

GFXBench2.7T-RexHDC24Z16-Offscreen(1080p) Frames 26 3124 3719 24434

GFXBench2.7T-RexHDC24Z16-Offscreen(1080p) Frames/s 0.5 55.8 66.4 436.3

GFXBench2.7T-RexHDC24Z16-Onscreen Frames 15 4274 2708 3358

GFXBench2.7T-RexHDC24Z16-Onscreen Frames/s 0.3 76.3 48.4 60

GFXBench2.7T-RexHDFixedTimestep-Offscreen(1080p) ms 1199830 56022 9117 1546

GFXBench2.7T-RexHDFixedTimestep-Offscreen(1080p) Frames/s 0.5 56022 61.4 1546

GFXBench2.7T-RexHDFixedTimestep-Onscreen ms 1699985 10 11734 362.2

GFXBench2.7T-RexHDFixedTimestep-Onscreen Frames/s 0.3 6761 47.7 9361

Trianglethroughput:Textured-Offscreen(1080p) MTriangles/s 1 59.8 2209 59.8

Trianglethroughput:Textured-Onscreen MTriangles/s 0 303 6246 2098

Trianglethroughput:Textured,fragmentlit-Offscreen(1080p) MTriangles/s 1 300 1770 2049

Trianglethroughput:Textured,fragmentlit-Onscreen MTriangles/s 0 303 4263 1678

Trianglethroughput:Textured,vertexlit-Offscreen(1080p) MTriangles/s 1 299 1780 1665

Trianglethroughput:Textured,vertexlit-Onscreen MTriangles/s 0 300 7310 1672

Host: Microsoft Windows Server 2012 Datacenter Edition,

Guest: Microsoft Windows Server 2012 Standard Evaluation,

Client: FreeRDP on Mac OSX * http://gfxbench.com/

Page 26



XenServer with VNC / RDP / Guacamole

Technology

GPU pass-through of Nvidia GRID K2 to Windows and Linux guests.

Two GPU devices available for two separate instances.

Physical GPU virtualization drivers promised by end of the year (n=64?).

Host / Hypervisor

XenServer 6.1 Platinum Edition (enables GPU license).

Guests

Linux: specify Windows 7 as server template then standard CentOS 6.4

install process. Good GPU performance using VNC and VirtualGL.

Windows: tried both RDP and TigerVNC do software OpenGL rendering.

• Unable to VNC or RDP to “console” display, possibly no monitor?

Clients

Windows and OSX Clients comparable to remote desktop to native boxes.

Tried RDP, VNC, and Guacamole (HTML5 VNC client).

OpenStack Status

XenServer Integrates with OpenStack using XenServer plugins.

Nova scheduler support for XenServer not on blueprint radar.

Page 27



Open Source Xen with VNC / RDP / Guacamole

Technology

GPU pass-through of Nvidia GRID K2 to Windows and Linux guests.

Two GPU devices available for two separate instances.

Unsure of GPU virtualization drivers status for n > 2.

Host / Hypervisor

Ubuntu 12.04, Xen (XCP 1.6), libvirt

Guests

Linux: Tested CentOS 6.4 image with latest Nvidia drivers (310.44).

Verified device functioned. Expect similar results to XenServer.

Windows: Did not test an image. Expect similar results to XenServer.

OpenStack Status

Xen supported within OpenStack using nova libvirt driver.

Nova specs support for libvirt exists (i.e. this host has a GPU).

Feasible to have support included in H or I release given lxc work.

Page 28



XenServer Linux Virtual Machine Installation Hack

Page 29



Specify GPU Type

Page 30



VirtualGL Architecture (VirtualGL.org)

Page 31



GLXINFO for Linux Instance on XenServer

Page 32



Guacamole HTML5 Browser VNC

Page 33



Windows Server 2012 on XenServer

Page 34



Heterogeneous Architecture Support Blueprints

https://wiki.openstack.org/wiki/Heter

ogeneousInstanceTypes

https://wiki.openstack.org/wiki/Sc

heduleHeterogeneousInstances

https://wiki.openstack.org/wiki/Heterog

eneousGpuAcceleratorSupport

Page 35



Summary

GPU-accelerated VDI pools within OpenStack are quite feasible.

Code support is very rough around the edges, not production ready.

We really need to get open community support organized for this feature.

Several vertical vendor solutions out there, but Xen and KVM baselines

should be possible as open-source.

Questions?

Joel Sciandra

Architecture

Tuesday, June 18, 2013

Disk IO Performance Opportunities in Virtualized

Environments

© Neustar, Inc. / Proprietary and Confidential

Neustar By The Numbers

37

Manage Almost 7 Billion Physical And Virtual Addresses

Enable 7 Billion Daily Text Messages

Manage 5.8 million Domain Names

Resolve over 18 Billion Daily DNS Queries

Enable 4 Billion Daily Phone Calls

Geo Locate nearly 3 Billion IP Addresses

Enable 3 Billion Daily Internet Location Searches

Answer 2 Billion Daily Customer Client Queries

Facts based on information available to us as of September 30, 2012 © Neustar, Inc.

Jira Issue.

© Neustar, Inc. / Proprietary and Confidential 38

» Random write I/O performance very bad on nova-

01-iad-01

Is OpenStack file IO fast enough?


» Last year, Neustar engineers

» ran a Postgres vs Oracle performance tests

» dedicated server w/ 24 cores, 32 gig RAM

» Highest-end SSD drives

» This year

» Ran the same Postgres tests

» OpenStack cloud w/ 8VCPU’s

» IO performance test tools:

» Bonnie++

» Fio

» Others including iozone, tiobench, filebench, dd, ffsb

What are our requirements.


» Standard Amazon EBS Volumes deliver approximately 100 IOPS on

average, with burst capability of up to hundreds of IOPS.

» Amazon Provisioned IOPS Volumes currently support up to 4000

IOPS per volume.

0

500

1000

1500

2000

2500

3000

3500

avg iops

amazon

openstack w/ local

openstack w/ Cinder w/ deadline

Linux I/O scheduler


» A Linux I/O scheduler, a.k.a.

elevator, is a piece of software

that manages a device’s I/O

queue.

» The goal is to make disk access

more efficient and reduce

process starvation.

» Standard Linux schedulers are:

» CFQ (Completely Fair Queuing)

» Deadline

» Anticipatory

» NOOP

io scheduler results.


2500

2600

2700

2800

2900

3000

3100

3200

openstack w/ deadline openstack w/ noop openstack w/anticipatory

openstack w/ cfq

avg iops

avg iops

Linux Page caches description, options, issues.


» Disk backed pages of memory to

speed up disk IO.

» Performance can be improved by

bypassing one of these caches.

» Some OS version dependencies.

» Some hardware dependencies such

as battery backup raid cache, etc.

» Options:

» Writethrough

» Writeback

» None

The VM disk’s format


» The VM disk’s format

» At the highest level, we have

to decide file-based images

or block devices.

» Block devices tend to

perform better because there

are fewer software layers.

» Disk image files have

different advantages such as

can be easier to clone or

backup. No preallocated

space saves disk.

Network hardware, drivers and configurations


» Paravirtualized network device, VirtIO is significantly faster

than the IDE driver.

» Vhost-net is a kernel based VirtIO driver.

» For network bandwidth to the Cinder (block) host

» use two NICs /w Bonding/LACP

» create with independent, dedicated iSCSI data paths for

network reliability and speed.

» TCP performance tuning

» iSCSI optimizations on the switch

» jumbo frames

» Txqueuelen (length of the tx queue)

Single Root I/O Virtualization (SR-IOV)


» New class of PCI devices that present multiple virtual devices

that appear as regular PCI devices

» These virtual adapters can be assigned directly to a VM

instead of relying on the hypervisor to arbitrate everything.

Disk drive / controller hardware


» “Disk” technology

» Single hard disk drive w/ 3 Gbit/s SATA

» ~100 to 200 iops

» Single high-end solid state drive w/ 6 Gbit/s SATA

» ~ 120,000 IOPS

» Single Fusion-io ioDrive2 - integrating flash into host systems

» Up to 9,608,000 IOPS

» RAID

» configurations that stripe (RAID 0, 10, 01, 0+1) will yield

significantly superior performance.

» RAID card w/ 1GB+ of cache, 6Gb/s PCI express 3.0 and

SATA+SAS ports

Disk storage system options.


» Cinder is the OpenStack blockstore service with interfaces

for many vendor disk subsystems such as

» EMC, NetApp, Ceph, etc.

» Swift is the default OpenStack object store service.

Ceph


» Ceph is a scalable, reliable Storage Service for Petabyte-scale

Storage Clusters. Ceph does block and object store.

Summary


» How do I get IO improvements?

» It depends.

» Big improvements require big dollars. Need

ROI calculations.

» Some are data center architecture choices.

» Most choices are workload/work mix

dependent.

» Your mileage may vary…

Questions?


Open Floor Discussion

Thank you for coming!

You will receive a survey & your feedback is greatly appreciated!

Stay tuned for our next Meetup!

Follow us on…

http://twitter.com/OpenStackDC

http://meetup.com/OpenStackDC

http://linkedin.com/groups/OpenStack-DC-4207039 http://www.meetup.com/OpenStackDC/suggestion/

http://www.meetup.com/OpenStackDC/messages/boards/






http://linkedin.com/groups/OpenStack-DC-4207039







http://www.meetup.com/OpenStackDC/suggestion/






Documents

Thank you to our sponsor,files.meetup.com/2979972/OpenStackDC_2013-06-18.pdf · 2013-06-26 · and highest traffic sites ... • Ryu • NEC OpenFlow ... GFXBench 2.7 T-Rex HD Fixed