Managing Heterogeneous Cloud Infrastructure with OpenStack · OpenStack with dedicated services (e.g., Nova, Cinder). ‘share everything ’ - All pools are managed by single OpenStack

1

Managing Heterogeneous Cloud Infrastructure with OpenStack

Example: Hosting ‘Hybrid’ Applications Comprising Bare-Metal and Virtualized Resources

OpenStack Summit, May 12th, 2014, Atlanta

IBM Research – Haifa

Alex Glikson

Ezra Silvera

2 © 2014 IBM Corporation

Outline

� Heterogeneous Clouds

‒ Background & Motivation

‒ Opportunities and Challenges

� Hosting ‘Hybrid’ (Bare-Metal + Virtualized) Applications

‒ Goals

‒ Scenarios

‒ Solution Approach

‒ Technical Challenges and Design Considerations

‒ Example

� Summary, Q&A

3

Background & Motivation


Evolution of IaaS Usage Patterns

� Early days of IaaS (‘infancy’)‒ Value proposition:

• Agility, efficiency, elasticity, self-service, pay-per-use

‒ Main usage patterns: • Dev & test, relatively simple ‘cloud-native’ web applications

‒ Offered hardware configurations:• Small set of standardized (virtual) hardware configurations

e.g., AWS:

– started with 1 instance type (2006),

– grew to 5 in 2008, …

Offered highly standardized virtualized infrastructure, used to host small set of applications


Evolution of IaaS Usage Patterns

� Now (‘adolescence’)‒ Value proposition:

• Agility, efficiency, elasticity, self-service, pay-per-use

‒ Usage patterns:

• Rapidly increasing spectrum of applications seeking the benefits of the cloud model

– E.g., ‘legacy’ business applications, HPC, analytics, etc

• Increasing adoption in large organizations, owning many ‘non-cloud native’ applications

‒ Offered hardware configurations:

• Greatly increased heterogeneity is expected

– E.g., AWS now offers >60 instance configurations

• Rise of bare-metal providers

– IBM/SoftLayer, Internap, etc

Evolving to a much larger variety of resources (including bare-metal),used to host much larger variety of applications


Underlying Heterogeneity

� Heterogeneity spectrum (examples)‒ Different CPU/memory/disk model & ratio’s‒ Compute:

• CPUs vs GPGPUs vs accelerators (e.g, FPGAs)

‒ Storage: • Local HDD vs SSD vs SAN vs NAS

‒ Network: • regular vs high-speed vs Infiniband

‒ Virtualization:• Bare-metal vs KVM vs LXC vs Xen vs VMware


Example: Collaborative 3D Design

� Example mapping considerations:

‒ Tier 1 (Web app): regular KVM-based ephemeral VMs

‒ Tier 2 (3D rendering): bare-metal with GPGPUs

‒ Tier 3 (Hadoop): LXC with SSDs, Infiniband interconnect

‒ Tier 4 (Designs DB): VMware VMs supporting DR with SRM, reliable and fast SAN-backed storage

Tier-1: web app

Tier-2: 3D rendering

Tier-4: Designs DB Tier-3: Hadoop


Two Main Drivers of Heterogeneity

� User-driven: new & challenging application requirements

� Provider-driven: natural evolution in hardware over time


New Application Requirements

� Key inhibitors preventing broader IaaS adoption‒ Special resource requirements

• Legacy applications, resource-intensive applications

‒ Isolation• Performance (e.g., due to strict SLAs)

• Security (e.g., due to compliance)

A modern cloud environment must offer HW configurations optimally designed for classes of applications with special requirements


Natural Evolution in Hardware

� HW models and technologies evolve over time

‒ E.g., EC2 Instance Types:

Original (since 2006)

New since December 2013

New since April 2014

Deprecated since April 2014

Model vCPUMem (GB) Storage (GB)

m3.medium 1 3.75 1 x 4 SSD

m 3.large 2 7.5 1 x 32 SSD

m3.xlarge 4 15 2 x 40 SSDm3.2xlarge 8 30 2 x 80 SSD

c3.la rge 2 3.75 2 x 16 SSD

c3.xlarge 4 7.5 2 x 40 SSD

c3.2xlarge 8 15 2 x 80 SSD

c3.4xlarge 16 30 2 x 160 SSDc3.8xlarge 32 60 2 x 320 SSD

r3.large 2 15 1 x 32 SSD

r3.xlarge 4 30 .5 1 x 80 SSD

r3.2xlarge 8 61 1 x 160 SSDr3.4xlarge 16 122 1 x 320 SSD

r3.8xlarge 32 244 2 x 320 SSD

g2.2xlarge 8 15 1 x 60 SSD

i2.xlarge 4 30 .5 1 x 800 SSD

i2 .2xlarge 8 61 2 x 800 SSDi2 .4xlarge 16 122 4 x 800 SSD

i2 .8xlarge 32 244 8 x 800 SSD

hs1.8xlarge 16 117 24 x 2048

t1 .micro 1 0.613 EBS Only

m1.small 1 1.7 1 x 160m1.medium 1 3.75 1 x 410

m 1.large 2 7.5 2 x 420

m1.xlarge 4 15 4 x 420

c1 .medium 2 1.7 1 x 350c1.xlarge 8 7 4 x 420

cc2.8xlarge 32 60 .5 4 x 840

cg1.4xlarge 16 22 .5 2 x 840

m2.xlarge 2 17 .1 1 x 420

m2.2xlarge 4 34 .2 1 x 850m2.4xlarge 8 64 .8 2 x 840

cr1.8xlarge 32 244 2 x 120 SSD

hi1.4xlarge 16 60 .5 2 x 1,024 SSD

A typical Cloud environment must support multiple ‘generations’ of HW

simultaneously


Two Main Drivers of Heterogeneity

� User-driven: new & challenging application requirements

� Provider-driven: natural evolution in hardware over time

Support for infrastructure heterogeneity is essential in any modern IaaS cloud

12

Opportunities and Challenges



� Business opportunities enabled by infrastructure heterogeneity

‒ Service/application providers:

• Cloud adoption in ‘traditional’ market segments � agility and efficiency

– E.g., manufacturing, finance, scientific, etc

– IDC: the (extended) HPC market alone is larger than IaaS ($15B vs $9B)

‒ Cloud/infrastructure providers:

• Ability to better fit to specific classes of applications � differentiation

– Especially relevant given the overall commoditization of the IaaS market, recent price wars, etc



Challenge: cloud infrastructure natively designed for:

‒ Underlying infrastructure heterogeneity

‒ Ability to seamlessly host applications with special requirements

‒ Still preserve the characteristics of the cloud model

• Agility, efficiency, elasticity, etc


Heterogeneous Cloud Requirements: Cloud User

� Cloud user perspective

‒ Access to a wider range of offered HW/SW configurations

• Performance, security/isolation

• Compute, storage, network

‒ Same or better user experience as with ‘regular’ cloud, e.g.:

• Same: single/unified catalog of configuration ‘flavors’, images, etc

• Same: seamless network connectivity, storage access, etc

• Same: higher-level services (orchestration, LBaaS, DBaaS, etc)

• Better: higher-level application abstraction, hiding the details of the underlying HW (critical with increased complexity of underlying HW)

– Specify goals/requirements, rather than specific resource types/models etc


Heterogeneous Cloud Requirements: Cloud Provider

� Cloud provider perspective

‒ Minimize management cost by unified processes and APIs

• Provisioning, inventory, monitoring, life cycle operations, visualization, etc

– Unified resource model

‒ Secure multi-tenancy across heterogeneous resources

‒ Elasticity & efficiency

• Smart scheduling, to overcome resource fragmentation

• Dynamic balancing between HW-compatible pools by “repurposing”(subject to compatibility)

– Capacity monitoring & prediction

– Automated evacuation & re-provisioning


Heterogeneous Cloud: OpenStack Perspective

� OpenStack is following a similar pattern‒ Started primarily with dev & test and cloud-native applications

• Simplistic approach to flavors, scheduling, performance optimization, etc

‒ Extensive efforts to enable adoption by additional applications, e.g.:

• Hadoop (Sahara)

• DBaaS (Trove)

• Bare-metal (Ironic, TripleO, Tuskar)

• Scheduling (Gantt)

• Reservations (Climate)

• Policies (Congress)

• Etc

‒ Growing community interest in supporting heterogeneous clouds, e.g.:• Multi-Tenant Bare Metal Provisioning (Mon, 11:15, B206; Tue, 2:50pm, B301)

• Deploying OpenStack in a Multi-Hypervisor Environment (Wed, 11am, B101)

• IBM, SoftLayer and OpenStack - Present and Future (Wed, 11:50am, B312)

• Many more related sessions…


Heterogeneous Cloud: OpenStack to the Rescue!

Claim: OpenStack provides a good basis for building heterogeneous clouds

• But few gaps yet to be addressed

19

Hosting ‘Hybrid’ (Bare-Metal + Virtualized) Applications


Hybrid Applications – Background and Goals

� Hybrid environment – includes both physical and virtual resources

‒ Pools: Resources are grouped into bare-metal and virtual pools

� Hybrid application – a composite (e.g., multi-tier) application that spans both virtual and physical server(s)

� Goals

‒ Admin: Minimize management complexity and overhead. Unified management across bare-metal and virtualized pools.

‒ User: Simplified ongoing management and application life cycle.

� Design guideline: If possible try to use OpenStack native solution

‒ E.g., no external coordinators, orchestration, management


Pool 3: Virt Type 2 Pool 4: Bare metalPool 1: Spare (BM) Pool 2: Virt Type 1

Tenant ATenant B

Hybrid Applications – Background and Goals

22

Scenarios


Scenario 1: Node Repurposing

Support native policies that dynamically balance the virtual andphysical pools (e.g., moving servers from one to another)

� Repurposing Virtualized node (Hypervisor) � Physical node

1. Detect resource congestion in the physical pool

2. Identify a candidate host to be re-purposed into physical

3. Trigger “evacuate“ for all VMs on the selected host

4. Assign physical host to the BM pool

� Preliminary network and storage configuration

‒ Server is ready to be deployed as bare-metal application (later on)


Scenario 2: Runtime Decision on Target Pool

Scenario: decide on the server type (BM/VM) upon deployment‒ E.g., according to performance requirements

1. User deploy an application without specifying the server type‒ Using generic definitions (e.g., image, flavor)

2. Automatically choose server type‒ Potentially using scheduler, Heat, ceilometers monitoring, etc.

3. Choose/construct actual image for deployment‒ Option 1: Maintain single base copy of the image and perform

necessary adaptation to match the target server type (BM/VM)

‒ Option 2: Maintain multiple ‘versions’ of the image and select the appropriate one according to target type.

4. Deploy using corresponding provisioning mechanism‒ E.g., leveraging Ironic

‒ Apply adaptation upon boot (e.g., leveraging Heat)

25

Solution Approach


Management Approaches

� Possible solutions range from 'share nothing to 'share everything'

‒ 'share nothing’ - Each pool is managed by a separate instances of OpenStack with dedicated services (e.g., Nova, Cinder).

‒ ‘share everything’ - All pools are managed by single OpenStack “instance”. Use single copy of each service

‒ Some of the services are shared

• Region and cells can be considered special cases for this

� OpenStack can natively handle bare metal and virtual hosts, so why not combining them together in a single OpenStack?

� We focused on the 'shared everything' architecture

‒ Defined as 'Integrated Management‘


Integrated Management: Basic Architecture

� Special resource types, mapped to specially designed resource pools� Use multiple host aggregates, including ‘bare-metal’

‒ Use aggregate filters for scheduling� Networking

‒ Admin network for provisioning and management of compute nodes‒ Separate management network for BM machines hosting user applications‒ Data network combining both VMs and BM nodes

Controller node

mysql-server

rabbit-server

neutron-server

glance

Network node

Neutron.*-plugin-agent

Neutron-l3-agent

Neutron-dhcp-agent


Neutron-l3-agent

BM computeCompute node

nova-compute


Compute node

nova-compute


Compute node

nova-compute


BM nodeBM node

BM node

External

Data/Guests

Management

API

Internet

BM MgtBM nodeBM nodeBM spare


Advantages of Integrated Management approach

� Native and simple OpenStack-based solution

‒ Doesn’t require external orchestration

• E.g., Dynamic repurposing of physical nodes can be driven by Heat

‒ No need to coordinate between services of different pools

• e.g., two Neutron instances to achieve unified NW domain

� Simplified administration

‒ Controlling all pools/resources from one place in a unified manner

‒ Reduce configuration complexity and overhead

• Reduce the number of services need to be managed

‒ Better diagnostics and root cause analysis

� Improved user experience and manageability

‒ Maintain/expose combined topology view for different server types

‒ Hide the underlying infrastructure details from the user (if needed)

29

Technical Challenges and Design Considerations


Challenges Examples

� OpenStack has some gaps when managing bare-metal and virtual machines simultaneously (**)

� Networking‒ Manage simultaneously both the virtual and physical networks

• Network isolation for tenants should combine both BM and VMs

• E.g., Use tunnels/overlays connecting physical nodes and VMs

• E.g., Use multiple plugins (e.g., OVS, openflow) simultaneously to gain advantages for each pool (if possible)

** Note: we focus on issues related to Hybrid management

‒ E.g., general BM issues should/are addressed as part of Ironic.


Challenges Examples (Cont.)

� Compute‒ Enhanced scheduler to better support heterogeneity

• Use different placement policies for different pools (e.g., look at GPU consumption rather then CPU/memory)

‒ Support instance management for BM

• Actions for BM instances may differ from actions on VMs

• E.g., implement: Console, View log. Explore applicability of: Pause, Suspend, Resize

‒ Support aggregates for specific BM nodes

‒ Add hierarchical relations between BM instance and the compute node running on it

• E.g., Propagate delete command to the VM level as well (mark VMs as deleted when deleting the BM instance).


Challenges Examples (Cont.)

� Image Management

‒ Enhance access control for images in Glance

• E.g., Don’t expose Ironic’s RAM disk images to users

‒ Support run-time image selection

• E.g., Display single instance of the image

‒ Support dynamic adaptation of images to match target pool type

� UI: extend UI to support Hybrid applications/instances

‒ Add bare-metal node management in Horizon

• Currently can be done through CLI (e.g., ironic node- create/delete)

‒ Physical machines are shown as Hypervisors

33

Hybrid Application example


Example – Deploying Hybrid WordPress

� Apache+WordPress on VM and MySQL on physical node

� Setup and configuration‒ A single HEAT template used for the two nodes

‒ Using host aggregates for scheduling

‒ Used diskimage-builder to build the baremetal images• cloud-init element installed


Example – Deploying Hybrid WordPress

� Example gaps reflected in Horizon:‒ BM nodes are shown as Hypervisors

‒ Deployment images are exposed to users

36

Summary


Summary

� Heterogeneous cloud environments are gaining momentum

‒ Critical in order to host a broad spectrum of applications

� OpenStack is a promising solution to manage hybrid clouds

‒ By using integrated management we can have

• Simple native OpenStack solution

• Simplified administration and enhanced user experience

� Need a careful design to ensure all requirements (e.g., security, isolation) are maintained across the hybrid environment

� Some gaps need to be addressed in OpenStack


Be sure to stop by the IBM booth to see some demos and get your rockin’ OpenStack t-shirt while they last.

Don’t miss Monday evening’s booth crawl where you can enjoy Atlanta’s own SWEET WATER IPA!

Thank you !


Q & A


Monday, May 12 – Room B314

12:05-12:45

Wednesday, May 14 - Room B312

9:00-9:40

9:50-10:30

11:00-11:40

11:50-12:30

OpenStack is Rockin’ the OpenCloud Movement! Who‘s Next to Join the Band ?Angel Diaz, VP Open Technology and Cloud LabsDavid Lindquist, IBM Fellow, VP, CTO Cloud & Smarter Infrastructure

Getting from enterprise ready to enterprise bliss - why OpenStack and IBM is a match made in Cloud heaven. Todd Moore - Director, Open Technologies and Partnerships

Taking OpenStack beyond Infrastructure with IBM SmartCloudOrchestrator.Andrew Trossman - Distinguished Engineer, IBM Common Cloud Stack and SmartCloud Orchestrator

IBM, SoftLayer and OpenStack - present and futureMichael Fork - Cloud Architect

IBM and OpenStack: Enabling Enterprise Cloud Solutions Now.

Tammy Van Hove -Distinguished Engineer, Software Defined Systems

IBM Sponsored Sessions


Monday, May 12

3:40 - 4:20

3:40 - 4:20

Tuesday, May 13

11:15 - 11:55

2:00 - 2:40

5:30 - 6:10

5:30 - 6:10

Wednesday, May14

9:50 - 10:30

2:40 - 3:20

Thursday, May 15

9:50 - 10:30

1:30 - 2:10

2:20 - 3:00

IBM Technical Sessions

Documents

Managing Heterogeneous Cloud Infrastructure with OpenStack · OpenStack with dedicated services (e.g., Nova, Cinder). ‘share everything ’ - All pools are managed by single OpenStack