SolidFire Mirantis Unlocked Reference Architecture · PDF fileSolidFire Mirantis Unlocked Reference Architecture : ... hardware in your data center or ... of existing data center assets

© 2005–2016 All Rights Reserved www.mirantis.com www.netapp.com Page 1

US HEADQUARTERS

Sunnyvale, CA 525 Almanor Ave Sunnyvale, CA 94085 +1-650-963-9828 Phone +1-650-963-9723 Fax

SolidFire Inc. Boulder, CO 1600 Pearl St, Suite 200 Boulder Colorado 80302 Phone: 720.523.3278 Fax: 303.557.0724

SolidFire Mirantis Unlocked Reference Architecture

Original Release December 2015 Revision 1 April 2016 Authors: Ed Balduf (Cloud Solutions Architect, SolidFire) Christian Huebner (Sr. Systems Architect, Mirantis)


Table of Contents 1 Abstract ................................................................................................................................... 5

1.1 About OpenStack ............................................................................................................... 5 1.2 Why OpenStack .................................................................................................................. 5 1.3 About SolidFire .................................................................................................................. 6 1.4 About Mirantis ................................................................................................................... 6 1.5 About this Reference Architecture ....................................................................................... 6

2 Target Audience ....................................................................................................................... 7 3 OpenStack ............................................................................................................................... 7

3.1 Compute Service (Nova) ..................................................................................................... 8 3.2 Block Storage Service (Cinder) ............................................................................................ 8 3.3 Image Service (Glance) ....................................................................................................... 9 3.4.3 Keystone ......................................................................................................................... 9 3.4 Other OpenStack Services .................................................................................................. 9

3.4.1 Neutron ....................................................................................................................... 9 4 OpenStack Storage ..................................................................................................................10

4.1 Storage Types in OpenStack ..............................................................................................10 4.1.1 Ephemeral Storage......................................................................................................10 4.1.2 Block Storage .............................................................................................................10 4.1.3 Object Storage ............................................................................................................11 4.1.4 File Storage ................................................................................................................11 4.1.5 Image Storage ............................................................................................................11

4.2 Solidfire Storage ...............................................................................................................12 4.3 SolidFire Element OS ........................................................................................................12 4.4 SolidFire Active Support ....................................................................................................13 4.5 SolidFire Active IQ ............................................................................................................13 4.6 SolidFire Storage Value for OpenStack ...............................................................................14

5 Reference Architecture ............................................................................................................15 5.1 Architecture ......................................................................................................................15 5.2 Physical Design .................................................................................................................16 5.3 Network layout .................................................................................................................16 5.3 OpenStack Storage ............................................................................................................24

5.3.1 Block Storage .............................................................................................................24 5.3.2 SolidFire .....................................................................................................................28 5.3.3 Ceph ...........................................................................................................................35 5.3.4 Image storage .............................................................................................................41 5.3.5 Object Storage ............................................................................................................45

6 Best Practices ..........................................................................................................................46 6.1 Mirantis OpenStack Best Practices .....................................................................................46 6.2 Networking Best Practices .................................................................................................47 6.3 Storage Best Practices .......................................................................................................48


6.4 Database Best Practices.....................................................................................................48 6.5 Hypervisor Operating System Tuning .................................................................................50

7 Implementation ......................................................................................................................52 7.1 SolidFire Fuel plugin .........................................................................................................52 7.2 Deployment Workflow .......................................................................................................52

8 Verification and Testing ...........................................................................................................70 8.1 Health Check ....................................................................................................................70 8.2 Functional testing .............................................................................................................72

8.2.1 Verify Backend Placement. ..........................................................................................72 8.2.2 Configure and Verify SolidFire Image caching ..............................................................74

8.3 System testing ..................................................................................................................79 8.4 Performing functional testing ............................................................................................79

9 Support ...................................................................................................................................83 10 Conclusion ............................................................................................................................83 11 Software and Documentation .................................................................................................84 12 Addendum - Network Config ..................................................................................................84

Configuring the network switch ...........................................................................................84


Table of Images Image 1: OpenStack Architecture Image 2: OpenStack and NetApp FAS network architecture Image 3: Control flow (black) and data flow (red) between participants Image 4: OpenStack and NetApp E Series architecture Image 5: Control flow (black) and data flow (red) between participants Image 6: Fuel Boot Image 7: Fuel Configuration Screen Image 8: Fuel Plugin Installation Image 9: OpenStack Environment Creation Image 10: Assign Roles in Fuel UI Image 11: Configure Networking Image 12: Enable and Configure NetApp Plugin Image 13: Deploy Changes


1 Abstract Cloud Computing has transformed the IT landscape and ushered in an era of Infrastructure-as-a-Service (IaaS). Applications running on cloud infrastructure can take advantage of its flexibility, cost-effectiveness, disaster-recovery options, and security. OpenStack is the leading free and open-source cloud computing IaaS platform and offers a rich set of features beyond traditional compute, storage and networking. This reference architecture was prepared jointly by Mirantis Unlocked and SolidFire. It details architectural considerations, best practices and deployment methodologies to create a highly available Mirantis OpenStack 7.0 (OpenStack Kilo Release) cloud using SolidFire storage. The architectural design described has been verified by Mirantis and SolidFire and can be deployed using the Mirantis 7.0 Fuel plugin for SolidFire.

1.1 About OpenStack OpenStack is an open-source project, released under the Apache 2.0 license, implementing services to support Infrastructure as a Service (IaaS) and with additional Platform as a Service (PaaS) components. The project is managed by the OpenStack Foundation, a nonprofit corporate entity established in 2012 to promote, protect, and empower OpenStack software and its associated community. OpenStack technology comprises a series of modular projects that control large pools of processing, storage, and networking resources throughout a datacenter, all managed through a single dashboard that gives administrators control and enables self-service resource provisioning by users.

1.2 Why OpenStack OpenStack provides the best mechanism for IT organizations and service providers to leverage flexible, cost-effective, enterprise-ready Infrastructure-as-a-Service (IaaS). Contributions made by Solidfire, Mirantis, and others can be deployed with existing hardware in your data center or proof-of-concept labs. In most cases, drivers for specific infrastructure solutions are provided open source and free of cost. OpenStack use cases include public cloud providers, private cloud deployments,


and hybrid clouds. For more information on OpenStack benefits and enterprise use-cases, see http://www.openstack.org/enterprise.

1.3 About SolidFire SolidFire delivers the industry’s most comprehensive OpenStack block storage integration. Combining this integration with SolidFire’s guaranteed performance, high availability, and scale, customers can now confidently host performance-sensitive applications in their OpenStack cloud infrastructure. SolidFire began contributing code to OpenStack in 2012, released our first Cinder driver with OpenStack Folsom in Fall of that year, and is an OpenStack Foundation Corporate Sponsor. Further information about SolidFire’s OpenStack integration and contributions can be found at www.solidfire.com/openstack.

1.4 About Mirantis Mirantis is the pure play OpenStack company. Mirantis delivers all the software, services, training, and support needed for running OpenStack. More customers rely on Mirantis than any other company to get to production deployment of OpenStack at scale. Mirantis is among the top three companies worldwide in contributing open source software to OpenStack, and has helped build and deploy some of the largest OpenStack clouds in the world, at companies such as Cisco, Comcast, Ericsson, NASA, Samsung and Symantec. Mirantis is venture-backed by August Capital, Ericsson, Intel Capital, Insight Venture Partners, Sapphire Ventures and WestSummit Capital, with headquarters in Sunnyvale, California. Follow us on Twitter at @mirantisit.

1.5 About this Reference Architecture OpenStack has steadily gained features and deployment tools that allow for high availability at the control plane, hypervisor, and storage tiers. Enterprises are now looking for a single solution that can meet the requirements of all their workloads, rather than piecing together various different solutions. In this reference architecture we focus on architectural considerations and deployment methodologies for a highly available Mirantis OpenStack 7.0 (MOS 7.0) cloud with SolidFire storage. We will also cover architectural considerations for multiple storage back-ends including

http://www.openstack.org/enterprise










SolidFire and perform some basic performance testing. We will cover the use of the Mirantis OpenStack 7.0 SolidFire Fuel plugin. The reference architecture described in this document is based on the SolidFire OpenStack Configuration Guide and the Mirantis OpenStack Operations Guide and User Guide.

2 Target Audience The target audience for this document includes the following groups:

● Systems Engineers, Solutions Architects, Consultants or IT administrators with basic configuration and proof of concept efforts.

● Technical decision makers. This document describes the value of using SolidFire storage solutions with OpenStack cloud services to create a cloud environment that enterprises and service providers can use to meet their respective needs.

● Service provider and Enterprise cloud architects. SolidFire storage provides an enterprise class underpinning for OpenStack-based clouds. This document describes architecture, best practices, and key design considerations for deploying OpenStack in a highly available manner.

● SolidFire and Mirantis partners and implementers. It is important for SolidFire and Mirantis partners to understand the solution architecture of OpenStack on SolidFire storage in order to meet and exceed customer requirements and expectations for their cloud solutions.

The document assumes the reader has an architectural understanding of OpenStack and has reviewed related content in the OpenStack documentation. The OpenStack documentation center can be found at: http://docs.openstack.org/

3 OpenStack OpenStack is a cloud operating system that controls pools of compute, storage, and networking. OpenStack consists of a number of different services, some of which are shown below.

http://www.solidfire.com/resources/configuration-guide-solidfire-and-openstack







https://docs.mirantis.com/










http://docs.openstack.org/









Image 1: OpenStack Architecture

3.1 Compute Service (Nova) The OpenStack Nova Compute Service manages the instance lifecycle from creation to scheduling, management, and eventually deletion. Nova manages compute resources and provides a common REST API, which developers can use to automate cloud operations and provide external orchestration. The API is also used by administrators to control and manage compute resources and instances. Nova is designed to scale horizontally on standard hardware.

3.2 Block Storage Service (Cinder) The OpenStack Cinder block storage service provides dynamic provisioning and portability of block storage devices for Nova instances. Cinder provides persistent block devices mapped to OpenStack compute instances (which use ephemeral, “transient” storage by default). Cinder manages the creation, attachment, and detachment of the block devices to instances. Persistent block devices provided by Cinder also support instance booting and provide mechanisms for creating Cinder snapshots. Cinder is built as a pluggable architecture, which permits using a wide range of storage back-ends, ranging from a simplistic Cinder reference back-end using Linux LVM, to enterprise class scale-out storage platforms such as SolidFire. Regardless of the back-end, Cinder provides a standard subset of functionality to Nova and the hypervisors. Some back-ends, including those from SolidFire, provide additional functionality, such as the Quality of Service features inherent to SolidFire block storage volumes. Although fully


integrated with Nova and Horizon, Cinder can also be used independently from OpenStack to provide a standardized abstraction for block storage provisioning.

3.3 Image Service (Glance) OpenStack Image Service (Glance) provides registration and delivery services for disk and server images. The ability to copy a server image and immediately store it away is a powerful feature of OpenStack. When multiple servers are being provisioned, a stored image can be used as a template to get new servers up and running more quickly and consistently than by installing a server OS and individually configuring additional services.

3.4.3 Keystone OpenStack Identity Service (Keystone) provides a central directory of users mapped to the OpenStack services they can access. It acts as a common authentication system across the cloud operating system, and it can integrate with existing back-end directory services such as Microsoft Active Directory (AD) and Lightweight Directory Access Protocol (LDAP).

3.4 Other OpenStack Services

3.4.1 Neutron OpenStack Network Service (Neutron) is a pluggable, scalable, and API-driven system for managing networks and IP addresses. Like other aspects of the cloud OS, it can be used by administrators and users to increase the value of existing data center assets. Neutron prevents the network from becoming a bottleneck or limiting factor in a cloud deployment and provides users with self service capability for their network configurations.

3.4.2 Horizon The OpenStack Dashboard (Horizon) offers administrators and users a graphical interface to access, provision, and automate cloud-based resources.

3.4.4 Ceilometer


OpenStack Telemetry (Ceilometer) provides a common infrastructure for collecting usage and performance measurements in an OpenStack cloud. Its primary initial targets are monitoring and metering.

3.4.5 Heat OpenStack Orchestration (Heat) implements a service to orchestrate multiple composite cloud applications by using the AWS CloudFormation template format, through an OpenStack-native and CloudFormation-compatible API.

4 OpenStack Storage Instances created in an OpenStack can be configured to use various forms of back-end storage, ranging from a simple LVM on compute node approach to enterprise class storage infrastructure. This reference architecture will focus on SolidFire scale-out, all-Flash, enterprise-class storage.

4.1 Storage Types in OpenStack

4.1.1 Ephemeral Storage Ephemeral storage can be used to boot instances on a compute node. It is controlled by OpenStack Nova. As it is not persistent beyond the instance’s lifecycle, it cannot be used to store any kind of persistent data. Ephemeral Storage in OpenStack is generally not shared between compute nodes, and when used in this manner, instances cannot be migrated among nodes. However, if the ephemeral storage location is a shared location and the proper configuration is in place, migration may take place.

4.1.2 Block Storage In OpenStack, the generic term ‘Block Storage’ refers to block storage which is persistent and survives when the instance is destroyed. As block storage resources are generally shared, the replacement instance may be on an arbitrary compute node. Cinder is the OpenStack component that provides access to and manages block storage. To OpenStack hosts, the storage appears as a


block device but uses iSCSI, Fibre Channel, NFS or a range of other proprietary protocols for back-end connectivity. Cinder serves two distinct roles around block storage: the first being allocation and control of block storage resources using any storage back-end with a proper driver. Cinder's second role is to provide an example reference implementation of a block storage back-end, using common Linux tools. The Cinder reference back-end is mainly used for testing during OpenStack development. Cinder is capable of addressing more than one back-end with the multi-back-end feature. This feature is useful for multiple performance tiers, e.g. SolidFire FlashArray//m as tier 1 for IOPS-heavy workloads, with NetApp FAS with NFS for general cloud storage.

4.1.3 Object Storage Object storage stores reference data as binary objects (rather than as files or LUNs), typically storing or retrieving the entire object in one command. Objects are stored and referenced using HTTP (web-based) protocols with simple commands such as PUT and GET.

4.1.4 File Storage The File Share Service, in a manner similar to Cinder, provides coordinated access to shared or distributed file systems. While the primary consumption of file shares would be across OpenStack Compute instances, the service is also intended to be accessible as an independent capability in line with the modular design established by other OpenStack services. The development name for this project is Manila. The Manila project has taken on the task of managing file system based storage (NFS, CIFS, HDFS) in an OpenStack context. Manila graduated incubation and was designated as a core service as of OpenStack's Kilo release (May 2015).

4.1.5 Image Storage The Glance component of OpenStack provides a service where users can upload and discover data assets that are meant to be used with other services. This currently includes images and metadata

http://searchenterprisedesktop.techtarget.com/definition/Network-File-System


definitions. While not strictly a storage subsystem, Glance is a core component of OpenStack and heavily relies on other storage subsystems for its actual storage needs1. Glance image services include discovering, registering, and retrieving virtual machine images. Glance has a RESTful API that allows querying of VM image metadata as well as retrieval of the actual image. Like Cinder and Manila, Glance works with multiple back-ends, among them Object Storage, Ephemeral storage, File storage and Cinder storage

4.2 Solidfire Storage SolidFire provides a scale-out all-Flash storage platform designed to deliver guaranteed storage performance to thousands of application workloads side-by-side, allowing consolidation under a single storage platform. Our scale-out system is built specifically for the Next Generation Data Center and has the unique ability to manage storage performance completely separately from storage capacity, so as to meet both technology pressures and business demands. Mix and match any SolidFire storage nodes within a system to take advantage of the most current and cost-effective Flash technology available. The SolidFire system can be combined together over a 10Gb Ethernet network or 8/16 Gb FC clients in clusters ranging from 4 to 100 nodes. The SolidFire primary storage solution scales up to 100 nodes, provides capacity from 35TB to 3.4PB, and can deliver between 200,000 and 7.5M guaranteed IOPS to more than 100,000 volumes / applications within a single system — increasing overall business productivity through consolidation, automation and granular scalability.

4.3 SolidFire Element OS SolidFire Element OS uniquely answers all five key storage requirements for a true next generation data center. This all-inclusive, industry-leading storage operating system is at the core of every SolidFire infrastructure, giving you capabilities no other storage technology can match. Element OS enables:

● Scale-Out Growth: Grow in response to business demands by seamlessly and granularly adding capacity and performance for each new business need.

1 Glance itself does not provide storage, but relies upon other facilities to store images. All file operations are performed using the glance_store library which is responsible for interactions with external storage back ends or local filesystems.


● Guaranteed Performance: Optimize your infrastructure by managing performance independent of capacity.

● Automated management: Set your infrastructure to automatic and respond to business demands rapidly with on-the-fly adjustments.

● Data assurance: Consolidate with confidence using SolidFire’s Helix data protection — assurance for protecting your most critical information.

● Global efficiencies: Maximize TCO with always-on deduplication, compression, and thin provisioning that reduce storage, space, and power costs.

4.4 SolidFire Active Support Support shouldn’t be something that happens every once in awhile. At SolidFire, support is an active process that happens consistently from the moment a SolidFire cluster is deployed. SolidFire Active Support continuously monitors and diagnoses systems, ensuring SolidFire products are maintained and operated at the highest possible level of availability and performance. SolidFire Active Support continuously monitors your systems, proactively alerting you when a problem is present. SolidFire support can remotely and securely log into systems to provide hands-on real-time support. SolidFire’s support is global; international offerings are available, with up to 4-hour break-fix SLAs. All calls and cases are handled by tier three support engineers who can resolve your issues or answer your questions the first time, and every time.

4.5 SolidFire Active IQ The SolidFire Active IQ SaaS platform, a key element of Active Support, provides real-time health diagnostics, historical performance, and trending from the system level all the way down to each individual volume. This holistic approach to infrastructure monitoring, combined with SolidFire’s unique abilities to upgrade, tune, and scale on-demand without disruption, redefines operational success for storage infrastructure.

● Anticipate business demands: Real-time usage modeling and granular system data increase agility — enabling you to proactively plan for evolving business demands, simplifying storage resource optimization.

● Increased productivity: Consolidated monitoring saves time when managing multiple clusters/sites, and comprehensive storage metrics reduce assessment efforts. This means you have continuous visibility into changing conditions, empowering you to avoid issues rather than react to them.


● Reduced risk: Customizable alerts notify you instantly of possible issues. Faster response reduces risks to the business.

4.6 SolidFire Storage Value for OpenStack Reliable and flexible infrastructure is critical to any business, and the era of cloud services has raised the bar on what it means to deliver IT services. When you deploy SolidFire storage with OpenStack, you can focus up the stack on application deployments, iterate and innovate more, and get back to the business of your business. SolidFire’s all-Flash, scale-out storage provides the following features to OpenStack:

● Provides one of the industry's best Cinder2 integrations and can be deployed for use in OpenStack in under a minute, by inserting a minimum of four lines of configuration into the Cinder configuration file3.

● Scales non-disruptively, enabling you to grow your Cinder block storage resources without impact as business needs demand.

● Guarantees performance to all tenants and applications in your cloud4. ● Ensures all your cloud data is available and protected against loss and corruption. ● Is an easily programmable resource through our full-featured API and complete

automation, reducing manual coding and errors. ● Is fully compatible with Swift object storage for integrated backup and restore.

SolidFire’s Cinder driver is elegant and straightforward, allowing customers to configure OpenStack for SolidFire quickly, without the need for additional configuration libraries or add-ons. The maturity of the driver reveals itself in its easy setup for users and the completeness of the feature set once setup is complete:

● Clone full environment with no impact to storage capacity ● Guarantee performance through full Quality of Service (QoS) ● Speed deployment with Glance image caching, and boot-from-volume capabilities ● Live-migrate instances between OpenStack compute servers completely non-disruptively

2 As defined by the Cinder driver support matrix ( https://wiki.openstack.org/wiki/CinderSupportMatrix ) and the lack of any additional installation requirements beyond the standard Cinder install. 3 May be dependent on how fast one can type. 4 Through use of Quality of Service parameters.

https://wiki.openstack.org/wiki/CinderSupportMatrix












● Easily triage issues through ○ 1:1 ID mapping between Cinder Vol-ID and SolidFire volume name ○ 1:1 mapping and automated creation of tenant/project IDs as SolidFire accounts

● Automate configuration of SolidFire’s Cinder driver through available Puppet module

5 Reference Architecture

5.1 Architecture For this use case, we have targeted the automated deployment and configuration of OpenStack services and multiple OpenStack storage back ends. With this goal in mind, we have only deployed two compute nodes, with a small SolidFire cluster and a small Ceph storage cluster. We will focus on the steps necessary to automate the deployment and utilize the two storage back-ends. The basic stack deployed will be as shown below, with the absence of the Blue 3rd party tools — although Fuel has many plugins to support these tools, we will leave the selection and implementation details for the tool vendors. Fuel will deploy the operating system components, OpenStack components, Ceph storage and connections amongst the components as necessary.


5.2 Physical Design The hardware used to build this reference architecture is shown below, configured in a rack. The hardware setup comprises eight Dell R620 servers (one deployment node, three controller nodes, three Ceph storage nodes and two compute nodes), a four-node SolidFire cluster, two 10 Gbps Top Of Rack (TOR) switches and two 1 Gbps management switches. The configuration consumes 16 Rack Units leaving vacant 26 rack units for growth. We have chosen to place the storage on the top and the bottom, and the compute node in the middle to allow for scaling of storage (both storage types) and compute.

5.3 Network layout Administrative / Provisioning TOR Access Switches The Dell s55 switches serve to provide 1 Gb/s connectivity for administrative in-band access to all infrastructure components, as well as dedicated connectivity for infrastructure bare-metal provisioning.


These switches are configured with stacking. In the stacked configuration, the switches provide both redundancy and a single point of switch administration. Stacked switches are seen as a single switch, and thus have the added benefit of allowing the directly connected systems to utilize LACP bonding for aggregated bandwidth, in addition to providing redundant connectivity. When connected to the data center aggregation layer, it’s recommended that each admin switch be connected to each upstream aggregation switch independently with at least two links. As the diagram below shows, this facilitates uptime and availability during an upstream aggregation switch outage or maintenance period. VLT is a different good/best practice in environments where the downstream devices like access switches need to build port channels/Link Aggregation groups across two separate upstream switches. The downstream switches view the VLT peers as a single logical chassis, thus allowing the LAG group to be split across the two peers. The other benefit is that due to distinct control planes, it is possible to perform maintenance one switch at a time without taking the entire fabric down.

Data and Storage TOR Access Switches


The Dell s4810 switches provide the necessary high bandwidth 10 Gb/s connectivity required to support OpenStack cluster operations and SolidFire storage access. Dual s4810 switches are configured to utilize virtual-link trunking (VLT). VLT facilitates enhanced high availability, and greater aggregated bandwidth to both the upstream data center aggregation/core layer, as well as downstream to the directly-connected hosts by allowing the switch pair to be seen as one switch, thus allowing the formation of link-aggregation-groups (LAGs) to combine multiple links into one logical link, and aggregating bandwidth. The VLT trunk utilizes 2 x 40GbE links to provide a total of 80Gb/s of total throughput between TOR switches. One or two of the remaining 40GbE ports may be used as uplinks to the data center aggregation switches. This configuration will deliver up to 160Gb/s total throughput between the TOR data/storage switches as the solution scales out rack by rack. The redundant link configuration as shown in the diagram below facilitates high availability and maintains a minimum number of network hops through the infrastructure in the event of an upstream outage.

Cabling Considerations This reference architecture utilizes copper cabling throughout the design. For 10G connectivity, copper direct-attach cables, also known as TwinAx cabling, is more cost effective than comparable optical options, and SFPs are directly attached to the cable. Depending on port availability,


connectivity to the aggregation layer switches can be via QSFP+ cables, or QSFP+ to 4x10Gbe SFP+ breakout cables. Both options deliver up to 40Gb/s per QSFP+ port. Fiber is also an option, however SFP+ direct-attach cables tend to be more cost effective than optical SFP+ fiber options. Physical Node Connectivity As a general design guideline, all Agile Infrastructure nodes (OpenStack and SolidFire) are dual connected to both the administrative TOR access switches and the TOR data/storage access switches. This provides switch fault-tolerance and increased aggregated bandwidth when needed via multiple NIC bonding options.

In this reference architecture, all nodes are configured to utilize NIC bonding to facilitate minimum downtime and added performance when needed. This provides simple, consistent physical connectivity across the deployment. However, different bonding modes were used between OpenStack nodes and SolidFire storage nodes, as explained below. Openstack nodes are configured to use NIC bonding mode 1, or Active-Passive. In addition to providing physical NIC failover in case of a failed link, Active-Passive bonding allows for simple predictable traffic flow over the designated active link. However, if bandwidth becomes a concern


at the compute layer or higher aggregated bandwidth is desired, changing to bonding mode 5 (TLB) is recommended to alleviate bandwidth constraints, while maintaining redundancy. Either bonding mode 1 (Active-Passive) or bonding mode 5 (TLB) is recommended for OpenStack nodes because switch-side port configuration is not necessary, making scaling compute nodes simple. SolidFire nodes are configured to use NIC bonding mode 4, or LACP bonding. This bonding mode provides both redundant switch connectivity, as well as aggregated bandwidth, with one main difference between bonding mode 5 (TLB) being that it uses the same mac address on all outbound traffic, and traffic is load-balanced inbound and outbound on all bonded links. LACP bonding suits the storage nodes well, giving up to 20 Gbp/s aggregated storage throughput, per node. Jumbo Frames For maximum performance, use of jumbo frames across the 10G network infrastructure is recommended. The design presented in this reference architecture relies on an MTU setting of 9212 or higher be configured end-to-end across the network path for proper operation and performance. OpenStack nodes and SolidFire nodes have all 10G interfaces set to use an MTU of 9000. This MTU is less than that defined on the switches because of the overhead required to process jumbo frames. The switches add additional information to each frame prior to transmission, and thus MTU must be higher than 9000 on the switch side. VLANs We have chosen to configure the four OpenStack networks using VLANs on the bonded 10Gbps network pairs (20 Gbps total throughput) as shown below, using four VLANs assigned by our lab administrator.


Logical Network Layout

Some notable points about our layout choices:

1) Using Neutron Network without the Virtual router option means that the public network is only required to be connected to the controller nodes.

2) The storage network needs to be connected to the Controller nodes to allow Cinder to manipulate data on storage volumes (i.e. for image to volume conversions).

3) Our corporate router/firewall complex does not pass DHCP requests from network to network; therefore, we do not need to have a completely private DHCP network for this architecture. We have a router port allowing access to the Fuel deployment node through the PXE network. The Fuel deployment node must still be the only DHCP controller on the network.

4) All of the storage on our lab is connected to its own control network for management, and we have chosen to route management traffic to it through our corporate router complex. We have moved the data path from the corporate storage network to this architecture’s Storage network to facilitate a fast storage network.

5) The PXE network and the Corporate Storage Control network are the only two networks running on 1Gbps infrastructure.


Networks Summary Networks used in this reference architecture are described below.

Network Purpose

PXE Boot Used by nodes to PXE boot from the Fuel Provisioning server to provision Operating Systems and OpenStack.

OpenStack (Public)

This network provides access between OpenStack VMs and the rest of the corporate or data center networks. It also provides portal / API access to OpenStack cloud controllers. Also known as the Public or Floating network, Floating IPs are assigned from this network to individual VMs to enable external connectivity to networks outside the OpenStack environment. The term “public” does not necessarily imply that the network is Internet accessible.

Management Provides isolated access for OpenStack Cloud internal communications such as DB queries, messaging, HA services, etc.

Storage Isolated network access between Openstack nodes and SolidFire cluster or other storage (Ceph).

Private (mesh) Internal network to provide internal connectivity between tenant VMs across the entire Openstack Cloud. Also known as a fixed network, fixed IPs are assigned to individual VMs.


Physical and Logical Interface Mapping

Role Network Physical Interface VLAN ID Logical Interface

OpenStack Controller PXE Port1 , Port 2 bond1

OpenStack Port 3 , Port 4 1001 bond0.1001@bond0

Storage Port 3 , Port 4 25 bond0.25@bond0

Management Port 3 , Port 4 1003 bond0.1004@bond0

Private Port 3 , Port 4 1004 bond0.1003@bond0

OpenStack Compute PXE Port1 , Port 2 bond1




Provisioning Server PXE Port1 , Port 2 bond1

SolidFire Storage Node Corporate Admin Port1 , Port 2 bond1G

Storage Port 3 , Port 4 25 bond10G

Ceph nodes PXE Port1 , Port 2 bond1





5.3 OpenStack Storage OpenStack Storage takes on several flavors in every OpenStack installation. In this section we will address each one in some detail, but we will focus on the Block Storage aspects first and in the most depth.

5.3.1 Block Storage Cinder the OpenStack block storage subsystem consists of 5 cooperating components, which provide block storage services to VMs managed mainly by the Nova compute system. The five components consist of:

● API service - This service provides a standardized RESTful interface to accept requests, may update the database and places requests on the message bus to other services within Cinder. The API service may send requests directly to the Volume Manager or Volume service for requests going to an existing known volume.

● Scheduler Service - The scheduler is responsible for determining where and when in storage resources should be allocated. Requests are routed here when decisions of where or when need to be made.

● Volume Manager service - The Volume Manager service is responsible for communication to the actual storage device through a driver. The Volume Manager service does not necessarily provide any storage, especially when a 3rd party storage device/cluster is used.

● Volume Service (optional) - The Volume service is the reference architecture for storage provided with Cinder. The Volume service uses standard Linux mechanisms (LVM and TGT) to provide storage. The Volume service does not provide highly available scalable storage as would be demanded by a production cloud, but instead, quick convenient storage for testing OpenStack. Unfortunately the Volume Service is included as part of the code in the volume manager, and therefore many people associate the Volume service and the volume manager as one unit; they are not. In this reference architecture we will use two third-party storage services and not the volume service.

● Backup Service (optional) - The Cinder backup service allows Cinder to coordinate backup of volumes under its control to OpenStack Swift Object stores. Cinder backup will mount those snapshots to the Cinder controller and copy the data to the Swift service.

The below diagram is a good representation of the architecture of Cinder and how we will utilize it in this reference architecture. It does however leave out the fact that Cinder (and Nova) are running in the Fuel High availability architecture utilizing HAproxy, Galera and Pacemaker, which means


there are 3 copies of each Cinder service, one of each on each of 3 Controller nodes. Notice that two drivers have been configured in the Cinder Volume Manager service to communicate with the two back-end storage devices (SolidFire and Ceph).

5.3.1.1 Data flow

As shown in the diagram above, the SolidFire driver communicates with the SolidFire cluster using a JSON-RPC RESTful interface and the bulk storage data flows from the VMs and Hypervisor via iSCSI to the Solidfire cluster without interference from Cinder. Within the SolidFire cluster data for each volume is spread across the nodes (load balanced) by the SolidFire Element Operating System. There is no configuration necessary to take advantage of this load balancing. Data is compressed and deduplicated in-line in 4KB blocks across the cluster. Nova requests and Cinder provides the connection information; the SolidFire driver provides the SVIP as the initial connection point. When the hypervisor contacts the SVIP, Cinder redirects the iSCSI session to the optimal location within the SolidFire cluster for the volume being requested. This initial contact with the SVIP interface allows the SolidFire cluster to also load balance iSCSI connections throughout the networking connections of the cluster. The redirection of the iSCSI connection to a node within the cluster means that all Hypervisors need high speed connection to


all SolidFire nodes -- that is why it is important to configure the networking based on the SolidFire networking best practices. In fact the SolidFire nodes also use the same network for internode communication.

5.3.1.2 Multi Back End and Message Routing

Cinder is designed to support multiple storage devices or back-ends. Cinder will make decisions on which back-end to assign volumes to based on characteristics of the back-end and the request. The Cinder scheduler is the mechanism to make the decisions. A key point about Cinder and multiple back-ends is that Cinder stores information in its database differently when it is configured for multiple back-ends than when it is configured for a single back-end. The change in information storage is related to routing of request to back-end devices. When there is a single back-end it is always accessed through the same host of the message bus and therefore little to no routing information is needed. Once multiple back-ends are configured additional routing information is required. In this reference architecture we will experience this as Fuel is not currently designed for multiple storage back-ends. Once deployment is complete, we will quickly (before provisioning any storage) configure the OpenStack instance for multiple back-ends. The definition of the old single back-end will remain, but be inactive. Because changing from single to multi-back-ends would require a database change for any storage previously configured, it is recommended to immediately change all configurations to multi-back-end constructs (before any allocations) even if only a single back-end is configured. In addition to supporting multiple back-ends the Cinder database contains routing information to the instance of Cinder-volume which supports the storage. In the case of the OpenStack reference implementation, cinder-volume is providing the storage (i.e. Cinder Volume service) and has locational demands (i.e. the storage lives on the host running cinder-volume and therefore all requests must go to that host). In the case of most 3rd party storage cinder-volume communicates via the network to orchestrate the storage, such that multiple cinder-volume services may be able to satisfy the request by communicating with the storage back-end over the network. In the first case, the routing information is generally the hostname or IP address of the cinder-volume service, but in the second case we want any and all cinder-volume services to be available to handle the request, we use a generic routing variable defined by the ‘host’ directive in the cinder.conf to indicate a common host name. An example may help:


In this screenshot, the last three sections of cinder.conf are shown along with the cinder service list. You can see that cinder uses the ‘host=’ configuration line to identify the host name. Any cinder-volume service with that hostname will be able to retrieve message requests for that back-end (the message bus enforces atomic operations so only a single host will process each request), but the third storage system will always to routed to the host 'node-25.' The ability of multiple hosts to process requests is important for high availability of the OpenStack controller nodes. The SolidFire plugin configures the host to always be ‘solidfire’ on all OpenStack controllers deployed by Fuel.

In addition, if we look at specific volumes, we can see their routing information in the database. Notice that the pool name has been appended to the routing information (after the ‘#’). If your services change from the volume entries in the database a migration of the data for the volumes is necessary.

5.3.1.3 Volume Types

Volume types provide a tagging system to characterize the storage OpenStack makes available to users. If volume types are not used, all block storage is considered to be the same and the OpenStack scheduler will select the least full back-end to place any requested volumes.


If, however, volume types are set up, sophisticated direction of volume requests to back-end characteristics, including quality of service (QoS), may be assigned. Volume types are given simple names to identify them for users -- but under the covers, key/value pairs provide direction to the storage subsystem and the scheduler to determine from which back-end to satisfy a request. The most common practice is to name various back-ends in cinder.conf, and then assign a specific volume type to the back-end by that name. Other options are to assign back-ends by vendor name or by some other characteristic, like replication.

5.3.1.4 Scheduling/Assignment

The Cinder scheduler takes information from the volume request and uses that information to find a placement for the volume on a configured storage back-end. Cinder automatically polls configured back-ends every 20 seconds to determine their type and capacity. The information from the poll and configuration information in the cinder.conf, mainly Pools and volume_back-end_name, are then utilized to determine on which back-end to place the requested volume. The scheduler has the ability for users to write their own filters, but most users interact with the scheduler through volume-types and volume_back-end_names. The volume_back-end_name parameter is configured in the stanza of the cinder.conf file and may be the same for multiple back-ends of the same type, but should be different for back-ends with different characteristics. When configuring a volume-type, the volume_back-end_name can and should be specified and the information will be carried along to the scheduler such that it can decide where to place the volume. If there are multiple back ends with the same volume_back-end_name (or all other characteristics lead to multiple back-ends) the scheduler will utilize fullness information it obtains from the 20 second poll operation and place the volume on the less full back-end.

5.3.2 SolidFire All configuration in this Reference Architecture is based on version 8.0.1.2 of the SolidFire Element Operating System.

5.3.2.1 OpenStack SolidFire integration

SolidFire’s integration with OpenStack is through the Cinder driver and the standard iSCSI protocol. The integration through iSCSI allows Cinder to perform any needed manipulations of data within volumes as needed for image to volume and backup operations. The SolidFire driver communicates with the SolidFire cluster using JSON-RPC on a HTTPS and TCP connection to the SolidFire


Cluster’s Management Virtual IP Interface (MVIP). The MVIP interface will move throughout the cluster in response to load and/or failure events. The SolidFire Cinder driver contains all functionality for volume manipulations, including create, delete, attach, detach, extend, snapshot and others. The SolidFire driver also implements the Cinder functionality for Quality of Service (QoS) designations on volumes through the volume-type specifications (see next section for details). SolidFire also includes a unique capability to carry the tenant information into the storage subsystem and create unique iSCSI CHAP credentials for each tenant. The example below details the integration of names between OpenStack and SolidFire. You can see in the display from OpenStack that the tenant/project ID “18e278....” has an account created (automatically by the driver) on the SolidFire cluster and that there is one volume (a3f654…”) for the project in OpenStack attached to a running instance (“36444…”)

The ability to filter on the list of volumes by account allows for simple location of the volume, and the attributes provide some vital information about the volume including creation date, cloned count, and the instance the volume is attached to. The volume name is the volume_id from OpenStack prepended with the ‘UUID-’ prefix. This information may also be obtained from the SolidFire API for script and program interaction with the SolidFire cluster and OpenStack API at the same time.


In SolidFire, accounts provide the means to specify iSCSI CHAP credentials. Therefore, if an administrator needed to track down connection information, they could access the CHAP credentials for the tenant/project as needed (See below). The CHAP credentials are randomly generated by the SolidFire OpenStack driver and applied to the Hypervisor by Nova after requesting from Cinder.


On the hypervisor, because the iSCSI protocol mounts the device, one can, if needed, investigate the connections to the SolidFire array utilizing standard Linux tools, including the following commands:

ls -al /dev/disk/by-path/ip* iscsiadm -m node --show -T <iSCSI target> iostat -mtx 5 fdisk -l

5.3.2.2 QoS

The SolidFire cluster implements Quality of Service metrics and enforcement for each and every volume. Through tight integration with OpenStack through the Cinder driver, OpenStack can and will allocate every volume on SolidFire with QoS metrics (if nothing else, the cluster default metrics will be applied). There are two possible ways to assign QoS levels to volumes in OpenStack. The first option involves the OpenStack administrator configuring OpenStack Volume Types for each disk service offering. This method allows the OpenStack administrator full control and tenants may only create volumes based on administrator-created types. The second option allows the tenant to either choose from pre-configured values or the tenant can also create new performance types. Use the second option with caution. Providing multiple volume types and utilizing OpenStack Ceilometer gives administrators the ability to present a self-service, enforced performance and chargeback-capable storage system to their users. SolidFire has worked with many customers to build public and private clouds which charge a differentiated price structure for various performance levels of storage. Many of these customers label their volume types with the expected performance (min. max and burst) and price per GB per month; for example, “Gold-1000/2000/2500-$0.75/GB/month.”

Administrator Assigning QoS with OpenStack Cinder QoS Specs

To assign QoS specifications to a SolidFire volume, the OpenStack QoS specs are used. The SolidFire driver in OpenStack will check the volume type and QoS spec information by default when the SolidFire volume is created. Following is an example of setting up and using Quality of Service settings with a SolidFire back-end. Notice that the name for the Volume Type can be of your choosing; however, specific keys are expected in the QoS specs (minIOPS, maxIOPS and burstIOPS). Note: Admin access is required.


Note: Key names are case sensitive. As the OpenStack admin user, navigate to Admin -> System -> Volumes -> Volume Types. Click on the ‘Create a QoS Spec’ button and the ‘Create QoS Spec’ dialog box will appear. Enter a name for the QoS specification and indicate that the ‘Consumer’ will be ‘back-end’. The consumer is indicating where the Quality of Service is going to be enforced, either the back-end storage array or the front-end Hypervisor (KVM, QEMU, ESXi), or both. Upon completion, a new line will appear in the QoS Specs list. Next, click the ‘Manage Specs’ button and the details for the Gold QoS Spec will be shown. Click the ‘+ Create’ button and the ‘Create Spec’ dialog box (shown below) will appear.


Use this combination to create the 3 QoS keys (minIOPS, maxIOPS, burstIOPS) and their respective values. Once complete the Gold QoS Spec should look like the following:

Click ‘Close’ and the ‘QoS Spec’ list on the Volumes -> Volume Types page should now show. Next create a Volume Type to associate these QoS specs with by clicking on ‘Create Volume Type’ in the upper right corner of the Volumes -> Volume Types page. Click ‘Create Volume Type’ and then in the Actions column of the list, pull down the menu and select ‘Manage QoS Spec Association.’ The Associate QoS Spec with Volume Type dialog will appear and allow selection of the QoS Specs assigned to this Volume Type. Once the QoS Specs have been selected, you will see word ‘Gold’ in the second column of the Volume Type list.

Tenants may change volume-types on volumes as they wish and Ceilometer will log the changes such that critical jobs (at month end, for example) may be accelerated for the appropriate fee.


5.3.2.3 SnapShots/Clones

The SolidFire Cinder driver implements OpenStack snapshots as clones and indicates as such through the use of the is_clone attribute on the volume (shown below). Clones on SolidFire clusters are lightweight and initially take only additional metadata space. Clones and snapshots (which are clones) are named using the same convention as Volumes by the driver with the ‘UUID-’ prefix.

5.3.2.4 Data Efficiency

SolidFire clusters provide full de-duplication and two levels compression across the cluster by default. The efficiency information may be obtained from the Reporting tab of the SolidFire UI and overall efficiency including thin provisioning savings is report in the bottom left corner

5.3.2.5 Image Caching

SolidFire image caching is used to eliminate the copy of glance images to volumes every time a bootable volume is needed. SolidFire image caching is built into the OpenStack Cinder driver and is enabled by default, but provides the administrator with the ability to indicate which images to cache with a special property placed on the image in the Glance database. The image cache, upon the first copy operation from Glance to a volume (if the property is set), will copy the image to a volume stored under the OpenStack administrative user, and then, using the lightweight cloning operation on the SolidFire cluster, clone a copy for the user who has requested the volume. On subsequent image to volume operations for the same image, the volume for the admin user will be checked to make sure it is still current, and if current, a clone operation will be performed to quickly present the next volume. Depending on the size of the image, and on network and Glance storage performance, this cloning operation has been known to save up to one-half hour of copy time from Glance, per image, thus


providing great workflow efficiency for Development and Test environments or production on-demand workloads.

5.3.2.6 Limits and Capabilities

The SolidFire Element 8.0 cluster has the following design specifications that should be adhered to:

System Design Capability

Nodes in a cluster 4 Minimum, 30 (validated), 100 Maximum

Volumes 2000 per account, 17,500 total volumes per cluster, 700 per node with 400 mounted with active I/O

Volume Size 8TB

Volume Snapshots 32 per volume

Node IOPS 50,000 (80% read, 4KB random) per node for SF3010, SF2405, SF6010, SF4805 and SF9605 75,000 (80% read, 4KB random) per node for SF9010

Volume IOPS 15,000 (80% read 4KB random) per volume

Sequential Read BW 350 MB/s per node

Sequential Write BW 190 MB/s per node

Cloning 2 simultaneous clones in progress per volume

Integrated Backup/Restore Operations

2 simultaneous bulk volume operations in progress per volume.

5.3.3 Ceph Ceph is based on RADOS: Reliable Autonomic Distributed Object Store. RADOS distributes objects across the storage cluster and replicates objects for fault tolerance. RADOS contains the following major components:


● Object Storage Device (OSD) Daemon. The storage daemon for the RADOS service, which interacts with the OSD (physical or logical storage unit for your data).

● Meta-Data Server (MDS). Stores metadata. MDSs build a POSIX file system on top of objects for Ceph clients. However, if you do not use the Ceph file system, you do not need a metadata server.

● Monitor (MON). A lightweight daemon that handles all communications with external applications and clients. It also provides a consensus for distributed decision making in a Ceph/RADOS cluster.

The components of Ceph are installed and configured by Fuel in accordance to the Best Practices detailed by Mirantis. In our reference design we have instructed Fuel to configure a minimal 3 node cluster with the OSD and MON demons. Our Ceph cluster consists of Dell R620’s with the same CPU and memory configuration as our Controller and Compute nodes, but with two additional 300GB SSDs. We are not using the Ceph Filesystem, which is still considered Beta at this writing. Ceph Clients include a number of service interfaces. These include:

● Object Storage: The Ceph Object Storage (a.k.a., RGW) service provides RESTful APIs with interfaces that are compatible with Amazon S3 and OpenStack Swift.

● Block Devices: The Ceph Block Device (a.k.a., RBD) service provides resizable, thin-provisioned block devices with snapshotting and cloning.

● Filesystem: The Ceph Filesystem (CephFS) service provides a POSIX compliant file system usable with mount or as a filesystem in userspace (FUSE).

5.3.3.1 Object Storage

The Ceph Object Storage daemon, radosgw, is a FastCGI service that provides a RESTful HTTP API to store objects and metadata. It layers on top of the Ceph Storage Cluster with its own data formats, and maintains its own user database, authentication, and access control. The RADOS Gateway uses a unified namespace, which means you can use either the OpenStack Swift-compatible API or the Amazon S3-compatible API. For example, you can write data using the S3-compatible API with one application and then read data using the Swift-compatible API with another application. Ceph’s Object Storage uses the term object to describe the data it stores. S3 and Swift objects are not the same as the objects that Ceph writes to the Ceph Storage Cluster. Ceph Object Storage objects are mapped to Ceph Storage Cluster objects. The S3 and Swift objects do not necessarily

http://docs.ceph.com/docs/master/glossary/#term-ceph-object-storage






http://docs.ceph.com/docs/master/glossary/#term-ceph-block-device






http://docs.ceph.com/docs/master/glossary/#term-ceph-filesystem




http://en.wikipedia.org/wiki/RESTful

http://en.wikipedia.org/wiki/RESTful


correspond in a 1:1 manner with objects stored in the storage cluster. It is possible for an S3 or Swift object to map to multiple Ceph objects.

5.3.3.2 Block Storage

A Ceph Block Device stripes a block device image over multiple objects in the Ceph Storage Cluster, where each object gets mapped to a placement group and distributed, and the placement groups are spread across separate ceph-osd daemons throughout the cluster. Striping allows RBD block devices to perform better than a single disk could, but does compete with all operations active in the Ceph storage cluster. In virtual machine scenarios, a Ceph Block Device with the rbd network storage driver is integrated in Qemu/KVM, where the host machine uses librbd to provide a block device service to the guest. The RBD devices and protocol -- which are open, but exclusive to Ceph -- have been integrated with libvirt and any hypervisors that use it. OpenStack can use thin-provisioned Ceph Block Devices with Qemu and libvirt. Ceph does not support hypervisors other than those utilizing libvirt at this time (example Xen or Hyper-V), but one may integrate with other hypervisors themselves with the rbd command line tool provided with Ceph.

5.3.3.3 FileSystem

The Ceph Filesystem (Ceph FS) provides a POSIX-compliant filesystem as a service that is layered on top of the object-based Ceph Storage Cluster. Ceph FS files get mapped to objects that Ceph stores in the Ceph Storage Cluster. Ceph Clients mount a CephFS filesystem as a kernel object or as a Filesystem in User Space (FUSE). The Ceph Filesystem is still considered to be in beta at the time of this writing.

5.3.3.4 RADOS Protocol

Ceph does not utilize a standards-based storage protocol such as iSCSI or Fibre Channel, but accesses storage via an exclusive, but open source protocol called RADOS, referred to sometimes simply as RBD. The RBD protocol specifies a set of operations which are carried out over the TCP protocol between the client and the various Daemons of the Ceph cluster. The Ceph Storage Cluster does not perform request routing or dispatching on behalf of the Ceph Clients. Ceph Clients make requests directly to Ceph OSD Daemons which means clients need


access to all nodes in a Ceph cluster. Ceph OSD Daemons perform data replication on behalf of Ceph Clients, which means replication and other factors impose additional loads on Ceph Storage Cluster networks. Ceph (RedHat/Inktank) offer several libraries which provide easier access to the RADOS protocol and integrate the protocol into the Linux kernel. RADOS utilizes ports in the 6800 - 7300 range and therefore firewall access must be allowed between the clients and this port range.

5.3.3.5 Server and Disk Layout

The key to the Ceph and the RADOS protocol is the CRUSH (Controlled, Scalable, Decentralized Placement of Replicated Data) algorithm and CRUSH map to determine data placement. The CRUSH algorithm determines how to store and retrieve data by computing data storage locations. CRUSH empowers Ceph clients to communicate with OSDs directly rather than through a centralized server or broker. With an algorithmically determined method of storing and retrieving data, Ceph attempts to avoid single points of failure, performance bottlenecks, and physical limits to scalability. CRUSH requires a map of your cluster, and uses the CRUSH map to pseudo-randomly store and retrieve data in OSDs with a uniform distribution of data across the cluster. CRUSH maps contain a list of OSDs, a list of ‘buckets’ for aggregating the devices into physical locations, and a list of rules that tell CRUSH how it should replicate data in a Ceph cluster’s pools. By reflecting the underlying physical organization of the installation, CRUSH can model—and thereby address—potential sources of correlated device failures. Typical sources include physical proximity, a shared power source, and a shared network. By encoding this information into the cluster map, CRUSH placement policies can separate object replicas across different failure domains while still maintaining the desired distribution. For example, to address the possibility of concurrent failures, it may be desirable to ensure that data replicas are on devices using different shelves, racks, power supplies, controllers, and/or physical locations. The CRUSH map for our reference architecture simply has 3 buckets, one for each server, and weights the two OSD devices relative to their equal sizes (i.e. weights are equal because the SSDs are equal-sized).


Weighting Bucket Items Ceph expresses bucket weights as doubles, which allows for fine weighting. A weight is the relative difference between device capacities. We recommend using 1.00 as the relative weight for a 1TB storage device. In such a scenario, a weight of 0.5 would represent approximately 500GB, and a weight of 3.00 would represent approximately 3TB. Higher level buckets have a weight that is the sum total of the leaf items aggregated by the bucket. A bucket item weight is one dimensional, but you may also calculate your item weights to reflect the performance of the storage drive. For example, if you have many 1TB drives where some have relatively low data transfer rate and the others have a relatively high data transfer rate, you may weight them differently, even though they have the same capacity (e.g., a weight of 0.80 for the first set of drives with lower total throughput, and 1.20 for the second set of drives with higher total throughput). The bucket weighting is one of the main tunables of Ceph and one should consult the Ceph documentation for details on Tuning.


5.3.3.6 OpenStack Block Storage Integration

Ceph is integrated with the OpenStack block storage subsystem through Cinder and exclusive coded integrations via the librados library. The Cinder driver for Ceph contains all of the functionality to create, delete, attach, detach, extend, snapshot and most other volume manipulations. Since Ceph is uses its own protocol in place of the standard OpenStack storage protocols (iSCSI, FC, iSER, NFS etc.), at the present time there is no functionality to perform volume data manipulations on Ceph volumes (backup, image to volume, image conversion) by Cinder. This limitation may change in future OpenStack releases, but as of this writing is not included. Name integration between Ceph and Cinder has been done utilizing the volume and snapshot IDs assigned by Cinder and are allocated from pools configured by Ceph and the cinder.conf file. This straightforward integration strategy makes it easy for administrators to track down volumes and snapshots from OpenStack to their locations in Ceph. As an example see the following listing of a volume and its snapshots from Ceph and Cinder:

Ceph has been integrated with libvirt and therefore VMs will access volumes through the RBD protocol directly from libvirt and will not be seen by the Hypervisor operating system. As an example, the following is the disk configuration of libvirt for 2 VMs.

as you can see the first VM is configured to use a disk path from the Hypervisor and the second is configured to use Ceph via the RBD protocol directly from libvirt. Since the access is from libvirt and not the hypervisor kernel there is no visibility to the hypervisor.


5.3.3.7 Image Caching

If the Glance image storage is configured to directly utilize Ceph, Ceph will then attempt to perform a clone operation for all image to volume operations. The use of cloning by the Storage subsystem expedites the operation, prevents the repeated copying of images and allows the storage subsystem to thin-provision and de-duplicate data amongst volumes and images where possible, which is important for Ceph since it has no ability to deduplicate after the images and volumes are created. The use of image to volume operations is limited to ‘raw’ images since there is no ability by Cinder to mount the Ceph volume, convert the image (from qcow2 for example) to a raw format and apply that raw image to the Ceph volume.

5.3.4 Image storage OpenStack image storage is actually a misnomer, in that the image service, known as Glance, does not provide any actual storage. Glance provides a catalog of images (VM templates) which contains location information and helper routines store and retrieve the actual images. Glance may also build an image cache which may be used on each or some hypervisors to speed up the creation of VMs with ephemeral storage. In the reference architecture we have chosen Ceph to be the primary storage location for image data (This is the Fuel default).

5.3.4.1 Standard Catalog Images

The standard image catalog consists of database entries and storage locations for each image. Glance will create the database entries and utilize its configured storage location to maintain the image. In this standard case for our reference architecture the image is located on the Ceph storage and is located under the ‘images’ folder within Ceph. The image information may be examined using the following command (example shown below):

rbd ls -l images


Note: Within Ceph, the image files are stored with the image_id assigned by Glance.

5.3.4.2 Snapshot Image

Image snapshots are designed to allow users to quickly snapshot an instance and upload it as a new image within the Glance catalog. To demonstrate this capability, create an instance (VM) with a bootable volume using Project (non admin) -> Compute -> Instances -> Launch Instance and select the Instance Boot Source as shown below.

Once the instance is booted, in the Admin -> System -> Instances -> <Instance> -> Create Snapshot and type in a name. You will be taken to the project -> Compute -> Images tab and a new image will have been created with your snapshot name. If we look at how this image is constructed via the command line, we can see the glance image with no Format or Size information as shown below. Looking at the image detail, we see the block_device_mapping property has a snapshot ID which corresponds to the snapshot on the volume as shown below.


The snapshot image capability is a great method to create golden images from a running instance. This functionality is limited to the Ceph back-end only. The images created this way may not be used with other back-ends, and OpenStack will fail silently when attempting to use these images with other back-ends.


The snapshot image functionality has only been partially implemented by the Glance project for storage other than Ceph in OpenStack releases up to and including Liberty. While this functionality appears to work for SolidFire (or any other back-end) it does not actually work. If one attempts to use this with iSCSI/FC back-ends, the image will appear in the catalog, but the snapshot on the storage will never appear and OpenStack will report the snapshot being in error. The recommended approach to this need for all storage back-ends is to utilize snapshots through the volume subsystem and then to clone those snapshots to bootable images through the volume subsystem as shown in this sequence. First create a volume from an image and boot the instance from the volume.

Next create a snapshot of the root volume of the running instance; in this case use the force option to force the snapshot while the instance is running (this will not be a consistent snapshot, if you want a consistent snapshot shutdown the VM). After creating the snapshot, boot another instance from the snapshot (this will create a new volume from the snapshot) as shown below.


If more Instances are needed continue to create instances from the Golden snapshot. NOTE: This method documented here will work for all storage back-ends and is therefore the recommended method for creating Instances from a Golden Image.

5.3.5 Object Storage Object storage (also known as object-based storage[1]) is a storage architecture that manages data as objects, as opposed to other storage architectures like file systems which manage data as a file hierarchy and block storage which manages data as blocks within sectors and tracks.[2] Each object typically includes the data itself, a variable amount of metadata, and a globally unique identifier. Object storage, stores and retrieves arbitrary unstructured data objects via a RESTful, HTTP based API. It is highly fault tolerant with its data replication and scale out architecture. Its implementation is not like a file server with mountable directories. OpenStack Object Storage is based on the Swift object model and may be implemented by a number of software packages including Swift, and in the case for this reference architecture, Ceph.

https://en.wikipedia.org/wiki/Object_storage#cite_note-1

https://en.wikipedia.org/wiki/Object_storage#cite_note-2


In this reference architecture we have allowed Fuel to configure Object Storage through Ceph for completeness, but is not the focus and therefore less documented.

6 Best Practices

6.1 Mirantis OpenStack Best Practices

● Planning is important for successful deployment. The Mirantis OpenStack Planning Guide is a recommended resource for the planning phase.

● The following set of hardware should be provided: One Mirantis OpenStack server is required for all Mirantis OpenStack deployments Three controllers are required for HA An adequate number of compute nodes for the projected workload Redundant power and networking is highly recommended If possible, nodes should be equipped with:

Four 10GbE interfaces for LACP and separation of storage and general networking

One GbE interface for admin/PXE network ● A larger number of cloud nodes is preferable to cutting edge CPUs and large amounts of

memory in a small number of nodes. Providing a larger number of nodes improves both performance and resilience to node failure while usually reducing cost. (Scale-out architecture is preferred to scale-up architecture.)

● While Mirantis OpenStack provides network verification, it is highly recommended to double check IP ranges to ensure a smooth deployment. IP ranges cannot be changed later without redeployment.

● Mirantis recommends adhering closely to the Mirantis OpenStack reference architecture for general Mirantis OpenStack configuration and deployment. Deviations from the reference architecture are feasible and in some cases required, but add complexity and require additional testing as well as operational procedures to support them.

● Changes after Fuel deployment should be kept to a minimum. If a major change is necessary, development of a Fuel plugin should be considered to provide repeatability of the deployment as well as reduce the risk of human error.


For further details on these and other best practices, please refer to the Mirantis OpenStack 7.0 documentation.

6.2 Networking Best Practices ● Implement a redundant, fault-tolerant network infrastructure. ● Implement a dedicated 10 gigabit Storage Network. ● Implement a dedicated, or shared network for SolidFire storage node management and

intra-cluster communications. ● Implement Jumbo Frames support end-to-end within the switching infrastructure, on all

SolidFire storage nodes, and all servers connected to the storage network. ● Implement redundant connectivity for all storage nodes. ● For optimal end-to-end availability and enhanced performance, configure redundant NIC

connectivity for all server-side connections utilizing NIC switch fault-tolerance and/or NIC bonding, where supported.

● Storage-side switches should have a non-blocking backplane architecture, and support full-duplex, line-rate forwarding on all ports. Backplane capacity should be greater than or equal to the following calculation: ((the total number of switch ports * 10Gb/s) * 2).

● SolidFire suggests a minimum of 512K packet buffers per port on all storage network switches. Some switches, by design, share the buffer allocation either between a group of ports, or for all ports as a whole on the switch. Best practice dictates using a switch that implements per-port packet buffers for dedicated high-bandwidth storage networks, otherwise performance may not be optimal. If the switch does not support per-port packet buffers, tune the settings, if possible, to allow adequate buffer space for the entire group of ports, and minimize the possibility of packet loss.

● Ethernet flow control must be implemented in a consistent manner across the network infrastructure to properly deal with network over-subscription situations that could impact performance. Failure to do so will likely cause packet loss and degraded network throughput.

● SolidFire storage nodes require availability of the following network services via the 1GbE Management interfaces and network to ensure proper SAN functionality: DNS - outbound access required for client lookups SNMP - inbound access for SNMP queries Syslog - outbound access for logging to an external log server SSH - inbound access required for node-level management


HTTPS - inbound access required for Cluster-level management and API communications

SolidFire Fast Deploy Virtual Appliance communications ● Disable Delayed ACK - it's mostly for low b/w environments ● Disable Large Receive Offload (LRO) on network cards – this is associated with TOE and is

known to cause similar issues with ISCSI traffic

6.3 Storage Best Practices

● When Cinder is deployed with SolidFire, Cinder snapshots are created by leveraging the Cloning feature on the SolidFire Cluster. An attribute “is_clone” is set to True on the cloned volumes.

● SolidFire does not provide traditional multipath HA from Linux perspective. The SolidFire cluster uses a Virtual IP address which initiators communicate with. In the event of a failure the Virtual IP address is moved throughout the cluster to provide resiliency. Therefore; SolidFire does not recommend configuring multi-path HA in Nova or Cinder.

● If live migration of VMs is a functional requirement of an OpenStack deployment, SolidFire recommends disabling Configuration drives in the OpenStack environment as the existence of Config Drives prevents Live Migration.

● VLAN Tagging is supported on the SolidFire SVIP interface. However, OpenStack Cinder and the SolidFire driver have no concept of VLANs, therefore it is not recommended to utilize VLANs. However port-based, Layer-2 VLANs may be implemented at the customer switch for creating an isolated storage network.

6.4 Database Best Practices ● If you choose to place the OpenStack MySQL database on SolidFire

512-byte emulation on the SolidFire volume is required. Use the InnoDB storage engine of MySQL. Restrict database network traffic to the OpenStack management network. Configure the database to require SSL for database clients to protect data “on the

wire.” SolidFire recommends breaking out the MySQL database components into

individual volumes, allowing the appropriate MySQL binary log files to be applied for any necessary recovery after a restore of the data.


MySQL has a default I/O block size of 16k, which SolidFire translates to 4k blocks in the Volume creation and modification interfaces. This default can be adjusted to 4k or 8k via the innodb_page_size configuration option, but this must be done before any InnoDB tablespaces are created. Existing databases will most likely be set to the default 16k page size.

To determine the correct QoS settings for your existing application, a good rule of thumb is to take your specific MySQL IOPS requirements (at 16k block size) and multiply that number by 2.7 to get the SolidFire IOPS setting (at 4k). This adjustment factor is determined from a non-linear algorithm that accounts for the conversion between the 16k and 4k block sizes.

Create application-consistent backups with SolidFire Snapshot copies by flushing the MySQL cache to disk.

With SolidFire Element 6 and above, Real-Time Replication can be used to create a Cold Standby DR copy of a MySQL database in a remote SolidFire Cluster. To make use of this feature, set up the target MySQL volume as a replication target and pair the volumes as per the Element 6 User Guide (Real-Time Replication section).

Follow the Galera configuration recommendations defined in the MySQL Galera cluster documentation.

Deploy a minimum of three nodes for the MySQL Galera cluster because Galera is a quorum-based system that requires at least three nodes to achieve quorum.

SolidFire highly recommends that database clients access MySQL Galera instances through a load balancer as set up by Fuel.

For appropriate MySQL and SolidFire settings, refer to the SolidFire Best Practices for MySQL document available from the SolidFire website.


6.5 Hypervisor Operating System Tuning To obtain maximum performance from your SolidFire volumes there are some Linux kernel parameters that can be tuned. Current tuning parameters for the volume can be checked by running a few commands. Procedure: To check tuning parameters, run the following commands: demo@demo:~# cat /sys/block/sdX/queue/rotational demo@demo:~# cat /sys/block/sdX/queue/scheduler demo@demo:~# cat /sys/block/sdX/queue/nr_requests demo@demo:~# cat /sys/block/sdX/queue/add_random

It is recommended for OpenStack deployments that you set the tuning parameters with udev rules. Procedure:

1. Create the following file as root: /lib/udev/rules.d/62-SFudev.rules 2. Edit the /lib/udev/rules.d/62-SFudev.rules file as root and add the following lines:

KERNEL=="sd*", \ SUBSYSTEM=="block", \ ENV{ID_VENDOR}=="SolidFir", \ RUN+="/bin/sh -c 'echo 0 > /sys/$devpath/queue/rotational && \

echo noop > /sys/$devpath/queue/scheduler && \ echo 1024 > /sys/$devpath/queue/nr_requests && \ echo 0 > /sys/$devpath/queue/add_random && \ /sbin/hdparm -Q <QD> /dev/$kernel \ ' \

"

a. See below “Volume I/O Queue Depth” for an explanation on setting the <QD> value shown.

3. Trigger the udev rules so the tunes are executed: demo@demo:~# sudo udevadm trigger. NOTE: This is only done when the rules are created or updated. Going forward the rules are matched/executed when the OS boots.


4. Verify that the tunes have been made by rechecking the tuning parameters (Checking Tuning Parameters). The values should be: rotational 0 scheduler [noop] nr_requests 1024 add_random 0

Volume I/O Queue Depth To optimize SolidFire QoS, it is necessary to update the volume queue depth to the appropriate value. If the queue depth is too high, then frames remain in the active queue too long. However, if the queue depth is too low, then the volume is unable to reach its desired performance levels. The following table assists you in evaluating what your queue depth should be:

Min IOPS Queue Depth

100-199 1

200-399 2

400-799 4

800-1599 8

1600-3199 16

3200-6399 32

6400+ 64

In addition, certain hypervisors and HBA’s may throttle queue depth; refer to the Configuring SolidFire's Quality of Service or Defining SolidFire's Quality of Service guides for additional details. If the hdparm command cannot complete, then check and update the settings on these devices accordingly. NOTE: The queue depth settings listed are suggestions only. They should be used as a starting point for tuning your OS and application performance.


7 Implementation

7.1 SolidFire Fuel plugin This reference architecture is deployed with a Mirantis Unlocked Validated Fuel plugin from SolidFire. The plugin provides a simple configuration of Fuel to configure OpenStack to utilize the Solidfire storage cluster. The plugin will eliminate the need to configure OpenStack manually if SolidFire is the only storage in the OpenStack environment. If other storage (such as Ceph) is to be used in addition to the SolidFire cluster, so small manual steps are required until Fuel is patched to support multi-back-end. The steps are all documented below. The plugin may be downloaded from the Fuel Plugin Catalog.

7.2 Deployment Workflow Deployment of the Fuel Master Node should follow directly from the Fuel User Guide. When arriving at the section Boot the node servers we have chosen to use a script which contacts the Dell Remote Access Controller (DRAC) and configures the Dell servers to PXE boot and then reboots the servers. In the following it is assumed that the DRAC has been configured with proper SSH keys such that passwords are not required. To configure a SSH key, using the root pasword: ssh [email protected] 'racadm sshpkauth -i 2 -k 1 "contents of the public key file"'

The servers DRACs we have been assigned for this reference architecture are consecutively numbered from 100-107 on their subnet, so a simple sequence of those IPs is used to drive the script. The script and its output are shown below. for i in `seq 100 107`; do echo $i ssh [email protected].$i "racadm config -g cfgServerInfo -o cfgServerBootOnce 0" ssh [email protected].$i "racadm config -g cfgServerInfo -o cfgServerFirstBootDevice PXE" ssh [email protected].$i "racadm serveraction powercycle" done

https://www.mirantis.com/products/openstack-drivers-and-plugins/fuel-plugins/






https://docs.mirantis.com/openstack/fuel/fuel-7.0/user-guide.html





https://docs.mirantis.com/openstack/fuel/fuel-7.0/user-guide.html#boot-the-node-servers








In the next section of the user guide, Install Fuel Plugins, install the SolidFire Fuel plugin as shown below by copying the plugin to the master node using: scp fuel-plugin-solidfire-cinder-1.1-1.1.0-1.noarch.rpm root@fuel-master-node:/tmp

and then installing with the command: fuel plugins --install /tmp/fuel-plugin-solidfire-cinder-1.1-1.1.0-1.noarch.rpm

https://docs.mirantis.com/openstack/fuel/fuel-7.0/user-guide.html#install-fuel-plugins






To verify that the plugin has been installed correctly, consult with the Fuel GUI and the ‘Plugins’ page as shown below.

At this point, we will create an environment for the reference architecture. The steps are documented below. Select ‘New OpenStack Environment’ and begin the wizard process by providing a name for the environment (we will call ours ReferenceArchitecture1). In Fuel 7.0 there are no other options for OpenStack Release than the Kilo on Ubuntu release.


Next select the compute virtualization options. We will be using hardware based virtualization based on KVM for this reference architecture.

For our hardware setup, we have been provided the necessary hardware networks, but not sufficient VLANs to provide each tenant with a VLAN for their private traffic, so we will let Fuel build an OpenStack environment with VXLAN segmentation.


We will be deploying the reference environment with SolidFIre and Ceph based storage, to demonstrate how to make the two platforms work together, so select Ceph as shown.

Finally we will install Ceilometer in case we decide to use the Heat AutoScaling features.


Upon completion of the environment wizard you will be presented with other configuration options for the environment including the node list. Personally, I like to work these backwards from the settings tab. For our environment, we change two things within the settings tab, one being under common to enable OpenStack Debug logging. The second change we need to make is to enable the SolidFire Cinder plugin and enter the appropriate configuration information for the SolidFire cluster. The SolidFire cluster Management Virtual IP (MVIP) address needs to be entered into the Plugin configuration along with the Cluster Admin login and password you wish to use. The default cluster admin may be used or a new cluster admin may be created with Reporting, Volumes and Account credentials minimum (shown below).


The remainder of the settings in the SolidFire plugin will be left at their defaults for the environment as described, but for convenience, they are described here

● Cluster endpoint Port - If your architecture requires a proxy between the SolidFire and the OpenStack controller nodes, you may need to change the Cluster Endpoint port.

● Enable Caching/Template Account - Image Caching is enabled by default at the driver/cluster level, if you wish to not use image caching you can disable it, or if you wish to have cache images contained under a different account on the Solidfire cluster, change the Template account.

● SolidFire Emulate 512 block size - SolidFire natively uses 4KB block size, but will emulate a 512B block device if requested. Qemu and other parts of OpenStack require 512B support therefore leave this enabled.

● SF account prefix - If multiple OpenStack environments will be using the same SolidFire cluster, an account prefix may be added to each account created. This should be set if you anticipate multiple OpenStack environments access the same SolidFire cluster.

Click ‘Save Settings.’


Moving on to the ‘Network’ tab next, fill in the information for the networks as designed. In our case, we will move the Public network to a VLAN and configure the appropriate VLANs and IP addresses for each network. If you are using an existing network (as we are for our storage network) and need to configure specific IP ranges for those networks, configure the base information here and in the next step we will reduce the range as appropriate.

In our storage network, we need to utilize a subset of the IP range on the VLAN25 subnet 172.27.25.0/24 so we will follow the procedure as described in the ‘Adding Network Ranges” subsection of the Using the Fuel CLI section of the Fuel User Guide, as shown below for convenience.

https://docs.mirantis.com/openstack/fuel/fuel-7.0/user-guide.html#using-fuel-cli









On the Fuel master node, we will run the commands shown below:

Once the networks are defined correctly, proceed to the ‘Nodes’ tab and add nodes to the various roles. For this reference architecture we will build three controller nodes, two compute nodes and three Ceph nodes. The Ceph nodes need to have enough disk space and must have similar disk configurations, so we’ll select the three nodes with the appropriate disk configurations and the other node roles will be assigned from the remaining nodes.


Once all the nodes have been defined, select all of them and click on the ‘Configure Interfaces’ button in the upper right. This screen is where you define the logical to physical network layout. For our design we will bond the two 10Gbps ethernet interfaces together with LACP bonding and place the 4 VLANs on top of the bond. There are 3 options available for the bonding interface, Balance-rr, Active-Backup and LACP. Both Balance-rr and LACP require specific switch configurations. Active-Backup is the safest for any switch configuration, but also limits to the bandwidth of a single interface. NOTE: We experienced server network problems when we selected Balance-rr by accident with our multi-switch setup. If your network is not properly configured, use Active-backup until the network is configured properly for Balance-rr or LACP. NOTE: We have also changed the Maximum Transmission Unit (MTU) of the bond interface to 9000 to support jumbo frames to increase throughput on the networks.


Click ‘apply’ and then jump back to the ‘Networks’ tab and scroll to the bottom and run the network verification tests as shown below.

Correct any errors discovered in your network configuration. Next, on the ‘Nodes’ tab select the three ceph nodes, click ‘Configure disks,’ and verify their disk configuration meets the design.


In our case, the default configuration selected by Fuel is sufficient.

We are now ready to deploy the environment. Select the ‘dashboard’ tab and verify that there are no messages that need to be addressed and click on the ‘Deploy Changes’ button. This process will take about one hour.


Monitor overall progress on the ‘Dashboard’ tab.

Or monitor process of individual nodes on the ‘Nodes’ tab.


When the deployment process is complete, you will see a message similar to the one shown below (Depending on which plugins you have installed). Notice the message indicating the environment has been deployed with the SolidFire plugin.


At this point, you will have functioning OpenStack environment which may be accessed by the “Proceed to Horizon’ button. However, it will be configured for SolidFire storage only. The next steps will take you through the process to enable SolidFire and Ceph storage together. Login to the Fuel master node and run the command:

fuel node list | grep controller | awk '{print $10}'

to get a list of the IP address of the 3 controller nodes. Login from the fuel master server to each of the controllers listed with the above command and then back up and edit the file /etc/cinder/cinder.conf. At the bottom of the file, create a new section called Ceph with the [ceph] syntax. Then move the following lines into this section from the [DEFAULT] section of the file (The UUID will vary, do not change the lines). [ceph] volume_driver=cinder.volume.drivers.rbd.RBDDriver volume_back-end_name=DEFAULT rbd_user=volumes rbd_max_clone_depth=5 rbd_ceph_conf=/etc/ceph/ceph.conf rbd_flatten_volume_from_snapshot=False rbd_pool=volumes rbd_secret_uuid=a5d0dd94-57c4-ae55-ffe0-7e3732a24455

Note: There is a similar ‘volume_driver’ line in the [solidfire] section, leave it in the [solidfire] section and make sure you move the one from the [DEFAULT] section. Once the [ceph] section has been created, enable it by modifying the enabled_back-ends directive in the [DEFAULT] section. It should now say: enabled_back-ends=solidfire,ceph

Save the file. Log out of the controller node and repeat on all controller nodes. (Note: There are differences in the cinder.conf on each controller node. Do not simply copy the file amongst controller nodes). Once the file has been changed on each controller and you are back on the Fuel master node for the last time, run the following: for i in `fuel node list | grep controller | awk '{print $10}'` ;do echo "--> " $i; for j in api backup scheduler volume;do ssh $i service cinder-$j restart;done;done


You should see the following:

Now verify that the services have been configured and started properly. Using Horizon with the admin login, select Admin -> System Information -> Block Storage Services, as shown below. Notice that there are two cinder-volume processes which are down. This is expected, since we have moved these services to a section other than DEFAULT. Note: The host column of the table shows the routing of requests to the various back-ends. The format is <driver:config>@<hostname>#<pool>. Both back-ends use a generic hostname such that any controller in the redundant group can service the request, which is the case when storage is in an independent back-end. If the storage is physically located on a single node, only that node can service that request (i.e. LVMdriver) and the hostname should be a physical host. Note: If you have previously created volumes in Ceph while it was configured in under the DEFAULT section, those volumes will need to be migrated in the database to their new section name (i.e. ceph). To perform the migration use: cinder-manage volume update_host --currenthost CURRENTHOST --newhost NEWHOST@BACKEND


At this point, we need to configure OpenStack volume-types for each storage back-end. Login to Horizon as the admin user and select the Admin -> Volumes menu from the left and then select the ‘Volume Types’ tab at the top and the ‘+ Create Volume Type’ on the upper right. A ‘Create Volume Type’ dialog will appear as shown below:

Type a name and click ‘Create Volume Type’. Do this once for SolidFire and once for Ceph. Upon completion, find the two types in the main table as shown below and use the pulldown on each one, selecting ‘View Extra Specs.’


The following will display:

Click Create:

... and fill in the key as ‘volume_back-end_name’ and the value as ‘solidfire’ (Note: be aware of the spelling and capitalization of key and value.) Do this again for Ceph: the ‘Value’ for Ceph should be ‘DEFAULT’ as shown below:

This value comes from the ‘volume_back-end_name’ definition in the cinder.conf file. Check the one for SolidFire in the [solidfire] section of cinder.conf.


8 Verification and Testing

8.1 Health Check Fuel provides a 6th tab called ‘Health Check’ which provides a very nice test of the functions of the deployed OpenStack environment. The Health Check may be run at any time after the deployment process, and will check the default storage back-end only. Select the ‘Select All’ button and ‘Run Tests.’ The tests take about 15 minutes to complete and each test has an expected run time listed.

In our reference environment, all tests passed, except for the last three which check to determine if the OpenStack environment is still using the default login credentials, which we are. Our Lab environment is very isolated and transient, so we will leave the default credentials in place.



8.2 Functional testing

8.2.1 Verify Backend Placement. In addition to the Health check tests, the reference architecture begs a test of both storage back-ends to verify that volumes are being created on the back-end requested. Start by creating a volume on each back-end by passing the appropriate volume-type to cinder create (as shown below) root@node-25:~$ cinder create --volume-type solidfire --display-name "solidfire" 20 +---------------------+--------------------------------------+ | Property | Value | +---------------------+--------------------------------------+ | attachments | [] | | availability_zone | nova | | bootable | false | | created_at | 2015-12-09T21:00:11.949334 | | display_description | None | | display_name | solidfire | | encrypted | False | | id | d27b9207-bae7-4c07-8c27-ae713ee0b28a | | metadata | {} | | multiattach | false | | size | 20 | | snapshot_id | None | | source_volid | None | | status | creating | | volume_type | solidfire | +---------------------+--------------------------------------+


root@node-25:~$ cinder create --volume-type ceph --display-name "ceph" 22 +---------------------+--------------------------------------+ | Property | Value | +---------------------+--------------------------------------+ | attachments | [] | | availability_zone | nova | | bootable | false | | created_at | 2015-12-09T21:00:33.660239 | | display_description | None | | display_name | ceph | | encrypted | False | | id | cca5b5a8-37ce-46c8-ae91-4fec922304bf | | metadata | {} | | multiattach | false | | size | 22 | | snapshot_id | None | | source_volid | None | | status | creating | | volume_type | ceph | +---------------------+--------------------------------------+

Then check the Ceph volumes pool using the rados ls command to make sure the volume is seen within Ceph. The Ceph volume name is the same as the volume ID created by Cinder. See below for an example. root@node-25:~# rados -p volumes ls rbd_directory rbd_id.volume-cca5b5a8-37ce-46c8-ae91-4fec922304bf rbd_header.69b9ceb4985

In order to verify the creation on the SolidFire cluster check the SolidFire UI and use the volume Filter to search for the created volume. The SolidFire volume name is created by pre-pending UUID- to the volume ID from Cinder. See example below.


8.2.2 Configure and Verify SolidFire Image caching SolidFire image caching is used to eliminate the copy of glance images to volumes every time a bootable volume is needed. In order to take advantage of SolidFire image caching, an additional parameter named ‘virtual_size’ must be placed on the Glance images. This additional parameter also functions as a method to enable caching on individual images within the glance catalog. To configure to configure the virtual_size parameter on a glance image perform the following as admin

root@node-25:~$ source ~/admin-openrc.sh root@node-25:~$ glance image-list +--------------------------------------+--------------------+-------------+------------------+-------------+--------+ | ID | Name | Disk Format | Container Format | Size | Status | +--------------------------------------+--------------------+-------------+------------------+-------------+--------+ | 4e9cae63-cb9a-4650-a596-55e952f56eaa | CentOS 6 | qcow2 | bare | 714473472 | active | | b983db69-3d44-4e5a-96bc-a5cda0a7762c | CentOS 7 | qcow2 | bare | 859111424 | active | | 22507675-fe29-4011-a019-7964e1cfabc8 | Fedora 23 | qcow2 | bare | 234363392 | active | | c8d5caa2-2a5c-4ecd-aa9b-0f1d19b184ee | GoldenImageSnap | | | | active | | 7efc0905-9fff-46ab-b7fa-664db088b65f | TestVM | qcow2 | bare | 13287936 | active | | 11e362ed-3285-4bad-add1-431ba270291b | Ubuntu Precise | qcow2 | bare | 263455232 | active | | 8c63e3b2-8cc1-4395-92b2-73a768297c89 | Ubuntu Trusty | qcow2 | bare | 258408960 | active | | 67465d23-29f3-46d7-b1e5-2bc8deb6e0d7 | Windows2012R2-Eval | qcow2 | bare | 17182752768 | active | +--------------------------------------+--------------------+-------------+------------------+-------------+--------+


Show the selected image to make sure the virtual_size property is not already set.

root@node-25:~$ glance image-show b983db69-3d44-4e5a-96bc-a5cda0a7762c +------------------+--------------------------------------+ | Property | Value | +------------------+--------------------------------------+ | checksum | 647723df10cd9c933ecc06056e52ca73 | | container_format | bare | | created_at | 2015-12-08T20:21:46.000000 | | deleted | False | | disk_format | qcow2 | | id | b983db69-3d44-4e5a-96bc-a5cda0a7762c | | is_public | True | | min_disk | 0 | | min_ram | 0 | | name | CentOS 7 | | owner | 231f33cc482f426eadf2165780e0379f | | protected | False | | size | 859111424 | | status | active | | updated_at | 2015-12-08T20:25:50.000000 | +------------------+--------------------------------------+

Then add the virtual_size property to the image using the image-update command to Glance. Use a virtual_size equivalent or larger than the minimum root disk of a running VM from the image. For example this CentOS 7 image requires a 16GB minimum root drive (Note: the compressed qcow2 image is 859MB)

root@node-25:~$ glance image-update --property virtual_size=1600000000 b983db69-3d44-4e5a-96bc-a5cda0a7762c +-------------------------+--------------------------------------+ | Property | Value | +-------------------------+--------------------------------------+ | Property 'virtual_size' | 1600000000 | | checksum | 647723df10cd9c933ecc06056e52ca73 | | container_format | bare | | created_at | 2015-12-08T20:21:46.000000 | | deleted | False | | deleted_at | None | | disk_format | qcow2 | | id | b983db69-3d44-4e5a-96bc-a5cda0a7762c | | is_public | True | | min_disk | 0 | | min_ram | 0 | | name | CentOS 7 | | owner | 231f33cc482f426eadf2165780e0379f | | protected | False | | size | 859111424 | | status | active | | updated_at | 2015-12-12T22:17:20.000000 | | virtual_size | None | +-------------------------+--------------------------------------+


To verify SolidFire Image caching is configured and working correctly, create a bootable volume through OpenStack by selecting Project (non-admin) -> Compute -> Volumes -> Create Volume and then selecting Volume Source = Image and selecting a Glance image from the pull down list, as shown below. Note: Make sure you size your volume for the image selected (i.e. if the image is 16GB the volume must be 16GB or larger depending on how the image is configured).

Next, login to the SolidFire UI and look in the Accounts tab to verify that an account called “openstack-vtemplate” has been created as shown below.


NOTE: The “openstack-vtemplate” account will be prefixed by any sf_account_prefix configured in the cinder.conf. Click on the number in the ‘Active Volumes’ column to filter the volume list by the account. You should see at least one volume under the ‘openstack-vtemplate’ account. In the example below, there are two.

Click on the ‘Volume details’ tab to see the Attributes assigned to the volumes. Within the attributes column you will notice the details of the Glance image that is contained in that cache entry including Image_id, Image_name, Image_info and creation date. This attribute information allows the admin to track back to glance each cache entry. If entries are no longer desired, they may be deleted from the SolidFire via the SolidFire UI (the OpenStack driver will re-create the cache entry as needed).


As an example of how SolidFire image caching will help during Bootable volume creation, the following example (shown below) was run on the reference architecture using the verification scripts available at the Solidfire Github site. In the example, we are using an image which is 17GB and the first time the bootable volume is created it takes 480 seconds, subsequent creation once the cache was created took just 60 seconds.

https://github.com/solidfire/solidfire-ai/tree/master/sfai-openstack/verification_scripts






8.3 System testing In addition to validating functionality, installation and deployment, modification of environment using the plugin, and uninstall were thoroughly tested and verified for production use with Mirantis OpenStack. Note: The SolidFire Cinder drivers and Fuel plugin integration are validated on Mirantis OpenStack based on Juno (Mirantis OpenStack 6.1) and Kilo (Mirantis OpenStack 7.0). Note 2: The Cinder Volume Manager service must be enabled on all Controller nodes.

8.4 Performing functional testing In addition to the testing performed by Fuel, SolidFire has a set of scripts to create a large number of boot from volume instances to confirm functionality of an Openstack setup. The scripts may found in our Agile Infrastructure github site. We will utilize five of the scripts available to 1) Build template and clone 40 volumes. 2) Boot 40 instances from those 40 volumes. 3) retype the volumes the instances are built on and observe the performance changes. 4) delete the 40 instances and 5) delete the 40 volumes. We have created a VM image which automatically runs a webserver type workload upon boot. The image has been loaded into the glance image catalog and is ready for use by Openstack. The VM will run the ‘fio’ testing utility on boot with the config file (below) and output its performance into a local file for future reference.

The workload of this ‘fio’ job will be run against the root disk of the instance, so it is important that the root disk be on the device under test (i.e. the SolidFire). We begin by creating a template from our image and then cloning 40 bootable volumes from the template.









Next we use the scripts to boot the instances from the volumes.

For this example, we will change the volume-type from webserver to bronze and watch the change in performance. below you can see the specification we are using for ‘webserver’ and ‘bronze’ types.

We’ll utilize our scripts to retype all 40 (not the template) volumes and confirm the retype has completed with a ‘cinder list’ command.

Upon changing the volume type we can observe the SolidFire UI for the performance of the entire cluster and see that a dramatic increase in performance has been observed (below).


The SolidFire UI also provides per volume statistics, and if we look at one of our volumes, we can see an appropriate increase in performance. The per volume performance graph also provides hard Current, Average and Peak numbers for each metric.


As a final check, we log into one of the running VMs and look at the last 20 lines of output from our load generator and see that the performance has gone up significantly (column 1: Read BW, Column 2: Read IOPS, Column 3: Write BW, Column 4: Write IOPS) when the volume retype was performed (see below).


Lastly we can clean up our volumes and instances.

9 Support Support for SolidFire’s storage systems with Mirantis OpenStack is provided jointly by SolidFire and Mirantis. Please call either SolidFire or Mirantis and have your issue resolved accordingly. In general, storage-related and driver-related issues will be resolved by SolidFire Support and Mirantis OpenStack-related issues will be resolved by Mirantis Support.

10 Conclusion OpenStack Storage is becoming increasingly complex. This reference architecture from SolidFire and Mirantis provides a stable, high-performance and affordable solution with many options for customers whose businesses depend on their ability to store, protect and serve data. The reference architecture should be easy to install and maintain. The combination of SolidFire and Mirantis


OpenStack expertise help simplify deployment and a tested and certified Mirantis OpenStack 7 plugin allows short time-to-value as well as a reliable and performant base for long term business success. The configurations in this reference architecture and the Mirantis OpenStack 7 plugin have been tested and verified by SolidFire and Mirantis and are presented jointly by both companies. Mirantis and SolidFire recommend implementing SolidFire storage in a Mirantis OpenStack context based on the recommendations in this reference architecture. Deviations from the reference architecture are not advised unless directed by Mirantis and SolidFire. On request, the expert teams at Mirantis and SolidFire can provide assistance with a reference architecture deployment or a custom architecture for use cases not covered by here. Please direct any SolidFire questions via this form and Mirantis-related questions via this Contact form.

11 Software and Documentation ● Mirantis OpenStack 7.0 ● Mirantis OpenStack 7.0 SolidFire Fuel Plugin ● Mirantis OpenStack 7.0 documentation

Mirantis OpenStack 7.0 Reference Architectures User Guide Operations Guide

● Mirantis Support ● SolidFire Resource Center ● SolidFire OpenStack Documentation ● SolidFire Support

12 Addendum - Network Config Configuring the network switch Key Considerations Ensure that all OpenStack and SolidFire node 10G switch ports are configured to meet the following requirements:

http://www.solidfire.com/contact/



https://content.mirantis.com/contact-us-landing-page.html



https://www.mirantis.com/products/mirantis-openstack-software/













https://docs.mirantis.com/openstack/fuel/fuel-7.0/#guides





https://docs.mirantis.com/openstack/fuel/fuel-6.1/reference-architecture.html







https://docs.mirantis.com/openstack/fuel/fuel-7.0/user-guide.html#user-guide



https://docs.mirantis.com/openstack/fuel/fuel-7.0/operations.html#operations-guide



https://www.mirantis.com/support/enterprise-support-services/



http://www.solidfire.com/resources/





http://www.solidfire.com/solutions/cloud-orchestration/openstack/





http://www.solidfire.com/support/




OpenStack node ports

● 802.1Q VLAN tagging enabled ● MTU 9212 or higher enabled ● Spanning-Tree edge-port (portfast) enabled ● Member of OpenStack Service, Public/DC, Private, and Storage Network VLANs

Solidfire node ports

● MTU 9212 or higher enabled ● Spanning-Tree edge-port (portfast) enabled ● Member of the storage VLAN/Network VLAN ● LACP port-channel membership configured

Switch Configuration Steps

1. Define VLAN IDs for each network defined in this reference architecture 2. Configure s55 TOR switches in stacking mode. These are the Admin/Provisioning switches. 3. On the s55 TOR switch, configure VLANs and VLAN port membership. This will be based on

the specific physical port allocations determined at the time of the physical cabling of each system. The only VLAN required on this switch should be the administrative VLAN.

4. Configure s4810 TOR switches in VLT mode. 5. On each s4810 switch, ensure that all physical port, VLAN and port-channel interfaces are

set according to the requirements listed above. 6. On each s4810 switch, configure VLANs and VLAN port membership according to the

network topology defined. This will be based on the specific physical port allocations determined at the time of the physical cabling of each system. For OpenStack node ports, ensure that VLAN tagging is configured in the Fuel environment pages.

1. On each s4810 switch, set up and configure SolidFire node switch ports for LACP bonding. Sample configuration templates are as follows:

a. Create a port channel for each individual node: interface Port-channel 31 description SF-OS-1 no ip address mtu 9212 switchport vlt-peer-lag port-channel 31 no shutdown


b. Assign the switch ports for a particular node to their respective port channel interface:

interface TenGigabitEthernet 0/1 description SF-OS-1:2 no ip address mtu 9212 flowcontrol rx on tx off ! port-channel-protocol LACP port-channel 31 mode active no shutdown

interface TenGigabitEthernet 0/1 description SF-OS-1:2 no ip address mtu 9212 flowcontrol rx on tx off ! port-channel-protocol LACP port-channel 31 mode active no shutdown

At this point, the network should be ready and you can move to the next step. If you are deploying this configuration to ultimately connect to the data center aggregation / core infrastructure, ensure that all VLANs are tagged on all uplink interfaces as required.

Documents

SolidFire Mirantis Unlocked Reference Architecture · PDF fileSolidFire Mirantis Unlocked Reference Architecture : ... hardware in your data center or ... of existing data center assets