GlusterFS and Openstack Storage

GlusterFS Openstack Integration Team

GlusterFS & Openstack Storage

GlusterFS MeetupNov. 2014

11/21/14 2GlusterFS Meetup

Openstack architecture


Openstack components

Manila(incubated)


OpenStack Swift + GlusterFS

Swift On File

Prashanth Pai

[email protected]: ppai

mailto:[email protected]


What is OpenStack Swift ?

● Object is { Data + Meta-data + Identifier }

● Alternative to Amazon S3

● Decoupled/Independent from OpenStack

● Unstructured data

● Scales horizontally

Swift is a highly available, distributed, eventually consistent object store.

But you can't..

● Mount it

● Have file hierarchies

HierarchyAccount -> Containers -> ObjectsHierarchyAccount -> Containers -> Objects


All operations are carried out by sending HTTP requests.

Each object’s access path consists of exactly three elements:/account/container/object

GET'X-Auth-Token: blah_blah'

http://127.0.0.1:8080/v1/AUTH_test/c1/a/b/c/d/z

Proxy-server

Account

ContainerObject

Requestmethod Request

header


Swift On File

● Allows objects created using Swift API to be accessed as files and vice-versa

● Map URL directly to file path

● Deployed as a Storage Policy

● Leverage distributed filesystem features

● Extend an existing Swift cluster you might already have

● Provide object interface to existing GlusterFS cluster

Example Usecases

Video transcoding, Analytics, Scientific Collaboration


GlusterFS Cinder Integration

Bharat Kumar K

[email protected]: BharatK



What is Cinder?

● Cinder is the code name for the open source project to develop OpenStack Block Storage.

● It is the Block-based storage component of OpenStack cloud computing.

● Cinder volumes provides persistent storage to guest virtual machine (aka instances), that are managed by OpenStack compute.

● Block storage is a fundamental requirement for virtual infrastructure.

● It is the foundation for storing virtual machines and data used by those machine.

● It provisions storage in the form of block devices know as Cinder volumes.

● Cinder provides APIs such as create/delete volume, backup/restore volume, snapshot/clone volume and many more. These APIs are implemented by the storage backends that wish to support themselves as part of Cinder


Cinder architecture


What are Cinder Backends?● Backend is a storage system which implement the APIs provided by

Cinder. Backend typically exports a storage pool which is used by Cinder to provision new storage (aka cinder volumes). Eg: GlusterFS volume, Ceph RBD pool etc.

● Cinder by default uses LVM (Logical Volume Manager) as its backend.

● There are open-source storage solutions from GlusterFS and Ceph that provide Cinder support using native GlusterFS protocol and Ceph RADOS Block Device (RBD) respectively.

● Many storage array suppliers now provide Cinder block device support. These include EMC, Hitachi Data Systems, HP, IBM and NetApp.

● Backend is configured in cinder via the cinder.conf file


Cinder - GlusterFS● Available upstream

● 1 Cinder volume == 1 GlusterFS file

● Support raw and qcow2 format cinder volumes

● GlusterFS backed cinder volumes can be attached to Nova instances in 2 ways

● Fuse

● Libgfapi

● Supports most of the Cinder APIs (including libgfapi based snapshots)

● cinder.conf - glusterfs_shares_config lists the glusterfs volumes to use in hostname:/volname format

● More info on configuring Cinder w/ GlusterFS is available @ http://docs.openstack.org/admin-guide-cloud/content/glusterfs_backend.html

● Support cinder multi-backend feature

http://docs.openstack.org/admin-guide-cloud/content/glusterfs_backend.html


GlusterFS as Cinder backend

Storage Backend

GlusterFS


Cinder – GlusterFS contd.● TODOs

● Manage/Unmanage support – WIP● Cinder - GlusterFS upstream CI – WIP● backup/restore for qcow2 and disks with snaps● Support for cinder volume replication● Perf improvements - WIP


Gluster Manila Integration

Ramana RajaDeepak C Shetty

[email protected]@redhat.comIRC: rraja, deepakcs




What is Manila● Shared FileSystem as a service

● Incubated since openstack Juno

● Provision file shares to Nova (openstack compute) instance(s)

● Strives to provide an API for management of shared filesystems with support for multiple protocols and backend implementations

● NFS and CIFS primarily supported

● Other protocols are encouraged too (eg: glusterfs)

● Supports Multi-tenancy

● Enables public cloud usecase

● Has framework to support storage backends that don't support multi-tenancy natively


Manila usecase


Manila - GlusterFSTwo approaches

1. GlusterFS native driver– protocol: 'glusterfs'– access type: 'cert' based

2. GlusterFS (NFS/CIFS) driver– protocol: NFSv3 (now), NFSv3/v4 and CIFS (later)– access type: IP based– mediated by GlusterNFS server (now),

NFS-Ganesha & SMB server (later)


Manila access types● IP

● Access control using IP address

● Takes IP as an argument

● Typically used in controlling access to NFS shares

● User

● Access control using user name

● Takes user name as argument

● Typically used in controlling access to CIFS shares

● Cert

● Access control using SSL certificates

● Takes SSL Certificate's CN (common name) as argument

● Certificate setup (aka trust setup) between client and server is out of band

● Currently implemented by GlusterFS native driver ('glusterfs' protocol)


GlusterFS Native Driver● Supports Certificate based access type of Manila

● Provision shares that use the 'glusterfs' protocol

● Instances directly talk with GlusterFS storage backend

● No service VM needed

● Secure access

● Only tenants with the right certificate will be able to access the share

● Multi-tenant

● Separation using tenant specific certificates

● Supports certificate chaining and cipher lists


GlusterFS Native Driver contd.● Available upstream

● 1 Manila share == 1 GlusterFS volume

● Pre-requisites

– GlusterFS volume(s) setup with Cert based access enabled

– Instance should have server signed client certificates pre-loaded

– Manila.conf – Provide list of glusterfs volume(s) to work with● TODOs

● Add documentation

● Snapshot support

● Dynamic creation of glusterfs volumes

● Data shredding as part of gluster volume delete

● Create share from snapshot


Current GlusterFS Driver (NFS)

● Flat Network Driver

● A Manila Share maps to a subdirectory within

a GlusterFS volume(Share layout may be changed in the future.)

● Shares exported by GlusterNFS (v3) residing

in the GlusterFS storage pool


GlusterFS Driver (Ganesha)

● WIP: Flat Network Ganesha + GlusterFS driver

- NFSv3 and v4

- Modular architecture of Ganesha driver allows easy integration of GlusterFS, and other drivers

● Future: Network Segmented Ganesha + GlusterFS driver

- will be a fully open source, multi-tenant Manila stack with a powerful distributed FS backend


GlusterFS Ganesha Driver (WIP)Flat Network Model

Storage Backend

GlusterFS

GlusterFS Driver

Tenant 1 Nodes Tenant 2 Nodes

Network

Gluster FSAL

NFS-Ganesha Server

Generic GaneshaDriver


GlusterFS Ganesha Driver (Future)Network-segmented Model

Storage Backend

GlusterFS

Ganesha Generic Driver

Tenant 1 NodesNetwork 1

Service VM

Gluster FSAL

NFS-Ganesha Server

Network 2

GlusterFS Driver

Ganesha Generic Driver

GlusterFS Driver

Service VM

Gluster FSAL

NFS-Ganesha Server


Summary

Deepak C Shetty

[email protected]: deepakcs



Nova Nodes(GlusterFS as

ephemeral storage)

SwiftObjects

Cinder Data

Glance Data

Swift Data

Swift API

Storage Server

(GlusterFS)

Storage Server

(GlusterFS)

Storage Server

(GlusterFS)

KVM

KVM

KVM

…Fuse/libgfapi

Manila Data

Storage Server

(GlusterFS)

Block Images FilesObjects

Openstack + GlusterFS – Current Integration


Cinder Data Glance Data

Compute + Storage

KVM

KVM

KVM

…Fuse/libgfapi

Manila Data

File

SwiftObjects

Swift Data

Swift API

Object

Compute + Storage(GlusterFS)

Block Images

Hyper converged storage using GlusterFS - WIP





Sahara Data

Hadoop



Thank You

Questions ?

Technology

GlusterFS and Openstack Storage