Quick Overview on OpenStack Block Storage Layer · Quick Overview on OpenStack Block Storage Layer . ... OpenStack has two types of persistent storage, Cinder and Swift. • Cinder:

© Hitachi, Ltd. 2014. All rights reserved.

Quick Overview on OpenStack Block Storage Layer

Hitachi, Ltd., Information & Telecommunication Systems Company IT Platform Division,

5/21/2014

Masaki Kimura


1. Introduction of OpenStack Block Storage layer

2. Cinder Plugin

Contents

1

3. Improvement

4. Summary


1. Introduction of OpenStack Block Storage layer

2


• OpenStack is a collection of software for setting up a massive IaaS(Infrastracture as a Service) environment. • OpenStack consists of seven main components below.

1-1 About OpenStack

3

Horizon (Dashboard)

Swift (Object Storage)

Cinder (Block Storage)

Nova (Compute)

Glance (Image)

Neutron (Network)

Keystone (Identity)

OpenStack


OpenStack has two types of persistent storage, Cinder and Swift. • Cinder:

• is a block storage, • provides requested size of volumes created in storage pools, so that users can use volumes as a hard drive like purpose. (Format as a file system, mount, read/write, and etc.)

•Swift: • is an object storage. • provides containers to store objects (files) in, so that users can use containers as a file server like purpose. (Users can get, put, copy, delete, head, or post objects.)

In this presentation, we will focus on Cinder, or a block storage.

1-2 About OpenStack Storage Layer

4


• Cinder provides persistent block storage resources to the virtual machines running on Nova compute. • Cinder is an abstraction layer to manipulate multiple types of backend storages as volumes by the same interfaces. Cinder uses plugin to support multiple types of backend storages. (Details of each Cinder plugin are explained in chapter #2.)

1-3 About OpenStack Block Storage Layer

5

qemu-kvm

VM1 VM2 VM3

qemu-kvm

VM4 VM5 VM6

Nova Compute1 Nova Compute2

Create/Delete a volume

Attach/Detach a volume

Snapshot

Nova

Cinder


1-4 Operation of Cinder

6

qemu-kvm

Image

Snapshot

Volume

(7)Create snapshot

VM1 VM2

snapshot

VM3

(8) Create volume from snapshot (2)Create volume from volume

(1) Create volume (5)Attach (5)Attach

# Category Function

(1)

Volume operation

Create volume (2) Create volume from volume (3) Extend volume (4) Delete volume (5)

Connection operation Attach volume

(6) Detach volume (7)

Volume-snapshot operation Create snapshot

(8) Create volume from snapshot (9) Delete snapshot (10)

Volume-image operation Create volume from image

(11) Create image from volume

Cinder

Nova

Glance

VM VM4 VM5

(10)Create volume from image

(11)Create image from volume


1-5 How Cinder Operations Are Utilized

7

Create VM

Launch VM

Operate VM

Stop VM

Delete VM Life cycle of VM

Required volume operations

Create/Attach volume

Extend/Snapshot volume

Detach/Delete volume

Requirements from users

Quickly prepare disk space needed.

Flexibly add or extend disk space when needed.

Safely preserve necessary important data.

Copy and reuse existing data.

Quickly backup/restore important data.

Safely delete unnecessary confidential data.

With the help of Cinder, users can manage data on VMs easily and efficiently.


2. Cinder Plugin

8


2-1 Cinder Plugin

9

Cinder Plugins

Software-based

File system based

DFS

NFS

Block based

Hardware-based

Fibre Channel

iSCSI

NFS

GlusterFS, GPFS, RBD(ceph)

LVM

Vendor specific plugins, such as, Dell, EMC, HP,

Hitachi and etc …

There are many kinds of Cinder plugins available.


2-1 Cinder Plugin

10

Cinder Plugins

Software-based

File system based

DFS

NFS

Block based

Hardware-based

Fibre Channel

iSCSI

NFS

GlusterFS, GPFS, RBD(ceph)

There are many kinds of Cinder plugins available.

2-2

2-4

How LVM plugin and Fibre Channel plugin work is explained from the next slides.

LVM

Vendor specific plugins, such as, Dell, EMC, HP,

Hitachi and etc …


2-2 Work Flow of Software-based Plugin

11

Control Node

API

Scheduler

Volume (LVM plugin)

iSCSI target

iSCSI initiator

Compute Nodes

qemu-kvm

VM1 VM2 VM3

/dev/sdx

Create/Delete Extend/etc

Attach/Detach via qemu-kvm

iSCSI initiator

qemu-kvm

VM4 VM5 VM6

/dev/sdx


Nova Compute1 Nova Compute2 Cinder node

LVM plugin case:

• Volumes are created as LV by using LVM. • Volumes are visible to compute nodes by using software iSCSI target.

iSCSI target

tgtd

VG1 LV1 LV2 LV3 LV4

LVM


2-3 Pros & Cons of Software-based Plugin

12

Control Node

API

Scheduler

Volume (LVM plugin)

iSCSI target

iSCSI initiator

Compute Nodes

qemu-kvm

VM1 VM2 VM3

/dev/sdx



iSCSI initiator

qemu-kvm

VM4 VM5 VM6

/dev/sdx


Nova Compute1 Nova Compute2 Cinder node

LVM plugin case:

iSCSI target

tgtd

VG1 LV1 LV2 LV3 LV4

LVM

[Cons] All I/Os are issued to a software iSCSI target on a single server, which can easily be a bottle neck of I/O performance.

[Pros] Volume operations are implemented by using LVM, which is light weight.

Volume operations

I/Os from VMs on compute nodes


2-4 Work Flow of Hardware-based Plugin

13

qemu-kvm


VM1 VM2 VM3 VM5 VM6 API

Scheduler

Cinder node

VM4



qemu-kvm Attach/Detach via qemu-kvm

Control Node Compute Nodes

Fibre Channel plugin case:

Volume (FC plugin)

/dev/sdx (LU1)

/dev/sdy (LU2)

/dev/sdx (LU3)

/dev/sdy (LU4)

• Volumes are created as LU by using storage functions. • Volumes are visible to compute nodes via Fibre Channel.

FC Storage

FC Storage

Storage Controller

LU1 LU2 LU3 LU4


2-5 Pros & Cons of Hardware-based Plugin

14

qemu-kvm



Scheduler

Cinder node

VM4





Fibre Channel plugin case:

Volume (FC plugin)

/dev/sdx (LU1)

/dev/sdy (LU2)

/dev/sdx (LU3)

/dev/sdy (LU4)

[Pros] I/Os are issued to a storage controller, which has enough scalability. [Cons] Volume operations are implemented by using storage function, which is heavy weight.

FC Storage Storage Controller

LU1 LU2 LU3 LU4

Volume operations

FC Storage I/Os from VMs on compute nodes


2-6 Comparison of Cinder Plugin

15

LVM FC

Implementation of volume

LV (managed by LVM)

LU (managed by storage)

Volume Operation

By software (LVM)

By hardware (Storage)

Supported Storage

Any storage (Storage independent)

Specific storage (Requires specific plugin)

Access Path

Via software iSCSI target

Directly via fibre channel

Light weight.

Better support

coverage.

Better I/O performance.


2-7 Which Plugin to Use for What Usecase

16

• LVM plugin is suitable for: • systems which does not require high reliability, • systems without I/O intensive workloads, • simple test environments, • no vendor specific storage plugin.

• Vendor specific plugin is suitable for: • high-end systems which need high I/O performance or high reliability • systems which require the storage functions, such as, asynchronous/copy-on-write snapshot and thin-provisioning for lower downtime and storage utilization.


3. Improvement

17


3-1 Proposed LVM Volume Plugin

18 (*) https://blueprints.launchpad.net/cinder/+spec/lvm-driver-for-shared-storage

Another implementation of LVM plugin(*) would be useful for some usecases.

This implementation is still under discussion.

FC Storage

FC Storage

LU1

Storage Controller

qemu-kvm



Scheduler

Cinder node

VM4

Proposed LVM volume Plugin





VG1 LV1 LV2 LV3 LV4

/dev/sdx (LU1) VG1 LV1 LV2 LV3 LV4



•Single LU is shared with all nodes. •Volumes are created as LV by using LVM on Cinder node. • Volumes are visible to compute nodes via Fibre Channel.


3-2 Work Flow of Proposed LVM Volume Plugin

19

FC Storage

FC Storage

LU1

Storage Controller

qemu-kvm



Scheduler

Cinder node

VM4

qemu-kvm


VG1

/dev/sdx (LU1) VG1

/dev/sdx (LU1) VG1

/dev/sdx (LU1) VG1

(0) State before LV is created and attached.


© Hitachi, Ltd. 2014. All rights reserved. 20

FC Storage

FC Storage

LU1

Storage Controller

qemu-kvm



Scheduler

Cinder node

VM4

qemu-kvm


VG1

/dev/sdx (LU1) VG1 LV1

/dev/sdx (LU1) VG1

/dev/sdx (LU1) VG1

Create

(1) Create LV on the Cinder node.




FC Storage

FC Storage

LU1

Storage Controller

qemu-kvm



Scheduler

Cinder node

VM4

qemu-kvm


VG1 LV1

/dev/sdx (LU1) VG1 LV1 LV2 LV3




(2) LV1 is written to the shared LU, as a result it appears in all nodes.




FC Storage

FC Storage

LU1

Storage Controller

qemu-kvm



Scheduler

Cinder node

VM4


qemu-kvm


VG1 LV1





(3) LV1 is activated on Nova compute1 and attached to VM1.




FC Storage

FC Storage

LU1

Storage Controller

qemu-kvm



Scheduler

Cinder node

VM4


qemu-kvm


VG1 LV1





(3) LV1 is activated on Nova compute1 and attached to VM1.



[Key points] • Volume operations are done only in cinder node, for exclusive access to the LVM meta data. • LV is activated on a single compute node where the VM connected is running, for exclusive access to the data.


3-3 Benefit of Proposed LVM Volume Plugin

24

FC Storage LU1

Storage Controller

qemu-kvm



Scheduler

Cinder node

VM4




VG1 LV1 LV2 LV3 LV4




Volume operations

I/Os from VMs on compute nodes FC Storage

• Volume operations are implemented by using LVM, which is light weight. • I/Os are issued to a storage controller, which has enough scalability.



3-4 Comparison of Proposed LVM Plugin

25

LVM Proposed LVM FC

Implementation of volume

LV (managed by LVM)

LV (managed by LVM)

LU (managed by storage)

Volume Operation

By software (LVM)

By software (LVM)

By hardware (Storage)

Supported Storage



Specific storage (Requires specific plugin)

Access Path

Via software iSCSI target



Light weight.

Better storage support

coverage.

Better I/O performance.


4. Summary

26


• Cinder is a block storage layer of the OpenStack. • There are many Cinder plugins and each plugin has its suitable usecases.

• LVM plugin: standard systems or test systems, • Vendor specific plugins: high-end systems.

• For the following usecase, proposed LVM volume plugin would be more useful.

• High-end systems with no vendor specific storage plugins

4 Summary

27


Quick Overview on OpenStack Block Storage Layer

5/21/2014

Masaki Kimura

END

Hitachi, Ltd., Information & Telecommunication Systems Company IT Platform Division,

28


Trademarks

29

• The OpenStack® Word Mark and OpenStack Logo are either registered trademarks/service marks or trademarks/service marks of the OpenStack Foundation in the United States and other countries and are used with the OpenStack Foundation's permission. We are not affiliated with, endorsed or sponsored by the OpenStack Foundation, or the OpenStack community. • GPFS is a trademark of International Business Machines Corporation in the United States, other countries, or both. • GlusterFS, the Gluster ant logo, and the Gluster Community logo are all trademarks of Red Hat, Inc. All other trademarks, registered trademarks, and product names may be trademarks of their respective owners. • Dell is a trademark of Dell Inc. • EMC and CLARiiON are registered trademarks of EMC Corporation. • HP is a trademark of Hewlett-Packard Development Company, L.P. in the U.S. and other countries. •Other company, product or service names may be trademarks or service mark of others.

Documents

Quick Overview on OpenStack Block Storage Layer · Quick Overview on OpenStack Block Storage Layer . ... OpenStack has two types of persistent storage, Cinder and Swift. • Cinder: