A Virtual Storage Environment for SSDs and HDDs in Xen Hypervisor

A Virtual Storage Environment for SSDs

and HDDs in Xen Hypervisor

Outline• Introduction• Motivation• Related Work• System Design and Architecture• Implementation• Experiment• Conclusion

雲端運算的定義

Cloud computing is Internet-based computing, whereby shared resources, software, and information are provided to computers and other devices on demand, like the electricity grid. [wiki]

Cloud computing describes a new supplement, consumption, and delivery model for IT services based on the Internet, and it typically involves over-the-Internet provision of dynamically scalable and often virtualized resources. [wiki]


• “... a pay-per-use model for enabling available, convenient, on-demand network access to a shared pool of configurable computing resources (e.g., networks, servers, storage, applications, services) that can be rapidly provisioned and released with minimal management effort or service provider interaction.”

[National Institute of Standards and Technology (NIST) ]


Introduction• Nowadays, virtualization is a popular technology

to provide an abstract hardware emulation due to the increasing hardware speed and capabilities.

• Advantages of virtualization technology:– Merge physical servers, reduce the space required for

the servers.– Simplify the management of the servers.– Enhance resource utilization.– Conserve energy to reduce carbon emission.

虛擬化為雲端運算的基礎技術

傳統單一主機 (low)

虛擬主機 (high)

低使用率

高使用率

現在的電腦效能相當強大，但卻有非常多的電腦平常只有使用不到20% 的運算資源，甚至更少。但因為種種需求，又無法降低電腦數量，因此把數台電腦“虛擬化”塞進一台功能較強大的實體主機，利用實體主機的運算資源，成了一種受歡迎的方法。

虛擬機的概念

• 虛擬機可允許一台實體主機同時執行多個作業系統。

• 一台強大電腦主機能做 30 台電腦的事，在佔用面積

、耗電量、空調、維護人事成本上，都能有效節約。

• 可在一台實體主機內執行多個虛擬主機，每一台虛擬主機既可以互相用線路連線、又可以獨立運作，互不干涉。

• 可將多台實體主機合併成一台大的資源池(Resource Pool) ，統籌管理、分配在上面的虛擬機。

虛擬機的概念

• 只要實體主機的硬體能力許可，便能靈活地建立或移除多台虛擬機，可保持高度彈性。

• 虛擬主機往往在映像檔內直接執行、存取檔案，使得備份、搬移虛擬機較實體主機容易許多。

• 虛擬化可使實體主機擴增、維修、更換不須關閉虛擬機，只要把裡面的虛擬機搬移到別的實體主機上即可。

• 可使用虛擬機測試不穩定的軟體，再利用快照完全還原整個系統 ( 僅需數秒鐘 ) ，而不須在實體主機測試。

Hardware

Virtualized World Physical World

Traditional x86 Architecture• 傳統是一台主機執行一套應用程式

且整體使用率偏低平均約 8%

Virtualization: • 新一代資訊中心是在單一主機執行多

種應用程式讓整體使用率提高到 80%

何謂虛擬化

Traditional Data Center – Fragmented and Inefficient Utilization

Fragmented and Inefficient IT Resources Server capabilities

wasted

0%

20%

40%

60%

80%

100%

All Servers

Server capabilities reach bottleneck

Num

ber o

f CPU

s re

quire

d

Time

Cost increases as new applications grow

Low utilization rate as too many servers are deployed for peak time

App4

OS

HW

App3

OS

HW

App2

OS

HW

App1

OS

HW

Virtual Data Center Resource Pools

New Virtual Data Center - Elastic Provisioning

Elastic scale up when demands increase

Sleeping

Elastic scale down when demands decrease

Running

Running Running Running Running Sleeping

Sleeping Sleeping Sleeping

Hardware

Virtualized World Physical World

Traditional x86 Architecture• 傳統是一台主機執行一套應用程式

且整體使用率偏低平均約 8%

Virtualization: • 新一代資訊中心是在單一主機執行多

種應用程式讓整體使用率提高到 80%

虛擬機的概念

虛擬化技術的好處• Test before we go:

Virtual machine just like a experiment box. You are able to test your application on different kind of system in same machine. It's easy to destroy, rebuild or backup your whole testing environments.

• Server consolidation: Many small physical servers could be replaced by virtual

machine, runs on virtualization environment provided by one powerful physical server. Although hardware is consolidated, typically OSs are not. Instead, each OS running on a physical server becomes converted to a distinct OS running inside a virtual machine. The large server can "host" many such "guest" virtual machines.

• Increase hardware utilization:To increase the utilization of costly hardware resources such as

cpu, memory or even like storage space.

• Rapid provisioning, dynamic fault tolerance..For example: Big system memory can partition as smaller portions for legacy

system runs within a virtual machine.

Virtualization Cloud Computing

Virtualization can Reduce cost for operation

and maintenance

Increase resource utilization

Enable quick deployment of applications and resources

Virtualization can NOT• Reduce system complexity

• Offer elastic resource provisioning

• Provide distributed computing capabilities

Introduction• Three parts of Xen:

– Xen Hypervisor– Domain 0– Domain U

Introduction• Xen Hyperviosr

– An abstraction layer between the guest domains and the physical hardware.

– Allocating and controlling resources.

Introduction• Domain 0

– A modified Linux kernel and is only one privileged virtual machine.

– Directly access the hardware resources, and contain drivers for the hardware resources.

Introduction• Domain U

– A general virtual machine, since domain U is a unprivileged virtual machine.

– Sending I/O requests through front-end driver, and back-end driver in domain 0 will handle and perform the I/O requests.

Introduction• Characteristics of SSD

– No mechanical latency– Low power consumption– Shock resistance– Provide better performance than HDD, especially for a

lot of random accesses– Smaller storage capacity than HDD

Introduction• I/O performance comparison between SSDs and

HDDs under various request sizes [using IOmeter benchmark]– Intel 320 series 160G– Intel 710 series 100G– Hitachi HDS721010CLA332 1TB

Introduction• Storage device combine SSD and HDD

– Seem SSDs to be a storage device.– Seem SSDs to be a secondary-level cache.

Virtual Storage Virtual Storage

HDDs HDDsSSDs

Mapping TableSSDs

Related Work• [1]Chai-Hao Tsai, Jerry Chou, Yeh-Ching Chung,

“Value-Based Tiering Management on Heterogeneous Block-Level Storage System,” IEEE 4th International Conference on Cloud Computing Technology and Science, 2012. – They formulate resource allocation problem into an ILP

(Integer Linear Programming), and propose data migration algorithms based on quantify the performance of difference access patterns on each type storage devices.

– They regard SSDs as storage space and storage cache of HDD.

Related Work• [2]Youngjae Kim, Aayush Gupta, Bhuvan

Urgaonkar, Piotr Berman, and Anand Sivasubramaniam, “HybridStore A Cost-Efficient, High-Performance Storage System Combining SSDs and HDDs,” 19th Annual IEEE International Symposium on Modelling, Analysis, and Simulation of Computer and Telecommunication Systems, 2011. – They formulate some constraints (such capacity,

performance, and lifetime) as a Mixed Integer Program to decide which storage to store.

– But they do not consider data migration in different storage devices.

Related Work• [3]Hannes Payer, Marco A.A. Sanvido,Zvonimir Z.

Bandic and Christoph M. Kirsch, “Combo Drive: Optimizing Cost and Performance in a Heterogeneous Storage Device,” WISH, 2009. – Combine SSD and HDD into a single heterogeneous

storage device.– Move some files to SSDs/HDDs based on file type.

• SSD:.exe, .dll, .sys, .msi, .cab, .drv, .jar, …• HDD:.jpg, .pdf, .wmv, .wma, .mp3, .doc, .xls, …

– Need modify file systems

HDDSSD

Logical Address

Virtual Address

Logical Address

Low high

Related Work• [4]Shih-Yu Liu, “An Efficient Data Management

Method for SSD-Based Storage Systems,” National Taiwan University of Science and Technology Master Thesis, 2012. – Propose a data allocation method for SSD-based

storage systems and consider append/update data.– Consider speed declining of SSDs under different space

utilization.

Related Work• [5]Chih-Kai Kang, “A Hybrid Storage Access

Framework for Virtual Machines,” National Taiwan University of Science and Technology Master Thesis, 2012.– Propose a hybrid storage framework in Xen hypervisor– Adopt potential value to determine which data should be

placed in SSDs or HDDs.– Only consider running one virtual machine.

Motivation• Virtualization technology

– Virtualization technology has ability to dynamically adjust resources in accordance with the demand of each virtual machine.

– Running multiple virtual machines in only one physical machine require high I/O performance.

– I/O is a serious bottleneck for overall performance. – SSDs can provide better I/O performance but higher cost

than HDDs.

Motivation• When space utilization of SSDs was 75%, write

performance of SSDs began to decline.• Read performance of SSDs do not change

significantly.

A Virtual Storage Environment for SSDs and HDDs in Xen Hypervisor

• Blktap in the kernel space – Can not directly access physical storage devices.– Receives all I/O requests of Domain U.– Transfers the I/O requests to Blktap in the user space of

Domain 0 to perform I/O requests to physical storage devices.


• Blktap in the user space – Provide Xen hypervisor an interface to access storage

devices.


• I/O Requests from Multiple Virtual Machines to Physical Storage Devices

• Monitor • Play an important role for

dynamic and adaptive adjustment in I/O performance

• Content mapping table• Record which data are

transferred to SSDs or HDDs

Data Distributed Management• Potential value [5]

– Data with smaller potential value is suitable to be placed in HDD.

– Data with bigger potential value has higher priority to be assigned to SSD.

• N : request size• RHDD (N) is the average time of random access whose size

is N in HDDs• RSSD (N) is the average time of random access whose size

is N in SSDs.• SHDD (N) is the average time of sequential access whose

size is N in HDDs• SSSD (N) is the average time of sequential access whose

size is N in SSDs

Potential Value• Maintain chunk info

– Chunk table

Potential Value• Maintain chunk info

Our method HD Size16G 32GB 64GB 128GB

Chunk Size

4k ≒12MB ≒24MB ≒48MB ≒96MB16k ≒3MB ≒6MB ≒12MB ≒24MB64k ≒0.75MB ≒1.5MB ≒3MB ≒6MB

256k ≒192kB ≒384kB ≒0.75MB ≒1.5MB1M ≒48kB ≒96kB ≒192kB ≒384kB

Potential Value• Identify high potential chunks

50

30

10

25

25

10

10

5

10

5

0

5

5

Quota : 10

Quota : 510*50/(50+25+25)

Quota : 35*30/(30+10+10)

Quota : 15*10/(30+10+10)

Data Distributed Management• Budget of SSDs [4]

– Smaller Budgetj means that the declining speed ratio for SSD storagej is low.

– Larger Budgetj means that the bandwidth of SSD storagej has been decreased.

– : Average writing speed for SSD storagej

– : Declining speed ratio for SSD storagej

– : Space utilization ratio for SSD storagej

Implementation• Mainly modify two components in Blktap driver:

– Tap-disk – Block-aio

Implementation• Receive and analyze I/O requests

– Tap-disk is a process which can handle I/O requests and provide an interface for virtual machine management.

– We can modify Tap-disk to analyze the I/O requests and then determine their storage destinations (i.e., the monitor mechanism).

– According to the analysis of I/O requests and SSD capacity utilization, we can transfer appropriate I/O requests to suitable SSDs to improve overall I/O performance.

Component File Function Functionality

Tap-disk Tapdisk-vbd.c tapdisk_vbd_create()tapdisk_vbd_issue_request()counter content mapping table()monitor()

1. Create and maintain a content mapping table2. Create a monitor mechanism3. Analyze I/O requests4. Deliver I/O requests and its locations to Block-aio component

Implementation• Perform I/O requests to physical storage devices

– Block-aio component is a process to handle each virtual machine image such as image open/close and I/O requests to the image.

– Block-aio is a bottom layer of Xen hypervisor to access physical storage devices.

• We modify Block-aio to perform I/O requests to the final storage devices (e.g., SSDs or HDDs).

Component File Function Functionality

Block-aio Block-aio.c tdaio_open()tdaio_complete()tdaio_queue_read()tdaio_queue_write()

1. Maintain virtual machine images in SSDs and HDDs2. Perform I/O requests to SSDs and HDDs3. Migrate data between SSDs and HDDs

Experimental Evaluation

Environment Configuration

Hardware

CPUIntel(R) Core(TM) i7 CPU 950 @

3.07GHz

Hard Disk Drive Hitachi HDS721010CLA332 1TB

Solid-state DriveSSD Intel 320 series 160GB

SSD Intel 710 series 100GB

Memory 8GB

SystemVirtualization Software Xen-4.1.2

Linux Kernel Linux 2.6.32

Domain 0Operating System Ubuntu 11.10

Memory 2GB

Domain U

Operating System Ubuntu 11.10

Memory 128MB

Storage space 32GB

vCPU 1

Experimental Evaluation• I/O traces file

Web search Financial

Read 99% Read 25%

Write 1% Write 75%

Request size 8kB Request size 512B~2kB

Workload Size 16.7GB Workload Size 7.2GB

Total Read 65.8GB Total Read 5.2GB

Total Write 8MB Total Write 28GB

Experimental Evaluation• Web search result

– Two Intel SSDs was about 8.44 times faster than two Hitachi HDD when running one virtual machine.

– When running two or more virtual machines, two Intel SSD was about 10 to 14 times faster than two Hitachi HDD.

Experimental Evaluation• Web search result

– Two Intel SSDs was about 1.38 times faster than one Hitachi HDD and two Intel SSDs when running one/two virtual machines.

– When running three/four virtual machines, one Hitachi HDD and two Intel SSDs are close to two Intel SSDs.

Number of VMs 2 Hitachi HDDs 1 Hitachi HD and 2 Intel SSDs

2 Intel SSDs

1 VM 1 21326 3502 2526

Average Execution Time (s) 21326 3502 2526

2 VM 1 45125 5848 4131VM 2 45709 5477 4146

Average Execution Time (s) 45417 5662.5 4138.5

3 VM 1 71893 6265 5796

VM 2 72639 6320 5809

VM 3 70458 6991 5809

Average Execution Time (s) 71663.33 6525.333 5804.667

4 VM 1 100057 6872 5839

VM 2 100959 7567 6862

VM 3 101053 6792 7346

VM 4 96269 7675 7046


1.38

1.36

1.12

1.06

Experimental Evaluation• Financial result

– The average execution time of two Intel SSDs was about 11 times faster than two Hitachi when running one virtual machine.

– When running two or more virtual machines, two Intel SSD was about 19 and 23 times faster than two Hitachi HDD.

Experimental Evaluation• Financial result

– Two Intel SSDs was 1.87 times faster than one Hitachi and two Intel SSDs,

– Running two or more virtual machines the result show two Intel SSDs was 1.4 times faster than one Hitachi and two Intel SSDs.

Number of VMs 2 Hitachi HDDs 1 Hitachi HD and 2 Intel SSDs

2 Intel SSDs

1 VM 1 14274 2387 1272


2 VM 1 35342 2635 1852

VM 2 35430 2587 1858


3 VM 1 56607 3591 2502

VM 2 58215 3575 2511

VM 3 56013 3577 2504

Average Execution Time (s) 56945 3581 2505.667

4 VM 1 77981 4298 3275

VM 2 82056 4214 3288

VM 3 80735 4285 3260

VM 4 68406 4288 3276


1.87

1.40

1.42

1.30

Conclusion• We focuses on the implementation and discussion

on how to revise Xen hypervisor to provide a platform using SSDs and HDDs in virtualization environment, and we also include a data management method for SSDs

• Xen hypervisor will play an important role of receiving and analyzing I/O requests from virtual machines, and performing I/O requests to SSDs and HDDs (i.e., the monitor mechanism and the content mapping table).

• The proposed virtual storage environment using SSDs and HDDs can not only provide better performance than only HDDs but also provide a more economical solution than only SSDs.

Reference• [1]Chai-Hao Tsai, Jerry Chou, Yeh-Ching Chung, “Value-Based

Tiering Management on Heterogeneous Block-Level Storage System,” IEEE 4th International Conference on Cloud Computing Technology and Science, 2012.

• [2]Youngjae Kim, Aayush Gupta, Bhuvan Urgaonkar, Piotr Berman, and Anand Sivasubramaniam, “HybridStore A Cost-Efficient, High-Performance Storage System Combining SSDs and HDDs,” 19th Annual IEEE International Symposium on Modelling, Analysis, and Simulation of Computer and Telecommunication Systems, 2011.

• [3]Hannes Payer, Marco A.A. Sanvido,Zvonimir Z. Bandic and Christoph M. Kirsch, “Combo Drive: Optimizing Cost and Performance in a Heterogeneous Storage Device,” WISH, 2009.

• [4]Shih-Yu Liu, “An Efficient Data Management Method for SSD-Based Storage Systems,” National Taiwan University of Science and Technology, Master Thesis, 2012.

• [5]Chih-Kai Kang, “A Hybrid Storage Access Framework for Virtual Machines,” National Taiwan University of Science and Technology, Master Thesis, 2012.

•

Documents

A Virtual Storage Environment for SSDs and HDDs in Xen Hypervisor