Evaluation of Delta Compression Techniques for Efficient Live

Migration of Large Virtual Machines

Petter Svärd, Benoit Hudzia, Johan Tordsson and Erik Elmroth

Umeå University, Dept of Computing Science

VEE 2011, Newport Beach, CA, USA

Live migration“Transfer a VM from one host to another without disrupting services.”

• The VM:s state (memory pages) is transferred in the background with the VM still running

• The file system is typically located on a NFS and is not moved

Live migration- Typical algorithm

The time between the VM is suspended and resumed is defined as the downtime

Our goal is to reduce the downtime

Live migration- Problems with the typical algorithm

When migrating memory intensive VM or over slow NW links:

1. Memory pages can be dirtied faster than they are transferred over the network

2. The VM has to be suspended for an extended period of time -> long downtime

3. Network connections time out and drop / triggers fail

Leads to disruption of services

Live migration- Problems with the typical algorithm (cont)

Problem

dirtying rate > migration throughput

Possible Solutions

Decrease dirtying rate or increase migration throughput

• Decreasing dirtying rate might hurt server performance and disrupts services

Increase migration throughput!

Delta compression- Increasing migration throughput

Overall idea: transfer changes to pages instead of the full page contents thus increasing migration throughput

• Store sent pages in a cache

• When transferring, if the page is cached, compute an XOR delta page

• Compress the delta page

Delta compression- continued

• Wasting time on cache misses

• Efficient caching scheme and compression algorithm is vital!

Vanilla (no compr.)

Delta compression

Delta compression- caching

Desired properties:• Lean• Constant seek time regardless of size

L2 caching scheme

Delta compression- compression

Desired properties:• Lean (low cpu usage)• Effective (high compression ratio)• General purpose

The XOR delta page is suitable for RLE compression

– (Symbol)(Repetitions) → AAAAABBBCCCCC = 5A3B5C

XOR BinaryRunLengthEncoding -> XBRLE

XBRLE compression- Source side algorithm

XBRLE compression- Destination side algorithm

XBRLE compression- conceptual illustration

ImplementationModified version of qemu-kvm userspace code to support the XBRLE migration algorithm.

Lean, ~500 LoC

Evaluation done on version 0.11.2

DemoMigrating streaming video over 10Mbit/s

Before migration:

DemoMigrating streaming video (cont)

After migration:

DemoMigrating streaming video (cont)

Evaluation- Test cases• Memory write benchmark (lm_bench)

– 1 GB RAM, 1 vcpu VM– Near ideal case

• Transcoded HD Video– 1 GB RAM, 1 vcpu VM– Real-world, non-ideal case

• SAP ERP application– 8 GB RAM, 4 vcpus VM– Large business application– Relies on transactions and is thus sensitive to

extended downtime

Evaluation- Experimental setupBenchmark and HD Video

2x 2,66GHz core2quad 16GB RAMNFS share on source machine100Mbit/s Network

SAP ERP

2x 3,0GHz Xeon dual-core 32GB RAM16TB Raid 5, 6Gbits/s trunked NFS server 1000Mbit/s Network

Evaluation- Benchmark

• Downtime reduced by a factor of 100• Throughput increased by 63 %

Evaluation- Streaming video

• UDP downtime reduced from 8 s to 1• Migration is transparent using XBRLE

Evaluation- SAP ERP

• The ERP application was non-responsive on resume using the vanilla algorithm but survived using XBRLE

• “Rule of thumb” is that more than 0.5 s of downtime might hurt the system. Measured downtime was 0.2 for XBRLE and 2 for vanilla.

Vanilla

XBRLE

ConclusionDelta compression works well migrating

• VMs running workloads with a highly compressible working set

• VMs running heavy workloads with large working sets

• and/or over slow networks (i.e., WANs).

Future work

Page priority algorithm

• Avoid re-sends of pages that are dirtied frequently

• Promising early results