VHPC'12: Pre-Copy and Post-Copy VM Live Migration for Memory Intensive Applications

Benoit Hudzia Sr. Researcher; SAP Research CEC Belfast (UK)

Aidan Shribman Sr. Researcher; SAP Research Israel

With the contribution of Roei Tell, Steve Walsh, Peter Izsak and Chaim Bendelac

Pre-Copy and Post-Copy VM Live Migration

for Memory Intensive Applications

© 2012 SAP AG. All rights reserved. 2

Agenda

Introduction

Live Migration

Memory Intensive Apps

Pre-Copy Live Migration

Post-Copy Live Migration

Summary

Introduction The challenge of live migrating memory intensive applications


What is live migration?

Moving of a guest VM running on host A (the source host) to

host B (the destination host) without noticeable service

disruption.

Typically used for dynamic resource load balancing in cloud

environments. So the faster to migrate and the less the service

disruption the better.

.


What is VM migration?

CPU state and device driver state migration

MBs of information can be transferred while VM is stopped

Memory state migration

GBs of information must (at least mostly) be transferred while VM is live

Requirement: is that memory can be migrated during guest execution

Storage state migration

TBs of information (requiring minutes up to hours to move)

Recommendation: use shared storage (accessed via NFS, etc)

NIC relocation

ARP announcement that IP address has been taken over

Requirement: both hosts reside on the same LAN


Memory Intensive Applications

An application dominated by high rate of memory read/write accesses

Why must memory be transferred mid execution?

16 GBs takes 12 seconds to migrate over 10 GbE (1.25 GB/s) interconnects

12 seconds downtime would cause severe service disruption

The challenge (of live migration implementations) is to transfer memory a mid high rates of

memory shurn

Pre-Copy Live Migration Reducing the amount of page re-sends and the cost of page re-sends


Pre-Copy Memory Transfer

Memory is transferred before VM relocation

The problem is how to copy memory while it is re-dirtied over and

over again by the guest VM

Solved by first copying all the memory followed by intervals of

copying newly dirtied pages until remaining state is small enough

Implemented by most (all) hypervisors VMWare, Xen, Qemu

Challenged by fast memory dirtying applications


Pre-Copy Live Migration (pre-migration)

Host A Host B

Guest VM

Iterative

Pre-copy X

Rounds

Stop

and

Copy

Commit

Total Migration Time

Downtime Live on A Live on B

Reservation Pre-migrate

Degraded on A


Pre-Copy Live Migration (reservation)

Host A Host B

Guest VM Guest VM

Iterative

Pre-copy X

Rounds

Stop

and

Copy

Commit




Degraded on A


Pre-Copy Live Migration (pre-copy)

Host A Host B

Guest VM Guest VM

Iterative

Pre-copy X

Rounds

Stop

and

Copy

Commit




Degraded on A


Pre-Copy Live Migration (stop and copy)

Host A Host B

Guest VM Guest VM

Iterative

Pre-copy X

Rounds

Stop

and

Copy

Commit




Degraded on A

}


Pre-Copy Live Migration (commit)

Host A Host B

Guest VM

Iterative

Pre-copy X

Rounds

Stop

and

Copy

Commit




Degraded on A


LRU Page Reordering (reducing number of pages transferred)

An LRU queue is used for tracking all historic page

dirty events (i.e. application wrote to memory)

QEMU (responsible for the live migration process)

queries the Linux Kernel KVM hypervisor for dirty

page events. It then inserts dirty page events into

the LRU queue (after a random shuffle)

Dirty pages (maintained in a separate LRU queue)

are inserted according to historic LRU queue and

transmitted head first

LRU reordering may have an adverse effect for

some workloads requiring a smarter cache (ARC?)


XBZRLE Delta Encoder (reducing cost of page resent)

Cache page with highest dirtying rate during send operation

If old page is found in cache delta compress it; Else transfer new page uncompressed

Implemented using an LRU cache configured to fit working set size

Classic Pre-Copy

Delta Compression Pre-Copy


XBZRLE Delta Encoder - Example

XBRLE (XOR + RLE) presented in VEE 2011: Evaluation of Delta Compression Techniques for

Efficient Live Migration of Large Virtual Machines by Benoit, Svard, Tordsson and Elmroth

XBZRLE (XOR + ZRLE) the current effort already released to QEMU community

0 1 2 3 4 5 6 7 Size

Old ‘a’ ‘b’ ‘c’ ‘d’ ‘e’ ‘f’ ‘g’ ‘h’ 8

New ‘a’ ‘b’ ‘c’ 0x00 0x01 ‘f’ ‘g’ ‘h’ 8

XOR 0x00 0x00 0x00 ‘d’ ‘f’ 0x00 0x00 0x00 8

RLE 0x00 0x03 ‘d’ 0x01 ‘f’ 0x01 0x00 0x03 8

ZRLE 0x00 0x03 ‘d’ ‘f’ 0x00 0x03 6


XBZRLE Delta Encoder - Micro Benchmark

Sparse Medium Dense Very Dense

Jump 1111 701 203 121

Dirty 12 33 41 43


Pre-copy Live Migration Evaluation

Host: 1 core; 2 GB

Benchmark: synthetic dirtying 512 MB; 1 T

Host: 2 core; 8 GB; Bandwidth 240 Mbps

Benchmark: appmembench dirtying 512 MB at 1 GB/s 8 T

Post-Copy Live Migration Fast live migration … but at a cost?


Post-Copy Memory Transfer

Memory is transferred only after VM relocation

The problem is how to ensure that VM performance is not degraded (after

relocation) due to expensive network bound page faults

Solved by using fast interconnects and improved page fault mechanism

References: Yobusame PoC for KVM Linux (using char device user mode

transfer)

Challenged by fast memory reading applications


Post-Copy Live Migration (pre-migration)

Host A Host B

Guest VM

Page Pushing

1

Round

Stop

and

Copy

Commit


Downtime Live on A Degraded on B Live on B



Post-Copy Live Migration (reservation)

Host A Host B

Guest VM Guest VM

Page Pushing

1

Round

Stop

and

Copy

Commit





Post-Copy Live Migration (stop and copy)

Host A Host B

Guest VM Guest VM

Page Pushing

1

Round

Stop

and

Copy

Commit





Post-Copy Live Migration (post-copy)

Host A Host B

Guest VM Guest VM

Page Pushing

1

Round

Stop

and

Copy

Commit




Page fault

Page push


Post-Copy Live Migration (commit)

Host A Host B

Guest VM

Page Pushing

1

Round

Stop

and

Copy

Commit





Reducing Effects of Network Bound Page Faults

Low latency RDMA page transfer protocol (reducing latency/cost of page faults)

Implemented fully in kernel mode OFED VERBS

Can use the fastest RDMA hardware available (IB, IWARP, RoCE)

Demand pre-paging (pre-fetching) mechanism (reducing the number of page faults)

Currently only a simple fetching of pages surrounding page on which fault occured

Full Linux MMU integration (reducing the system-wide effects/cost of page fault)

Enabling to perform page fault transparency (only pausing the requesting thread)

Hybrid post-copy live migration (reducing the number of page faults)

Moving some of the pages to the destination in a bounded pre-copy phase prior to relocation


Evaluation Pre-Copy vs. Post-Copy

Specification

Benchmark: Google Stress Application Test (SAT)

with a 1 GB working set

Communication: softIWARP over 1 GbE

Guest VM: 2 x vCPU; 1 GB – 14 GB memory

Results

Roughly the total migration time between pre-copy

to post-copy is 1:10

Summary


Key takeaways

Pre-Copy and Post-Copy optimizations enhace the

support of live migrating memory intessive applications

To be released as open source under GPLv2 and LGPL

licenses to Qemu and Linux communities

Developed by SAP Research Technology Infrastructure

(TI) Practice

Thank you

Benoit Hudzia; Sr. Researcher;

SAP Research CEC Belfast

[email protected]

Aidan Shribman; Sr. Researcher;

SAP Research Israel

[email protected]

mailto:[email protected]

mailto:[email protected]


Legal Disclaimer

The information in this presentation is confidential and proprietary to SAP and may not be disclosed without the permission of

SAP. This presentation is not subject to your license agreement or any other service or subscription agreement with SAP. SAP

has no obligation to pursue any course of business outlined in this document or any related presentation, or to develop or

release any functionality mentioned therein. This document, or any related presentation and SAP's strategy and possible future

developments, products and or platforms directions and functionality are all subject to change and may be changed by SAP at

any time for any reason without notice. The information on this document is not a commitment, promise or legal obligation to

deliver any material, code or functionality. This document is provided without a warranty of any kind, either express or implied,

including but not limited to, the implied warranties of merchantability, fitness for a particular purpose, or non-infringement. This

document is for informational purposes and may not be incorporated into a contract. SAP assumes no responsibility for errors or

omissions in this document, except if such damages were caused by SAP intentionally or grossly negligent.

All forward-looking statements are subject to various risks and uncertainties that could cause actual results to differ materially

from expectations. Readers are cautioned not to place undue reliance on these forward-looking statements, which speak only as

of their dates, and they should not be relied upon in making purchasing decisions.