KVM/ARM: The Design and Implementation of the Linux ARM Hypervisor Christoffer Dall Department of Computer Science Columbia University [email protected]

KVM/ARM: The Design and Implementation of the Linux ARM HypervisorChristoffer Dall Department of Computer Science Columbia University [email protected] Jason Nieh Department of Compouter Science Columbia University [email protected]

32091700 김해천

ARM

~1.2 billion ~300 million

ARM

http://www.v3.co.uk/v3-uk/news/2397366/arm-aims-at-net-work-infrastructure-with-many-core-socs-and-open-software-stack

ARM Server ARM Network infrasturcture

Key Challenges

ARMVirtualization Extensions

intelVT-x

No PC-standard on ARM

!=

Hypervisor Layering in software stack

Above part

HyperOne, Xen, PikeOS, OKL4,

Hyper V, Vmware ESX

Lower part

KVM, VirtyalBox, Virtual PC,

Parallels, BlueStacks

ARM Virtualization Extensions

Provides virtualization in 4 key areas:

CPU Virtualization

Memory Virtualization

Interrupt Virtualization

Timer Virtualization

ARM Virtualization Extensions CPU Virtualization

Hyp mode was introduced as a trap and emulate mechanism to sup-port virtualization

User

Kernel

Hyp

H/w

User

Kernel

System call, page faultTo reduce virtualization overhead

ARM Virtualization Extensions memory virtualization

Hardware support to virtualize physical memory: stage 2 Page Tables

ARM Virtualization Extensions Interrupt virtualization

One distributor in a system, but each CPU core has a cpu Interface

Distributor is used to configure the GIC

CPU interface is used to acknowledge(ACK) and to signal End-Of-

Interrupt(EOI)

Interrupt can be configured to trap to either Hyp or Kernel mode

Trap to kernel : avoiding the overhead of going through Hyp mode

Trap to Hypervisor : hypervisor retain control, but big cost

GIC v2.0 include H/W virtualization (VGIC)

Virtual CPU interface, List RegisterPPI: Private Peripheral InterruptsSPI: Shared Peripheral InterruptsSGI: Soft Generated InterruptGIC : Generic Interrupt Controller

VGIC


Generic Interrupt Controller : Trapping Interrupt in Hyp Mode

Vm

H/W

Hypervi-sor

1) interrupt2) trap

3) Emulate Virtual Interrupt By signal

Cumbersome & Expensive


Generic Interrupt Controller (V2.0) , Virtual GIC : Trapping Interrupt in Kernel Mode

Vm

H/W

Hypervi-sor

1) interrupt

2) trap Good

ARM Virtualization Extensions Timer virtualization

ARM define the Generic Timer Architecture

The timers used by the hypervisor cannot be directly configured and manipulated by guest OSes.

Such timer accesses from guest OS would need to trap to Hyp mode, incurring additional overhead

counter

Timer 0

Timer 2

Timer 1

Timer 3

CPU 0

CPU 1

CPU 2

CPU 3

Virtual counter

Virtual Timer 0

Virtual CPU 0

Virtual CPU 3

Virtual CPU 1

Virtual CPU 2

Virtual Timer 1

Virtual Timer 2

Virtual Timer 3

Virtual counterVirtual counterVirtual counter

Accessible from Hyp mode

Accessible from VMsARM provides ☞

Hypervisor Architecture

KVM/ARM builds on KVM and leverages existing infrastructure in the Linux kernel

Bare metal hypervisor(xen) vs KVM/ARM

ARM platform designs are non-standard ways by different manufactures

Samsung exynos, qualcomm snapdragon, Apple A series

But, Linux is supported across almost all ARM platform ☞ by integrating KVM/ARM with Linux

PL0 User

PL1 Kernel

PL2 Hyp Linux kernel KVM

Hypervisor Architecture Split-mode Virtualization

Running KVM/ARM in Hyp mode implies running the Linux kernel in Hyp mode

This is problematic

Low-level architecture dependent code in Linux is written to work in kernel mode

Running the entire kernel in Hyp mode would adversely affect native performance

Kernel mode

Hyp mode

Kernel

Kernel ?


KVM/ARM introduces split-mode virtualization

It runs across different privileged CPU mode to take advantage offered by each CPU mode

Two components, the lowvisor and the highvisor

Lowvisor takes advantage of the H/W virtualization support available in Hyp mode

Set up the correct execution context by configuration of the H/W

Enforce protection and isolation between different execution context

Switch from a VM execution context to the host execution, vice versa

Provide a virtualization trap handler

Highvisor can directly leverage existing Linux functionality

Scheduler, kernel data structure, locking, memory allocation functions

Highvisor

Lowvisor

HypervisorOS Kernel

Kernel mode

Hyp mode

Handles Low level Functionality

Handles High level Functionality


Switching between a VM and the highvisor

Highvi-sor

Lowvisor

HypervisorOS Kernel VM

Run VM

Trap

Kernel mode

Hyp mode

Trap


Switching between a VM and the highvisor

Highvi-sor

Lowvisor


Kernel mode

Hyp mode

Function call

Trap Trap

As a result, split mode virtualization incurs a double trap cost in switching to and from the high-visor

Hypervisor Architecture CPU Virtualization

VMHypervisor

ARM

Virtualized cpu

Controlled by the Hypervisor

S/W in the VM must have persis-tent access to same register state as S/W running on the physical CPU

physical H/W state associ-ated with the hypervisor and its host kernel is persistent across running VMs

Context switch register during world-switch

Performs trap and emu-late on sensitive instruc-tion and when accessing H/W state

trap

Hypervisor Architecture Memory Virtualization

KVM/ARM provides memory virtualization by enabling Stage-2 translation

When running in a VM

Completely transparent to the VM

The highvisor manages the Stage-2 translation page tables to only allow access to memory allocated for a VM

Other accesses will cause stage-2 page faults which trap to the hypervisor

Stage-2 translation is disabled when running in the highvisor and lowvisor


Configuring page tables is a high level Functionality

Highvi-sor

Lowvisor


Kernel mode

Hyp mode

Configures Stage-2 Page Tables


LowVisor has hardware access as it runs in Hyp Mode

Highvi-sor

Lowvisor


Kernel mode

Hyp mode

Enables Stage-2 Translation


Highvi-sor

Lowvisor


Kernel mode

Hyp mode

Disables Stage-2Translation

get_user_pages()

Page fault

Hypervisor Architecture Interrupt Virtualization

When running in a VM or Host & highvisor

All, H/W interrupt processing is done in the host by using Linux’s existing interrupt handling func

However, VM must receive notifications in the form of virtual interrupt from emulated devices

KVM/ARM uses the VGIC

Multicore guest Oses musts be able to send virtual IPIs

to others

Highvisor

Lowvisor

Hypervisor

OS Kernel

VMKernel mode

Hyp mode

H/WTrap

Trap

Hypervisor Architecture Timer Virtualization

KVM/ARM leverage ARM’s H/W virtualization features of the generic timer

Unfortunately, due to architectural limitations, the virtual timers cannot di-rectly raise virtual interrupts, but always raise hardware interrupt, which trap to the hypervisor

KVM/ARM detects when a Virtual timer expires Injects

a corresponding virtual interrupt to the VM

KVM/ARM performs all hardware ACK and EOI operations

Highvisor

Lowvisor

Hypervisor

OS Kernel

VMKernel mode

Hyp mode

timer

VM

trap

Enable virtual timer

Prepare

Experimental Setup

http://www.cs.columbia.edu/~cdall/pubs/KVMARM_talk.pdf

Experimental Setup


Experimental Results

Table 3 presents costs of virtualization using KVM/ARM on ARM and KVM x86 on x86

Measured in cycle units

bg

Hypercall - Cost of two world switch

- Saving & restore VGIC state is quite expensive on ARM.- x86 provides H/W supportTrap

- Cost of switching the h/w mode from the into the cpu mode

bg

bg


Figure 3,4 show normalized performance for running lmbench in a VM versus Host

UP: uni-processorSMP:symmetrical multi-processing

Caused by updating therun-queue clock

KVM/ARM has less overhead than KVM x86 fork & exec

Repeatedly sending an IPI Cost of KVM is higher than KVM/ARMBecause this require tapping to the hypervisor on x86 but not on ARM


Figure 5,6 show normalized performance for running application workloads

More mature KVM x86 system has significantly higher virtualization overheads,KVM/ARM’s split-mode virtualization design allows it to leverage ARM H/W sup-port with comparable performance to traditional hypervisor


Figure 7 shows normalized power consumption of using virtualization

As well as ARM, mac air’s i7 is one of Intel’s more power optimized proces-sors

Both workloads are not CPU bound & the power consump-tion is not significantly affected by the virtualization layer

http://www.linux-kongress.org/2010/slides/KVM-Architecture-LK2010.pdf

http://gigglehd.com/zbxe/4440629

http://www.arm.com/



esc2014chhypforarmv7-140829040738.pdf

kvm/arm experiences building the linux arm hypervisor





http://www.arm.com/

http://www.arm.com/





Virtual CPU Interface

Virtual CPU Interface

VIRQ

ACK

Documents

KVM/ARM: The Design and Implementation of the Linux ARM Hypervisor Christoffer Dall Department of Computer Science Columbia University [email protected]