Unit 2. “VMMs and hypervisors”2966-Network and Services Virtualisation2013-2014 – First semesterAssistant professor: Katja GillyDepartament: Physics and Computer Architectures

Outline

• Introduction• Xen project• KVM• Vmware• Usages in server platforms• Usages in client platforms

Outline

• Introduction• Xen project• KVM• Vmware• Usages in server platforms• Usages in client platforms

• System virtual machines are capable of virtualising a full set of hardware resources, including a processor (or processors), memory and storage resources and peripheral devices.

• Constructed at ISA level

• Allow multiple OS environments, or support time sharing.

• Examples

IBM VM/360

VMware

Xen

KVM

OKL4

Introduction

Main Theorem

• A virtual machine monitor can be constructed if the set of sensitive instructions is a subset of the set of privileged instructions

• Proof shows

1. Equivalence

• by interpreting privileged instructions and executing remaining instructions natively

2. Resource control

• by having all instructions that change resources trap to the VMM

3. Efficiency

• by executing all non-privileged instructions directly on hardware

• A key aspect of the theorem is that it is easy to check

Introduction

• Emulation seems a good way to implement VMM.

We can run a guest OS above emulator.

Emulators can manage all hardware resource and arrange sharing resource to the guest OS

• However, there are rarely people using emulator as VMM.

• WHY?

Because emulation is quite SLOW!

It’s not Efficient!

• How to let it faster?

Don’t emulate everything. Just emulate some sensitive instructions which will directly access hardware resource.

Execute all non-privileged instructions directly on hardware

Emulation & virtualisation

• Definition:

We run the guest OS without modified.

The guest OS doesn’t realise that it is running above VM rather than physical machine.

• Pro:

User can use any OS what he/she wants to install as guest OS

For those OS which is hard to patch (ex: Windows, because it’s hard to get its source code), user can only install them in full-virtualisation environment

• Con:

For non-virtualisable CPU, running guest OS without patched critical instruction need use Dynamic Binary Translation in the hypervisor. It costs a lot.

Even for the virtualisable or hardware assistant CPU, running an OS which doesn’t find out it's in VM rather in physical machine, it is still lots of computing resource. Using patched guest OS can avoid these resource wasting. And using full-virtualisation environment cannot gain the performance optimisation.

Full-virtualisation

• Definition:

Run the guest OS which is patched for virtualisation.

The guest OS realises that it is running above VM rather than physical machine.

• Pro:

For non-virtualisable CPU, running guest OS with patched critical instruction can reduce lots of work for hypervisor. Let guest OS run faster.

Even for the virtualisable or hardware assistant CPU, running an OS which doesn’t find out it's in VM rather in physical machine, it is still lots of computing resource. Using patched guest OS can avoid these resource wasting.

• Con:

User cannot use any OS what he/she wants to install as guest OS

For those OS which is hard to patch (ex: Windows, because it’s hard to get its source code), user cannot install them as guest OS.

Para-virtualisation

• According to the category from Popek and Goldberg in 1974, virtual machine monitor can be separate into two major type which are majorly category from where the hypervisor is.

• Type 1

a.k.a. “Bare-metal VMM”

• Type 2

a.k.a. “Hosted VMM”

Several types of VMM

Bare-Metal VMM

Bare-Metal VMM

• VMM is responsible for scheduling and managing the allocation of HW resources

• Example:

Xen

Hyper-V

VMware workstation

Hosted VMM

Hosted VMM

• VMM is built on top of an existing OS

• Installation process is similar to the installation of an APP

• Let the host OS to provide device drivers and other low-level services

• Can patch privileged instructions to VMM calls (traps), or using DBT techniques

• Example:

VMware player

KVM

Parallels

Comparison with native and hosted VMs

Non-privilegedmodes

PrivilegedMode

Applications

OS

Traditionaluniprocessor

system

Hardware

VirtualMachine

VMM

Hardware

VirtualMachine

Host OS

Hardware

VMM

VirtualMachine

Host OS

Hardware

VMM

NativeVM system

User-modeHosted

VM system

Dual-modeHosted

VM system

Outline

• Introduction• Xen project• KVM• Vmware• Usages in server platforms• Usages in client platforms

04/02/14 17

Xen Project bio

• Xen project was created in 2003 at the University of Cambridge Computer Laboratory in what's known as the Xen Hypervisor project – Led by Ian Pratt with team members Keir Fraser, Steven Hand, and Christian

Limpach.

– This team along with Silicon Valley technology entrepreneurs Nick Gault and Simon Crosby founded XenSource which was acquired by Citrix Systems in October 2007

• The Xen® hypervisor is an open source technology, developed collaboratively by the Xen community and engineers (AMD, Cisco, Dell, HP, IBM, Intel, Mellanox, Network Appliance, Novell, Red Hat, SGI, Sun, Unisys, Veritas, Voltaire, and of course, Citrix)

• Xen is licensed under the GNU General Public License

• Xen supports Linux 2.4, 2.6, Windows and NetBSD 2.0

• Name from neXt gENeration virtualization

04/02/14 18

Xen Components

A Xen virtual environment consists of several modules that provide the virtualisation environment:• Xen Hypervisor - VMM

• Domain 0

• Domain Management and Control

• Domain User, can be one of:– Paravirtualised Guest: the kernel is aware of virtualisation

– Hardware Virtual Machine Guest: the kernel runs natively

04/02/14 19

Hypervisor - VMM

Domain 0

Domain Managemen

t and Control

Domain UParavirtual Guest

Domain UParavirtual Guest

Domain UParavirtual Guest Domain U

HVM Guest

Domain UHVM Guest

Domain UHVM Guest

Xen Components

04/02/14 20

Xen Components

• When Xen boots, one of the first things it does is load a Domain 0 (dom0) guest kernel. Domain 0 is the first guest to run, and has elevated privileges.

• Domain 0 is very important to a Xen system. Xen does not include any device drivers by itself, nor a user interface. These are all provided by the operating system and userspace tools running in the dom0 guest

• The most obvious task performed by the dom0 guest is to handle devices. This guest runs at a higher level of privilege than others, and so can access the hardware

• In contrast, other domains are referred to as domain U (domU )—the “U” stands for unprivileged. However, it is now possible to delegate some of dom0’s responsibilities to domU guests, which blurs this line slightly.

04/02/14 21

Xen Components

04/02/14 22

Xen Components

• There are three components to any driver:

– The split driver

– The multiplexer

– The real driver

• The split driver is typically as simple as it can be. It is designed to move data from the domU guests to the dom0 guest, usually using ring buffers in shared memory.

• The real driver should already exist in the dom0 operating system, and so it cannot really be considered part of Xen.

• The multiplexer may or may not. In the example of networking, the firewalling component of the network stack already provides this functionality. In others, there may be no existing operating system component that can be pressed into use.

04/02/14 23

Xen Hypervisor - VMM

• The hypervisor is Xen itself. • It goes between the hardware and the operating systems of the various

domains. • The hypervisor is responsible for:– Checking page tables– Allocating resources for new domains– Scheduling domains. – Booting the machine enough that it can start dom0.

• It presents the domains with a VirtualMachine that looks similar but not identical to the native architecture.

• Just as applications can interact with an OS by giving it syscalls, domains interact with the hypervisor by giving it hypercalls. The hypervisor responds by sending the domain an event, which fulfills the same function as an IRQ on real hardware.

• A hypercall is to a hypervisor what a syscall is to a kernel.

04/02/14 24

Xen Hypervisor - VMM

• Thin, privileged abstraction layer between the hardware and operating systems

• Defines the virtual machine that guest domains see instead of physical hardware:

– Grants portions of physical resources to each guest

– Exports simplified devices to guests

– Enforces isolation among guests

04/02/14 25

• The hypervisor executes privileged instructions, so it must be in the right place:

• x86 architecture provides 4 privilege levels / rings

• Most OSs were created before this implementation, so only 2 levels are used

• Xen provides 2 modes:– In x86 the applications are run at ring 3, the kernel at ring 1

and Xen at ring 0

– In x86 with VT-x, the applications run at ring 3, the guest at ring non-root-0 and Xen at ring root-0 (-1)

Restricting operations with Privilege Rings

04/02/14 26

3

0

3

1

0

3

0

The Guest is moved to ring 1

Native Paravirtual x86

HVM x86

Applications Guest kernel (dom0 and dom U)

Hypervisor

The Hypervisor is moved to ring -1

Restricting operations with Privilege Rings

04/02/14 27

• Domain 0 is a Xen required Virtual Machine running a modified Linux kernel with special rights to:– Access physical I/O devices• Two drivers are included in Domain 0 to attend requests from

Domain U PV or HVM guests

– Interact with the other Virtual Machines (Domain U)– Provides the command line interface for Xen daemons

• Due to its importance, the minimum functionality should be provided and properly secured

• Some Domain 0 responsibilities can be delegated to Domain U (isolated driver domain)

Domain 0

04/02/14 28

Domain 0

Network backend driver

Block backend driver

Communicates directly with the local networking hardware to process all virtual machines

requests

Communicates with the local storage disk to read and write data from the drive based upon Domain

U requests

PV

HVM

Qemu-DMSupports HVM guests for

networking and disk access requests

Domain 0

04/02/14 29

Domain Management and Control - Daemons

The Domain Management and Control is composed of Linux daemons and tools:• Xm– Command line tool and passes user input to Xend through XML RPC (creates and manages

guest VMs)

• Xend– Python application that is considered the system manager for the Xen environment (Daemon

to communicate with the hypervisor)

• Libxenctrl– A C library that allows Xend to talk with the Xen hypervisor via Domain 0 (privcmd driver

delivers the request to the hypervisor)

• Xenstored–Maintains a registry of information including memory and event channel links between

Domain 0 and all other Domains

• Qemu-dm– Supports HVM Guests for networking and disk access requests

04/02/14 30

Domain U – Paravirtualised guests

• The Domain U PV Guest is a modified Linux, Solaris, FreeBSD or other UNIX system that is aware of virtualisation (no direct access to hardware)

• No rights to directly access hardware resources, unless especially granted

• Access to hardware through front-end drivers using the split device driver model

• Usually contains XenStore, console, network and block device drivers

• There can be multiple Domain U in a Xen configuration

04/02/14 31

Domain U – Paravirtualised guests

Domain U - PV

Network front-end driver

Block front-end driver

Communicates with the Network backend driver in Domain 0

Communicates with the Block backend driver in Domain 0

Console driver

XenStore driver

Similar to a registry

04/02/14 32

Domain U – HVM guests

• The Domain U HVM Guest is a native OS with no notion of virtualisation (sharing CPU time and other VMs running)

• An unmodified OS doesn’t support the Xen split device driver, Xen emulates devices by borrowing code from QEMU

• HVMs begin in real mode and gets configuration information from an emulated BIOS

• For an HVM guest to use Xen features it must use CPUID and then access the hypercall page

Domain U - HVM

Xen virtual firmware

Simulates the BIOS for the unmodified operating system to

read it during startup

04/02/14 33

Pseudo-Physical to Memory Model

… …

… …

… …

Application

Kernel

Hypervisor

Virtual

Pseudo-physical

Machine

• In an operating system with protected memory, each application has it own address space. A hypervisor has to do something similar for guest operating systems.

• The triple indirection model is not necessarily required but it is more convenient from the performance point of view and modifications needed in the guest kernel.

• If the guest kernel needs to know anything about the machine pages, it has to use the translation table provided by the shared info page (rare)

04/02/14 34

Pseudo-Physical to Memory Model

• There are variables at various places in the code identified as MFN, PFN, GMFN and GPFN

PFN (Page Frame Number) It means “some kind of page frame number”. The exact meaning depends on the context

MFN (Machine frame number) Number of a page in the (real) machine’s address space

GPFN (Guest page frame number)

These are page frames in the guest’s address space. These page addresses are relative to the local page tables

GMFN (Guest machine frame number)

This refers to either a MFN or a GPFN, depending on the architecture

04/02/14 35

Virtual Ethernet interfaces

• Xen creates, by default, seven pair of "connected virtual ethernet interfaces" for use by dom0

• For each new domU, it creates a new pair of "connected virtual ethernet interfaces", with one end in domU and the other in dom0

• Virtualized network interfaces in domains are given Ethernet MAC addresses (by default xend will select a random address)

• The default Xen configuration uses bridging (xenbr0) within domain 0 to allow all domains to appear on the network as individual hosts

04/02/14 36

OFF RUNNING

SUSPENDED

PAUSED

Turn on

Turn off

Resume

PauseStart (paused)

Stop

Turn off

Wake Sleep

Migrate

The Virtual Machine lifecycle

Xen provides 3 mechanisms to boot a VM:

• Booting from scratch (Turn on)

• Restoring the VM from a previously saved state (Wake)

• Clone a running VM (only in XenServer)

Outline

• Introduction• Xen project• KVM• Vmware• Usages in server platforms• Usages in client platforms

04/02/14 38

KVM Overview

• It is a VMM built within the Linux kernel– The name stands for Kernel Virtual Machines– It is included in mainline Linux, as of 2.6.20

• It offers full-virtualisation– Para-virtualisation support is in alpha state

• It works *only* in platforms with hardware-assisted virtualisation– Currently only Intel-VT and AMD-V – Recently also s390, PowerPC and IA64

• Decision taken to achieve a simple design– No need to deal with ring aliasing problem,– Nor excessive faulting avoidance– Nor guest memory management complexity– Etc

04/02/14 39

Why KVM?

• Today’s hardware is becoming increasingly complex–Multiple HW threads on a core–Multiple cores on a socket–Multiple sockets on a system– NUMA memory models (on-chip memory controllers)

• Scheduling and memory management is becoming harder accordingly• Great effort is required to program all this complexity in hypervisors– But an operating system kernel already handles this complexity– So why no reuse it?

• KVM makes use of all the fine-tuning work that has gone (and is going) into the Linux kernel, applying it to a virtualised environment

• Minimal footprint– Less than 10K lines of kernel code– Implemented as a Linux’s module

04/02/14 40

How does it work?

• A normal Linux process has two modes of execution: kernel and user– KVM adds a third mode: guest mode

• A virtual machine in KVM will be “seen” as a normal Linux process– A portion of code will run in user mode: performs I/O on behalf of the

guest

– A portion of code will run in guest mode: performs non-I/O guest code

04/02/14 41

guest mode

With its own 4 rings

How does it work?

04/02/14 42

Key features

• Simpler design: Kernel+Userspace (vs. Hypervisor + Kernel + Userspace)– Avoids many context switches– Code reuse (today and tomorrow)– Easy management of VMs (standard process tools)

• Supports Qcow2 and Vmdk disk image formats– “Growable” formats (copy-on-write)

• Advanced guest memory management– Increased VM density with KSM (under development)

• KSM is a kernel module to save memory by searching and merging identical pages inside one or more memory areas

– Guest’s page swapping allowed

04/02/14 43

Future trends

• Para-virtualisation support (Windows & Linux)– virtIO devices already included in Linux’s mainline as of

2.6.25

• Storage– Many similar guests cause a lot of duplicate storage

– Current solution: baseline + delta images• Delta degrades overtime (needs planning)

• Disk-in-file is overheady

Outline

• Introduction• Xen project• KVM• Vmware• Usages in server platforms• Usages in client platforms

04/02/14 46

VMware

• In 1998, VMware created a solution to virtualise the x86 platform, creating the market for x86 virtualisation

• The solution was a combination of binary translation and direct execution on the processor

• Nonvirtualisable instructions are replaced with new sequences of instructions

• User level code is directly executed on the processor• Each VMM provides each VM with all the services of the physical

system, including a virtual BIOS, virtual devices and virtualised memory management

04/02/14 47

VMware

04/02/14 48

VMware

Basic properties:• Separate OS and hardware –

break hardware dependencies

• OS and Application as single unit by encapsulation

• Strong fault and security isolation

• Standard, HW independent environments can be provisioned anywhere

• Flexibility to choose the right OS for the right application

04/02/14 49

Vmware major products

VMware Server • A free-of-charge virtualisation-software server suite• Run multiple servers on your server• Hosted architecture• Available for Linux hosts and Windows hosts

VMware ESX Server• An enterprise-level computer virtualisation product• Quality of service• High-performance I/O• Host-less architecture ( bare-metal )

VMware ESX architecture

04/02/14 51

VMware ESX architecture

• Datacenter-class virtualisation platform used by many enterprise customers for server consolidation

• Runs directly on a physical server having direct access to the physical hardware of the server

Parts:• Virtualisation layer (VMM/VMKernel): implements the idealised hardware

environment and virtualises the physical hardware devices • Resource Manager: partitions and controls the physical resources of the

underlying machine • Hardware interface components: enable hardware-specific service delivery• Service Console: boots the system, initiates execution of the virtualisation

layer and resource manager, and relinquishes control to those layers• Add – Virtual Centre / Lab manager

04/02/14 52

VMware ESX architecture

04/02/14 53

Primary method of interaction with

virtual infrastructure

(console and GUI)

Virtualisation layer that abstracts the

processor, memory, storage, and networking resources of the physical

host intomultiple virtual

machines

Centrallymanages the VMware ESX Server hosts

Organises all theconfiguration data

for the virtual infrastructure environment

AuthorisesVirtualCenter Servers and

ESX Server hosts appropriately for the

licensingagreement

VI Client from theVirtualCenter Server or ESX Server hosts

VMware default deployment

04/02/14 54

VMware for free

• VMware provides freeware Server and Workstation virtualisation solutions• VMware Server:

– Is a free desktop application that lets you run virtual machines on your Windows or Linux PC

– Lets you use host machine devices, such as CD and DVD drives, from the virtual machine

– Datasheet or FAQ page is available

– Different Virtual Appliances are provided for free

• VMware Player:– Similar to VMware Server but limited to run pre-built virtual appliances

Outline

• Introduction• Xen project• KVM• Vmware• Usages in server platforms• Usages in client platforms

04/02/14 56

Virtualisation in servers

04/02/14 57

Workload Consolidation description

HWn

…

HW0

VM1 VMn

OS

App

OS

App …

HW

VM1 VMn

VMM

OS

App

OS

App

Hardware is more stressed:

- More intense IO- More intense mem usage

Focus in Equivalence

prop

• Too many servers: Hot and underutilised– Server virtualisation consolidates many systems onto one physical platform– Higher physical resource utilization

• Improved utilization: Several users with different OS requirements can more easily share a virtualised server– It allows to run legacy and new operating systems concurrently

04/02/14 58

Workload Consolidation pros/cons

• Pros:– Each application can run in a separate environment delivering true isolation– Cost Savings: Power, space, cooling, hardware, software and management– Ability to run legacy applications in legacy OSs– Ability to run through emulation legacy applications in legacy HW

• Cons:– Disk and memory footprint increase due to multiples OSs– Performance penalty caused by resource sharing management

Workload consolidation provides the basis most usages/benefits of virtualisation

04/02/14 59

…

HW

VM1 VMn

VMM

OS

App

OS

App

HWn

…

OS

App AppFocus in

Efficiency prop

(P&G)

Workload Isolation

• Virtualisation can improve overall system security and reliability by isolating multiple software stacks in their own VMs

– Security: intrusions can be confined to the VM in which they occur

– Reliability: software failures in one VM do not affect the other VMs

– As a side effect, if the hypervisor or drivers are compromised, the whole VMs can be compromised (equivalent to BIOS attack)

04/02/14 60

Workload migration for dynamic load balancing

…

HW0

VM1VM1 VMnVMn

VMM

OS

App

OS

App …

HWn

VM1VM1 VMnVMn

VMM

OS

App

OS

App

What happens with the established connections, mem pages, disk?

• If a given application needs more resources, it could be easily moved to other physical host with more power

• Several application replicas can be started in different physical machines and split the workload among them–While the applications should be engineered for this behaviour, the

deployment is simplified

04/02/14 61

Workload migration for disaster recovery

HW0

VMM

VM1VM1

OS

App

HW0

VMM

VMn

OS

App

HW

VMM

VMn’

OS

App

VM1’VM1’

OS

App

How often should the VMM store the VM’s status?

• High Availability: if an application goes down, it is not necessary to wait for the reboot of the operating system/application–Virtualisation allows to immediately put another replica up–State-less applications can be immediately restored from snapshots–Disk images should exist in external network storage (SAN, NAS)

04/02/14 62

HW0

VMM

VM1

OS

App

HW0

VMM

VM2

OS

App

OS

App

Gold ImageGold Image

OS

App

HW1

VMM

VMn

OS

App

VM3VM3

OS

App

OS

App

OS

App

OS

App

Deployment of standard environments

• Service providers usually offer some standard services– Standard images can be provided instantaneously– Simplifies deployment procedures: everything is stored in a file that

represents the VM– Easier backward compatibility (Gold Image 1, 2, 3, etc)

Deployment of standard environments

04/02/14 63

HW

VMM

OS

App

OS

App

Test and deployment

• Development and testing environments– A VM with standard tools is distributed amongst developers– Releasing new revisions of tools, patches, etc. is very simple

• Business Agility and Productivity– It allows to easily transform environments (Development to test, back to

development, etc)

• Deployment of Patches in controlled environments• Allows for testing in production hardware before formal activation

04/02/14 64

Virtu

al A

pp

lian

ce

Virtual Machine

Soft

ware

Ap

plia

nce

App.

Middleware

Fully pre-installed and pre-configured application

and operating system environmentVirtual Platform

OS

A software appliance is a software

application combined with a tailored

operating system

Virtual appliances

• The convergence of virtual machine technology and a new initiative by several tool vendors is giving birth to this new form of software packaging called Virtual Appliance

– Software appliance that includes necessary components to run in a virtual computing environment (VMware, Xen, Virtual Iron, …)

– Both, software and virtual appliances, are aimed to eliminate the installation, configuration and maintenance costs associated with running complex stacks of software

Outline

• Introduction• Xen project• KVM• Vmware• Usages in server platforms• Usages in client platforms

04/02/14 66

HW

VMMOS

App

HW

OS

Game

Windows games emulation in Mac

• Games are very dependant of the platform for which they were programmed– Creating games for each platform is too costly

• Minimal emulation is needed for minimal performance degradation

04/02/14 67

HW

VMM

OS

App

OS

App

Test and Development – also for clients

• Development and testing environments– A VM with standard tools is distributed amongst developers– Releasing new revisions of tools, patches, etc. is very simple

• Business Agility and Productivity– It allows to easily transform environments (Development to test, back to

development, etc)

• Deployment of Patches in controlled environments• Simplifies testing of P2P applications

04/02/14 68

HW

OS

VM1’HW (new)

VMM

VM2

OS

App

VM1VM1

OS

App

App

HW

OS

VM2’ App

Connectivity becomes the critical path

Desktop infrastructure management

• Migration to new hardware due to OS compatibility is too costly– Windows Vista slow adoption is an example– Gradual upgrade can be implemented

• Desktops are underutilised (as opposed to previous)– Consolidate in the server, run remotely in the client

04/02/14 69

HW (personal)

VMMHW (corporation)

VMM

VM2

OS

App

VM1VM1

OS

App

HW (personal)

VMM

OS

Personal

OS

Corp

Connectivity becomes the critical path

OS

Personal

OS

Corp

My corporate PC within my PC

• Users sharing their own PC with the corporate– A VM for own work

– A VM for corporate usages

– Can be extended to more profiles (secure, entertainment, etc)

• Temporary remote employees are given a VM with expiration date

Bibliography

• Books :– The definite guide to Xen Hypervisor. David Chisnall.

Prentice Hall

• Other resources :– Lecture slides of “Virtual Machine” course (5200) in

NCTU

– KVM web page: http://www.linux-kvm.org

– VMware web page: http://www.vmware.com