Enhancing Hypervisor and Cloud

Solutions Using Embedded Linux Iisko LappalainenMontaVista

2

Setting the Stage

This presentation will discuss the usage of Linux as a base component of hypervisor

components particularily in the context of creating a base for a cloud hosting

framework.

The basis for the discussion is formed by combining technologies like KVM/QEMU,

OpenVSwitch, OpenFlow and OpenStack to form a technical base for the solution, and

digging into particular problem areas to see where the experience from the embedded

world can help in creating a more competent experience.

The presentation also discusses the general characteristics of the above technologies

to create an overview of the problem area. In particular the issues that are key to

creating a competetive solution are the responsiveness and throughput of the system,

the capability for providing logging and tracing facilities across the hypervisor and

guest domains, the seamless integration of the components to provide stability and

maintainability and providing a suitable level of security throughout the layers of SW

in the system.

Agenda

Key Components and Architecture

Main Architecture

OpenStack

OpenvSwitch

QEMU/KVM

Virtualization Problems and Solutions

Further Work on the Topic

Example Hypervisor Solution Architecture

Hardware

Carrier Grade

Services

(HA, Tracing,

Logging,

Security,

CGL)

OS layer

and base services

KVM/QEMU incl. HW

accelerationLXC

Management

and

Provisioning

SW and

Services

OpenStack

OpenFlow

Guest VMGuest

Container

OpenVSwitch

QEMU and KVM

-based

hypervisor solutions

At least it will be looking forward

Features

QEMU: Comprehensive architecture support

X86, ARM, MIPS, PPC, etc

QEMU: Support for particular board configurations

QEMU :Live Migration of guests

KVM: Integration with HW virtualization capabilities for x86,

PPC, ARM

Best support on x86 currently

ARM seeing a lot of work currently

OpenVSwitch

OpenVSwitch features

Virtual switching ports and connection to physical switch HW

Support for OpenFlow control

Support for standards such NetFlow, sFlow, SPAN

What does it provide

Facilitates live-migration

Much easier to replicate state between hosts

Abstraction API

Similar interfaces to all switches accessed

Performance

OpenFlow

Software Defined Networking (SDN) means to separate

the control plane from the data plane in network

switches and routers to different components

The expectation is that this leads to massively more

scalable network infrastructure

The initiative for the standardization of OpenFlow is

being led by major end-users like Google and Facebook

and not really the equipment vendors

OpenFlow is split into target and controller domains

API is in essence a adaptation layer

Both types of OSS SW for Linux available

Open Stack Cloud Management

OpenStack is a cloud management system that controls large pools of

compute, storage, and networking resources throughout a datacenter, all

managed through a dashboard that gives administrators control while

empowering their users to provision resources through a web interface.

OpenStack features

Control APIs for managing VM images and setups

Integration to multiple cloud subsystems

User interface, dashboard

OpenStack Components

Quantum-network, Nova-compute, Keystone-identity

Cinder-block storage, Swift-object storage

Glance-image service

Savanna-Hadoop service

What does it provide

OpenStack is a key higher-level components in our Cloud solution

of today

A Cloud Framework like Openstack is usually required

in all real Cloud deployments

OpenStack

Agenda

Key Components and Architecture

Main Architecture

OpenStack

OpenvSwitch

QEMU/KVM

Virtualization Problems and Solutions

Further Work on the Topic

Hypervisor Pain Points

Hardware

Tracing,

Logging,

Security

functionality

Kernel Layer

KVM/QEMU

incl. HW acceleration

PIC

Guest VM

Application

HW-independent

acceleration API

Guest Network

Throughput

Real-time

Latency

Memory Access Speed

NIC Memory

Hardware

Tracing,

Logging,

Security

functionality

Kernel Layer

KVM/QEMU

incl. HW acceleration

PIC

Guest VM

Application

HW-independent

acceleration API

NIC Memory

Security and

Isolation

Multi-OS

support

Forward

portability

Hypervisor Pain Points

Hardware

Tracing,

Logging,

Security

functionality

Kernel Layer

KVM/QEMU

incl. HW acceleration

PIC

Guest VM

Application

HW-independent

acceleration API

NIC Memory

Full-stack

Management

Interfaces

Custom HW

adaptation

Hypervisor Pain Points

Background: Real-Time / Context Switching

Hardware

Tracing,

Logging,

Security

functionality

Kernel Layer

KVM/QEMU

PIC

Guest VM

Application

NIC

1) The interrupt event will enable a

thread for running, usually

invoking the host-side scheduler

first

2) The scheduler in the guest-side

will get activated to actually

thread inside the guest domain

3) The application running in the

thread will process the event

-> the critical components are the

context switching time and the

RT latency related to switching to

the right thread

Scheduler

ISR

Scheduler

Use-case: Real-Time / Context Switching

Hardware

Tracing,

Logging,

Security

functionality

Kernel Layer

KVM/QEMU

PIC

Guest VM

Application

NIC

Scheduler

ISR

Scheduler

Video Stack 1) The streaming video protocol

stack and endpoint have to

process packets at a certain

continuous rate to avoid dropping

the connection or having low

quality picture transfer

2) The application running the stack

is dependant on the scheduling

mechanism on the guest and

host

3) The timer interrupts are usually

controlled by the HW and are

used to control the timing,

sometimes a network

synchronized time source is used

Page 16

Solution: Real-Time / Context Switching

Hardware

Tracing,

Logging,

Security

functionality

Kernel Layer

KVM/QEMU

PIC

Guest VM

Application

NIC

Scheduler

ISR

Scheduler

1) Enhance latency response by

integrating Real-Time Linux to

the hypervisor

2) Improve the context switching

response by customizing the IPI

interrupt handling for the VM

guest

Video Stack

Page 17

Background: Network Throughput

Hardware

Tracing,

Logging,

Security

functionality

Kernel Layer

KVM/QEMU

Guest VM

Application

NIC

Guest Network

Throughput

Host Network

Throughput

1) The application inserts data into

the OS mechanisms that control

the HW packet transmission

inside the guest VM

2) The network handler in the guest

OS will feed the data to the

virtual interface (for example

virtio)

3) The host OS will handle

transmission from the virtual

interface to the actual HW,

possibly multiplexing the HW IP

interface

-> the critical components are the

propagation time from the guest

to the host and the effectiveness

of the host IP stack

Use-Case: Network Throughput

Hardware

Tracing,

Logging,

Security

functionality

Kernel Layer

KVM/QEMU

Guest VM

Application

NIC

Guest Network

Throughput

Host Network

Throughput

Control Plane

Server1) A control plane server application

is used to provide large amount

2) It uses standard networking to

send data from inside a

virtualized guest OS, which is

used to provide load-balancing

and failover facilities in the total

system.

3) The packet traffic is routed

through the guest and host OS

and if affected by delays at each

of the points, therefore affecting

the total capacity of the server

Solution: Network Throughput

Hardware

Tracing,

Logging,

Security

functionality

Kernel Layer

KVM/QEMU

Guest VM

Application

NIC

Guest Network

Throughput

Host Network

Throughput

Control Plane

Server1) Implement packet handling on

the host to bypass userspace

and enter the kernel or dedicated

HW directly

Eliminate processing for

traffic copy

Eliminate context transitions

2) Implement control mechanisms

for prioritizing and balancing

traffic in case of multiple VMs

Allow better QoS capabilities

for higher-priority functions,

like streaming data

Several Cloud users especially targeting embedded-

type deployments have significant SW legacy

HW consolidation of legacy OSs

Multi-version support for forward portability

future versions of the platform

Issues faced

Real-time

IO performance

However, usually new HW can do much better in relation..

Multi-OS Support

Hypervisor and Guest-to-Guest vulnerabilities are seen

as new fruitful ground for hackers

Multi-tenancy

Multi-node consolidation

Guests of lesser importance can be used to gain access

to the hypervisor and the more important Guest

domains

Hypervisor priviledge escalation

Careful management of CVEs needed

SELinux - sVirt

Not only exploits, also important data must not be

visible to non-authorized Guests

Packet traffic

Memory, IO

Security and Isolation

Key usability characteristic of a Cloud solution is the

accessibility to the provisioning and monitoring

systems

Custom GUIs for end solutions

Need access to APIs exposed by underlying facilities

Monitoring Solutions

Hypervisor and Guest logging

OpenStack interface adaptation

OpenStack plugin adapation

Libvirt as a key underlying facility

Full-Stack Management Interfaces

Agenda

Key Components and Architecture

Main Architecture

OpenStack

OpenvSwitch

QEMU/KVM

Virtualization Problems and Solutions

Further Work on the Topic

Co-operative Scheduling

Paravirtualized

Easy Adaptation of Applications

Across HW and Middleware

Adaptation layer for DPDK, SE, LWE, RTE etc

Performance and Manageability

OpenStack

Value-add-solutions

The Way Forward

Thank you!

Questions?

Comments on chosen components?