Dealing with Hardware Heterogeneity Using a Virtualization Framework Tailored to ARM Based Embedded Systems

Prof. Daniel Rossier, PhD

HEIG-VDInstitut REDS, Reconfigurable & Embedded Digital Systemsrte Cheseaux 1, 1400 Yverdon-les-Bainshttp://www.reds.ch/

REDS@HEIG-VD

Outline• Background

• Overview of EmbeddedXEN

• Protection & Memory Isolation

• Domain Interactions

• Device Heterogeneity

• Conclusions & Future Work

2

REDS@HEIG-VD

Background

• HEIG-VD • University of Applied Sciences in Yverdon, Switzerland (CH)

• Reconfigurable Embedded Digital System Institute

Hardware Design, FPGA

Embedded Execution Environment, Drivers, BSP, OS/RTOS, etc.

• Applied Research & Development• ARM based Microcontrolers (v4, v5, v6, v7)

• Low-level interactions between computing cores (CPU, DSP, FPGA, etc.)

• Towards Cortex-A15 HVM

3

REDS@HEIG-VD4

Background• Embedded Virtualization on ARM

• XEN: free & easy access to a stable & evolving hypervisor source code

• Early port of XEN on ARM in 2007 in the context of a Diploma Project

• Necessity to get a simple, thin, fast, robust, easy-to-deployed virtualization framework

• Re-use of Linux file organization & build system (Makefiles, scripts, …)

• Focus on realtime aspects (Linux/Xenomai as RTOS)

• Different sources of inspiration• "Fast Secure Virtualization for the ARM Platform", Daniel

Ferstay, Master Thesis, The University of British Columbia, 2006

• XEN ARM port, George G. Davis, MontaVista, 2007

• Secure Xen on ARM Project, Sang-bum SUH, Samsung, 2007

REDS@HEIG-VD

Background

• Focus on heterogeneity of embedded devices• Idea to re-use OS & applications from old devices to recent

devices

• Time-to-market migration to new hardware generation

• Dealing with various cross-compiled binaries (ARM v5-v7)

• Dealing with different peripherals

• Less emphasis on security aspects

• Publicly available• https://sourceforge.net/projects/embeddedxen

5

REDS@HEIG-VD6

Background• Hardware constraints

• Low latency, reactivity (response time)

• ARM cores do not support virtualization mechanisms

not easy to deal with various levels of execution modes

• Para-virtualization remains attractive• About 30 files to be (slightly) adapted

• Low execution overhead

• Efficient processing of downcalls/upcalls with support of domain interactions

Physical interrupts are quickly processed in dom0.

REDS@HEIG-VD7

Overview of EmbeddedXEN

• Limited use of hypercalls

• ARM execution modes (USR/SVC) – pseudo-user mode with double stack handling / downcalls & upcalls are simple jumps to specific (callback) addresses

• No subtle use of domain access control (ARM DACR) to protect memory

• Memory sharing facilitated between primary and secondary OS

Domain Creation and

Setup

Secondary Guest OS known as DomU

VCPU sched_flipScheduler

Migration

Manager

Hardware

Primary Guest OS known as Dom0

EmbeddedXEN Hypervisor Hypercalls

Handling

Xen-guest(pv)

Xen-guest(pv)

PrioritizedUpcalls

Handling

- Full privilege- DomU original

drivers- Frontend drivers

- Full privilege- Dom0 original drivers- Backend drivers- Native drivers

REDS@HEIG-VD

Overview of EmbeddedXEN• Linux-like source tree and build system (Makefiles)

8

embeddedxen

hypervisor-4.0.2

linux-2.6.32-dom0

arch/arm

linux-2.6.26-domU

xen-guest

xen-guest

tools

xen

arch/arm

environnement.conf

include

xen-guest

include

xen-guest

xen-guest

mach-mx35

arch

arm

mach-msm

mach-msm

xenbus

core includ

earch

arm

mach-mx35

mx35_fab4.c

kernel

mm

console

include

xen-guest

Hypervisor

Commonpart

Primary OS

SecondaryOS

Config & build system

REDS@HEIG-VD9

Overview of EmbeddedXEN

• Single binary (multi-kernel) image

• Automatic parsing & image relocation during hypervisor bootstrap

uImage

Boot Head

EmbeddedXEN

Hypervisor

DOM-0EOD

0 (Dom-0)

DOM-UEOD

1 (Dom-U)

vmlinux.dom0 vmlinux.domUvmlinux

Dom-0 Filesystem

Dom-U Filesystem

/home/root/squeezeos.rootfs.img: • Stored separately in sdcard, for example

REDS@HEIG-VD

Protection & Memory Isolation• Memory isolation between domains relies on different

address space isolation.• No further advanced mechanisms to protect hypervisor and guest memory

• The guest OS kernel runs at the same privilege as the hypervisor.

• Each domain receives its own (contiguous) physical memory region during domain set up.

• No pagination of the kernel linear address space is performed.

• Paravirt of memory management is kept minimal.

• Guest OS kernel has access to the whole memory.• No protection mechanism for strong isolation of VM (but is it really necessary?)

10

REDS@HEIG-VD

Protection & Memory Isolation

• Virtual & Physical Memory Layouts

11

3GiB

PAGE_OFFSET 0xC0000000

0x00000000

Initial Virtual Address Space Physical RAM

boot allocator

XEN heap (2 MiB)

frame table

I/OVMALLOC_END

Interrupt Vectors0xffff0000

0xFF000000

Hypervisor

dom0

Hypervisor code

1st-level page table

domU

dom0domU

Linearmapping

Linearmapping

3GiB

dom0

boot allocator

XEN heap (2 MiB)

frame table

I/O

Interrupt Vectors

Hypervisor code

1st-level page table

dom0 Virtual Address Space

REDS@HEIG-VD

Domain Interactions• Virtualization of peripherals in EmbeddedXEN is quite

similar to the existing mechanisms in XEN.• Driver split with frontend & backend drivers

• Communication with xenbus

• Use of grant tables for sharing/copying pages between domains

• However, a revisited (simplified) implementation of these mechanisms have been achieved in EmbeddedXEN.

• XEN store is dynamically allocated at boot time of guest OSes.

• No user space tools are required to manage XEN store entries or peripherals configs.

• Hotplugs & dynamic configs of peripherals are less relevant to embedded systems.

12

REDS@HEIG-VD

Domain Interactions• domU is passed information during bootstrap under control

of dom0.

13

xen-guestsubsys

Hypervisor

OS Subsys

start_info->store_mfnstart_info->store_evtchn

domU

xenstore

xen-guest subsysxenbus

thread

xenbus thread

xenstore thread

frontend drivers

backend drivers

event_channel C/D

event_channel A/B

Page Dom0

prod

cons

Page DomU

via unpause_domU(store_mfn, store_evtchn)

prod

cons

OS Subsys

REDS@HEIG-VD14

Domain Interactions• Grant tables are used in a different way

• Shared pages are possible only in the vmalloc'd area.

• Kernel linear addresses are not shareable; contents needs to be copied using temporary mappings.

3GiB

0xC0000000

0x00000000

RAM

VMALLOC_END

Hypervisor0xFF000000

Hypervisor

dom0

domU

VMALLOC_START

3GiB

Hypervisor

0xC0000000

0x00000000

VMALLOC_END

0xFF000000

VMALLOC_STARTgnttab(dom0)

gnttab(dom0)

mem_map_foreign(domU)

gnttab(dom0) gnttab_foreign[0]

gnttab

mem_map_foreign mem_map_foreign(dom0)

shared pages

Dom0 DomU

gnttab(domU) gnttabgnttab(domU)gnttab_foreign[1]

gnttab(domU)

shared pages

mem_map_foreign

has references to

has references to

has references to

has references to

REDS@HEIG-VD

Device Heterogeneity• Different levels of heterogeneity

• At CPU level: various instructions sets (locks, cache, etc.), various PTEs (MMU) flags, various co-processors, etc.

Compatibility ensured via hypercalls

• At peripherals level: not the same hardware

Compatibility ensured via backend driver processing

15

hypervisor-4.0.2

arch/arm

xen

arch/arm

mach-mx35

mach-msm

kernel

mm

include

xen-guest

mm

built-in.o cache-v6.S cache-v7.S Kconfig Makefile mm.h proc-macros.S tlb-v6.S tlb-v7.S

REDS@HEIG-VD

Device Heterogeneity• Example of ARMv6 running on ARMv7 CPU (iMX35 -> HTC

Desire HD)

16

linux-2.6.26-domU/arch/arm/mm/cache-v6.S:ENTRY(v6_flush_kern_cache_all) mov r0, #0#ifdef HARVARD_CACHE mcr p15, 0, r0, c7, c14, 0 @ D cache clean+invalidate#ifdef CONFIG_SMP mcr p15, 0, r0, c7, c5, 0 @ I+BTB cache invalidate#else b v6_icache_inval_all#endif#else mcr p15, 0, r0, c7, c15, 0 @ Cache clean+invalidate#endif

mov pc, lr

linux-2.6.26-domU/xen-guest/hypervisor.c:void xen_flush_kern_cache_all(void){ struct mmuext_op op;

op.cmd = MMUEXT_FLUSH_CACHE; HYPERVISOR_mmuext_op(&op, 1, NULL, DOMID_SELF);}

hypervisor-4.0.2/arch/arm/mm/cache_v7.S:ENTRY(xen_flush_kern_cache_all) stmfd sp!, {r4-r5, r7, r9-r11, lr} bl xen_flush_dcache_all @ much more complex!! mov r0, #0 mcr p15, 0, r0, c7, c5, 0 @ I+BTB cache invalidate ldmfd sp!, {r4-r5, r7, r9-r11, lr} mov pc, lrENDPROC(xen_flush_kern_cache_all)

linux-2.6.26-domU/arch/arm/mm/cache-v6.S:.extern xen_flush_kern_cache_allENTRY(v6_flush_kern_cache_all) b xen_flush_kern_cache_all#if 0 /* paravirt */ mov r0, #0

…

mov pc, lr#endif /* 0 */

Original version Paravirt version

REDS@HEIG-VD

Device Heterogeneity• Example of framebuffer device heterogeneity

• Configuration of framebuffer is retrieved from Dom0 via xenbus

17

Dom0 DomUxen-guest

xenfb backend xenfb frontend

User space applications(/dev/fb0)

framebuffercore

User space applications(/dev/fb0)

framebuffercore

dom0 fb memory(allocated by dom0)

fb_event_dom_register fb_event_dom_switch

Query framebuffer params

xenbus thread

domU fb memory(allocated by domU)

FB config is retrieved from dom0

device-specific

framebuffermsmfb_pan_update()

xen-guest

xenstore

xenbus thread

REDS@HEIG-VD

DomUDom0xen-guest

Device Heterogeneity• Example of audio device heterogeneity

• DomU audio buffers are accessed from Dom0 via shared pages.

18

xen-audio backendxen-audio frontend

Sound device (headset, speakers, etc.)

xenstore

User space applications

/dev/msm_pcm_out - or -/dev/pcm/snd/pcm0c0d0p

xenbus thread

User space applications

/dev/pcm/snd/pcm0c0d0p

pcm subsystem

arch-specific audio driver

pcm subsystem

xenvaud_pcm_start xenvaud_pcm_stop

Audio buffers allocated in domU

xen-guest

xenstore

xenbus thread

REDS@HEIG-VD

Conclusions & Future Work• EmbeddedXEN is an embedded virtualization framework

which puts emphasis on efficient and heterogeneous hardware.

• Application environments can be re-used "as such" on modern platforms (Android-based for example) taking advantage of last generation hardware.

• EmbeddedXEN relies on the main principles of XEN, with a revisited lightweight, but less secure architecture.

• A single multikernel binary image, easy to deploy on the target platform without additional tools, makes EmbeddedXEN well tailored to embedded systems.

19

REDS@HEIG-VD

Conclusions & Future Work

• Further investigation projects:

• Elaboration of a domU using a graphical desktop for user applications

• Support of multicore ARM CPUs (cortex-A9, cortex-A15)

• Live migration of domU using remote NFS-filesystem (migration within a cloud)

• Support of hard realtime OS (RTEMS-paravirt)

20

REDS@HEIG-VD21

• Thanks for your attention!

• Further information: daniel.rossier@heig-vd.ch