ARMvisor, more details

ARMvisorPeter Chang

This side is licensed under CC-BY-NC-SA姓名標示─非商業性─相同方式分享

(http://creativecommons.org/licenses/by-nc-sa/3.0/tw/legalcode)

http://creativecommons.org/licenses/by/3.0/tw/legalcode

http://creativecommons.org/licenses/by/3.0/tw/legalcode

Who am I?

Who am I?

• Peter Chang

Who am I?

• Peter Chang

• One of the current developers of ARMvisor

Who am I?

• Peter Chang

• One of the current developers of ARMvisor

• http://tw.linkedin.com/in/peterchangtw

http://tw.linkedin.com/in/peterchangtw

http://tw.linkedin.com/in/peterchangtw

What is ARMvisor?

ARM

ARM

ARM

What is KVM?

What is KVM?

• A.k.a. Kernel-based Virtual Machine

What is KVM?


• Type-II Virtual Machine Monitor

What is KVM?


• Type-II Virtual Machine Monitor

• A module of Linux kernel

What is KVM?

What is KVM?

• Officially support x86/x64, PowerPC, S390

What is KVM?

• Officially support x86/x64, PowerPC, S390

• No official support for ARM architecture

What is ARMvisor?

What is ARMvisor?

• ARM架構上的KVM

What is ARMvisor?


• Para-virtualization

What is ARMvisor?



• Trap & Emulation

What is ARMvisor?




• Dynamic Memory Allocation

What is ARMvisor?




• Dynamic Memory Allocation

• virtio & IRQchip-in-kernel

Hardware: ARM Cortex-A8

Host OS: Linux 2.6.38

ARMvisorDriver

QEMU 0.14

Device

Driver

Guest OS: Linux 2.6.35

2012

2011

2010

2009

2012

2011

2010

2009

2012

2011

2010

2009

Starting Point

2012

2011

2010

2009

Starting Point

2012

2011

2010

2009

Starting Point

ARMvisor Prototype

2012

2011

2010

2009

Starting Point

ARMvisor Prototype

2012

2011

2010

2009

Starting Point

ARMvisor Prototype

CPU Opt & Mem Opt

2012

2011

2010

2009

Starting Point

ARMvisor Prototype

CPU Opt & Mem Opt

2012

2011

2010

2009

Starting Point

ARMvisor Prototype

CPU Opt & Mem Opt

I/O Opt

Supported Hardware

ARM Realview-eb

ARM11

ARMv6 ISA

Supported Hardware

TI BeagleBoard

Cortex-A8

ARMv7 ISA

Supported Software

• Patched Host OS:

• Linaro Linux 2.6.38

• Host Root Filesystem

• Ubuntu/Debian RFS (CLI or GUI)

• QEMU 0.14

Supported Software

• Patched Guest OS:

• Linux 2.6.35 (running on ARMv6 ISA)

• Guest Root Filesystem:

• Ubuntu/Debian RFS (CLI or GUI)

System model of ARMvisor

CPU virtualization

CPU virtualization

• ARM is non-virtualizable CPU

• Patch guest OS

• “Trap and emulation”

ARMv6 ISA1.Branch instructions2.Data-processing instructions3.Multiply instructions4.Parallel addition and subtraction instructions5.Extend instructions6.Miscellaneous arithmetic instructions7.Other miscellaneous instructions8.Status register access instructions9.Load and store instructions10.Load and Store Multiple instructions11.Semaphore instructions12.Exception-generating instructions13.Coprocessor instructions


Sensitive Instructions ?


Data-processing instructions S-BIT: MOVS, ...

Status register access instructions MRS, MSR, CPS, SETEND

Load and store instructions T-BIT: LDRT, STRT, ...

Load and Store Multiple instructions LDM(2), LDM(3), STM(2)

Exception-generating instructions SWI, BKPT

Coprocessor instructions MCR, MRC, MCRR, ...

Sensitive Instructions

Observation from Guest Linux Code

• MOVS• (MRS, MSR, CPS) • (LDRBT, LDRT, STRBT, STRT)• (LDM(2), LDM(3), STM(2))• SWI• (MCR, MRC, MCRR)

15 sensitive instructions used in the guest linux code

“Trap & emulation”

User%space%%

Kernel%space�

Guest%OS�

VMM�

trap�

How to “trap”?

…mov r0, r0add sp, spmovs pc, lr…

How to “trap”?

…mov r0, r0add sp, spvirt_svc_movs “movs pc, lr”…

How to “trap”?


How to “trap”?


.macro virt_svc_movs, instSWI 0x190\inst.endm

How to “emulate”?

oxffff0000

oxffff1000

Kernel Vector0xffff001c

oxffff0000

oxffff1000


oxffff0000

oxffff1000


The KVM trapInterface

KVM Trap Entry

KVM/Guest Context Switch

UnitHost Trap Handler

Instruction Emulation

Exception/Interrupt Emulation

MMU Emulation

QEMU I/OEmulation

KVM Trap Dispatcher

UND ABORT SWI IRQ/FIQ

Kernel space

KVM

User space

QEMU

Guest Mode

Guest OS

Kernel space

KVM

User space

QEMU

Guest Mode

Guest OS

1. VM initialization

Kernel space

KVM

User space

QEMU

Guest Mode

Guest OS

2. Return to QEMU


Kernel space

KVM

User space

QEMU

Guest Mode

Guest OS

2. Return to QEMU


3. Run VM

Kernel space

KVM

User space

QEMU

Guest Mode

Guest OS

2. Return to QEMU


3. Run VM4. Enter Guest

Kernel space

KVM

User space

QEMU

Guest Mode

Guest OS

2. Return to QEMU



5. Exit Guest

Kernel space

KVM

User space

QEMU

Guest Mode

Guest OS

2. Return to QEMU



5. Exit GuestLightweight trap

Kernel space

KVM

User space

QEMU

Guest Mode

Guest OS

2. Return to QEMU



5. Exit Guest

6. Enter Guest

Lightweight trap

Kernel space

KVM

User space

QEMU

Guest Mode

Guest OS

2. Return to QEMU



5. Exit Guest

6. Enter Guest

7. Exit Guest

Lightweight trap

Kernel space

KVM

User space

QEMU

Guest Mode

Guest OS

2. Return to QEMU



5. Exit Guest

6. Enter Guest

7. Exit Guest8. Return to QEMU

Lightweight trap

Kernel space

KVM

User space

QEMU

Guest Mode

Guest OS

2. Return to QEMU



5. Exit Guest

6. Enter Guest


Lightweight trap

Heavyweight trap

Kernel space

KVM

User space

QEMU

Guest Mode

Guest OS

2. Return to QEMU



5. Exit Guest

6. Enter Guest


9. Run VM

Lightweight trap

Heavyweight trap

Kernel space

KVM

User space

QEMU

Guest Mode

Guest OS

2. Return to QEMU



5. Exit Guest

6. Enter Guest


9. Run VM

10. Enter Guest

Lightweight trap

Heavyweight trap

oxffff0000

oxffff1000



VCPU Register

FileSync

oxffff2000

oxffff0000

oxffff1000



Shadow Register FileVCPU

Register File

Sync

oxffff2000

oxffff0000

oxffff1000



Shadow Register File

mcr cpsr, r1

VCPU Register

FileSync

oxffff2000

oxffff0000

oxffff1000



Shadow Register File

mcr cpsr, r1

VCPU Register

FileSync

oxffff2000

oxffff0000

oxffff1000



Shadow Register FileRead/WriteInstructions

mcr cpsr, r1

VCPU Register

FileSync

oxffff2000

oxffff0000

oxffff1000



Shadow Register FileRead/WriteInstructions

mcr cpsr, r1

VCPU Register

FileSync

oxffff2000

Memory virtualization

Virtual Address

Physical Address

PTB

GVA

GPA

HVA

HPA

Guest PTBGVA

GPA

HVA

HPA

Guest PTBGVA

GPA

HVA

HPA

Guest PTBGVA

GPA

HVA

HPA

Host PTB

Guest PTBGVA

GPA

HVA

HPA

Host PTB

New SPTE !!!

PABT/DABT trap

PABT/DABT trap

guest page table walker

PABT/DABT trap

True Translation fault


Guest permission checker

PABT/DABT trap




PABT/DABT trap


True permission fault



MMIO access checker

PABT/DABT trap





MMIO access checker

PABT/DABT trap

MMIO emulationTrue Translation fault




Shadow page table

mapping

MMIO access checker

PABT/DABT trap





Shadow page table

mapping

MMIO access checker

PABT/DABT trapHidden protection fault





Shadow page table

mapping

MMIO access checker


Hidden translation fault





Shadow page table update

Shadow page table

mapping

MMIO access checker


Hidden translation fault




I/O virtualization

I/O virtualization

• Emulation by QEMU

• virtio

• IRQ chip in kernel

Emulate by QEMU



ARMvisor

Driver

QEMU 0.14

Device

Driver




ARMvisor

Driver

QEMU 0.14

Device

Driver


1



ARMvisor

Driver

QEMU 0.14

Device

Driver


1

2



ARMvisor

Driver

QEMU 0.14

Device

Driver


1

23



ARMvisor

Driver

QEMU 0.14

Device

Driver


1

23

4



ARMvisor

Driver

QEMU 0.14

Device

Driver




ARMvisor

Driver

QEMU 0.14

Device

Driver


5



ARMvisor

Driver

QEMU 0.14

Device

Driver


6

5



ARMvisor

Driver

QEMU 0.14

Device

Driver


76

5



ARMvisor

Driver

QEMU 0.14

Device

Driver


8

76

5

virtio

virtio

Vir,o AMBA Controller

Vir,o Driver Guest

Vring Transport

Vir,o AMBA Controller

Vir,o DeviceQEMU

irq_chip in kernel

Interrupt ControllerOpera,ng System

Deliver Interrupt

Get IRQ number

Ack IRQ number1

Mask IRQ2

End of IRQ3

Unmask IRQ4

irq_chip in kernel

ARMvisor�

GIC�

Device� Device� Device�

QEMU�

Guest�

IOCTL7

Deliver7IRQ7

Control7GIC7

irq_chip in kernel

GIC$in$Kernel�

Device� Device� Device�

QEMU�

Guest�

KVM�

Deliver$IRQ$

Control$GIC$

Future Works

Future Works

Future Works

• Support for ARM’s Virtualization extension

Future Works


• Cortex-A15 and beyond

Future Works



• SMP for host and guest

Future Works




• AArch64 Support

Future Works




• AArch64 Support

• ARMv8

By the way, ...

OpenSource

OpenSource

• We HAVE opened source in late August, 2012.

OpenSource


• GNU GPLv2

OpenSource


• GNU GPLv2

• Source code of Host and Guest OS

OpenSource


• GNU GPLv2

• Source code of Host and Guest OS

• https://github.com/SSLab-NTHU

https://github.com/SSLab-NTHU

https://github.com/SSLab-NTHU

Q & A

Technology

ARMvisor, more details