XPDS13 : PVH Technical Deep Dive - George Dunlap, Citrix

Intro PV and HVM PVH and Xen PVH and Linux PVH Issues Spare slides

PVH technical deep-dive

George Dunlap

Edinburg – 21-23 October, 2013


Intro: PVH

Edinburg – 21-23 October, 2013 PVH technical deep-dive 2 / 25


Goal

I Give you a technical overview of PVH, so that...

I you can understand the characteristics, advantages, anddisadvantages...

I and approach the code to improve and fix



Goal






Goal






Overview

I PV, HVM, and PVH

I PVH from Xen’s perspective

I PVH from Linux’s perspective

I Issues in PVH



Overview

I PV, HVM, and PVH



I Issues in PVH



Overview

I PV, HVM, and PVH



I Issues in PVH



Overview

I PV, HVM, and PVH



I Issues in PVH



PV and HVM

I Issues with PV

I PVMMU in LinuxI 64-bit hypercalls

I Issues with HVM (PVHVM)

I Qemu processI Legacy bootI Devices emulated in Xen



PV and HVM

I Issues with PVI PVMMU in Linux

I 64-bit hypercalls





PV and HVM

I Issues with PVI PVMMU in LinuxI 64-bit hypercalls





PV and HVM






PV and HVM


I Issues with HVM (PVHVM)I Qemu process

I Legacy bootI Devices emulated in Xen



PV and HVM


I Issues with HVM (PVHVM)I Qemu processI Legacy boot

I Devices emulated in Xen



PV and HVM


I Issues with HVM (PVHVM)I Qemu processI Legacy bootI Devices emulated in Xen



PVH

I “PV in an HVM container”

I Goal: take the best aspects of PV and HVM

I Written by Mukesh Rathor @ Oracle

I Significant revisions by George Dunlap @ Citrix



PVH







PVH







PVH







PVH from Xen’s perspective

I Begin with HVM guest

I Disable HVM-specific things not needed

I Start & pin in 64-bit paged mode

I Enable PV path for a handful of things
























Xen: Things disabled

I Device model (qemu)

I Emulated hardware (APIC, PIT, etc)

I Nested HVM

I MSIX






I Nested HVM

I MSIX






I Nested HVM

I MSIX






I Nested HVM

I MSIX



Xen: Pinning in 64-bit

I Set initial values ofr cr0, cr4, EFER

I Certain paging things have to happen at start of day

I Disable writes to EFER

I Don’t allow guest to change paging-related cr0 bits
























Xen: PV paths

I Enable PV hypercalls

I PV e820 map

I PVH VCPU boot

I PV CPUID

I PV PIO



Xen: PV paths


I PV e820 map

I PVH VCPU boot

I PV CPUID

I PV PIO



Xen: PV paths


I PV e820 map

I PVH VCPU boot

I PV CPUID

I PV PIO



Xen: PV paths


I PV e820 map

I PVH VCPU boot

I PV CPUID

I PV PIO



Xen: PV paths


I PV e820 map

I PVH VCPU boot

I PV CPUID

I PV PIO



PVH from Linux’s perspective

I xen hvm domain() false, xen pv domain() true

I Act natural (get rid of PV special cases)

I Autotranslate side effects

I Use PVHVM callback vector setup

I VCPU bring-up








I VCPU bring-up








I VCPU bring-up








I VCPU bring-up








I VCPU bring-up



Linux: Things disabled

I No PV IDT

I No PV irq ops

I No PV CPUID

I Native Syscall/sysenter

I No VM assists

I No event / failsafe callbacks

I No need to set IOPL




I No PV IDT

I No PV irq ops

I No PV CPUID


I No VM assists






I No PV IDT

I No PV irq ops

I No PV CPUID


I No VM assists






I No PV IDT

I No PV irq ops

I No PV CPUID


I No VM assists






I No PV IDT

I No PV irq ops

I No PV CPUID


I No VM assists






I No PV IDT

I No PV irq ops

I No PV CPUID


I No VM assists






I No PV IDT

I No PV irq ops

I No PV CPUID


I No VM assists






MMUOPS:

I No need to pin pagetables

I No pfn/mfn conversion

I No need to special-case PT protections

I Only pvmmu op: flush tlb others




MMUOPS:








MMUOPS:








MMUOPS:







Linux: Autotranslate

I PV

I Pagetables controlled by XenI Real MFN in pagetablesI “Map this page here” hypercall

I PVH

I Pagetables controlled by guestI gpfn in pagetablesI Every mapped page must be in the p2m




I PVI Pagetables controlled by Xen

I Real MFN in pagetablesI “Map this page here” hypercall

I PVH





I PVI Pagetables controlled by XenI Real MFN in pagetables

I “Map this page here” hypercall

I PVH





I PVI Pagetables controlled by XenI Real MFN in pagetablesI “Map this page here” hypercall

I PVH






I PVH






I PVHI Pagetables controlled by guest

I gpfn in pagetablesI Every mapped page must be in the p2m





I PVHI Pagetables controlled by guestI gpfn in pagetables

I Every mapped page must be in the p2m





I PVHI Pagetables controlled by guestI gpfn in pagetablesI Every mapped page must be in the p2m




I “Special pages”: grant frame, make a hole in the p2m

I Need to add p2m entries to map foreign pages

I Grant table: similar to HVM (but not the same)















Linux: VCPU bring-up

I Brought up via hypercalls (PV)

I Guest in control of IDT, Xen can’t guarantee loading properly

I Only set GS by default

I All other state must be set in on-cpu boot code
























PVH: Not yet working

I 32-bit

I vtsc

I shadow mode

I vcpu hotplug




I 32-bit

I vtsc

I shadow mode

I vcpu hotplug




I 32-bit

I vtsc

I shadow mode

I vcpu hotplug




I 32-bit

I vtsc

I shadow mode

I vcpu hotplug



Issues: HVM paths

I Original idea: “Lightweight” container

I Reality: Large amount of code necessary for minimumfunctionality

I Very large amount of code duplication

I Current patch: use HVM paths, with special cases for PVH



Issues: HVM paths







Issues: HVM paths







Issues: HVM paths







Issues: PIO instructions

I Direct access permitted

I pv PIT

I cmos idx

I pci config spaceI “Abusing” IO with SMM

I Execute exact instruction with guest GPRs





I pv PIT

I cmos idx







I pv PIT

I cmos idx







I pv PIT

I cmos idx

I pci config space

I “Abusing” IO with SMM






I pv PIT

I cmos idx







I pv PIT

I cmos idx






I Problems

I Ugly changes to allow PV paths to emulate for PVH guestsI Race condition because of double checking

I Possibilities

I Need 2 sets of access controls: For user procs, and for guestI PV only has oneI PVH: Have two, so may not need to emulate anymore




I ProblemsI Ugly changes to allow PV paths to emulate for PVH guests

I Race condition because of double checking

I Possibilities





I ProblemsI Ugly changes to allow PV paths to emulate for PVH guestsI Race condition because of double checking

I Possibilities






I Possibilities






I PossibilitiesI Need 2 sets of access controls: For user procs, and for guest

I PV only has oneI PVH: Have two, so may not need to emulate anymore





I PossibilitiesI Need 2 sets of access controls: For user procs, and for guestI PV only has one

I PVH: Have two, so may not need to emulate anymore





I PossibilitiesI Need 2 sets of access controls: For user procs, and for guestI PV only has oneI PVH: Have two, so may not need to emulate anymore



HAP vs Shadow

I Page table update

I HAP: Memory writeI Shadow: Trip through Xen

I TLB effectiveness

I HAP: Superpages allow 2MiB / 1GiB TLB entriesI Shadow: Still only 4k TLB entries

I TLB replacement cost

I Shadow: Worst case, 4 memory readsI HAP: 9-16 memory reads

I Does this matter?

I Kernel build: Shadow 30% slower than HAPI SpecJBB: Shadow 30% faster than HAP



HAP vs Shadow

I Page table updateI HAP: Memory write

I Shadow: Trip through Xen

I TLB effectiveness




I Does this matter?




HAP vs Shadow

I Page table updateI HAP: Memory writeI Shadow: Trip through Xen

I TLB effectiveness




I Does this matter?




HAP vs Shadow


I TLB effectiveness




I Does this matter?




HAP vs Shadow


I TLB effectivenessI HAP: Superpages allow 2MiB / 1GiB TLB entries

I Shadow: Still only 4k TLB entries



I Does this matter?




HAP vs Shadow


I TLB effectivenessI HAP: Superpages allow 2MiB / 1GiB TLB entriesI Shadow: Still only 4k TLB entries



I Does this matter?




HAP vs Shadow





I Does this matter?




HAP vs Shadow



I TLB replacement costI Shadow: Worst case, 4 memory reads

I HAP: 9-16 memory reads

I Does this matter?




HAP vs Shadow



I TLB replacement costI Shadow: Worst case, 4 memory readsI HAP: 9-16 memory reads

I Does this matter?




HAP vs Shadow




I Does this matter?




HAP vs Shadow




I Does this matter?I Kernel build: Shadow 30% slower than HAP

I SpecJBB: Shadow 30% faster than HAP



HAP vs Shadow




I Does this matter?I Kernel build: Shadow 30% slower than HAPI SpecJBB: Shadow 30% faster than HAP



Overview

I PV, HVM, and PVH



I Issues in PVH



Overview

I PV, HVM, and PVH



I Issues in PVH



Overview

I PV, HVM, and PVH



I Issues in PVH



Overview

I PV, HVM, and PVH



I Issues in PVH



Goal






Goal






Goal






Questions

Questions?

Feedback? [email protected]

Check out our blog: http://blog.xen.org/



Xen: PVH hypercalls

I memory op

I console io

I vcpu op

I mmuext op



Xen: PVH hypercalls

I memory op

I console io

I vcpu op

I mmuext op



Xen: PVH hypercalls

I memory op

I console io

I vcpu op

I mmuext op



Xen: PVH hypercalls

I memory op

I console io

I vcpu op

I mmuext op


Technology

XPDS13 : PVH Technical Deep Dive - George Dunlap, Citrix