Upload
ngoduong
View
223
Download
0
Embed Size (px)
Citation preview
connect.linaro.org
SFO17-410: NEVE: Nested Virtualization Extensions for ARMJintack Lim, Christoffer Dall, Shih-Wei Li, Jason Nieh, and Marc Zyngier
ENGINEERS AND DEVICESWORKING TOGETHER
Nested Virtualization
Hardware
Hypervisor
VM
VM
Kernel
App App
Hypervisor
VM
Kernel
App App
VM
Kernel
App App
ENGINEERS AND DEVICESWORKING TOGETHER
Nested Virtualization
Hardware
Hypervisor
VM
VM
Kernel
App App
Hypervisor
VM
Kernel
App App
VM
Kernel
App App
• IaaS hosting private clouds
• Develop and test your hypervisor
ENGINEERS AND DEVICESWORKING TOGETHER
Terminology
Hardware
Host Hypervisor
VM
VM
Kernel
App App
Guest Hypervisor
Nested VM
Kernel
App App
Nested VM
Kernel
App App
ENGINEERS AND DEVICESWORKING TOGETHER
ARM Virtualization Extensions
Hypervisor
VM
OS Kernel
App AppApp
EL1
EL0
EL2
ENGINEERS AND DEVICESWORKING TOGETHER
ARM Nested Virtualization
Host Hypervisor
VM
EL1
EL0
EL2
Guest Hypervisor
Nested VM
OS Kernel
App AppApp
ENGINEERS AND DEVICESWORKING TOGETHER
ARM Nested Virtualization
Host Hypervisor
VM
EL1
EL0
EL2
Guest Hypervisor
Nested VM
OS Kernel
App AppApp
Trap and Emulate
ENGINEERS AND DEVICESWORKING TOGETHER
Host Hypervisor
VM
EL1
EL0
EL2
Guest Hypervisor
Nested VM
OS Kernel
App AppApp
??? and Emulate
ARMv8.0
ENGINEERS AND DEVICESWORKING TOGETHER
ARMv8.3
Host Hypervisor
VM
EL1
EL0
EL2
Guest Hypervisor
Nested VM
OS Kernel
App AppApp
Trap and Emulate
ENGINEERS AND DEVICESWORKING TOGETHER
Performance Evaluation
• Problem: No ARMv8.3 hardware available.
• Solution: Use ARMv8.0 hardware with the software modification
ENGINEERS AND DEVICESWORKING TOGETHER
Emulating v8.3 on v8.0
Host Hypervisor
VM
EL1
EL0
EL2
Guest Hypervisor
Nested VM
OS Kernel
App AppApp
ARMv8.0 Hardware
SW Modification
ENGINEERS AND DEVICESWORKING TOGETHER
Hypercall MicroBenchmark
Hypervisor
VM
OS Kernel
App AppApp
EL1
EL0
EL2Hypercall Return
Host Hypervisor
VM
EL1
EL0
EL2
Guest Hypervisor
Nested VM
OS Kernel
App AppApp
Hypercall Return
ENGINEERS AND DEVICESWORKING TOGETHER
Hypercall MicroBenchmark
ARMv8.3
VM Nested VM
Cycle counts 2,729 422,720
Ratio to VM 1 155x
ENGINEERS AND DEVICESWORKING TOGETHER
Nested VM Hypercall
Host Hypervisor
VM
EL1
EL0
EL2
Guest Hypervisor
Nested VM
OS Kernel
App AppApp
ReturnHypercall
…
ENGINEERS AND DEVICESWORKING TOGETHER
Hypercall MicroBenchmark
ARMv8.3
VM Nested VM
Cycle counts 2,729 422,720Ratio to VM 1 155xTrap counts 1 126
ENGINEERS AND DEVICESWORKING TOGETHER
All Traps are Necessary?
Host Hypervisor
VM
EL1
EL0
EL2
Guest Hypervisor
Nested VM
OS Kernel
App AppApp
ReturnHypercall
…
ENGINEERS AND DEVICESWORKING TOGETHER
NEVE: NEsted Virtualization Extensions for ARM
• Improves performance of nested virtualization
• Key Mechanisms
1. Redirects register accesses to memory
2. Redirects register accesses to other registers
ENGINEERS AND DEVICESWORKING TOGETHER
VNCR_EL2
• Virtual Nested Control Register
• Enable bit
• BADDR (Base Address)
ENGINEERS AND DEVICESWORKING TOGETHER
#1 Redirection to MemoryRegister
msr x0, TTBR0_EL1
VNCR_EL2.Enable == 0 Trap!
Memory
ENGINEERS AND DEVICESWORKING TOGETHER
#1 Redirection to MemoryRegister
msr x0, TTBR0_EL1
VNCR_EL2.Enable == 1
TTBR0_EL1
MemoryVNCR_EL2.BADDR
ENGINEERS AND DEVICESWORKING TOGETHER
Accessing EL2 State
Host Hypervisor
VM
EL1
EL0
EL2
Guest Hypervisor
Nested VM
OS Kernel
App AppApp
ReturnHypercall
…
ENGINEERS AND DEVICESWORKING TOGETHER
#2 Redirection to RegisterTTBR0_EL2
msr x0, TTBR0_EL2
TTBR0_EL1
VNCR_EL2.Enable == 0 Trap!
ENGINEERS AND DEVICESWORKING TOGETHER
#2 Redirection to Register
msr x0, TTBR0_EL2
TTBR0_EL1
VNCR_EL2.Enable == 1
TTBR0_EL2
ENGINEERS AND DEVICESWORKING TOGETHER
Hypercall MicroBenchmark
ARMv8.3 NEVE
VM Nested VM Nested VM
Cycle counts 2,729 422,720 92,385
Ratio to VM 155x 34x
Trap counts 1 126 15
ENGINEERS AND DEVICESWORKING TOGETHER
Application Workloads
Application Description Application Description
Kernbench Kernel compile Netperf TCP_RR Network performance
Hackbench Scheduler stress Netperf TCP STREAM Network performance
SPECjvm2008 Java Runtime Netperf TCP MAERTS Network performance
MySQL Database management Apache Web server stress
Memcached Key-Value store Nginx Web server stress
ENGINEERS AND DEVICESWORKING TOGETHER
Experimental Setup• Hardware
• APM X-Gene (ARMv8.0)
• 8-way SMP
• 64 GB RAM
• 10 Gb Ethernet
• Software
• KVM on KVM
• v4.10
• Native/VM/Nested VM Setup
• 4-way SMP
• 12 GB RAM
• Virt I/O(VM/nested VM)
ENGINEERS AND DEVICESWORKING TOGETHER
Application Benchmarks
01020304050 ARMv8.3 VM
ARMv8.3 NestedNEVE Nested
012345
Kernbench
Hackbench
SPECjvm2008TCP RR
TCP STREAM
TCP MAERTSApache
Nginx
MemcachedMySQL
Normalized overhead (lower is better)
ENGINEERS AND DEVICESWORKING TOGETHER
Application Benchmarks
01020304050 ARMv8.3 VM
ARMv8.3 NestedNEVE Nestedx86 Nested VM
012345
Kernbench
Hackbench
SPECjvm2008TCP RR
TCP STREAM
TCP MAERTSApache
Nginx
MemcachedMySQL
Normalized overhead (lower is better)