Xen on ARMA success story
Stefano Stabellini - Citrix Xen Project Team
Achievements of one year
11/11 08/12 11/12 03/13 07/13
First Xen on ARM talk atXen Summit 2012
Xen support for ARM upstream in Linux 3.7
Xen running on real ARM hardware
09/12
Xen 64-bit on ARM64
01/13
Citrix announces that will be joining Linaro
Xen support for ARM64 upstream in Linux 3.11
06/13
Xen 4.3 released with ARM and ARM64 support
Part-time Xen ARM hacking starts
You are here
Xen-devel ARM traffic from August 2012: ● 4685 emails: 360 emails per month!● 39% of which are not from Citrix
A growing community
Hardware supportUpstream:● Versatile Express Cortex A15● Arndale board● ARMv8 FVP
In progress:● Calxeda “Midway”● Applied Micro “Mustang”● Cubieboard2● Broadcom Brahma-B15● OMAP5
Upstream featuresXen v4.3:● basic lifecycle operations● memory ballooning● scheduler configurations and vcpu pinning
Linux v3.11:● dom0 and domU support● 32-bit and 64-bit support● SMP support● PV disk, network and console
Coming in Xen 4.4● 64-bit guest support● live-migration● SWIOTLB
Coming in Xen 4.4● 64-bit guest support● live-migration● SWIOTLB
The problem
virtual address
physical address
machine address
hardware
Linu
xX
en
2 st
age
1 S
tage
The problem: dom0 DMA
virtual address
physical address
machine address
Device DMA
Linu
xX
en
2 st
age
1 S
tage
The best solution: IOMMU
virtual address
physical address
machine address
Device DMA IOM
MU
MM
U
Linu
xX
en
2 st
age
The workaround:Dom0 1:1 mapping
virtual address
physical address = machine address
Device DMA
Linu
xX
en
The workaround:Dom0 1:1 mapping● rigid solution● no ballooning in dom0● no page sharing in dom0● does not work with foreign grant table
mappings
UNHAPPY
The alternative: SWIOTLB
virtual address
physical address
machine address
Device DMA
DM
A o
psM
MU
Linu
x
The alternative: SWIOTLB● use memory_exchange_and_pin hypercall
○ create a contiguous buffer in machine memory○ retrieve the machine address of the buffer
● introduce an additional memcpy
● remove the need for the 1:1 workaround
STILL UNHAPPY
SWIOTLB:the improved versionpin and unpin hypercalls:● dynamically retrieve P2M mappings● pin a mapping for DMA● remove additional memcpy
map_page
XENMEM_pin
pin
pfn
pfn
mfn
mfn
SWIOTLB:the “improved” version● Linux rbtree maintenance is expensive● too many uncached address translations in
Xen○ guest virtual to machine○ guest physical to machine
cpu utilization increase
NOT AN IMPROVEMENT
SWIOTLB: the compromise● keep the dom0 1:1 workaround
○ dom0 without ballooning and page sharing is the default configuration in XenServer x86 today
● use the swiotlb only to handle DMA involving foreign grants○ we already know the p2m mappings of grants
■ no need for pin and unpin hypercalls○ can take shortcuts: avoid many tree lookups○ tree lookups are much faster○ avoidable with IOMMU support
SWIOTLB: the compromiseTesting platform:● 1.5Ghz quad-core Cortex A15● 1 Gbit link
Benchmark results:● same network throughput as native (line
rate)● < 2% cpu usage increase
THAT’S BETTER
SWIOTLB: where to find itThe patches (swiotlb-xen v8):http://marc.info/?l=linux-kernel&m=138203180707683&w=2
The kernel tree:git://git.kernel.org/pub/scm/linux/kernel/git/sstabellini/xen.git swiotlb-xen-8
Xen 4.5+● IOMMU support in Xen● device assignment● UEFI booting● ACPI support
DEMO
Questions?
Recommended
