Xen summit spring2010_tom_woller_amd

Virtualization Futures

Tom Woller – Principal MTSXen Summit Spring 2010AMD Corporation

| Xen Summit | April, 2010 2

Outline

Virtualization Trends

� Virtualization technology today

� Virtualization technology tomorrow

I/O virtualization with AMD IOMMU

� AMD IOMMUv1 review

� AMD IOMMUv2 definition and uses

Summary


Today


Trends that are driving virtualization

Reduce total cost of ownership (TCO)

� Increased systems utilization (current servers have less than 10% average utilization, less than 50% peak utilization)

� Reduce hardware (25% of the TCO)

� Space, electricity, cooling (50% of the operating cost of a data center)

Management simplification

� Dynamic provisioning

� Workload management/isolation

� Virtual machine migration

� Reconfiguration

Virtualization is a true scalable multi-core work load


CPU Virtualization Trends

� The key trend is to eliminate the overhead of virtualization

– Reduce overall world-switch times

– Reduce world-switch frequencies

� Reduce world-switch times

– Tag TLB by ASID

– Better caching of VMCB state

� Reduce world-switch frequencies

– Nested paging (up to 70%)

– Direct device assignment

– Implement more functions in the guest OS through paravirtualization

VM World-switch Times

0

200

400

600

800

1000

1200

1400

1600

1800

Processor

Cycles

Fam0F Fam10 Future


The Future


Virtualization Futures

Virtualization is becoming increasingly pervasive

� Server consolidation is still the primary reason

� Client virtualization (Xen leading efforts)

� Platform virtualization capabilities are rounded out and exceed the capabilities of mainframes


Virtualization Futures - Changing Landscape

Novel uses of virtualization

� security, high-availability, manageability

Live Migration

� key component towards enabling the dynamic datacenter

I/O Virtualization is driving platform, adapter and software stack changes


What Else Can You Expect?

A few more virtualization acceleration widgets

� Virtualized interrupt controller (interrupts, IPIs)

Additional hardware RAS capabilities

� Putting all your eggs in one basket

� High-availability / Disaster recovery

� VM failover

– SMP

Nested/recursive virtualization

� Embedded hypervisors are driving this

Virtualization aware devices supporting I/O Virtualization

� NICs, storage


AMD IOMMU (version 1)

H/W help for I/O Virtualization is already here…

IOMMU is to Devices

as

MMU is to CPUs


CPUCPU

CPUCPU

CPUCPU

CPUCPU

DRAMDRAMDRAMDRAM

DRAMDRAMDRAMDRAM

Virtualizing The Platform IOMMU Version 1

ATC = Address Translation Cache(ATC a.k.a. IOTLB)

HT = HyperTransport™ linkPCIe = PCI Express™ linkATS = Address Translation Service

HT

DRAMDRAM

IOMMU

IOMMU PCI Express™

PCI Express™

devices,

devices,

switches

switches

CPUCPU

DRAMDRAM

HT

PCI, LPC, etcPCI, LPC, etc

HTHT

PCIebridge

CPUCPU

DeviceATC

optional optional remote ATCremote ATC

TunnelTunnel

PCIePCIebridgebridge

ATS

ATS

PCIebridge

IO HubIO Hub

IOMMU

IOMMU

DeviceDeviceDeviceA

TC


AMD IOMMU V1 - Uses

I/O Virtualization

� Direct device assignment for efficient I/O

� I/O interrupt steering avoids hv interaction

� Legacy devices – avoid “bounce buffers”

� PCI-SIG

– PCIe IOV – using SR-IOV

– PCIe ATS 1.0 - Address Translation Services

RAS

� Device DMA containment

� Denial-of-service protection -- interrupt flood or MSI spoofing


AMD IOMMUv1 Limitations

• No ATS 1.1 PRI (Page Request Index) support

• No Hypervisor memory overcommit

• No Demand Paging, no Page Faults for devices

• Only 1 level of page table walking supported

• Page table supported are not AMD64 compatible

• Very difficult to share Page Tables with OS

• Must be managed by software (Hypervisor)

• Direct device assignment to guest applications in virtualized systems requires 2nd level (gv to gp)

• No nested virtualization guest device assignments


Introducing AMD IOMMU Version 2

• IOMMU version 1 compatibility • ATS 1.1 PRI support (Page Request Index)

• Supports “Page Faults” for devices• Allows Hypervisor memory overcommit for guests (Demand paging)

• RDMA usage without pinning memory• Nested Page Tables

• 2nd levels of page table walking supported• L1: Guest virtual to Guest Physical (AMD64 compatible)

• L2: Guest Physical to System Physical (v1 compatibility)

• 100% AMD64 compatible level• Allows direct device assignment in virtualized systems to use guest virtual address

• Share OS PTs in assigning User Level I/O to devices in native environments


IOMMUv1 (ATS 1.0) Caching Address Translations

Peripheral (ATC)

Processor

ATS request

ATS response

Use ATSresponse

IOMMU

TLB lookup& 1 lvl PT walk


IOMMUv2 (ATS 1.1 PRI) Page Fault & Overcommit

Peripheral (ATC)

Processor

ATS request

ATS response

PRI request

PRI response

ATS request

ATS response

PPRqueue

Cmdqueue

SW

Evaluate ATSresponse

Evaluate PRIresponse

• Swap in page

• Alloc new page

• Reject request

• Upgrade privs

• Copy-on-write

• Etc.

IOMMU

TLB lookup& 2 lvl PT walk


IOMMUv2 changes

IOMMUv2

IOMMUv1 base

AMD64 long

page tables (guest)

Guest CR3 table

Peripheral page service requests (PRI)

IOMMU page tables

(host)

Interrupt remapping tables (host)

Perf counters & RAS info

Command & Event buffers


Example: Smart NIC RDMA Use Case

Current

� Overhead of managing pinned

buffers

� Lack of demand-paging

support

What do we want?

� Eliminate need for Pinned

memory

� Smart NIC operates on

unpinned region directly

using ATS PRI and Page

Faults

User LevelPinned Memory

RDMA

NICIniniband/

VIA

User Level

Unpinned Memory

NICInfiniband

/VIA

RDMA +

IOMMUv2

ATS 1.1 PRI+

Page Faults


IOMMUv2 Direct Guest MappingUser-level I/O

User-level I/O

User process

X86 nested paging

System memory

RDMA Buffers

IOMMUv2nested paging

� x86 PTE, IOMMU nested paging PRI+ATS

� Advanced memory model• Demand paging• Swapping• Copy-on-write

� Shared Virtual addresses among smart devices

�Direct access to devices at user-level reduces I/O overhead

Shared virtual address space


• Virtualization is a critical part of the future “dynamic“ data center and making considerable inroads into the client space

• I/O Virtualization is an integral part of the next generation server and client

• Next generation AMD IOMMU provides another level of I/O Virtualization functionality

• Demand Paging for smart devices (NICs, GPGPU, …)

• Two levels of Page Table walking

• Guest User Level I/O direct access to devices

Summary


Enjoy the Summit! Any Questions?

Source: infosecurity.com


Trademark Attribution

AMD, the AMD Arrow logo and combinations thereof are trademarks of Advanced Micro Devices, Inc. in the United States and/or other jurisdictions. Other names used in this presentation are for identification purposes only and may be trademarks of their respective owners.

©2010 Advanced Micro Devices, Inc. All rights reserved.

Technology

Xen summit spring2010_tom_woller_amd