Upload
others
View
2
Download
0
Embed Size (px)
Citation preview
Copyright © 2006 Intel Corporation
Symerton – Using Virtualization to Accelerate Packet Processing
Aaron R. KunzeStephen D. Goglin
Erik J. Johnson
Communications Technology LabCorporate Technology Group
December 4, 2006
December 4, 2006 Symerton - Kunze, Goglin, Johnson - ANCS 20062
*Other names and brands may be claimed as the property of others.
Copyright © 2006 Intel Corporation
S D
Cisco AS5800 SERI ES
Po wer
CISCO SYSTEMS
Complexity at the Network Edge
SD
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17
8260
SD
Enterprise LAN
SD
Cisco 1720
BRIS/T
CONSOLE
AUXWIC 0 OK
OK
B2
B1
WIC 1 OK
DSUCPU
LNK100FDX
S3
LOOP
LP
Access Network
MAN/WAN• VPN Gateway• Firewall• Intrusion Detection• XML & SSL acceleration
• L4-L7 switching• Application acceleration
• Compression• Monitoring (billing, QoS)
December 4, 2006 Symerton - Kunze, Goglin, Johnson - ANCS 20063
*Other names and brands may be claimed as the property of others.
Copyright © 2006 Intel Corporation
Problem
Network edge packet processing on a general-purpose OS does not perform well
• Buffer copies
– Required to share network devices between applications...
– ...but network edge applications don’t need to share network devices
• Interrupt-based device management
– Allows processor to stay busy when no packets arriving...
– ...but when no packets are arriving, network edge apps have no work
• Virtual memory
– Allows appearance of more memory than physically available...
– ...but page faults are an eternity for network edge apps
– Allows protection between applications in different trust domains...
– ...but network edge devices are embedded devices with one trust domain (today)
December 4, 2006 Symerton - Kunze, Goglin, Johnson - ANCS 20064
*Other names and brands may be claimed as the property of others.
Copyright © 2006 Intel Corporation
Existing Solutions
Pros
• High performance
• Rich ecosystem of non-performance-critical code
Cons
• High maintenance costs
• License, IP, and upgrade issues
user
kernel
Application (perf-critical)
Heavily-modified off-the-shelf operating
system
Application (non-performance-critical)
= customer developed
= off-the-shelf
Real-time OS
Application
Pros
• High performance
• No need to modify/maintain OS
Cons
• Much smaller ecosystem for skills/code
“The Kernel Hack”
“The RTOS”
December 4, 2006 Symerton - Kunze, Goglin, Johnson - ANCS 20065
*Other names and brands may be claimed as the property of others.
Copyright © 2006 Intel Corporation
Intel® Virtualization Technology Primer
Hardware acceleration for virtual machines (VMs)
Unmodified operating system
App
kernel
user
Virtual machine monitor (VMM)
App App
Unmodified operating system
App
kernel
userApp App
root
non-root
VM enter VM exit VM enter VM exit
kernel
user
December 4, 2006 Symerton - Kunze, Goglin, Johnson - ANCS 20066
*Other names and brands may be claimed as the property of others.
Copyright © 2006 Intel Corporation
Symerton Approach
Special-purpose networking operating system
Application (performance-critical)
Special-purpose networking operating system
No buffer copiesPolled network
interfacesNo paging
= off-the-shelf
= customer developed
December 4, 2006 Symerton - Kunze, Goglin, Johnson - ANCS 20067
*Other names and brands may be claimed as the property of others.
Copyright © 2006 Intel Corporation
Application (performance-critical)
Special-purpose networking operating system
= off-the-shelf
Performance Partition
Symerton Approach
= customer developed
Virtual Machine Monitor
non-root
root
Special-purpose networking operating system
Special-purpose networking operating system
Application (performance-critical)
Special-purpose networking operating system
December 4, 2006 Symerton - Kunze, Goglin, Johnson - ANCS 20068
*Other names and brands may be claimed as the property of others.
Copyright © 2006 Intel Corporation
= off-the-shelf
Performance Partition
Symerton Approach
= customer developed
Virtual Machine Monitor
non-root
root
Special-purpose networking operating system
Application (performance-critical)
Special-purpose networking operating system
General-purpose Partition
Application (non-performance-critical)
Off-the-shelf operating system
December 4, 2006 Symerton - Kunze, Goglin, Johnson - ANCS 20069
*Other names and brands may be claimed as the property of others.
Copyright © 2006 Intel Corporation
= off-the-shelf
Performance Partition
Symerton Approach
= customer developed
Virtual Machine Monitor
non-root
root
Special-purpose networking operating system
Application (performance-critical)
Special-purpose networking operating system
General-purpose Partition
Application (non-performance-critical)
Off-the-shelf operating system
Performance partition maintains dedicated access to network
devices
December 4, 2006 Symerton - Kunze, Goglin, Johnson - ANCS 200610
*Other names and brands may be claimed as the property of others.
Copyright © 2006 Intel Corporation
= off-the-shelf
Performance Partition
Symerton Approach
= customer developed
Virtual Machine Monitor
non-root
root
Special-purpose networking operating system
Application (performance-critical)
Special-purpose networking operating system
General-purpose Partition
Application (non-performance-critical)
Off-the-shelf operating system
Network driver
Packets passed between performance partition and general-
purpose partition using network driver
December 4, 2006 Symerton - Kunze, Goglin, Johnson - ANCS 200611
*Other names and brands may be claimed as the property of others.
Copyright © 2006 Intel Corporation
What About Virtualization Overhead?
Page faults are more expensive!
• ...we can reduce/eliminate paging
Interrupts are more expensive!
• ...we can use polling
Passing data between VMs is expensive!
• ...only a small fraction of traffic should pass between VMs
December 4, 2006 Symerton - Kunze, Goglin, Johnson - ANCS 200612
*Other names and brands may be claimed as the property of others.
Copyright © 2006 Intel Corporation
Evaluating the Approach
Questions to answer:
• How much performance gain in removing general-purpose OS overheads?
– Are applications faster with no copies, interrupts, paging?
– Is it worth the extra effort?
• How much of that lost due to virtualization?
– Does having an extra software layer erase the gains achieved by resolving general-purpose OS issues?
December 4, 2006 Symerton - Kunze, Goglin, Johnson - ANCS 200613
*Other names and brands may be claimed as the property of others.
Copyright © 2006 Intel Corporation
Application (performance-
critical)
Xen Domain 0Performance Partition
Symerton Proof-of-Concept (POC)
Modified Xen*
XenoLinux*
non-root
root
Modified FreeBSD*
user
kernel
Application (non-performance-critical)user
kernel
December 4, 2006 Symerton - Kunze, Goglin, Johnson - ANCS 200614
*Other names and brands may be claimed as the property of others.
Copyright © 2006 Intel Corporation
Application (performance-
critical)
Xen Domain 0Performance Partition
Symerton Proof-of-Concept (POC)
Modified Xen*
XenoLinux*
non-root
root
Modified FreeBSD*
user
kernel
Xen modified to allow direct access to NIC for
guests with modified drivers
Application (non-performance-critical)user
kernel
FreeBSD modified to poll the NIC and provide zero-
copy access to user space applications
Using Xen means can’t readily turn paging off or
use large pages
December 4, 2006 Symerton - Kunze, Goglin, Johnson - ANCS 200615
*Other names and brands may be claimed as the property of others.
Copyright © 2006 Intel Corporation
POC Evaluation Setup
System Setup
• 3.0GHz Intel® Pentium® D 930 Processor
• Intel® PRO/1000 PF PCI Express (Only one port used)
• Linux* configs are RedHat* Enterprise Linux* 4 Advanced Server Update 2
• FreeBSD configs are FreeBSD 6.0
• Xen 3.0
Two test applications
• Null forwarder (forwards packets)
• Snort (intrusion detection system)
Packet traces
• Traces from real networks, including NLANR traces and Intel IT traces
December 4, 2006 Symerton - Kunze, Goglin, Johnson - ANCS 200616
*Other names and brands may be claimed as the property of others.
Copyright © 2006 Intel Corporation
Null Forwarder on Native and Symerton
0102030405060708090
100
0 200 400 600 800 1000 1200 1400 1600
Packet Size
Per
cent
of M
axim
um L
ine
Rat
e
RedHat FreeBSD
Modified FreeBSD Symerton
Evaluation Results – Packet Forwarding
Large performance increase for small
packets
No noticeable overhead from
Xen/VT
Source: Intel
December 4, 2006 Symerton - Kunze, Goglin, Johnson - ANCS 200617
*Other names and brands may be claimed as the property of others.
Copyright © 2006 Intel Corporation
Evaluation Results – Snort
Snort Throughput w/ Traces
0
20000
40000
60000
80000
100000
120000
1999
0513
1900
1999
0514
1000
1999
0515
0000
AN
L-11
0739
0954
AN
L-11
0740
2013
AN
L-11
0741
3100
MR
A-
1104
7108
88
MR
A-
1104
7219
46
MR
A-
1104
7348
30
plat
o02-
035a
qp
ww
w12
-00
1ggi
ww
w12
-01
5ggi
Trace
Pac
kets
Per
Sec
ond
FreeBSD RedHat SymertonSource: IntelSnort performance improves 22% on average!
December 4, 2006 Symerton - Kunze, Goglin, Johnson - ANCS 200618
*Other names and brands may be claimed as the property of others.
Copyright © 2006 Intel Corporation
Evaluation Results – Snort w/ Non-perfCritical Code
Snort Throughput w/ Traces
0
20000
40000
60000
80000
100000
120000
1999
0513
1900
1999
0514
1000
1999
0515
0000
AN
L-11
0739
0954
AN
L-11
0740
2013
AN
L-11
0741
3100
MR
A-
1104
7108
88
MR
A-
1104
7219
46
MR
A-
1104
7348
30
plat
o02-
035a
qp
ww
w12
-00
1ggi
ww
w12
-01
5ggi
Trace
Pac
kets
Per
Sec
ond
Sym erton Sym erton + Slowpath Sym erton + Slowpath Shared CoreSource: IntelSnort performance drops when VMs share a core
December 4, 2006 Symerton - Kunze, Goglin, Johnson - ANCS 200619
*Other names and brands may be claimed as the property of others.
Copyright © 2006 Intel Corporation
Evaluation Results – Effects of Paging
Can’t turn paging off or use large pages on Xen
When running one particularly bad trace through Snort, observed 374 TLB misses per packet
Using large pages or no paging has potential for more gains
Source: Intel
December 4, 2006 Symerton - Kunze, Goglin, Johnson - ANCS 200620
*Other names and brands may be claimed as the property of others.
Copyright © 2006 Intel Corporation
Conclusions
Currently, designers using mainstream hardware platforms for packet processing face a dilemma:
• High performance of a specialized OS, or
• Rich software ecosystem of a general-purpose OS
Symerton approach offers best of both worlds with low overheads in most cases
December 4, 2006 Symerton - Kunze, Goglin, Johnson - ANCS 200621
*Other names and brands may be claimed as the property of others.
Copyright © 2006 Intel Corporation
Potential Research Topics
Could one design a VMM specifically for this usage model?
• Better support for real-time?
• Compromise some inter-VM security concerns for performance?
What does a special-purpose networking OS look like?
• Better scheduling algorithms?
• Better memory allocation?
• How best to use domain knowledge?