A Comparative Study on Secure Network Virtualization Serdar Cabuk, Chris I. Dalton, Aled Edwards, Anna Fischer HP Laboratories HPL-2008-57 May 21, 2008* virtualization, security, networking

Secure and efficient network virtualization is a key building block for virtualized environments such as data centers or enterprise networks. While machine virtualization alone provides immediate isolation of computing resources such as memory and CPU between guest domains, the network remains to be a shared resource as all traffic from guests eventually pass through a shared network resource and end up on the shared physical medium. As a result, we need mechanisms to (1) control the information flow between virtual machines (e.g., who can communicate with whom), (2) configure virtual and physical network resources, and (3) separate network resources used by each networking domain. Within HP Labs we have successfully developed and deployed technologies that enable secure networking within virtualized infrastructures. In this report, we present the findings of a comparative study that we conducted to evaluate the security, performance, and manageability of these approaches. We further report our experiences with prototype implementations on Xen platforms.

External Accession Date Only Approved for External Publication

© Copyright 2008 Hewlett-Packard Development Company, L.P.

A Comparative Study on Secure Network Virtualization

Serdar Cabuk, Chris I. Dalton, Aled Edwards, Anna FischerHewlett-Packard Laboratories

Bristol, United Kingdom

{serdar.cabuk, cid, aled.edwards, anna.fischer}@hp.com

ABSTRACT

Secure and efficient network virtualization is a key build-ing block for virtualized environments such as data centersor enterprise networks. While machine virtualization aloneprovides immediate isolation of computing resources such asmemory and CPU between guest domains, the network re-mains to be a shared resource as all traffic from guests even-tually pass through a shared network resource and end upon the shared physical medium. As a result, we need mech-anisms to (1) control the information flow between virtualmachines (e.g., who can communicate with whom), (2) con-figure virtual and physical network resources, and (3) sep-arate network resources used by each networking domain.Within HP Labs we have successfully developed and de-ployed technologies that enable secure networking withinvirtualized infrastructures. In this report, we present thefindings of a comparative study that we conducted to eval-uate the security, performance, and manageability of theseapproaches. We further report our experiences with proto-type implementations on Xen platforms.

1. OUTLINEThis report is organized as follows: In Section 2, we list

the functional and security requirements for secure networkvirtualization in data centers. In Section 3, we present threenetworking designs that are currently under developmentin HP Labs. We present the findings of our comparativestudy in Section 4. In Section 5, we report our experienceswith two prototype implementations on Xen platforms. Wepresent related work in Section 6 and conclude in Section 7.

2. AIMS AND REQUIREMENTSSecure network virtualization allows parties to define vir-

tual topologies on top of physical infrastructure in a securemanner. In this section, we list the networking, security, andperformance requirements for enabling and securing networkvirtualization on virtualized data centers.

2.1 Networking AimsMachine virtualization allows a single physical machine

to run several (perhaps different) operating systems concur-rently creating the illusion of multiple machines each run-ning their own operating system. In a similar fashion, net-work virtualization allows groups of related virtual machinesrunning on separate physical machines to be connected to-gether as though they were on their own separate networkfabric. Virtual network extensions further enable the cre-ation of arbitrary virtual network topologies in virtualizedinfrastructures independent of the underlying physical net-work topology. As an example, Figure 1 illustrates how a

simple computer infrastructure consisting of two machinesconnected together via a single physical LAN segment can bemapped to virtual machines and a virtual LAN segment. Inthis setting, physical machine A hosts two virtual machines.One of them (A2) is connected into virtual LAN segment 1.Physical machine B also hosts two virtual machines. Oneof them (B1) is also connected into virtual LAN segment1. Virtual machine A2 and Virtual machine B1 appear asthough they are connected together via a single LAN seg-ment even though in reality the physical network connectingthem consists of multiple LAN segments interconnected viarouters.

Virtual machines C2 and D1 are also connected togethervia virtual LAN segment 2. Traffic is isolated between vir-tual LAN segments so machines C2 and D1 cannot see thetraffic passing between A2 and B1 even though they aresharing the underlying physical network infrastructure. Fig-ure 2 shows a more sophisticated topology, this time con-sisting of two LAN segments interconnected by a routingdevice. It also shows a possible instantiation of this usingvirtual network extensions.

In some circumstances, it may be that the virtual topol-ogy is much simpler than the physical topology. This isthe case in the first example where two machines appearas though they are connected directly together on the sameLAN segment even though in reality they may be separatedby many networking components, including potentially be-ing at opposite ends of a Wide Area Network (WAN) link.On the other hand, as in the second example, it may bedesirable that a particular virtual topology is more compli-cated than the underlying physical topology. For example,network segmentation (such as a DMZ arrangement) is oftenused for reasons of security to protect services running onmachines connected to that network. It should be possibleto offer those same kind of security properties and protec-tions to the services hosted on virtual machines by the useof segmented virtual networks, irrespective of the underlyingphysical network arrangement.

Of course, isolated virtual networks just linking virtualmachines with each other are of limited use. We also wantto allow interaction with traditional non-virtualized entitiesanywhere on the Internet, for example, to offer third-partyhosts occasionally to be connected to the datacenter envi-ronment. Therefore, virtual network extensions must alsoallow for bridging between the virtual and the physical net-work.

2.2 Security RequirementsWe describe the security objectives for network virtualiza-

tion through security policies that define confidentiality, in-tegrity, information flow control, and isolation requirements.

1

Figure 1: Virtual to Physical network mapping.

Figure 2: Physical mapping of router connected vir-tual LAN segments.

In our work, we treat each virtual LAN segment as a secu-rity domain with necessary mechanisms in place that enforcethe aforementioned security policies within domains (intra-domain policies) as well as between domains (inter-domainpolicies). In particular, the confidentiality requirement isgeared towards the separation of traffic that originates fromdifferent virtual LAN segments. This separation is essentialbecause in most cases the physical infrastructure is sharedamong various virtual LAN segments. In cases where thephysical infrastructure is trusted (e.g., in a controlled data-center) no additonal scheme is required. In other cases (e.g.,on a WAN link) appropriate measures need to be taken toensure the confidentiality inline with each domain policy.

In addition to the confidentiality requirement, a securitydomain must be able to enforce intra-domain policies thatdetermine the conditions domain entities need to adhere toprior to being admitted as members. In virtual networkingspeak, all entities that request membership to the particu-lar virtual LAN segment should behave as expected – i.e.,as defined by the segment policy. A straightforward way toensure this is to verify the hardware and software configu-ration of the prospective members either statically prior tojoining or dynamically in an ongoing fashion.

The traffic flow between security domains may also be sub-ject to strict controls as defined by inter-domain policies.These bidirectional flow control policies determine whichhosts belonging to different security domains are allowedto communicate and under what conditions. These policiesare usually enforced at the security domain perimeter bytrusted entities with hardened configurations. One exam-ple is a trusted dual-homed host that connects two securitydomains while enforcing traffic flow control policies (e.g.,firewall rules that determine the traffic types that are al-

DOMAIN A DOMAIN B

Figure 3: An example dual-homed host with twoNICs for inter-domain flow control between two se-curity domains.

lowed to be forwarded). As we illustrate in Figure 3, thishost uses two physical NICs (one for each domain) and en-forces the inter-domain flow policy for the information thatflows between the NICs.

Another inter-domain requirement is the isolation of vir-tual resources used by each security domain. In particular,isolation refers to the requirement that resources used bytwo security domains are logically separated and there isno unintended direct information flow using such logical re-sources1. Note that resource-based flow control is differentfrom the aforementioned traffic-based flow control betweensecurity domains. The former flow control type prevents themisuse of shared resources to leak information (e.g., listen-ing to virtual NICs of other virtual machines on a physicalhost) whereas the latter controls the network traffic betweendomains.

2.3 Performance RequirementsVirtual networking is central to virtualized data center

operations. Hence, the added network virtualization logicshould match the required operational levels of the datacen-ter. Poorly performing virtualized networking componentscan potentially affect all datacenter operations ranging fromVM deployment to inter-VM communications. Additionally,the logic should be scalable to support the sheer number ofVMs in a virtualized data center. Certain availability lev-els should be guaranteed for virtual networking components(e.g., virtual switches) lack of which can potentially jeop-ardize data center operations and lead to Denial of Service(DOS) attacks. Lastly, it is important to isolate and localizeperformance requirements to each security domain. Withsuch guarantees in place, each domain can operate inde-pendently in concordance with the Service Level Agreement(SLAs) as required by the domain owner (i.e., a data centercustomer). These SLAs may define availability requirementsfor the underlying virtualized network infrastructure, butmay also include network bandwidth guarantees or limita-tions, so that data center customers that require high band-width communication can be charged more than customersthat are running low bandwidth applications. Virtual net-work extensions need to provide mechanisms to control vir-tual network performance in a secure and reliable fashion.

Eventually, it is desired that entities of different securitydomains should be allowed to talk to each other. In this

1Indirect communication flows such as covert channels areoutside the scope of this paper.

2

case the solution ideally has to take into account the per-formance requirements of all involved domains. Also, it isimportant that inter-domain communication does not createany bottlenecks in the infrastructure – which is a challengewhen using an entity that becomes a single point of controlfor cross-domain communication (e.g., the dual-homed hostillustrated in Figure 3). Overall, it is important that ananalysis of performance requirements considers the actualusage model of the system. However, one can argue that inmany scenarios most network traffic will be within a singledomain rather than across domain boundaries, and in thesecases a slight decrease in inter-domain performance mightbe acceptable to ensure better security and manageabilityof the solution.

Added network virtualization logic imposes a certain levelof overhead on datacenter operations. In this paper, weempirically assess the overhead of each network virtualiza-tion technology in comparison to each other and to the non-virtualized equivalent. We additionally investigate the avail-ability, scalability, and isolation (in relation to performance)properties that can be guaranteed by each technology.

3. NETWORK VIRTUALIZATIONNetwork virtualization abstracts away the underlying net-

work infrastructure and topology. In this section, we presentvarious schemes we have designed that enable this abstrac-tion in a security preserving way. In Sections 3.1 and 3.2, weintroduce the virtual switch approach that employs EtherIPand VLAN tagging, respectively. In Section 3.3, we presentMAC rewriting that follows a different virtualization ap-proach from the former two. We highlight the differencesbetween the three approaches in Section 4.

3.1 Virtual Switch and EtherIPOne option for virtual networking is to virtualize at the IP

level. However, to avoid problems with supporting non-IPprotocols and IP support services such as ARP that sit di-rectly on top of the Ethernet protocol, we chose to virtualizeat the Ethernet level. Figure 4 shows the main abstractionof our virtual network extensions in which we divide a vir-tual network into virtual LAN segments. For our purposes,a LAN segment is considered to be an Ethernet broadcastdomain.

3.1.1 Networking Design

Each virtual LAN segment is represented by a virtualswitch. A virtual machine appears on a particular virtualLAN if one of its (virtual) network interface devices is “plugged”into one of the switch ports on the virtual switch formingthat segment. The virtual switch behaves like a normalphysical switch. Ethernet broadcast traffic generated by avirtual machine connected to the switch is passed to all vir-tual machines connected to that switch. The virtual switchbuilds up a forwarding table based on observed traffic sothat non-broadcast Ethernet traffic can be delivered in apoint-to-point fashion, as in a real switch.

The virtual switch is designed to operate in a distributedfashion. The privileged domain (or host OS, depending onthe virtualization technology in use) on each physical ma-chine managing a VM connected to a particular virtual LANsegment, runs part of the virtual switch forming that virtualLAN segment. A component of the privileged domain cap-tures the Ethernet frames coming out of a VM virtual net-

Figure 4: Abstract view of the Virtual Network Ex-tensions.

work device. This component is configured to know whichparticular virtual switch the VM is supposed to be connectedto.

EtherIP encapsulation. The VM Ethernet frames areencapsulated in IP packets using EtherIP. EtherIP is a stan-dard protocol for tunneling Ethernet and 802.3 packets viaIP datagrams and can be employed to expand a LAN overa Wide or Metropolitan Area Network [12]. In this setting,each tunnel endpoint uses a special network device providedby the operating system that encapsulates outgoing Eth-ernet/802.3 packets in new IP packets. We insert virtualLAN membership information (i.e., the virtual LAN identi-fier, which is unique for each virtual LAN segment withina virtual network) into the EtherIP header of each encap-sulated packet. The encapsulated packets are then trans-mitted to the other side of the tunnel where the embeddedEthernet/802.3 packets are extracted and transmitted to thedestination host that belongs to the same virtual LAN seg-ment.

Address mapping. The virtual switch component on aVMM maps the Ethernet address of the encapsulated Eth-ernet frame to an appropriate IP address. This way, theencapsulated Ethernet frame can be transmitted over theunderlying physical network to physical machines hostingother VMs connected to the same LAN segment that wouldhave seen that Ethernet traffic had the VMs actually beenon a real LAN together. The IP address chosen to routethe encapsulated Ethernet frames over the underlying phys-ical network depends upon whether the encapsulated Eth-ernet frame is an Ethernet broadcast frame or not and alsowhether the virtual switch has built up a table of the lo-cations of the physical machines hosting other VMs on aparticular LAN segment based on observing traffic on thatLAN.

Broadcast / multicast mapping. IP packets encapsu-lating broadcast and multicast Ethernet frames are given amulticast IP address and sent out over the physical network.Each virtual LAN segment has an IP multicast address as-sociated with it. All the physical machines hosting VMs ona particular virtual LAN segment are members of the mul-ticast group for that virtual LAN segment. This mechanismensures that all VMs on a particular virtual LAN segmentreceive all broadcast Ethernet frames from other VMs onthat segment.

3

Non-broadcast mapping. Encapsulated Ethernet framesthat contain a directed Ethernet address destination are ei-ther flooded to all the VMs on a particular LAN segment(using the IP multicast address as in the broadcast case) orsent to a specific physical machine IP address. This dependsupon whether the virtual switch component on the encapsu-lating VM has learned the location of the physical machinehosting the VM with the given Ethernet destination addressbased on traffic observation through the virtual switch.

3.1.2 Requirements Revisited

Layering over arbitrary physical networks. The de-cision to encapsulate Ethernet frames from VMs within IPpackets allows us to connect different VMs to the same vir-tual LAN segment as long as the physical machines hostingthose VMs have some form of IP based connectivity betweenthem. This also enables the connection of virtual networksegments over a WAN link, even though that requires someform of tunnelling and appropriate multicast support - whichis not always a given. In general there are no restrictions onthe topology of the underlying physical network, meaning itcould be a fully switched network or have any kind of routedconfiguration.

Routing within virtual networks. Currently, a routerwithin a virtual network is provided by the use of virtualmachine with multiple virtual network interface cards. Theinterface cards are plugged into ports on the different vir-tual switches that it is required to route between. Standardrouting software is then configured and run on the virtualmachine to provide the desired routing services between theconnected LAN segments.

Gatewaying to non-virtualized systems. To allow forcommunication with systems that live in the non-virtualizedworld we provide a gateway. The gateway is simply a virtualmachine with two virtual network interface cards. One of thecards is plugged into a port on a virtual switch. The othervirtual network card is bridged directly on to the physicalnetwork. The gateway has two main roles. First, it adver-tises routing information about the virtual network behindit so that hosts in the non-virtualized world can locate thevirtual machines residing on a virtual network. Second, thegateway converts packets to and from the encapsulated for-mat required of our virtual networks.

3.2 Virtual Switch and VLAN TaggingVLAN Tagging is a well-established network virtualiza-

tion standard for isolation on physical network equipment.The standard is described in IEEE 802.1Q and uses tag-ging of Ethernet packets for isolation between networks [11].VLAN tagging is used in non-virtualized environments where,e.g., a host in the VLAN 42 uses a special network deviceprovided by the operating system to apply a VLAN tag,which contains the VLAN ID 42, to outgoing packets andremove the tag from incoming packets before they are pro-cessed by the upper network stack. VLAN-capable physi-cal switches ensure that by default packets only flow withinVLANs.

3.2.1 Networking Design

We employ VLAN tagging as an alternative to EtherIPencapsulation for efficiency purposes. As an example, in acontrolled data center environment where WAN connectiv-ity may not be required, the use of efficient VLAN-enabled

switches that provide sufficient isolation on the wire yieldsperformance gains over EtherIP encapsulation as we reportin Section 5. In this case, VMs are assigned to one or moreVLAN(s), and each VLAN segment employs its own VLAN-capable virtual switch module to tag Ethernet frames. Thismodule resides within the host OS or privileged domain thatfacilitates the networking capabilities, captures packets com-ing from VMs and tags those with the ID of the VM’s VLANbefore sending them onto the physical wire. On the receiv-ing side, the module removes the VLAN tag and passes thepackets untagged into the destination VM(s). Packets areonly tagged when they have to be transmitted over the phys-ical network. VMs are unaware of the VLAN tagging andsend/receive packets without any VLAN information.

To handle VLAN tagged packets, the physical networkequipment needs to support IEEE 802.1Q and be config-ured accordingly. As an example, if a machine hosts aVM that is part of VLAN 42, then the switch port that isused by that machine needs to be assigned to that specificVLAN 42. Of course, a machine might host multiple VMswhich can be in different VLANs, and therefore a switchport might be assigned to multiple VLANs (which creates aVLAN trunk between the host and the switch port). When-ever a host deploys a new VM or removes a VM, the switchport might need to be reconfigured. Ideally, this can bedone in a dynamic and automated fashion, e.g., throughnetwork management protocols such as, for example, theGARP VLAN Registration Protocol (GVRP). As the phys-ical switches only pass packets between machines within thesame VLAN, those provide an additional isolation mecha-nism to our VLAN-capable virtual switch that is deployedon all of the hosts.

Address mapping. In contrast to both the encapsu-lation and the MAC Rewriting approach, the VLAN tag-ging solution does not require any address mapping mech-anism. VMs discover address information of other VMs byusing standard discovery protocols in the same way as ina non-virtualized environment. However, the virtual switchmodule that runs on each physical machine learns Ethernetaddresses attached to the virtual switch ports by inspectingpackets (in the same way as physical switches do) and buildsup lookup tables (one table per virtual switch / VLAN) thatstore information about the location of VMs based on theirEthernet addresses. The virtual switch uses this table todecide if a packet has to be passed to a local VM or ontothe physical network to be delivered to a remote machine.

Broadcast / multicast mapping. There is no explicitmapping of broadcast / multicast addresses as in the case ofEtherIP encapsulation or MAC Rewriting. Instead, physi-cal switches that manage the underlying network infrastruc-ture ensure that broadcast and multicast traffic never crossVLAN boundaries. Broadcast and multicast packets thatare tagged with a VLAN ID are passed to all switch portsthat are associated with that particular VLAN, but no otherports. When those packets enter the physical machine thatruns our virtual switch module, the packets are only passedinto VMs that are attached to virtual switch ports that areassigned to the VLAN matching the ID in the packets.

3.2.2 Requirements Revisited

Layering over arbitrary physical networks. TheVLAN tagging solution can be used over arbitrary layer 3networks. However, in contrast to the encapsulation ap-

4

proach, a solution based on pure VLAN tagging is limitedto a LAN environment and cannot be deployed over WANlinks. VLAN tagging is highly dependent on support fromthe physical network equipment that is managing the un-derlying infrastructure. E.g., switches need to support theVLAN tagging standard that we use when tagging our pack-ets in our virtual switch module (IEEE 802.1Q) and need tobe configured to handle tagged packets in order to provideappropriate isolation between VLANs.

Routing within virtual networks. A VLAN is a log-ical network segment and by default network traffic suchas broadcast messages or ARP communication is limited toa single VLAN. However, it is also possible to allow com-munication between (virtual) machines of different VLANsthrough Inter-VLAN routing. There are multiple well-knownand standardized solutions to allow this. For example, mostof today’s network switches facilitate fast layer 3 routingbetween multiple VLANs. This solution offers high perfor-mance, but requires that routing policies can be configuredon the physical network devices – ideally in an automatedfashion. As an alternative, we can also deploy specific VMsthat have multiple network interfaces in multiple VLANsand route packets between those – as in the EtherIP encap-sulation approach.

Gatewaying to non-virtualized systems. In VLANtagging, communication with non-virtualized systems is straight-forward. This is because VLAN tagging is a widely usedstandard that is deployed within infrastructures where phys-ical machines do not run any (network) virtualization soft-ware. There is no need for a gateway VM as in the EtherIPencapsulation approach. Instead, physical switches can beconfigured to remove VLAN tags from packets when trans-mitting on a port where the connected endpoint is not VLAN-capable, and add tags whenever packets are received on thatspecific port. In that case those endpoints are completelyunaware of VLANs, and receive and transmit packets with-out any VLAN information.

3.3 MAC RewritingThe MAC Rewriting approach focuses on a closed data

center environment. It allows the grouping of virtual ma-chines into isolated virtual network segments and enforcespolicies to control network packet flow between and insidethese groups. The underlying physical network is a com-pletely switched network and assumed to be a “constrainedworld” hosted by a single owner who controls the infrastruc-ture. However, certain virtual machines may be allowed tocommunicate with external systems through a NAT gate-way.

3.3.1 Networking Design

In contrast to both VLAN Tagging and EtherIP encap-sulation that virtualize the network at layer 2, the MACRewriting approach virtualizes the network based on layer3 network-level information and provides the abstraction offarms which consist of several subnets. For example the IPaddress space is segmented by assigning IP addresses of theformat 10. < FARM > . < SUBNET > . < HOST > tovirtual machines. By default, VMs inside a subnet can com-municate with each other without any restrictions. However,communication between VMs of different subnets has to beexplicitly allowed by the farm owner. Communication acrosstwo different farms is only permitted if both sides have mu-

Figure 5: Farms, subnets and virtual gateways.

tually agreed on such a communication. At the core of thefarm network is a notional gateway connecting all of the sub-nets within the farm. The gateway mediates inter-subnetcommunications within the farm and inter-farm communi-cation across multiple farms. However, the gateway does notexist; we only provide the illusion of it to virtual machines.In fact its actual functionality is distributed amongst all thephysical machines hosting VMs within the farm. Figure 5visualizes this architecture.

MAC rewriting. As in the virtual switch approach, eachhost OS of the virtualized infrastructure runs a componentthat intercepts packets emitted by virtual machines. Thiscomponent takes care of eventually rewriting Ethernet ad-dresses of packets. Addresses are rewritten to (1) providethe illusion of a gateway, (2) not allow VM MAC addresseson the physical wire, and (3) map broadcast and multicastpackets into farm- or subnet-specific multicast traffic.

To provide the illusion of a gateway, if the sending and re-ceiving VM are located on different subnets, then the gate-way’s MAC address is set as source MAC address inside thepacket before it is passed into the destination VM. There-fore VMs never see MAC addresses of VMs that are hostedon a different subnet.

To prevent MAC addresses from appearing on the wire, ifa packet has to go on the wire to a VM that is hosted on aremote system, then the virtual source Ethernet addressesin the packet header is replaced by the Ethernet address ofthe sending physical machine. Further, the virtual destina-tion Ethernet address is replaced by the Ethernet address ofthe physical machine that is hosting the destination VM. Onthe destination host, the physical MAC addresses are sub-stituted by the virtual MAC addresses before the packet ispassed into the destination VM. This means that no virtualMAC addresses will ever be seen on the wire, and VMs neversee MAC addresses of physical machines. Figure 6 illustrateshow a packet is typically processed under this scheme.

Lastly, to do the mapping, when a VM emits broadcastor multicast packets, MAC addresses have to be rewrittento preserve isolation.

Address mapping. Each host OS manages a lookuptable that maps a virtual IP address to a virtual MAC ad-dress. It also stores the physical MAC address of the ma-chine hosting that particular virtual IP. For this purpose anARP solicitation engine runs on each physical machine to

5

Figure 6: MAC Rewriting process.

quickly discover the location of virtual machine IP addresses.Additionally it takes care of keeping involved machines up-to-date when IP addresses move between VMs or when aVM is relocated to a another host within the infrastructure.When a packet arrives on a destination machine, it mighthappen that this host does not have a virtual MAC addressbinding for the packet’s virtual source IP address, becausevirtual MAC addresses get lost through the MAC Rewritingprocesses when a packet has to go out on the physical wire.However, the destination host can quickly find out aboutthe unknown binding by sending a ”backwards” unicast re-quest to the machine on which the sending VM is located.This machine is identified by the packet’s source Ethernetaddress.

Broadcast / multicast mapping. Broadcast IP pack-ets have to be delivered to all VMs in the same subnet, sothey possibly have to be passed to multiple physical ma-chines. We use Ethernet multicast addresses for that pur-pose - each subnet is represented by a multicast group, andphysical machines that host VMs on that subnet have to reg-ister to that particular multicast address. Whenever a VMemits a broadcast packet, the host OS makes use of MACRewriting and replaces the broadcast destination MAC ad-dress with the subnet’s multicast address before letting thepacket on the wire. The same is applicable for farms, andpackets that have to be delivered to a whole farm are sentto the multicast group of that particular farm. IP multicasttraffic coming from virtual machines is either mapped intoa subnet multicast address and delivered only subnet-wide,or handled as a farm-wide multicast distribution. This de-pends on whether the IP packet is allowed to pass subnetboundaries - which is determined from its TTL value.

Packet filtering. Network packet flow policies are man-aged by a packet filtering component that is distributedamongst all physical machines. Packet filtering is enforcedwithin the host OS and therefore cannot be bypassed by anyVM. We use packet filtering to control and restrict inter-subnet or inter-farm communication. Additionally, throughthis a farm owner can specify more sophisticated networkpolicies on a per-VM basis.

3.3.2 Requirements Revisited

Layering over arbitrary physical networks. Cur-rently we rely on having a flat layer 2 network as under-lying physical infrastructure and do not support any other

network topologies. This is mainly because the solicitationmechanism is based on ARP and therefore does not crosssubnet boundaries.

Routing within virtual networks. Communicationbetween different subnets is possible if this path has beenpermitted by the farm owner. There is no need for a routingVM as in the VLAN Tagging or EtherIP encapsulation ap-proach, because we provide the illusion of that intermediatehop through MAC Rewriting. VMs of two different farmsmay also communicate with each other - if both farm ownershave mutually agreed on such a communication. Throughthe MAC Rewriting process packets appear to a VM as ifthey have traversed a gateway, but in reality packets arepassed between VMs with just a single network hop. Sub-net and farm boundaries are enforced by a packet filteringengine that runs on each host and controls packet flow forvirtual machines.

Gatewaying to non-virtualized systems. Virtual ma-chines can communicate with non-virtualized systems thatdo not run our network virtualization technology (assumingthat this communication is not restricted through packet fil-tering). The ARP engine takes care of discovering locationsof any kind of machine within the infrastructure. Packetscoming from these non-virtualized systems are processed us-ing MAC Rewriting in the same way as explained before,there is no special treatment necessary.

4. COMPARATIVE STUDYIn this section, we assess the advantages and disadvan-

tages of each approach in different usage scenarios. In Sec-tion 4.1, we discuss the networking features provided by eachapproach. In Section 4.2, we compare the approaches withrespect to their ability to support network isolation. In Sec-tion 4.3, we present a number of scenarios that each tech-nology can be best targeted for. We argue that, in manyways, our approaches are complementary and can be incor-porated into one solution that provides a secure and efficientLAN / WAN virtual networking service. We report a sum-mary of the advantages and disadvantages of the differentapproaches in Table 1.

4.1 Networking Features And LimitationsAll three approaches enable network virtualization; to do

so, however, they use different mechanisms that require dif-ferent assumptions and dependencies.

4.1.1 Support From Physical Infrastructure

Our network virtualization approaches are mostly software-based; however, the VLAN tagging approach requires theuse of hardware devices for network separation. In particu-lar, VLAN tagging requires support from physical networkswitches to isolate virtual machine network traffic. Thus,hardware switches need to be able to understand the VLANinformation and process tagged packets accordingly. Theswitches need to be (re-)configured with VLAN informationwhenever VMs are deployed, removed or migrated – whichcan happen frequently within dynamic and flexible infras-tructures. It is thus a challenge to manage and alter theconfiguration of physical switches in an automated manner.Another issue with VLAN tagging is that modern NICs sup-port VLAN offloading to accelarate network packet process-ing. As a result, they can potentially strip the tags frompackets on reception. This feature needs to be disabled on

6

all hosts because it interferes with our technology as we ex-plain in Section 5. In contrast, both EtherIP encapsulationand MAC Rewriting are purely software-based and do notrequire any specific configuration of physical network devicesor NIC drivers.

Each approach can be deployed on a restricted set of net-work types. Network virtualization based on MAC Rewrit-ing or EtherIP encapsulation relies on having an IP networkdeployed between physical machines of the virtualized in-frastructure. The MAC Rewriting approach further needsto run on a flat layer 2 network within a LAN environmentand VMs have to be given IP addresses of a predefined for-mat. The EtherIP encapsulation approach in contrast cantransmit packets over any arbitrary physical network topol-ogy that supports multicast. The VLAN tagging technologycan be deployed within IP and non-IP networks, but is lim-ited to a LAN infrastructure. Note that multiple approachescan be deployed on a data-center in a complimentary fash-ion, e.g., to enable fast LAN connectivity using VLAN tag-ging and a relatively slow WAN connectivity using EtherIP.

Lastly, the approaches differ in the way they handle vir-tual machine MAC addresses. In VLAN tagging, MAC ad-dresses of virtual machines are allowed to appear on thephysical network which increases the amount of entries withinnetwork switch tables when deploying large numbers of VMs.This problem is eliminated in networks that use the MACRewriting or the EtherIP encapsulation approach.

4.1.2 Networking Support for Virtual Machines

Each approach supports a restricted set of network proto-cols. In particular, MAC Rewriting only supports IP-basednetworking for VMs. Further, it currently requires that allVMs have IP addresses of a defined format, for example,we currently assign addresses of the form 10. < FARM >

. < SUBNET > . < HOST > to virtual machines. Whilewe do require some addressing structure that does not allowcompletely arbitrary user-selectable IP addresses, we do notimpose any restrictions on the prefixes or address ranges touse. For example, if a customer provides us with his or herown class B network, then we can use those addresses in-stead of a 10. < FARM > . < SUBNET > . < HOST >

scheme. EtherIP and VLAN tagging do not pose any re-strictions on layer 3 networking protocols; they can be usedwhen virtual machines want to run non-IP based services.

The particular advantage with the MAC Rewriting ap-proach is that every communication path between two vir-tual machines only ever involves a single network hop. Forexample, if two communicating VMs are located on differentsubnets (or even on different farms), network traffic does nothave to pass a router (or a routing VM) as in the EtherIPapproach. This yields better network performance and addi-tionally decreases the management effort for deploying andconfiguring routing entities. Similarly, in VLAN tagging wecan avoid the deployment and management of routing VMssince the physical network switches are configured to per-form fast inter-VLAN routing.

Lastly, MAC Rewriting does not require any form of packetencapsulation and therefore allows VMs to use full MTUsized packets that can potentially yield higher network through-put. EtherIP significantly reduces the MTU of virtual ma-chine network packets: when encapsulating an Ethernet framewithin an IP packet it reduces the MTU available to VMsby 38 bytes per packet. In an ideal setup, VLAN tagging

does not reduce the MTU of packets at all, because theIEEE 802.1Q VLAN tagging standard allows the additionof 4 bytes per VLAN header to the normal MTU. In gen-eral, most of today’s network switches support much higherMTUs, and so in many cases the problems stated here caneliminated2.

4.1.3 Manageability

Unlike MAC rewriting, both EtherIP and VLAN taggingneed external network entities to be managed accordingly.The EtherIP approach requires that routing VMs are set upwithin the infrastructure. These need to be deployed justlike other virtual machines, but also need to run specificrouting software which needs to be programmed and con-trolled. One drawback of this is that a routing VM can onlyinterconnect as many VNETs as the virtualization softwareallows a VM to have virtual network interfaces, e.g. whenusing Xen this limit is 3 vNICs per guest and when usingVMWare the limit is 4 (Server 1.0 and ESX version). TheVLAN tagging approach does not necessarily require routingVMs, but instead relies on physical switches to be configuredwith VLAN support. Hence setting up a new virtual net-work always involves deploying new switch configurationswithin the physical infrastructure. To enable automated de-ployment of virtual networks this means that managementsoftware needs to be able to communicate with hardwareswitches (of possibly different manufacturers) and VNETcomponents on host machines.

Configuring new virtual networks when using the MACRewriting approach does not involve configuration of net-work devices or setting up of routing VMs. However, IPaddresses of virtual machines need to be managed and allo-cated in a global and secure manner to ensure that virtualnetwork boundaries and packet flow restrictions can be en-forced properly - which is only possible in a very restrictedand highly controlled data center environment. However,binding VMs to use IP addresses of a specifically definedformat significantly reduces this management effort.

4.1.4 Scalability

All three approaches pose scalability limitations due totheir particular designs. In the MAC Rewriting approach,VMs are assigned to a farm/subnet combination throughtheir IP addresses which have the form 10. < FARM >

. < SUBNET > . < HOST >. Because the IP addressis only a four-byte value, the number of farms and subnetsthat we can encode in an IP address is limited, and thisdefines the number of farms and subnets that we can hoston a single network. The current design provides up to 14ksubnets to customers. Additionally, it is possible to connectmultiple physical networks together in a federated manner atthe expense of additional hops in the network path which canhave a significant impact on the performance. Extending theMAC Rewriting approach to an IPv6 solution will break thecurrent limits of 32-bit IP addresses, and hence will providea much larger number of farms and subnets within a singlenetwork.

We currently use the 802.1Q standard ([11]) when de-ploying our VLAN tagging solution. 802.1Q defines a four-byte header that is placed on Ethernet frames. Within this

2In addition to support from the physical network infras-tructure, the network cards on all systems also need to beable to handle larger MTU sizes.

7

header there is a 12-bit field for the VLAN identifier. Thismeans that a solution that is based on VLAN tagging canonly provide up to 4096 VLANs (or VNETs) within a sin-gle local network. A possible option to overcome this limitwould be to stack multiple VLAN tags in order to expandthe VLAN space - this is supported by some hardware de-vices (mainly Cisco) through the IEEE 802.1q-in-q (QinQor stackable VLANs) technology.

The EtherIP approach poses a similar scalability limita-tion in terms of the number of VLANs that can be sup-ported. EtherIP uses the standard in [12] that adds a 16-bitheader to an Ethernet or 802.3 frame. This header includes afour-bit field that specifies the EtherIP version number, andleaves 12 bits reserved which we use to encode the VNETidentifier. This implies that the EtherIP network virtual-ization approach is limited to 4096 VLANs within a singleadministrative domain. Another scalability issue of this ap-proach is the use of routing VMs. Routing VMs interconnectvirtual networks by having multiple network interfaces (oneper VLAN) and routing traffic between those. The num-ber of network interfaces that a routing VMs can have islimited by the virtualization technology in use (e.g., Xen orVMware). Additionally, a routing VM represents a singlepoint of failure and can potentially become a bottleneck forinter-VNET communication because all traffic has to tra-verse that VM. Therefore, in a large-scale virtualized infras-tructure it is critical that traffic is efficiently load-balancedacross various routing VMs.

4.2 Security AnalysisMachine virtualization alone provides immediate isolation

of computing resources such as memory and CPU betweenguest domains. However, the network remains to be a sharedresource as all traffic from guests eventually pass through ashared network resource (e.g., a virtual switch) and end upon the shared physical medium. As a result, we need mech-anisms to (1) control the information flow between virtualmachines (e.g., who can communicate with whom), (2) con-figure virtual and physical network resources, and (3) sep-arate network resources used by each networking domain.In this section, we revisit the security requirements outlinedin Section 2 and compare each approach in terms of theircapability to enforce and support domain policies. In par-ticular, we describe how each approach reinforces networkseparation and isolation within and across network domains.

Our solutions mostly rely on VLAN separation and ad-dress mapping techniques to control the information flowwithin and across networking domains. The approaches dif-fer in ways the address mapping and packet forwarding de-cisions are taken. In particular, EtherIP and VLAN tagginguse a single virtual switch per VLAN segment that forwardsand accepts traffic to and from member hosts. MAC rewrit-ing implements a distributed virtual gateway to (re)directtraffic within and between networking domains.

Within a networking domain (i.e., a VLAN segment or asubnet), EtherIP uses a single (distributed) virtual switchto decide on which network packet to forward and accept.Similarly, MAC rewriting uses a single (distributed) virtualgateway to do same. In contrast, in VLAN tagging a phys-ical switch is additionally involved in the decision process.Thus, the former two have an advantage over the latter be-cause the decision is taken on an end-to-end basis and nointermediate node (e.g., the physical switch) is involved. In-

volving the physical switch, however, can provide better net-work separation as compared to software-only approaches.As a result, VLAN tagging can provide network isolationnot only between virtual machines, but also between physi-cal machines.

Another basis for comparison is the way each approachhandles the communication across different networking do-mains. In MAC rewriting, all decisions on whether to allowcommunication across domains (i.e., subnets and farms) aretaken by a distributed virtual gateway. This yields an effi-cient scheme and allows VMs to communicate over a singlehop regardless of their domain membership. However, thevirtual gateway becomes the single point of decision thatneeds to be trusted by all domains. In contrast, in EtherIPand VLAN tagging, a virtual switch is designated to a par-ticular domain and controls communication within that do-main only. Communication across domains is handled bygateway routers or firewalls that are trusted by both do-mains. This yields a less efficient scheme than MAC rewrit-ing. However, it can potentially provide a better separationof networking resources across domains. Further, the vir-tual switch design allows the disaggregation of componentsinto separate, highly isolated domains. For example, eachvirtual switch can run in a separate purpose-build stripped-down VM. This creates an advantage for providing high as-surance about the security properties of a system and en-ables the separation of the configuration and managementof VLANs that are not under a single administrative en-tity. This is especially beneficial for systems that are sharedamongst competing parties.

While the vSwitch approaches focus on strong separa-tion of virtual networks, the MAC Rewriting technologyaddresses secure inter-farm communication from a differentperspective. In particular, this approach fits best when cus-tomers owning different farms want to share services in acontrolled but efficient manner. This is, for example, thecase with the Service Utility Platform (SUP) project whichis currently developed within HP Labs. We present a MACRewriting prototype that runs on SUP in Section 5.

Lastly, all schemes support confidentiality and can pro-vide encryption over untrusted physical medium. Further,EtherIP can protect the VLAN membership information us-ing end-to-end encryption. In VLAN tagging, however, thetag needs to be revealed to the physical switch. Hence, end-to-end encryption of the tagging information is not possible.MAC rewriting does not involve a tag, hence no extra pro-tection is needed.

4.3 Applications and Use CasesEach networking technology we have described in the pre-

vious sections has specific advantages in different environ-ments with varying use cases and scenarios. Here, we presenta number of scenarios that each technology can be best tar-geted for.

Customers might want to use virtual machine technol-ogy to run legacy applications, simulate proprietary net-work solutions, or run non-IP based protocols. The MACRewriting approach cannot be used for these specific ap-plication requirements, because it only supports IP-basednetwork communication. Note that most mainstream datacenter solutions will fit for the MAC Rewriting approach inthis regard (as IP is a widely used protocol). However, itmight not be the best applicable solution for customers that

8

Table 1: Summary of the advantages and disadvantages.

Features MAC RW EtherIP VLAN Tagging

Arbitrary L3 protocols for VMs N Y YForces reduced MTU sizes N Y Y/Na

User-selectable IP addresses for VMs N Y YSingle-hop networking Y N N

Arbitrary L3 protocols for hosts N N YArbitrary physical network topology N Y Y

Requires programming of physical switches N N YRequires support from NICs N N Y

Runs over WAN link if tunneling available Y Y YVM MAC addresses on the wire N N YRequires external routing entities N Y Y

Automatic network policy verificationb Y N NIntegrated NAC frameworkc N Y Y

Number of subnets per single LAN (by default) 14K 4K 4K

aAs defined in the IEEE 802.1Q standard, a VLAN-tagged Ethernet frame can be larger than the standard 1500 byte EthernetMTU size. However, many older network interfaces cannot handle frames with sizes of over 1500 bytes, so in practice thestandard MTU size has to be reduced for most older network cards.bAs described in 3.3cThe NAC framework is a Network Admission Control mechanism that we have implemented in order to control whether ornot a VM is allowed to join a specific virtual network segment.

require customized network configurations (e.g., non-IP).Currently, the VLAN approach is the best fit when run-

ning over a shared, untrusted physical network infrastruc-ture as it facilitates the separation of network traffic on thephysical wire – enforced by network switches within the in-frastructure. However, both EtherIP and MAC Rewritingcan be easily enhanced to securely run over untrusted net-works by using encryption or network access control mech-anisms such as, for example, IEEE 802.1X ([1]). As well asbeing able to use 802.1X network access control to restrictphysical machine access to a particular physical LAN, thevirtual switch implementation supports 802.1X authentica-tion of virtual machines before they are allowed to join aparticular virtual LAN.

Lastly, the MAC Rewriting prototype implementation fa-cilitates a scheme to automatically verify network policiesand provide assurance about the actual state of the com-plete system from a network point of view. This is an im-portant feature that can help guarantee fulfilling certain se-curity requirements (e.g., as defined in Service Level Agree-ments (SLAs)).

5. PROTOTYPE IMPLEMENTATIONSIn this section, we provide a brief overview of two proto-

types we implemented on Xen and VMware platforms usingthe virtual networking technologies we introduced in Sec-tion 3. In Section 5.1, we present the virtual switch imple-mentation that uses a combination of EtherIP encapsulationand VLAN tagging. In Section 5.2, we present the virtualgateway implementation that uses MAC Rewriting. Lastly,in Section 5.3, we empirically assess the performance of eachapproach.

5.1 Security-enhanced NetworkVirtualizationWe describe a Xen-based [3] prototype implementation of

the virtual switching framework. Figure 7 shows the imple-

mentation of virtual switches that manage two vLAN seg-ments, V LANα and V LANβ . The policy engine, also shownin the figure, implements the policies corresponding to thesecurity domains formed by the VLAN segments.

5.1.1 Implementation Details

Our implementation is based on Xen-unstable 3.0.4, aVMM for the IA32 platform, with the VMs running theLinux 2.6.16 operating system. Our networking extensionsare implemented as kernel modules in Dom0, which alsoacts as driver domain for the physical NIC(s) of each physi-cal host. A driver domain is special in the sense that it hasaccess to portions of the host’s physical hardware, such as aphysical NIC.

The virtual network interface organization of Xen splitsa NIC driver into two parts: a front-end driver and a back-end driver. A front-end driver is a special NIC driver thatresides within the kernel of the guest OS. It is responsiblefor allocating a network device within the guest kernel (eth0in Dom1 and Dom2 of hosts A and B, shown in Figure 7).The guest kernel layers its IP stack on top of that device asif it had a real Ethernet device driver to talk to. The back-end portion of the network driver resides within the kernelof a separate driver domain (Dom0 in our implementation)and creates a network device within the driver domain forevery front-end device in a guest domain that gets created.Figure 7 shows two of these back-end devices, vif1.0 andvif2.0, in each of the two hosts A and B. These back-enddevices correspond to the eth0 devices in Dom1 and Dom2,respectively, in each host.

Conceptually, the pair of front-end and back-end devicesbehaves as follows. Packets sent out by the network stackrunning on top of the front-end network device in the guest

9

Host A

Dom 1Dom 0

αencap.

module

VLAN

taggingmodule

PolicyEngine

vSwitches

eth0 eth0vif2.0vif1.0eth0

front-end devices

back-end

devicesphysicalNIC

VLAN-capablephysical switch

Host B

Dom 1Dom 0

αencap.

module

VLAN

taggingmodule

PolicyEngine

vSwitches

eth0 eth0vif2.0vif1.0eth0

front-end devices

back-end

devicesphysicalNIC

Host C

eth0

physical NIC(VLAN un-aware)

CA1 CB1

CA1 CB1

A1C B1C

A1C B1C

Dom 2Dom 2

β β

Figure 7: Prototype implementation of VLANs as security domains.

domain appear as packets received by the back-end networkdevice in the driver domain. Similarly, packets sent out bythe back-end network-device by the driver domain appear tothe network stack running within a guest domain as packetsreceived by the front-end network device. In its standardconfiguration, Xen is configured to simply bridge the driverdomain back-end devices onto the real physical NIC. By thismechanism, packets generated by a guest domain find theirway onto the physical network and packets on the physicalnetwork can be received by the guest domain.

The Xen configuration file is used to specify the particularvSwitch and the particular port in the vSwitch to which aXen back-end device is attached. We use additional scriptsto specify whether a particular vSwitch should use one orboth of VLAN tagging and encapsulation mechanisms forisolating separate virtual networks.

The vSwitches for V LANα and V LANβ are each imple-mented in a distributed fashion (i.e., spread across hosts Aand B) by a kernel module in Dom0, which maintains a ta-ble mapping virtual network devices to ports on a particularvSwitch. Essentially, the kernel module implements EtherIPprocessing for packets coming out of and destined for theVMs. Each virtual switch (and hence VLAN segment) hasa numeric identifier associated with it. The Ethernet pack-ets sent by a VM are captured by the kernel module im-plementing part of the vSwitch as they are received on thecorresponding back-end device in Dom0. The packets areencapsulated using EtherIP with the network identifier fieldset to match the identifier of the vSwitch that the VM issupposed to be plugged into. The EtherIP packet is giveneither a multicast or unicast IP address and simply fed intothe Dom0 IP stack for routing onto the physical network.The kernel module also receives EtherIP packets destinedfor the physical host. The module un-encapsulates the Eth-ernet frames contained in the encapsulated EtherIP packetsand transmits the raw frame over the appropriate virtualnetwork interface so that it is received by the intended guest

vNIC.In addition to the kernel module for EtherIP processing,

we have also implemented a kernel module for VLAN tag-ging in Dom0 of each virtualized host. Ethernet packets sentby a VM are grabbed at the same point in the Dom0 net-work stack as in the case of EtherIP processing. However,instead of wrapping the Ethernet packets in an IP packet,the VLAN tagging module tags the packet with the ID ofthe VLAN that the VM is supposed to be connected to. Thetagged packet is then sent straight out onto the wire throughthe physical NIC. The VLAN tagging module also interceptsVLAN packets arriving on the physical wire destined for aVM. The module then removes the VLAN tags and, basedon the tag, maps packets to the appropriate vSwitch (α orβ) which, in turn, maps them to the corresponding back-enddevice (vif1.0 or vif2.0) in Dom0. The packets eventually ar-rive at the corresponding front-end device (eth0 in Dom1 orDom2) as plain Ethernet packets.

5.1.2 Implementation Issues

Below are some implementation issues we had to tackle inrealizing the VLAN and encapsulation approaches.

1. Some Ethernet cards offer VLAN tag filtering and tagremoval/offload capabilities. Such capabilities are use-ful when running just a single kernel on a physical plat-form, in which case there is no need to maintain thetags for making propagation decisions. However, forour virtual networking extensions, the hardware de-vice should not strip the tags from packets on recep-tion over the physical wire; instead, the kernel mod-ules we have implemented should decide to which VMthe packets should be forwarded. For this purpose,we modified the Linux kernel tg3.ko and forcedeth.konetwork drivers so as to disable VLAN offloading.

2. For efficiency reasons, the Xen front-end and back-end driver implementations avoid computing check-

10

sums between them for TCP/IP and UDP/IP packets.We modified the Xen code to also handle our EtherIP-encapsulated IP packets in a similar manner.

3. The EtherIP encapsulation approach relies on map-ping a virtual Ethernet broadcast domain to a IP mul-ticast domain. While this works in a LAN environ-ment, we encountered problems when creating VLANsegments that span WAN-separated physical machines.We resolved this issue by building uni-directional mul-ticast tunnels between successive LAN segments.

5.2 Network Virtualization in Datacenters

5.2.1 Providing Secure Service Environments

A prototype of the MAC Rewriting approach has beenimplemented as part of the Service Utility Platform (SUP)project within HP Labs. The SUP is a secure, automatedmanagement system that facilitates the unified aggregationof virtual resources into secure service environments. TheSUP focuses on a local virtualized datacenter. In this con-text the MAC Rewriting approach securely separates andisolates customers’ virtualized elements to provide the illu-sion of dedicated network resources. It also separates cus-tomer networks from the utility network. Additionally, cus-tomer farms should be allowed to implement and selectivelyexpose services to other farms and consume services pro-vided by other farms, to allow a rich ecosystem of interact-ing services to develop. The MAC Rewriting approach is themost suitable choice to enable network virtualization in thisparticular scenario, because unlike the other two approaches,the communication that spans over farm boundaries is real-ized as a single-hop network path.

5.2.2 Security Assumptions

In implementing MAC rewriting, we have assumed thatthe following conditions are present:

1. The datacenter network is separated from the Internetby a firewall and NAT, only a subset of VMs will bevisible externally.

2. A layer 2 connection exists between all machines thatare part of the virtualized datacenter. However, tun-neling could allow federation of multiple instances ofthese datacenters.

3. The virtualized datacenter is controlled by a singleowner who manages physical machines and networkdevices of the infrastructure. All hosts run standardsoftware to provide basic system-level virtualization.We use Xen and VMWare for this purpose.

4. Access to the host OS on VMWare systems as wellas to Xen’s most privileged domain is assumed to begranted only to authorized users, as in here one hasfull control over the configuration of virtual networkingcapabilities.

5.2.3 Implementation

Every physical machine of the infrastructure runs a vir-tualization layer - in our cases Xen 3 or VMWare Server.

The virtual networking capabilities are provided by a Linuxkernel module (the VNET module). On a Xen host, thismodule runs in domain 0 (or in an isolated driver domain)and on a VMWare system it is loaded within the host OS.Here each VM’s network card is represented by a virtualNIC, and also the driver for the physical NIC resides here.In the current design all network traffic has to pass throughthis domain, and it is impossible for VMs to circumvent this.

The VNET module (1) intercepts packets coming fromvirtual machines as well as coming from the physical wire,(2) hands them to the filtering engine, (3) replaces MAC ad-dresses, and (4) passes them on to either one or more VMsor onto the network to a remote machine. The kernel mod-ule hooks into the network stack at the Ethernet layer forpackets from / to the physical network and from / to VMs.Packets from the host OS (or Xen domain 0 respectively)are intercepted on the IP layer and also passed back in atthat level. This means that the host OS does not do anyARP discovery on its own, but uses the shared ARP engineprovided by the VNETLinux kernel module instead.

The kernel module exports certain configuration param-eters and options to user-space tools through the LinuxSYSFS filesystem. Only authorized users or processes canaccess this interface to configure virtual network settings.SYSFS cannot be directly accessed from remote machines,therefore there is a control daemon running on each systemto enable automatic remote deployment.

The current implementation supports all features of theMAC Rewriting approach that have been described in Sec-tion 3. Although, we do not provide transparent encryptionof network traffic yet.

Packet filtering is currently realized using standard IPT-ables and a proprietary matching module that allows us toeasily filter traffic to enforce farm and subnet boundaries.Network policies have to be set up within the host OS us-ing standard IPTables user-space tools. It is also possibleto configure more advanced network policies on a per-VMbasis - here we support most filtering options that IPTablesprovides. Policies can only be set up by the owner of theVM.

Global packet filtering rules that enforce farm and subnetboundaries are distributed amongst all participating hostmachines. Each host runs a rules management componentthat knows about illegal packet flows and immutable, globalpolicies (these static rules are part of the configured TrustedComputing Base (TCB) that all physical machines boot upwith). All new network policies have to be validated andapproved by this component. Additionally, policies that af-fect multiple farms with potentially different owners alwayshave to be mutually agreed.

5.3 Performance AnalysisAs part of our comparative study, we have empirically

assessed how current prototypes fare when running basicnetworking applications within a virtual machine. We com-pare our solutions against a virtualized system (such as Xenor VMWare) in standard bridging mode and also againsta non-virtualized system. We first report and analyze theperformance figures for each VNET technology and comparethe results. We then provide an overview of potential per-formance improvements that we are currently investigating.

We obtained the throughput results using the netperf

network benchmark and latency results using the ping tool.

11

(a) Summary of relative throughput performance results. (b) TX/RX throughput benchmark results in Mbps.

Figure 8: Intra-subnet Throughput Results.

While our VNET technologies run with both VMWare andXen systems, we only provide performance results for a Xensetup in this report as both technologies perform very simi-lar. Our test systems run Xen 3.0.4 (release version) with a2.6.16.33 Linux kernel in domain 0. We used HP ProLiantBL25p G2 blade servers each fitted with two AMD Opteronprocessors running at 2 GHz, 8GB system memory and aGigabit Ethernet card. For throughput measurements wehave configured netperf with a confidence level of 99% anda confidence interval of 5% to ensure that our results onlyinclude consistent samples. As netperf is a client/server-based tool, we ran one instance of the benchmark on oneguest VM as a server process and another instance on thesecond guest VM to do the actual benchmark.

We have measured network performance for two differentscenarios:

1. Intra-subnet communication - both communicating VMsreside on the same virtual network segment (or sub-net), and

2. Inter-subnet communication - communicating VMs re-side on different subnets.

5.3.1 Intra-subnet communication

Figure 8(a) shows a summary of the throughput resultsthat we have obtained for a netperf TCP STREAM testusing a message size of 8192 bytes and a socket size of65536 bytes. The figure evaluates throughput performanceof a standard Xen bridged configuration, VLAN tagging,EtherIP encapsulation and MAC Rewriting and comparesthem to the performance that a native (non-virtualized)Linux system achieves. These results take into account howmuch CPU resources each approach consumes during thetests. This is a critical characteristic of any solution thatneeds to be assessed, mainly because it is important toclearly account for resource utilization in a virtualized sys-tem where resources like CPU are shared across multipleVMs. The results show that in this setup all approachesperform very similarly.

In Figure 8(b), we report more detailed throughput mea-surements that break down the implementation specific dif-

ferences between our three network virtualization approaches.These results show the actual achieved throughput reportedby the netperf tool in Megabits per second (Mbps). Wedo not report CPU figures in these graphs as utilizationbetween these three technologies is very close and there-fore does not give any additional insight. The graphs showthat the VLAN tagging extension achieves the best perfor-mance overall while EtherIP encapsulation yields the worstthroughput.

All VNET extensions perform better on the Tx path -except the EtherIP method where the major cost is havingto allocate a fresh socket buffer (skb) and copy the originalbuffer data into the fresh skb. During the initial alloca-tion of the skb, the Linux network stack allocates a fixedamount of headroom for the expected headers that will beadded to the packet as it goes down the stack. However,not enough space is allocated initially to accommodate theEtherIP header; thus, we have to copy the data, which is acostly operation. However, there is some spare headroomspace, which is sufficient for the extra VLAN tag. As a re-sult, the VLAN tagging method does not suffer from thepacket copying overhead. It is important to mention herethat the small headroom is an implementation specific issuethat could be fixed with a specifically patched Linux kernel.The MAC Rewriting approach does not change the packetin any way other than eventually replacing source and des-tination Ethernet addresses. Therefore, it does not need toallocate a new socket buffer. However, it has more complexdecision paths in order to find out how to rewrite the packetwhich results in more processing overhead. Additionally,packets need to be processed up to the IP header while bothVLAN tagging and EtherIP encapsulation inspect packetson Ethernet level only.

In the Rx path, there is no packet-copying overhead forthe EtherIP approach; the extra EtherIP header merely hasto be removed before the packet is sent to a VM. However,as compared to VLAN tagging and MAC Rewriting in whichpackets are grabbed from the Linux network stack immedi-ately after coming into the OS from the network driver, theEtherIP solution requires that packets are passed into andprocessed by the host OS IP stack before they are handedover to the EtherIP packet handler of the virtual switch

12

code. Lastly, the MAC Rewriting approach performs simi-lar for both Tx and Rx due to similar packet processing -e.g. the same lookups and the same packet rewriting in bothdirections.

An important difference between MAC Rewriting and theother approaches including standard Xen bridging is thatMAC Rewriting virtualizes the network on a higher level(network layer). This has two significant implications froma performance point-of-view:

1. MAC Rewriting manages larger lookup tables. In factthe VLAN / encapsulation implementation facilitatesa table per vSwitch/VNET while the MAC Rewritingprototype manages only one table per physical host(as it operates in a distributed router fashion).

2. Communication between VMs which are deployed onan infrastructure that runs MAC Rewriting only everinvolves a single network hop. All other approachesrequire a routing entity in the network path in the casewhere communicating VMs are on different VNETs.

Lastly, in Table 2 we report the round-trip times betweentwo guest VMs on a physical host for the bridged, VLAN,EtherIP encapsulation, and MAC Rewriting cases obtainedusing the ping -c 1000 host command, i.e., 1000 packetssent. The results show that on average Xen bridging has thelowest round-trip time.

Table 2: Intra-subnet Round-trip Times using Ping.Minimum Average Maximum Mean

DeviationBridged 0.1355 0.18 0.294 0.0235

MAC RW 0.1582 0.2062 0.3182 0.0276VLAN 0.1395 0.21225 0.35675 0.0295EtherIP 0.151 0.246 0.378 0.0335

5.3.2 Inter-subnet communication

We have additionally measured and analyzed the per-formance that our different approaches achieve when thecommunicating VMs reside on different virtual network seg-ments (or subnets). While all technologies offer similar per-formance for communication within a subnet, the resultsfor inter-subnet communication demonstrate a major de-sign difference between MAC Rewriting and the other twoextensions, and its impact on network performance. Thisis because MAC Rewriting operates at L3 and it does notrely on external routing entities, whereas both VLAN tag-ging and EtherIP encapsulation introduce a routing entityon the network path for inter-subnet communication.

We cam deploy routing entities in various forms for boththe EtherIP encapsulation and the VLAN tagging approach.These can mainly be distinguished by the degree of flexibil-ity, manageability and performance that they can provide.For VLAN tagging we enforce routing between virtual net-work segments through the integration of layer 3 switchesthat are capable of carrying out Inter-VLAN routing. Thisapproach allows high-performance inter-subnet communica-tion. For EtherIP we can deploy Routing VMs that routepackets between VMs that reside on different subnets. How-ever, the use of Routing VMs introduces a huge performance

Figure 9: Inter-subnet Throughput Results inMbps.

drop for cross-subnet traffic. This is demonstrated in Fig-ure 9 which compares cross-subnet throughput performanceof MAC Rewriting and EtherIP encapsulation. These re-sults have been recorded with the same netperf test con-figuration and process that we have used for intra-subnetcommunication as described in Section 5.3.1. While bothapproaches achieve similar performance for network trafficthat stays within a single subnet, we see a drop in per-formance of about 40% in case of EtherIP encapsulationwhen traffic has to go across subnet boundaries and only asingle VM is transmitting through the Routing VM at thesame time (shown as ”single VM TX” in Figure 9). Theperformance suffers even more when multiple VMs commu-nicate through the a Routing VM at the same time: wehave recorded a drop in throughput of almost 60% whenonly two VMs are transmitting at the same time (shown as”double VM TX” in Figure 9). In contrast, the L3 approachof MAC Rewriting eliminates the need for any kind of exter-nal routing entities, and therefore it provides more constantand reliable network performance within and across subnets.

The introduction of Routing VMs on the network pathalso shows a significant increase in latency which we reportin Table 3. The results show that latency is almost doubledwhen using a Routing VM for the EtherIP encapsulationextension, whereas no increased latency is reported whenusing MAC Rewriting.

Table 3: Inter-subnet Round-Trip Times using Ping.Min Avg Max Mean

DevMAC RW 0.1582 0.2062 0.3182 0.0276EtherIP 0.287 0.37 0.4975 0.387

The poor performance of Routing VM network communi-cation is mainly due to the fact that the overhead of virtual-ization hits at least twice. Also, it is clear that Routing VMscan quickly become the bottleneck of inter-subnet network-ing, and additionally a single point of failure. To overcomesome of these issues we can also deploy routing entities di-

13

rectly on physical host machines.

5.3.3 Future Performance Enhancements

Network virtualization provides a flexible scheme to sepa-rate customer networks in virtualized platforms; usually atthe expense of performance. A recent development we areinvestigating in this area is the enhancement of hardwarewith better support for virtualization. This is especiallyrelevant to networking because the virtualization layer isinvolved in all network I/O processing which significantlydegrades performance as we have reported in Figure 8(a).Advanced hardware virtualization support can remove theprivileged domain or host OS from the main data I/O pathwhich will enable direct access to the network hardware fora virtual machine.

We are also investigating network resource control mech-anisms that can allow better performance isolation acrossvirtual machines. Briefly, this scheme enables throttlingnetwork bandwidth for VMs which in turn can improve theoverall network performance within a virtualized infrastruc-ture. Further, it protects the infrastructure in the pres-ence of malicious/buggy VMs that could use up networkresources and slow down other VMs or network services.

6. RELATED WORKPrevious work on virtualizing physical networks can be

roughly grouped into two categories: those based on Eth-ernet virtualization (Layer 2) and those based on TCP/IP-level virtualization (Layer 3). Virtualization technologiessuch as Xen and VMWare also provide basic virtual net-working in these two categories – these are referred to asbridged and routed configuration modes. However, thesebasic approaches alone do not provide sufficient isolation ofnetwork traffic between virtual network segments. Further,they lack the mechanisms that can enforce more sophisti-cated intra-VNET and inter-VNET network policies.

Although both categories include a substantial amount ofwork (e.g., [12, 2, 4, 8, 9, 16, 19, 20, 21]), few studies havean explicit security focus:

Ethernet Virtualization: Ethernet virtualization aims attransporting multiple Ethernet connections over a single phys-ical medium. There are a large number of Ethernet tunnel-ing protocols [9]. Local transport over a “trusted” wire isusually multiplexed using the well-established VLAN stan-dard IEEE 802.1Q-2003. It adds virtual LAN tags to eachEthernet segment and enables separation of multiple net-works. An example for high-performance Infiniband VLANsis given in [10]. In wide-area networks, VLAN tags are of-ten not preserved. To overcome these restrictions, Ethernetencapsulation has been proposed as an alternative [12, 19,8, 9]. Ethernet packets (including tags) are wrapped intoTCP/IP packets. This enables the embedding of a virtualEthernet network into a wide-area network. Unfortunately,the performance and scalability of the resulting system arelimited.

TCP/IP-level Virtualization: TCP/IP-based virtualiza-tion is for example used for Overlay Networks that provideapplication-level network virtualization among participatinghosts. An overlay network typically consists of hosts (phys-ical or virtual), routers, and tunnels that serve as virtuallinks between the hosts. Several overlay designs have beenintroduced in the literature: PlanetNet VNET [16, 4], X-Bone [20], Resilient Overlay Networks [2], and the JXTA

project [21]. The designs share the common goal of creat-ing a virtualized network layer with a customized topologymapped onto the actual physical infrastructure. They dif-fer in the underlying technology that enables the mapping,management of the technology, and the terminology used.

Overlay networks are most useful for implementing a vir-tual network topology on top of the physical topology. How-ever, they are not suitable for systems with strong separa-tion, isolation, and flow control requirements. As an exam-ple, although the PlanetLab VNET provides separation ofnetwork packets originating from different slices, the sepa-ration is merely enforced using the OS network services [4].Similarly in JXTA, peergroups are used to group networkpeers and enforce certain isolation properties [21]. However,it is the network administrator’s responsibility to enforceflow control policies across group boundaries as JXTA doesnot impose any specific flow control schemes for the sake offlexibility. Other shortcomings of overlay networks are com-plex management models, binary intra-group flow policies,and lack of inter-group flow control policies.

The VIOLIN project addresses a number of these deficien-cies and enhances the traditional TCP/IP overlay networksto create mutually isolated distributed environments [13,17]. The main idea is to provide each subsystem with avirtual IP world having its own address space. In particu-lar, a VIOLIN is created on top of an overlay network (suchas PlanetLab [4]) and consists of virtual hosts, switches,and routers. Communication between these entities is en-abled through a User-Mode Linux (UML) implementationenhanced with UDP-tunneling for inter-host communica-tion3. The VIOLIN model provides isolation between dif-ferent VIOLINs, which in turn enhances mobility throughlocation-independent addressing. Further, the model en-ables the customization of each VIOLIN with the desiredtechnology (e.g., IPv6) without requiring a global deploy-ment. A major disadvantage of VIOLIN is that the modelcompletely disallows inter-VIOLIN communication ratherthan adopting a policy-based flow control scheme. In prac-tice, it may be desirable for VIOLINs belonging to differentorganizations to interact with each other under certain flowcontrol policies enforced at each VIOLIN boundary.

Previous solutions also offered network virtualization schemesthat do not rely on overlay networking. Spawning networks,employ nested programmable networks to form a hierarchyof virtual networks that are isolated from each other [6, 7,14]. The main idea is to enable parent networks to spawnchild networks that utilize the parents’ resources. The childnetworks then may or may not choose to inherit certain char-acteristics from their parents. The advantages are that thechild networks can employ a specialized networking tech-nology (e.g., a mobile-IP network) while inheriting basicnetwork functionality from their parent. Further, they canspawn child networks of their own, forming a forest of net-works.

Spawning networks utilize the Genesis network kernel [14]that enables the life-cycle management of each spawned net-work including the spawning capability. The Genesis kernelis a complex virtual networking kernel that needs to be in-stalled on every physical domain that will potentially hostspawning networks. The major downside is that this re-quires major changes to the existing network infrastructure.

3A Xen-based solution has recently been introduced [18].

14

7. CONCLUSIONSIn this report we have presented and analysed three HPL

VNET technologies that we have successfully developed anddeployed in different application scenarios. We have assessedthe advantages and disadvantages of each approach and dis-cussed that the choice of which network virtualization tech-nology should be based on application and security require-ments, and also on how much control we have over the un-derlying infrastructure and what kind of features it provides.We have shown that our approaches show measurable differ-ences from a performance point of view. Further, they im-pose different restrictions on network applications and relyon different external capabilities in order to operate.

In the light of our findings, we deduce that an optimalsolution would interoperate multiple network virtualizationtechnologies in different settings. E.g., one could use MACRewriting or VLAN tagging within a LAN / data centerenvironment, but use EtherIP encapsulation in cases thepackets need to traverse a WAN link. Ideally, we want to beable to automatically deploy the underlying network virtu-alization approach based on a high-level specification thatdescribes networking and application needs.

We are currently investigating emerging networking andvirtualization technologies that we will employ to furtherenhance our approaches regarding performance, manage-ability and security. These enhancements include hardware-assisted virtualization and networking, advanced network re-source controlling, network access control mechanisms, andautomated configuration management of network equipment.

Acknowledgements

We thank our colleagues at IBM Zurich for collaboration insecure network virtualization within the OpenTC project [15].More information on this work can be found in [5].

8. REFERENCES[1] 802.1x: IEEE standard for local and metropolitan

networks — port-based network access control. IEEEStandards, 2004. Revision of 802.1X-2001.

[2] D. Andersen, H. Balakrishnan, F. Kaashoek, andR. Morris. Resilient Overlay Networks. In Proc. 18thACM Symposium on Operating Systems Principles(SOSP-2001), pages 131–145, New York, NY, USA,2001. ACM Press.

[3] P. T. Barham, B. Dragovic, K. Fraser, S. Hand, T. L.Harris, A. Ho, R. Neugebauer, I. Pratt, andA. Warfield. Xen and the Art of Virtualization. InProc. 19th ACM Symposium on Operating SystemsPrinciples (SOSP-2003), pages 164–177, October2003.

[4] A. Bavier, M. Bowman, B. Chun, D. Culler, S. Karlin,L. Peterson, T. Roscoe, and M. Wawrzoniak.Operating Systems Support for Planetary-ScaleNetwork Services. In Proc. 1st Symposium onNetworked Systems Design and Implementation(NSDI’04), San Francisco, CA, USA, 2004.

[5] Serdar Cabuk, Chris Dalton, HariGovind V.Ramasamy, and Matthias Schunter. Towardsautomated provisioning of secure virtualized networks.In Proc. 14th ACM Conference on Computer andCommunications Security (CCS-2007), October 2007.To appear.

[6] A. T. Campbell, M. E. Kounavis, D. A. Villela, J. B.Vincente, H. G. De Meet, K. Miki, and K. S.Kalaichelvan. Spawning Networks. IEEE Network,13(4):16–29, July-August 1999.

[7] A. T. Campbell, J. Vicente, and D. A. Villela.Managing Spawned Virtual Networks. In Proc. 1stInternational Working Conference on Active Networks(IWAN ’99), volume 1653 of LNCS, pages 249–261.Springer-Verlag, 1999.

[8] C. I. Dalton. Xen Virtualization and Security.Technical report, HP Security Office Report, August2005.

[9] R. Davoli. VDE: Virtual Distributed Ethernet. InProc. 1st International Conference on Testbeds andResearch Infrastructures for the Development ofNetworks and Communities (Tridentcom 2005), pages213–220. IEEE Press, February 2005.

[10] S. W. Hunter, N. C. Strole, D. W. Cosby, and D. M.Green. BladeCenter Networking. IBM Journal ofResearch and Development, 49(6), 2005.

[11] IEEE. Virtual Bridged Local Area Networks.Technical Report ISBN 0-7381-3662-X.

[12] IETF. EtherIP: Tunneling Ethernet Frames in IPDatagrams. RFC 3378.

[13] X. Jiang and D. Xu. VIOLIN: Virtual Internetworkingon OverLay INfrastructure. In Parallel and DistributedProcessing and Applications, volume 3358 of LNCS,pages 937–946. Springer-Verlag, Berlin, 2004.

[14] M. E. Kounavis, A. T. Campbell, S. Chou,F. Modoux, J. Vicente, and H. Zhuang. The GenesisKernel: A Programming System for SpawningNetwork Architectures. IEEE Journal on SelectedAreas in Communications, 19(3):511–526, March 2001.

[15] The Open Trusted Computing Project, 2006.Available from http://www.opentc.net/.

[16] L. Peterson, T. Anderson, D. Culler, and T. Roscoe. ABlueprint for Introducing Disruptive Technology intothe Internet. SIGCOMM Comput. Commun. Rev.,33(1):59–64, 2003.

[17] P. Ruth, X. Jiang, D. Xu, and S. Goasguen. VirtualDistributed Environments in a Shared Infrastructure.IEEE Computer, 38(5):63–69, May 2005.

[18] P. Ruth, J. Rhee, D. Xu, R. Kennell, and S. Goasguen.Autonomic Live Adaptation of Virtual ComputationalEnvironments in a Multi-Domain Infrastructure. InProc. IEEE International Conference on AutonomicComputing (ICAC-2006), June 2006.

[19] A. Sundararaj and P. Dinda. Towards VirtualNetworks for Virtual Machine Grid Computing. InProc. 3rd USENIX Conference on Virtual MachineTechnology (VM 04), pp. 177-190, 2004.

[20] J. Touch. Dynamic Internet Overlay Deployment andManagement using the X-bone. Computer Networks,36(2-3):117–135, 2001.

[21] B. Traversat, A. Arora, M. Abdelaziz, M. Doigou,C. Haywood, J-C. Hugly, E. Pouyoul, and B. Yaeger.Project JXTA 2.0 Super-Peer Virtual Network, 2003.http://www.jxta.org/project/www/docs/JXTA2.

0protocols1.pdf.

15