Identity: a data center network fabric to enableco-existence of identical addresses

Vijay Mann, Kalapriya Kannan, Anilkumar Vishnoi and Shivkumar KalyanaramanIBM Research - India

{vijamann,kalapriya,avishnoi,shivkumar-k}@in.ibm.com

Abstract—Seamless virtual machine (VM) mobility within andacross data centers brings its own set of problems. One of theseproblems is enabling co-existence of identical or overlappinglayer-2 and layer-3 addresses in a single data center network. Themotivation for this problem comes from a number of compellingscenarios. These include the need to backup and restore orreplicate multi-tier applications that comprise of multiple VMsfrom one data center to another or within the same datacenter. This requires significant network reconfiguration costsas IP addresses of replicated VMs may clash with other existingIP addresses in the data center or with other replicas of thesame VMs. Similarly, when multiple data centers need to beconsolidated through a single data center interconnect, theiraddress ranges may overlap. Lastly, cloud providers need toensure that various customers can backup and restore theirVMs which can have potentially conflicting addresses with othercustomers’ VMs without requiring time consuming networkreconfiguration efforts. In this paper, we present Identity - adata center network fabric that enables co-existence of hostsor VMs with identical layer 2 and layer 3 addresses. We usepseudo addresses to uniquely identify each host or VM andemploy address resolution and duplicate detection techniques toenable co-existence of hosts and VMs with identical addresses.We leverage the centralized programmable control plane offeredby OpenFlow and present the design and implementation of ourscheme in Mininet. We provide an experimental evaluation ofour scheme and validate that its average latency and throughputperformance is as good as a default setup.

I. INTRODUCTION

Enterprise computing is evolving at a rapid pace due toadvances in the virtualization technology. The biggest changeis around mobility of software resources. Entire softwarestacks in the form of virtual machines can now be moved,live (i.e., without a downtime) or offline (backup and restore),anywhere within the same data center or from one datacenter to another with relative ease. This evolving landscapebrings its own set of problems. One of these problems is theco-existence of identical or overlapping layer-2 and layer-3addresses within a single data center network. In the pre-virtualization or pre-cloud computing days, this problem waslargely ignored since there were no compelling use casesthat required such a facility. However, with more and moreenterprises looking towards adopting cloud computing, andconsolidation of multiple data centers, this requires a revisit.

The motivation for enabling co-existence of physical ma-chines (PMs) or virtual machines (VMs) with duplicate layer-2 and layer-3 address in a single data center network comes

from a number of compelling scenarios.

Backup/Restore of multi-tier applications. Enterprises spenda significant portion of their IT costs on configuration anddeployment of large multi-tier applications. Typically, multipleinstances of such applications are required due to a varietyof reasons. First, there are multiple environments (such asproduction, staging or test, dev) in an enterprise that need tohost identical (or scaled down) instances of these applications.Second, as enterprises grow and new acquisitions are madein different geographical locations, multiple geographicallydistributed but interconnected data centers emerge that need tohost identical applications. For both these scenarios, significantredeployment and reconfiguration costs can be saved if thesame multi-tier application instance (comprising of severalvirtual machines) can be cloned or replicated. This practiceis still not wide-spread because the IP address range for theoriginal replicated instances might be completely differentfrom the new network where these replicas are being restored.Worse, these IP address ranges may be overlapping or identi-cal. Due to this, typically IP addresses need to be reconfiguredfor the restored instances. This effort is non-trivial as theseIP addresses are embedded at various places (various OS,middleware configuration files) in these virtual machines.Significant time and IT costs involved in this reconfigurationcan be saved if the network fabric can enable co-existence ofthese replicated VMs with identical addresses.

Data center consolidation. As mentioned above, as en-terprises grow and new acquisitions are made in differentgeographical locations, multiple geographically distributeddata centers emerge. Since these data centers are designedindependently, they may have overlapping network addressranges. In order to interconnect these data centers as part ofa single network fabric, the network fabric must recognizethese overlapping addresses and allow them to co-exist andcommunicate with each other as much as possible.

Cloud on-boarding. A cloud provider must allow its cus-tomers to restore their backed up applications in the formof VMs without any change onto its network. These variouscustomer VMs may have IP addresses that clash with IPaddresses of other customers’ VMs or with the IP addressesused by the provider itself. The network fabric used by thecloud provider must enable co-existence of these VMs with978-1-4673-0298-2/12/$31.00 c© 2012 IEEE

identical addresses.

The majority of data center networks today have no sup-port for identical addresses. This results in either manual orautomated [1] network reconfiguration efforts of replicatedor restored VMs. Both have huge IT costs and limitations interms of wide spread applicability. Recent distributed virtualswitch based solutions such as VMware vNetwork distributedswitch [2] and Cisco Nexus 1000V [3] and research effortssuch as Violin [4] provide policy based VM connectivity inthe form of a virtual network and may enable co-existence ofoverlapping address ranges. However, these solutions assumea virtualized infrastructure and a largely homogeneous envi-ronment (e.g. all Intel machines running similar version ofVMware hypervisor, etc). In practice, data centers compriseof heterogeneous platforms (Intel machines with or withoutvarious version of VMware hypervisor, IBM PowerPC serverswith power hypervisor, etc) and a network fabric based solu-tion that handles this heterogeneity is desirable. Furthermore,the number of servers or hosts in a data center are at leastan order of magnitude more than the number of networkswitches. Hence, the problems associated with manageabilityand scalability of a host based solution may make it a lesspreferred option. Finally, both approaches can be viewed ascomplementary and can coexist since there will be scenarioswhere end users may prefer a host based solution over a switchbased solution and vice versa.

In this context, this paper presents Identity: a data center net-work fabric that enables co-existence of physical machines orvirtual machines with identical layer 2 and layer 3 addresses.We make the following contributions:

1) We present the design and architecture of our schemefor enabling co-existence of identical layer 2 and layer3 addresses in a single network.

2) We use pseudo addresses, similar to the ones proposedin prior research [5] [6], to uniquely identify each hostor VM and employ address resolution and duplicate de-tection techniques in conjunction to enable co-existenceof hosts and VMs with identical addresses.

3) We leverage the centralized programmable control planeoffered by OpenFlow [7] and present the design andimplementation of our scheme in Mininet [8] - anOpenFlow platform for rapid prototyping of softwaredefined networks.

4) We present an extension to our scheme that automat-ically detects duplicate addresses and assigns privatenetwork identifiers to new hosts or VMs that join thenetwork.

5) We provide an experimental evaluation of Identity andvalidate that its average latency and throughput perfor-mance is as good as the default network fabric. We alsovalidate that the overheads of Identity are limited toconnection and flow setup.

The rest of this paper is organized as follows. We givean overview of related research in Section II. Section IIIpresents a description of the problem and gives an overview of

the key technical challenges. Section IV describes the designand architecture of our scheme. We provide an experimentalevaluation of our scheme in Section V. Section VI describesan extension to our scheme to automatically detect hosts orVMs with duplicate addresses and automatically assign privatenetwork identifiers to them. Finally, we conclude the paper inSection VII.

II. BACKGROUND AND RELATED WORK

In this section, we give some background and an overviewof related research.

A. Address resolution using ARP

ARP (or Address Resolution Protocol) is used for resolutionof network layer (e.g. IP) addresses into link layer (e.g.MAC) addresses during packet transmissions from one host toanother. For data center networks using Ethernet, ARP requestsand responses become the payload of ethernet packets. Whena host, A , needs to send a packet to another host, B (knownby its IP IP-B) it first checks its local ARP cache to seeif the corresponding link layer (MAC) address for host Bexists. If it exists in the ARP cache, it uses it to construct thepacket. If it does not exist in its local cache, it sends an ARPbroadcast request of the form “Who has IP-B”. This requestgets broadcast to all hosts that are on the same network (i.e.same subnet) as host A. If host B exists on the same network,it receives the ARP request and responds with a unicast ARPresponse specifying its link layer (MAC) address in the ARPresponse payload. Identity uses ARP redirection along withheader rewriting for resolving duplicate layer 2 (MAC) andlayer 3 (IP) addresses.

An ARP announcement (or gratuitous ARP) is sent atsystem startup by operating systems. It contains the IP andMAC address of the sender and is used to update otherhosts’ mapping of a hardware address when the sender’s IPaddress of MAC address has changed. A gratuitous ARP isnot intended to solicit a reply; instead it updates any cachedentries in the ARP tables of other hosts that receive thepacket. Identity captures gratuitous ARPs and handles themsuch that hosts with identical addresses can coexist in differentprivate networks. Furthermore, in an extension to Identity(refer section VI), we use gratuitous ARPs to detect newhosts while automatically assigning private network identifiersto new hosts that join the network for the first time.

An ARP probe is an ARP request with an all zero senderIP address. According to IPv4 Address Conflict Detectionspecification (RFC 5227) [9], a host must check if an addressis already in use, by broadcasting ARP probe packets. Identitycontroller uses ARP probes to check how many hosts or VMsexist in the network with an identical IP address.

B. Location independence - Portland and VL2

Recent research [5], [6] has proposed several schemesfor location independence that enable any IP address to existanywhere in the data center network. This is typically achievedthrough an overlay network comprising of the application IP

addresses used for identifying applications, while routing hap-pens based on location IP addresses [5] or pseudo addresses[6]. VL2 [5] employs a host based approach where theyinstrument the operating system to capture packets and useIP-in-IP encapsulation to create an overlay network. Portland[6], on the other hand, employs a switch based approach anduses the abstraction of a large layer 2 network. It interceptsARP requests packets and redirects them to a network fabricmanager who responds with a pseudo identifier (pseudo MACor pseudo IP). All routing takes placed using these pseudoidentifiers and egress switches rewrite the ethernet headerreplacing pseudo MAC with a real MAC. However, both theseefforts are mainly focused on providing location independence(e.g. for VM migration), loop free and fault free routing anddo not attempt to solve the problem of overlapping or identicallayer 2 and layer 3 addresses.

The architecture and design of Identity draws its inspirationfrom Portland and extends some of the same concepts tosolve the problem of supporting identical layer 2 and layer 3addresses through the creation of private networks. We reusethe idea of using pseudo MAC for routing from Portland, aswell as its scheme for redirecting ARP requests to a centralcontroller. However, unlike Portland we use ARP requests forunique host identification by replacing the source MAC in anARP request with the source pseudo MAC. We also create andmaintain a mapping table at the controller that maps pseudoMACs with the private network identifier (PNID) of which aparticular host or VM is part of. This helps us create multipleprivate networks or overlays within a single network and eachprivate network comprises of machines or hosts with uniqueaddresses. We also employ duplicate detection schemes toautomatically assign private network identifiers.

III. KEY TECHNICAL CHALLENGES

To enable co-existence of hosts or VMs with identical layer2 and layer 3 addresses, there are many technical challengesthat need to be overcome.

Problem 1: Uniquely identifying a host or a VM. Sincethe network will comprise of hosts and VMs with identicallayer 2 and layer 3 addresses, the first challenge is to uniquelyidentify a host or a VM. This is done by assigning a pseudoidentifier (such as the pseudo MAC in Portland [6] or thelocation address in VL2 [5]) that is unique for each host orVM in the network and is used by the network elements forrouting packets. Since one of our requirements is to not touchthe end hosts, the assignment and maintenance of this uniqueidentifier must be handled by network elements (switches) ora network controller.

Problem 2: Configuration and addition of new hosts orVMs. As new hosts or VMs are added to the network, thenetwork fabric must quickly detect a duplicate address, assigna pseudo identifier and take steps to ensure that the newhost or VM is added to a private network within whichit can communicate without any ambiguity. This must be

done quickly before the newly added host or VM startscommunicating. It is desirable that the configuration requiredfor this should be automated as far as possible (“plug andplay”).

Problem 3: ARP Resolution. We must ensure that the ARPrequests from the source host are resolved unambiguouslyand with the MAC address of the intended destination. Asdiscussed in section II, when an IP packet is to be sent outfrom a host A to host B, the network stack in host A first sendsout a broadcast Address resolution protocol (ARP) request tofind out the MAC or Ethernet (layer 2) address of host Bgiven the IP (layer 3) address of host B. If host B resides inthe same network, it gets the ARP request and responds with aunicast ARP response to host A which contains host B’s MACaddress. However, if there are multiple hosts in the networkwith the same IP address as host B (lets say, host C and hostE), then they will also get the same ARP request broadcastmessage from host A and all of them will respond to hostA with an ARP response message containing their respectiveMAC addresses. If all the MAC addresses in ARP responsesreceived at host A are unique (i.e. only the IP addresses ofhost B, C and E are identical, but each host has its own MACaddress), then it results in ambiguity, since host A can notdecide which MAC should be used. On the other hand, if allMAC addresses in ARP responses received at host A are alsoidentical (i.e. both the IP and MAC addresses of host B, C andE are identical), then the intended destination of the packet cannot be identified at all.

Problem 4: Gratuitous ARP handling. Most operating sys-tems also send ARP announcements or “Gratuitous ARP” atsystem startup. A gratuitous ARP is an ARP announcementcontaining the IP and MAC address of the sender and is usedto update other hosts’ mapping of a hardware address when thesender’s IP address of MAC address has changed. A gratuitousARP is not intended to solicit a reply; instead it updates anycached entries in the ARP tables of other hosts that receive thepacket. For example, let us assume that a given host - Host Aboots up and sends a gratuitous ARP to all other hosts. It ispossible that a network card might have been changed on hostA (with a different MAC) or the IP address of host A mighthave been updated (followed by a reboot). Upon receiving agratuitous ARP, most hosts in the network will just update theircache and will not send any responses. If any of the other hostsin the network (lets say host B), detects that the IP addressis the same as the one in gratuitous ARP from host A, it willrespond with an ARP response with its own IP and MAC. Thiswill be detected as a conflict by the operating system on hostA, which might alert the user of an IP address conflict. Someoperating systems like Windows might just shutdown uponreceiving such a conflict [10]. We must ensure that gratuitousARP requests are handled appropriately since we will haveduplicate layer 2 and layer 3 addresses in the network bydesign.

� �

2

3

3

1

1

0

Switch

ID

10.0.2.16

10.0.1.16

10.0.2.1

10.0.2.16

10.0.2.1

10.0.1.16

IP

2

2

2

1

1

1

PNID

11:11:E3:EA:82:21

00:10:F2:EA:92:21

00:92:89:DA:11:43

00:26:92:FA:88:E2

00:11:A9:DA:82:00

00:19:B9:FA:88:E2

AMAC

00:01:02:0F:00:00

00:01:03:00:00:00

00:01:03:0F:00:00

00:00:01:0F:00:00

00:00:01:00:00:00

00:00:00:0F:00:00

PMAC

15

0

15

15

0

15

Port

TORTORTORTOR TORTOR TORTOR

� � �

Core SwitchCore

SwitchCore

SwitchCore

Switch

Agg. switchAgg.

switchAgg.

switchAgg.

switchAgg.

switchAgg.

switchAgg.

switchAgg.

switch

�

���������

�� � �� �

� �

�

��

�

��

�

��

�

��

��������

�����������������

��������

�����������������

��������

�����������������

��������

�����������������

��������

�����������������

��������

�����������������

Fig. 1. Identity Architecture - hosts with similar color or shades indicatehosts with duplicate addresses

IV. ARCHITECTURE AND DESIGN

We give an overview of our architecture in Figure 1. Itleverages the centralized programmable control plane providedby OpenFlow [7]. The architecture comprises of a networkcontroller which is connected to all the switches. The networkcontroller forms the centralized control plane of the networkand takes most decisions related to routing and forwardingof packets (the data plane). The network controllers have theability to install rules on all switches according to whichincoming packets can be modified or routed.

Identity has been implemented as a python applicationrunning on top of the NOX [11] OpenFlow controller. NOX isan event driven controller and applications running on top mayregister callbacks to get notified of various events. Identitycontroller registers callback for 3 types of events : arrivalof a packet (packet in), a switch joining the network (datap-ath join), and a switch leaving the network (datapath leave).

We now describe how the various technical challengesmentioned in the previous section are handled systematicallyby our architecture. We also give relevant code snippets inNOX.

A. Uniquely identifying a host or a VM

We reuse the idea of using a pseudo MAC [6] to uniquelyidentify every host and VM in the network. OpenFlow con-troller assigns a 48 bit Pseudo MAC (PMAC) to each host orvirtual machine of the form:

pod.position.port.vmidwhere pod (16 bits) is a POD identifier where the ToR

switch resides, position (8 bits) identifies the ToR switch’sposition within the POD, port (8 bits) is the port numberon the ToR switch to which the physical host is directlyconnected, and vmid (16 bits) is the VM identifier on aparticular host. This PMAC format embeds the hierarchical

network topology within the pseudo identifier and is used forall routing.

The controller keeps a mapping of a VM’s (or a host’s)PMAC to its actual MAC address (AMAC), its IP address aswell a private network identifier (PNID) to which a particularhost or VM belongs. VMs or host within the same privatenetwork can communicate freely and the system ensures thatthere are no duplicate addresses within a private network.External hosts (outside the data center across the Internet)can also communicate with hosts within this private networkusing a gateway service which has a publicly known unique IPaddress and is configured to communicate with a given privatenetwork. The gateway can be one of the hosts or VMs withinthe private network itself which is assigned a unique publicIP in addition to its original address. By default all hosts andVMs are provided with a PNID of 0, which represents theentire data center network. Subsequent hosts with duplicateaddresses are assigned PNIDs in increasing order.

B. Addition of new hosts and address assignment

Pseudo MACs (PMACs) can be statically assigned at startupor the topology may be learnt using a protocol such asLocation Discovery Protocol (LDP) in Portland [6] or DAC(Data center Address Configuration) [12]. Since the overallnetwork topology (governed by network switches and theirinterconnect) changes very infrequently in enterprise datacenter network, we use static assignment. A mapping ofappropriate PMACs for each port for all the switches iscreated based on the known network topology. The last 16 bitsin the PMAC (representing the VM identifier) are staticallyassigned in increasing order to different VMs or hosts basedon their known actual MAC addresses. The mapping betweenPMAC, AMAC,IP, PNID, switchID, and switch port numberis statically defined and read by the Identity controller from aconfiguration file at startup. The python code for creating thecontroller tables is given in Figure 2.

# reads a configuration file mac_to_pmac that has the format# <Switch ID, AMAC, PMAC, Output Port,IP,PNID> and populates# the controller mapping table and ip tabledef create_controller_tables():

f = open(’./mac_to_pmac’,’r+’)mapping_table={}ip_table={}for line in f:

# parse each line for switchid,amac,pmac,outport,ip# and pnid......# popoulating the controller mapping table which is# hashed on pmacmapping_table[pmac] = [amac,ip,pnid,switchid,outport]# ip table is a table hashed on IPs and stores# pmac and pnid for duplicate IPs as a listif(ip in ip_table):

ip_table[ip].append([pmac,pnid])else:

ip_table[ip]=[[pmac,pnid]]f.close()

Fig. 2. Creating mapping table and ip table at the Identity controllerWe discuss an extension to our scheme that enables au-

tomatic generation of PNIDs and their mapping to PMACs,

AMACs, IPs, switchIDs and switch port numbers based onduplicate address detection in section VI.

C. ARP Resolution

Whenever a network switch joins the network, Identitynetwork controller gets notified. Identity network controller,in turn, consults its mapping table to find the list of all hostsand VMs that are supposed to be connected to that switch. Foreach of those hosts and VMs, it then generates and installs tworules on the switch:

• Rule 1: This rule redirects all ARP requests (includinggratuitous ARP requests) that the switch gets on all itsinput ports from the connected hosts or VMs to thecontroller. Since the PMACs for all ports of a switchare in the controller mapping table, the controller alsospecifies in the rule that the source ethernet (AMAC)address of the ARP packet must be replaced with thesource PMAC. The python code for setting this rule isgiven in Figure 3.

# reads a configuration file arp.cfg that has the format# <Switch ID, Port Number, PMAC># and installs rules in the switch for# replaciing ethernet source address with source PMAC# in ARP requests and redirecting them to controllerdef install_flows_arpredirection(dpid):

f=open(’./arp.cfg’,’r’)val=str(dpid)for line in f:

# if switch id matches, extract the corresponding# port_number and pmac......flow ={}#arp packets have EtherType 0x806flow[core.DL_TYPE]=0x0806flow[core.IN_PORT] = int(port_numaber)actions = [[openflow.OFPAT_SET_DL_SRC,pmac],[openflow.OFPAT_OUTPUT,

[1500, openflow.OFPP_CONTROLLER]]]inst.install_datapath_flow(dpid,flow,0,0,actions)

f.close()

Fig. 3. Rule 1 for redirecting ARP requests• Rule 2: This rule replaces the destination PMAC in an

IP packet with the destination AMAC for all incomingpackets, so that it can be accepted by the destination hostor VM. The python code for setting this rule is given inFigure 4.

# installs rules in the switch for# replacing PMAC in the ethernet destination address# with AMAC and forwarding it to the correct output portdef install_flow_reversal_headerrewriting(dpid):

for pmac,value in mapping_table.iteritems():#switch id is at index 3, amac is at index 0,#output port is at index 4 for each pmac#in the controller mapping_tableif(value[3] == str(dpid)):

flow={}flow[core.DL_DST]=pmac

#IP packets have ether type 0x800flow[core.DL_TYPE]=0x0800

actions=[[openflow.OFPAT_SET_DL_DST,value[0]],[openflow.OFPAT_OUTPUT,[0,value[4]]]]

inst.install_datapath_flow(dpid,flow,0,0,actions)

Fig. 4. Rule 2 for replacing PMAC with AMAC for incoming IP packets

In addition to the above 2 rules, several other rules that arerequired to route a packet based on PMACs are also installedon the switch. Once these rules are installed on the switches,every time an ARP broadcast request is generated from a hostor a VM, it is redirected to the controller by the switch afterreplacing the source AMAC in ethernet header with the sourcePMAC. Upon receiving an ARP request, the controller checksthe PMAC of the source in the ethernet header. It, then consultsits mapping table, retrieves the PNID corresponding to thesource PMAC from its mapping table. It then chooses thedestination PMAC based on a combination of the destinationIP address and the source PNID. This resolves the issue ofidentifying the correct destination in presence of duplicate IPaddresses. The controller then creates an ARP response withthe correct destination PMAC. The actual code snippet forthese steps is shown in Figure 5.

def send_arp_reply(dip,reqpkt,inport):reqh = reqpkt.find(’arp’)#mapping table is hashed using pmac and#pnid is at index 2 of the returned list#ip table is hashed using ip and each entry#is a list of (pmac, pnid)pnid=0if array_to_octstr(reqpkt.src) in mapping_table:

pnid = mapping_table[array_to_octstr(reqpkt.src)][2]else:

print "src pmac not found - a new host"return

for ip in ip_table[ip_to_str(reqh.protodst)]:if(pnid == ip[1]):

dest_pmac = ip[0]if(dest_pmac == None):print "No matching dest_pmac found"return

replyh=arp()replyh.hwsrc = octstr_to_array(dest_pmac)replyh.hwdst = reqh.hwsrcreplyh.hwlen = reqh.hwlenreplyh.opcode = arp.REPLYreplyh.protolen = reqh.protolenreplyh.protosrc = reqh.protodstreplyh.protodst =reqh.protosrcreplypkt =ethernet()replypkt.set_payload(replyh)replypkt.type = ethernet.ARP_TYPE#Identity controller’s PMAC is set as the source#in ARP responsereplypkt.src = octstr_to_array("00:00:00:00:00:23")replypkt.dst = reqh.hwsrcinst.send_openflow_packet(dip,replypkt.tostring(),inport)

Fig. 5. Finding the destination PMAC from controller tables and respondingto an ARP request

Illustrative example:For the purpose of illustration, let us consider an example

where there are 3 VMs (VM1, VM2 and VM3) in the networkeach with their own unique IPs (IP-1, IP-2 and IP-3), uniquePMACs (PMAC-1, PMAC-2, PMAC-3) and AMACs (AMAC-1, AMAC-2, AMAC-3). Let us also assume that these 3 VMsare connected to a switch with ID=2 and on port numbers 1, 2and 3, respectively. These VMs will be given a default PNIDof 0. These VMs may be running a typical 3 tier applicationwith a web tier, an application tier, and DB tier. Let us assumethat each of these VMs is copied to create another instanceof this multi-tier application. These 3 replicated VMS (VM1’,

VM PMAC AMAC IP PNID Switch PortID

VM1 PMAC-1 AMAC-1 IP-1 0 2 1VM2 PMAC-2 AMAC-2 IP-2 0 2 2VM3 PMAC-3 AMAC-3 IP-3 0 2 3VM1’ PMAC-4 AMAC-1 IP-1 1 5 1VM2’ PMAC-5 AMAC-2 IP-2 1 5 2VM3’ PMAC-6 AMAC-3 IP-3 1 5 3

TABLE IMapping table at the controller

VM2’ and VM3’) will have the same IP and MAC addressesas the original 3 VMs. However they will have their ownunique PMACs (PMAC-4, PMAC-5, PMAC-6). Let us alsoassume that these 3 replicated VMs are connected to a switchwith ID=5, and on port numbers 1, 2 and 3, respectively. Thecontroller would have assigned a PNID of 1 to these replicatedVMs. At this point, the mapping table at the controller willbe as shown in table I (column 1 with VM names has beenadded only for the purpose of illustration). The first 3 linesrepresent the entries for the original 3 VMs whereas the last3 lines represent the entries for the replicated VMs.

When VM1’ attempts to send a packet to VM2’ for thefirst time, an ARP request (of the form “who has IP-2”) isgenerated and goes to switch with ID=5. The switch redirectsthe ARP request to the controller after replacing the sourceaddress in the ethernet header (AMAC-1) with the sourcePMAC (PMAC-4) (according to Rule 1 above). The controllerreceives the ARP request and checks the source address inthe ethernet header. Since the source address is PMAC-4, itconsults the mapping table and detects that the PNID is 1.It scans its mapping table to find out the PMAC addresscorresponding to IP-2 and PNID=1. Since PMAC-5 is thePMAC matching IP-2 and PNID=1, the controller respondswith an ARP response specifying the destination address in theARP payload as PMAC-5. VM1’ then creates a packet withdestination ethernet address as PMAC-5. The routing rulestake care of ensuring that packet reaches the correct switch(with SwitchID=5). Switch 5 receives the packet and replacesthe destination ethernet address back to AMAC-2 (accordingto Rule 2 above) and outputs it to the correct port (port 2).

One of the limitation of this architecture is that the hosts orVMs that have identical AMACs, can not be connected to thesame edge or ToR switch (since the ToR switch outputs basedon AMAC). This is not a problem for VMs that only have aduplicate IP address but a unique AMAC. One workaround forthis limitation is to use an OpenFlow enabled vswitch (such asopenVswitch [13]) in the hypervisor and install all OpenFlowrules in the vswitch instead of the ToR switch. This will movethis restriction to the hypervisor or host level (no VMs withduplicate AMACs on a host). However, this solution can notbe used for non-virtualized hosts and will require the hosts tohave support for an OpenFlow enabled vswitch.

D. Gratuitous ARP handling

As mentioned above, a gratuitous ARP is an ARP announce-ment, sent by the operating system on a host at boot time,containing the IP and MAC address of the sender and isused to update other hosts’ mapping of a hardware addresswhen the sender’s IP address of MAC address has changed.Identity needs to handle gratuitous ARPs for a couple ofreasons. First, it needs to ensure that gratuitous ARP is notsent as a broadcast to all hosts, since it will generate manyARP responses because of existence of duplicate IP addresses.Identity network fabric should broadcast the gratuitous ARPonly to hosts that reside within the same private network sothat they can update their ARP-cache. This ensures that allhosts within a private network do not have any stale ARPentries. It also helps in detecting any duplicate IP addresseswithin a private network.

Second, gratuitous ARP can also be used for detectingduplicate IP addresses when they appear for the first time in thenetwork, assign PMACs and assign private network identifiersautomatically. This is discussed in detail in section VI.

Gratuitous ARPs (like all ARPs) are redirected to Identitycontroller (according to Rule 1) with the source AMACreplaced with source PMAC in the ethernet packet header.Identity controller first detects a gratuitous ARP as follows:An ARP packet is detected as a gratuitous ARP if

• the ARP packet is a request, with the source IP addressthe same as destination IP address, and the destinationMAC is set to 0.

• the ARP packet is a reply, with both the source IP addressand MAC address the same as destination IP address andMAC address.

Note that for both the cases, the ethernet packet is a broadcast(i.e. ethernet destination is ff:ff:ff:ff:ff:ff)

After identifying an ARP packet to be a gratuitous ARP, thecontroller finds out the private network to which the host orVM sending the gratuitous APR belongs. This can be easilydetected using the source PMAC in the ethernet header ofgratuitous ARP packet and consulting the mapping table.

Identity controller, then creates a set of unicast gratuitousARP messages for all the hosts or VMs that share the sameprivate network (PNID). While creating these gratuitous APRmessages, the controller uses a couple of rules. First, it usesthe source PMAC in the ethernet packet for source addressand destination PMAC in the ethernet packet for destinationaddress so that the packet can be easily routed using PMACs.

Second, it uses the source PMAC as the source MACaddress in the ARP packet (payload of ethernet packet) andset the destination MAC to 0, and use the source IP forboth source and destination IP address. This helps in ensuringthat that the receiving hosts will detect the message as agratuitous ARP message and will update their ARP-cachewith the corresponding source PMAC and its IP. If the IPis duplicate within the private network, they can respond withan ARP response to the sender (the one who originally sendthe gratuitous ARP), thereby reporting the conflict.

TORTORTORTOR TORTOR TORTOR

Core SwitchCore

SwitchCore

SwitchCore

Switch

Agg. switch

Agg. switch

Agg. switch

Agg. switch

Agg. switch

Agg. switch

Agg. switch

Agg. switch

����� �����

� �� �

� � � � � � � �

����� ��������

Fig. 6. Network topology in Mininet: VMs with same color or shades indicateVMs belonging to a private network and share their IP addresses with VMsof different colors or shades

V. EXPERIMENTAL EVALUATION

In this section we evaluate the various overheads of usingIdentity. As mentioned earlier, we have implemented Identityas a NOX application that controls an OpenFlow network.For experimental evaluation, we simulated a network topologycomprising of 8 hosts and 4 ToR switches, 4 aggregateswitches and 2 core switches in Mininet [8] (refer Figure6.

Mininet uses process-based virtualization and networknamespaces to rapidly prototype OpenFlow networks. Sim-ulated hosts (as well as switches) are created as processes inseparate network namespaces. Connectivity between hosts andswitches, between various switches, and between switches isobtained using virtual interfaces and each host, switch and thecontroller has a virtual interface (half of a veth pair). We runthe Identity controller on top of NOX as a separate processoutside Mininet.

Identity installs two rules at each switch (when the switchjoins the network) for each host or VM. These rules are forARP redirection and header replacement and for replacing thedestination PMAC with AMAC for all incoming IP packets. Inaddition, rules are also preinstalled for routing packets basedon PMACs. The main overheads of Identity lie in

• redirecting the ARP requests to the controller,• replacing the source MAC in an ARP request with the

source PMAC, and• replacing the destination PMAC in an incoming IP packet

with the destination AMAC.We compare the performance of Identity network fabric

with a “Default” fabric version in which ARP requests areflooded as usual without any header rewriting, and no PMACsare being used. No duplicate IP or MAC addresses are usedand routing takes place based on preinstalled rules using IPaddresses.

We realize that Mininet provides only a simulation envi-ronment in software. However, we are really interested in therelative performance of Identity and the “Default” fabric. Any

Ping RTT variation

for first 4 packets

0.01

0.1

1

10

100

1000

10000

1 2 3 4

Packet number

RT

T i

n m

s

DefaultIdentityTime Spent in Identity Controller

Fig. 7. Variation of latency for initial ping packets - most of the overheadfor Identity is is in the initial couple of packets

relative performance difference between these two fabrics islikely to be less in a real testbed where the rules will be appliedin hardware in OpenFlow switches and the controller is likelyto run on a separate heavy duty machine.

We conducted 4 types of experiments to evaluate the relativeperformance of Identity and Default fabric.

Experiment 1: Flow setup overheadOur first experiment was to measure the overhead of flow

setup in Identity. This can be indirectly measured using asingle ping test, since this involves the time taken for ARPresolution. ARP resolution in Identity involves a header rewrit-ing step at the OpenFlow switch followed by a redirect to theIdentity controller. The Identity controller then searches itstables to find the correct destination PMAC, creates an ARPreply and sends its back. We measured the latency incurredby the first four packets of a ping request (using the defaultpacket size of 84 bytes, and interval time of 1 second) forIdentity and Default network fabrics. We also measured thetime spent in Identity Controller code for ARP resolution andsending an appropriate ARP reply. The results are shown inFigure 7.

The first two ping packets suffer huge delays (around 2000ms and 1000 ms, respectively) in Identity as compared tothe Default fabric (which is consistently around 1 ms orless). However, only a small portion of this (around 1.2ms) is being spent in the Identity controller code for ARPresolution and ARP reply, and that too only for the first ping.Most of this extra delay is probably a combination of thecommunication latency to the controller, additional time takenby NOX infrastructure to deliver an event to the application,time taken at the ingress switch for rule matching and rewritingthe ARP request header, and the time taken at the egress switchto rewrite the incoming IP packet header. However, after thefirst couple of packets, the latency drops to the Default fabriclevel.

In order to verify how much of the latency was due to ARPheader rewriting at the switch, and communication latency in

ARP performance

0.1

1

10

100

Average Time (ms) Max Time (ms) Std Dev (ms)

Metric

Mil

liseco

nd

sDefault 2 Hosts Identity 2 HostsDefault 4 Hosts Identity 4 HostsDefault 8 Hosts Identity 8 Hosts

Fig. 8. ARP performance - Even though Identity performs worse than Defaultas expected, ARP redirection and header rewriting only explains the 20-35ms of the latency delay in initial ping packets

redirecting ARP requests to Identity controller, we conducteda series of ARP performance tests using “ARPing” [14]. Weused ARPing to generate unicast ARP requests messages toanother host. Each one of these ARP requests will match withthe rule R1 at the ingress switch, which in turn, will replace thesource MAC with source PMAC in the ethernet header beforeredirecting it to the controller. The controller will search itstable for the destination PMAC, create an ARP response andsend it back. We conducted tests using 2 hosts, 4 hosts and 8hosts which resulted in 2, 4 and 8 simultaneous ARP flows,respectively. For each flow, ARP request count was set to 1000.The results are shown in Figure 8.

As expected, Identity’s ARP performance is worse thanDefault - both for average and max times. The average ARPresponse time for Identity was around 1.7 ms for all testswhich was around 1ms more than the average ARP responsetime for Default. The difference was even bigger for maximumARP response times, with maximum Identity ARP responsetime ranging from 5-35 ms as compared to 2-2.8 ms forDefault. However, even though Identity performs worse thanDefault as expected, ARP redirection and header rewritingonly explains the 20-35 ms of the latency delay in initial pingpackets.

Even though the ARP performance of Identity is worse thanDefault and the initial couple of ping packets incur a veryhigh latency, it is unlikely to affect the overall throughput andlatency of flows when millions of packets are being sent. Wetest this hypothesis by conducting more experiments at variousnetwork layers - at the IP layer using flood ping, at the TCPlayer using Iperf [15] and at the application layer using httperf[16]).

Experiment 2: Flood ping latency testsOur second experiment is to test the overall effect on latency

while using Identity when a large number of packets are beingsent from multiple hosts. We conducted flood ping tests (aflood ping output packets as fast as they come back or 100

Ping Average and

Max Latency

0.01

0.1

1

10

100

1000

Avg RTT (ms) Max RTT (ms) Std. Dev RTT (ms)

Metric

Mill

isec

onds

2 Hosts Default 2 Hosts Identity4 Hosts Default 4 Hosts Identity8 Hosts Default 8 Hosts Identity

Fig. 9. Average latency measurement - Identity performs as good as theDefault fabric in terms of average latency and variation in latency

Number of Default Identityhosts (Mbps) (Mbps)

2 331.8 334.84 243.4 236.98 136.8 128.9

TABLE IIAvailable TCP bandwidth measured by Iperf - Identity performs as good as

Default

times per second) for 2 minute intervals, using a packet sizeof 1024 bytes. We conducted tests using 2 hosts, 4 hostsand 8 hosts which resulted in 2, 4 and 8 simultaneous flows,respectively. The results are shown in Figure 9.

The average RTT values are very similar for Identity andDefault. The average RTT is reported to be around 0.05 ms forall the tests. Even the standard deviation values are also verysimilar, ranging between 0.011 (2 hosts, Default) to 0.387 ms(8 hosts, Identity). We counted the number of ARP packetsgenerated. There were only 40 ARP packets in a flow ofaround million packets. This explains the reason for Identity’sgood average performance. The max RTT values range from30 ms (2 hosts) to 178 ms (8 hosts) for Identity and from 9ms (2 hosts) to 252 ms (4 hosts) for Default. The max RTTvalues for these tests are not high as in experiment 1 as theARP resolution had already taken place before the start ofthis experiment. The flood ping latency tests reconfirmed thataverage latency of IP flows does not get affected much whenIdentity is being used, even though the first couple of packetsduring flow setup incur very high latency.

Experiment 3: Iperf bandwidth testsOur third experiment is to test the overall effect on available

bandwidth observed by TCP flows while using Identity. As inexperiment 2, we conducted Iperf tests using 2, 4 and 8 hostswhich resulted in 2, 4, and 8 simultaneous flows, respectively.The bandwidth numbers for these tests are given in table II.The TCP bandwidth values measured by Iperf for various testsare again very similar for both Identity and Default. The valuesare slightly lower for Identity (by around 7 Mbps) for 4 and

Httperf connection and

response times

0.1

1

10

100

Avg Connection Time (ms) Avg Response TimeMetric

Tim

e in

ms

Default - 700 conn Identity - 700 conn

Default - 1000 conn Identity - 1000 conn

Fig. 10. Application level latency reported by httperf - Identity preforms asgood as Default in terms of average response time and most of its overheadlies in TCP connection setup

8 hosts and it is slightly higher for Identity (by 3 Mbps) for 2hosts. Again, this test reconfirms, that TCP bandwidth is notaffected significantly while using Identity.

Experiment 4: Httperf web server testsOur fourth experiment is to test the effect of using Identity

on user applications such as web servers and their clients. Weinstalled the apache web server and ran it simultaneously ontwo hosts. We ran the httperf client driver on the two otherhosts and simultaneously stressed the two web severs. Wecreate a synthetic workload comprising of 5 different file sizes(1KB, 10KB, 30KB, 50KB and 70 KB) which were servedby the web server. Httperf client was configured to issue 5requests for each file size before moving on to the next file.

For our experiments, we used a request rate of 10 re-quests/second and varied the number of connections from 700to 10000. This resulted in almost 90% CPU utilization on ourMininet host. Each connection was used to send exactly onerequest, after which it was closed and a new connection wascreated. This helped us maintain the same request rate, whileincreasing the number of TCP connections initiated from ahost. TCP send buffer and recv buffer were set to 4096 and16384, respectively. The result for this experiment are shownin Figure 10.

The average response time reported by httperf is almost thesame for Identity and Default version (between 1 and 2 ms),indicating that Identity provides as good average performanceas the Default fabric. Most of the overhead for Identity is inthe TCP connection setup as shown by the average connec-tion setup time. Average connection setup time for Identityranged from 55 ms (for 700 connections) to 78 ms (for 1000connections). However, the average connection time decreasedwith increasing number of connections, which indicates thatthe high connection setup time was really high only for thefirst few attempts (that involved ARP resolution). The averageconnection setup time for Default fabric was around 1 ms forboth the tests.

Controller

Switch

Host

Receive arp packet

Extract ethernet.src

ethernet.src in mapping table

(hash of PMACs)Existing host

Nono

Send ARP Probebroadcast

Wait for ARPresponse

Compute PNIDRedirect Gratuitous

ARP

Rewrite header ethernet.src=PMAC

Application – generate Gratuitous ARP

Host – sendpacket with AMAC

Is known host (based on AMAC)

Receive packetsyes

yes

Generate PMAC

Install rule R1

(ARP redirection with

header rewriting on the

switch for this host)

Fig. 11. A flow chart explaining automatic PNID assignment and duplicatedetection

VI. AUTOMATED DUPLICATE DETECTION AND PNIDASSIGNMENT

In this section we present an extension to our scheme toautomatically detect hosts or VMs with duplicate addressesand assign private network identifiers to them.

As part of this extension, the Identity controller installsa rule Rule 1’ on all edge switches (ToRs) that redirect allARPs requests to the controller (without replacing the sourceMAC with source PMAC (as described in Rule 1 in IV-C)).A flow chart explaining the scheme is given in Figure 11.As new hosts or VMs are added to the network, they send agratuitous ARP at boot time (this can also be done manuallyusing “arping” command [14] if the VM is being restoredfrom a checkpointed state). The gratuitous ARP is a broadcastmessage and gets redirected to the controller because of thepreinstalled rule Rule 1’. The controller, upon receiving agratuitous ARP, retrieves the source address from the ethernetheader (which is an AMAC) and checks its mapping table(which is hashed using PMACs) as given in Figure 5. For newhosts, the AMAC will not be found in the mapping table andat this point the controller will send a broadcast ARP probeto all hosts specifying the IP address and the actual MAC(AMAC) address of the sender who generated the gratuitousARP.

The controller waits for a predetermined time interval(configured to 2 minutes) for any responses. If no ARPresponses are received, the controller assumes that there are noduplicates, and it assigns a private network identifier (PNID)0 to the new host. However, if one or more ARP responsesare received, it retrieves the source ethernet address (whichshould have been replaced with the source PMAC as per Rule1) from all the responses and detects the PNIDs correspondingto all the PMACs. It then generates a unique PNID (inmonotonically increasing fashion) for the new host.

Having determined the PNID for a new host, the controllerthen assigns the appropriate PMAC based on the switch ID,

and input port number of the switch where the gratuitous ARPgenerated. It also adds this new entry to its mapping table withthe PMAC, AMAC, PNID, Switch ID and Port information.

Using this time order, hosts or VMs that comprise onebacked up/restored application can be added one by one tothe network and they will all be assigned the same PNID(assuming that all of them either have a duplicate IP or aunique IP).

Once the controller adds a new entry to its mapping table(either for a new IP address or a duplicate one), it installsRule 1 on the switch for the new host. This rule will redirectall future ARP request from this host to the controller afterappropriate header rewriting (source AMAC is replaced withsource PMAC). If the number of rules on a switch is not aconcern, then this rule can be installed on all edge switches.This will help in cases, when an existing host or VM isdisconnected from one switch and moved to a different switchin the same network. Since the rule will replace AMAC withPMAC in the ethernet header of gratuitous ARP, the controllerwill be able to find the PMAC in its mapping table and willnot do duplicate detection.

One of the limitations of this scheme is that if any hostsare down or disconnected and a new host with an identicalIP address joins the network, it may be assigned a PNID thatis already being used (especially if the host that was downhappened to have the highest PNID for that IP).

VII. CONCLUSION

This paper presented Identity, a data center network fabricthat enables co-existence of hosts or VMs with identicaladdresses. We presented the design and architecture of ourscheme that uses pseudo addresses, similar to the ones pro-posed in prior research, to uniquely identify each host orVM and employs address resolution and duplicate detectiontechniques to enable co-existence of hosts and VMs withidentical addresses. We leverage the centralized programmablecontrol plane offered by OpenFlow and present the designand implementation of our scheme in Mininet. We presentedan experimental evaluation of Identity and validated that itsaverage latency and throughput performance at is as good asthe default network fabric. We also validate that the overheadsof Identity are limited to connection and flow setup.

We also presented an extension to our scheme that automat-ically detects duplicate addresses and assigns private networkidentifiers to new hosts or VMs that join the network. Thisextension still has some limitations on detecting duplicatehosts that are temporarily down and hence, it can not bereadily deployed. We are currently working on removing theserestrictions. We also plan to work on implementation of thisscheme on a hardware OpenFlow switch and experiment withreal hosts and VMs with duplicate addresses.

REFERENCES

[1] M. Sethi, K. Kannan, N. Sachindran, and M. Gupta, “Rapid Deploymentof SOA Solutions via Automated Image Replication and Reconfigura-tion,” in IEEE SCC, 2008.

[2] “VMware Vnetwork distributed switch,”https://www.vmware.com/products/vnetwork-distributed-switch/overview.html.

[3] “Cisco Nexus 1000V Series Switches,”http://www.cisco.com/en/US/prod/collateral/switches/ps9441/ps9902/data sheet c78492971.html.

[4] X. Jiang and D. Xu, “Violin: Virtual internetworking on overlay in-frastructure,” in International Symposium on Parallel and DistributedProcessing and Applications (ISPA), 2005.

[5] A. Greenberg, N. Jain, S. Kandula, C. Kim, P. Lahiri, D. Maltz, P. Patel,and S. Sengupta, “VL2: A Scalable and Flexible Data Center Network,”in ACM SIGCOMM, August 2009.

[6] R. Mysore, A. Pamboris, N. Farrington, N. Huang, P. Miri, S. Radhakr-ishnan, V. Subramanya, and A. Vahdat, “PortLand: A Scalable Fault-Tolerant Layer 2 Data Center Network Fabric,” in ACM SIGCOMM,August 2009.

[7] N. McKeon, T. Anderson, H. Balakrishnan, G. Parulkar, L. Peterson,J. Rexford, S. Shenker, and J. Turner, “OpenFlow: Enabling Innovationin Campus Networks,” in ACM SIGCOMM Computer CommunicationReview, April 2008.

[8] “Mininet: rapid prototyping for software defined networks,”http://yuba.stanford.edu/foswiki/bin/view/OpenFlow/Mininet.

[9] “RFC 5227 - IPv4 Address Conflict Detection,”http://tools.ietf.org/html/rfc5227.

[10] “Detection of duplicate IP addresses by Microsoft TCP/IP,”http://support.microsoft.com/kb/120599.

[11] N. Gude, T. Koponen, J. Pettit, B. Pfaff, M. Casado, N. McKeown,and S. Shenker, “NOX: Towards an Operating System for Networks,”in ACM SIGCOMM Computer Communication Review, July 2008.

[12] K. Chen, C. Guo, H. Wu, J. Yuan, Z. Feng, Y. Chen, S. Lu, andW. Wu, “Generic and Automatic Address Configuration for Data CenterNetworks,” in ACM SIGCOMM, August 2010.

[13] “Open vSwitch - An Open Virtual Switch,” http://www.openvswitch.org.[14] “ARPing command,” http://linux.about.com/library/cmd/

blcmdl8 arping.htm.[15] “Iperf Bandwidth Measurement Tool,”

http://sourceforge.net/projects/iperf/.[16] “Httperf Web Server Performance Tool,”

http://sourceforge.net/projects/httperf/.