© 2016 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 1 of 8

White Paper

Cisco Application Virtual Switch and VMware vSphere Distributed Switch Failover Convergence in Cisco Application Centric Infrastructure: A Cisco IT Case Study

Cisco’s Business Need for Migration

Cisco® IT is a large global IT organization with multiple data centers distributed throughout the world. The

infrastructure for each data center is huge: for example, Cisco’s Allen, Texas, data center includes more than 5000

applications, 8000 virtual machines, and 1700 Cisco Unified Computing System™

(Cisco UCS®) blades. As Cisco’s

data centers grow, quick and agile application deployment becomes increasingly challenging. Cisco IT realized the

solution to this challenge is to make the infrastructure application aware. So Cisco turned to Cisco Application

Centric Infrastructure (Cisco ACI™

).

Cisco ACI enables Cisco IT to use a common application-aware policy-based operating model across their physical

and virtual environments. This approach simplifies and accelerates the entire application deployment lifecycle.

Cisco IT hence is actively investigating mapping existing network elements and applications to the Cisco ACI

model and migrating its infrastructure entirely to Cisco ACI.

Cisco ACI Migration Options

The Cisco Application Policy Infrastructure Controller (APIC) integrates with a VMware-based virtual environment

by creating a distributed virtual switch mapped to the Cisco ACI environment in VMware vSphere vCenter. Cisco

ACI customers need to choose the distributed virtual switch that they want to use with Cisco ACI. Two options are

available: Cisco Application Virtual Switch (AVS) and VMware vSphere Distributed Switch (VDS).

AVS is a purpose-built virtual network-edge switch for Cisco ACI. It is based on the highly successful Cisco Nexus®

1000V Switch platform, which is the industry’s first and leading multihypervisor virtual switch. The Cisco Nexus

1000V has more than 10,000 customers and has been deployed by several large-scale service providers and large

enterprises, some with more than 20,000 virtual machines in production. AVS brings the Cisco ACI policy model to

virtual infrastructure, thus providing policy consistency across physical and virtual workloads. AVS tightly integrates

into Cisco ACI under APIC control, bringing networking features widely used in the physical world to the hypervisor

environment with better performance, greater scalability, and faster convergence than VDS.

Cisco IT tested both Cisco AVS and VMware VDS and choose Cisco AVS for its simplified management, greater

scalability, better performance, flexible deployment options, and faster convergence. Although AVS provides a

number of advantages, such as flexible deployment options using VLAN and Virtual Extensible LAN (VXLAN)

modes, detailed policy control for virtual workloads using microsegmentation, and greater security using the

stateful distributed firewall, this case study focuses its advantages for failover convergence.

© 2016 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 2 of 8

Cisco IT Requirements

AVS can use either VLAN or VXLAN encapsulation to forward traffic between the leaf switch and the VMware ESXi

host. Connectivity to the Cisco ACI leaf through VDS is only VLAN-based in the absence of VMware vShield

Manager. If VLAN encapsulation is used, each endpoint group (EPG) maps to a port group in the distributed virtual

switch and receives a user-friendly name and a VLAN ID. The Cisco ACI fabric is configured to translate VLAN

tags at the VMware port-group level into EPGs for policy application. If VXLAN encapsulation is used, AVS acts as

a VXLAN tunnel endpoint (VTEP) on the hypervisor, providing the capability to perform VXLAN tunneling over the

infrastructure VLAN between the leaf switch and the host. In VXLAN mode, only the infrastructure VLAN needs to

be defined on the network infrastructure (if any) between the host and the leaf, resulting in simpler configuration.

This approach is particularly useful in environments with blade servers in which blade switches lie between blade

servers and leaf switches. AVS in VXLAN mode thus offered Cisco IT more flexibility in deployment compared to

AVS in VLAN mode.

Although deployment flexibility is an important consideration, Cisco IT also has very strict criteria for failover

convergence in its infrastructure. For instance, network, computing, and storage IP downtime in various network

failover scenarios (using high availability) must be less than 5 seconds now, and less than 2 seconds within a year.

Hence, Cisco IT evaluated the two distributed virtual switch options, Cisco AVS in VXLAN mode and VMware VDS,

to see which would offer the best failover time compliant with requirements.

Test Topology

Cisco IT used the Cisco ACI computing and storage topologies shown in Figure 1 to measure failover times. Four

spine switches and six leaf switches were used in the Cisco ACI fabric. Three Cisco UCS domains were used, with

three tenants spread across them. Each Cisco UCS domain had two fabric interconnects. VDS and AVS both had

200 virtual machines at the network edge (DMZ) and 800 virtual machines in internal Virtual Routing and

Forwarding (VRFs) instances. Storage for the virtual machines was on separate network-attached storage (NAS)

servers and not on the Cisco UCS blades themselves. The topology was made highly available through port

channels and virtual port channels (vPCs) wherever applicable to facilitate measurement of failover time in various

scenarios.

© 2016 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 3 of 8

Figure 1. Test Topology

Test Methodology

The main objective of Cisco IT was to measure how long the network took to converge through high availability

when certain links in the Cisco ACI fabric were shut down and then brought up again. Continuous ping tests were

run on all the applicable virtual machines and logged during each scenario to accurately measure the packets lost

during each failover scenario. Packet loss was measured during both link shutdown and link startup. Secure Copy

(SCP) tests on the virtual machines were conducted in parallel to mimic storage read-and-write operations during

each failover scenario. Testing occurred separately for AVS and VDS hosted virtual machines. The following

failover scenarios were tested to measure the convergence.

© 2016 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 4 of 8

● Failure of a single port of a port channel to the fabric interconnect

● Failure of a single vPC to the fabric interconnect

© 2016 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 5 of 8

● Failure of vPC to the computing switch (fabric interconnect)

● Failure of a single switch to the computing switch (fabric interconnect): one on each vPC leaf pair, in pods 1

through 3 separately

© 2016 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 6 of 8

● Failure of a single Cisco UCS fabric interconnect (A or B): one on each Cisco UCS instance, in pods 1

through 3 separately

● Failure of a single border leaf

© 2016 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 7 of 8

● Failure of a single spine switch

Test Results

Table 1 shows the test results. A test case was marked pass only if it resulted in virtual machine downtime of less

than 2 seconds. These results clearly show the superiority of AVS compared to VDS in the following failover

scenarios:

● Failure of vPC to computing switch (fabric interconnect)

● Failure of a single Cisco UCS fabric interconnect (A or B): one on each Cisco UCS instance, in pods 1

through 3 separately

Table 1. Test Results

Test Case Result (AVS) Result (VDS)

Fail single port of a port-channel to fabric interconnect (FI) PASS PASS

Fail single port-channel of VPC to FI PASS PASS

Fail VPC to Compute (FI) PASS FAIL (> 60 seconds)

Fail single switch to Compute (FI) - one on each VPC leaf pair, pods 1-3 separately PASS PASS

Fail single UCS FI (A or B) - one on each UCS, pods 1-3 separately PASS FAIL (> 60 seconds)

Fail single border leaf PASS PASS

Fail single spine switch PASS PASS

Cisco AVS Benefits

A number of multicast, Internet Group Management Protocol (IGMP), and Address Resolution Protocol (ARP)

enhancements in AVS, particularly during failover scenarios, substantially reduced virtual machine downtime.

Some of these enhancements are listed here:

● AVS proactively sends IGMP join messages during failover rather than just replying to IGMP query

messages, improving convergence time.

© 2016 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 8 of 8

● AVS sends a gratuitous ARP (GARP) message during VXLAN load balancing failover, accelerating IP

learning and decreasing convergence time.

● AVS uses IGMPv3 instead of IGMPv2 for greater efficiency and reduced demands on the network.

● AVS allows 30 seconds for stabilization before adding links to an uplink port channel to help ensure reliable

behavior during failover recovery.

These enhancements helped ensure that Cisco IT not only met the current downtime criterion of 5 seconds but

also the future one of 2 seconds. Cisco IT decided to proceed with AVS in VXLAN mode instead of VDS in VLAN

mode. Cisco IT has completed migration of its Research Triangle Park (RTP), North Carolina, data center to AVS

and is on track with migrating others.

Conclusion

The testing discussed here of the two virtual switches clearly shows that Cisco AVS offers better failover

convergence than VMware VDS in at least two scenarios. AVS also offers the advantages of policy consistency

across physical and virtual workloads, tighter integration into Cisco ACI, incorporation of networking features widely

used in the physical world into the hypervisor environment, flexible deployment options using VLAN and VXLAN,

detailed policy control of virtual workloads using microsegmentation, greater security using the distributed firewall,

better performance, and greater scalability. AVS thus is an excellent choice for Cisco ACI customers as the virtual

switch for a Cisco ACI deployment.

For More Information

http://www.cisco.com/c/en/us/products/switches/application-virtual-switch/index.html

Printed in USA C11-736554-00 01/16