High Availability

1 © Copyright 2013 EMC Corporation. All rights reserved.

Twitter: @drjmetz

Designing High Availability in a Cisco Converged Networking Environment

J Metz, Ph.D Product Manager, Storage, Cisco Systems

Co-Sponsored by Intel®


Agenda

• High Availability

– How we got here

– For FC, for Ethernet, and for both

– Isolation and Redundancy

• HA Topologies

– Access-Layer

– End-To-End

– Future HA Topologies?


What is High-Availability (HA)?


What is "HA"?

• Redundancy


What is "HA"?

• Redundancy

• No Single Points of

Failure

– Control and

Switching Elements

have redundant

components


Why HA?

• Switches were small (8-16

ports) and not very powerful

• Easily have an outage

• Switch can go down

FC

SAN

(not drawn to scale)


Why HA?

• Meshes were created for adding

ports and redundancy

• Many of the ports (up to half)

were being used for ISLs

SAN

FC



Why HA?

• Moved to high-density, highly-

available Director Class systems

• 80% of storage environments use

Director Class Fibre Channel

Switches


SAN

FC


Why HA?

• Add separate Fabric

• SAN A/B introduced for redundancy

purposes

• Still have an outage if change control

procedures are not followed


SAN A SAN B

FC


Best Practices for High-Availability

Ethernet and Fibre Channel: Common Elements

Fibre Channel HA

Ethernet Storage HA


Best Practices for HA

• There are elements common to

both FC and Ethernet

– Director Class

• Redundant control plane,

switching, power, and cooling

– No single spoke network

topologies

• There are specific needs to

both Ethernet and FC


Network and Fabric

• Ethernet/IP

– Goal: Provide any-to-any connectivity

• Unaware of packet loss (“lossy”) – relies on

Upper Layer Protocols (ULPs) for retransmission

and windowing

• Provides the transport without worrying about the

services –

– Services provided by upper layers

• East-west vs. north-south traffic ratios are

undefined

• Network design has been optimized for

– High Availability from a transport perspective

by connecting nodes in mesh architectures

– Service HA is implemented separately

– Takes into account control protocol interaction

(STP, OSPF, EIGRP, L2/L3 boundary, etc…)

?

?

?

?

? ? ?

?

?

?

? ?

Switch Switch

Switch

?

Client/Server

relationships are not

pre-defined

? ?

?

Fabric topology and traffic flows are highly flexible


Network and Fabric

• Servers typically dual homed to two or more

access switches

• Redundant connections to the next layer

• Distribution and Core can be collapsed into

a single box

• L2/L3 boundary typically deployed in the

aggregation layer

– Spanning tree or advanced L2 technologies

(e.g., virtual link aggregation) used to prevent

loops within the L2 boundary

• Services deployed in the L2/L3 boundary of

the network (load-balancing, firewall, etc.)

L2

L3

Core

Aggregation

Access

Outside Data Center “cloud”

STP

STP


Network and Fabric

• Fibre Channel SAN

– Transport and Services are on the same layer

in the same devices

– Well-defined end device relationships (initiators

and targets)

– Does not tolerate packet drop – requires

lossless transport

– Only north-south traffic, east-west traffic mostly

irrelevant

• Network designs optimized for scale and

availability

– High availability of network services provided

through dual fabric architecture

– Edge/Core vs Edge/Core/Edge

– Service deployment Client/Server

Relationships are pre-defined

I(c)

I(c) T(s)

Fabric topology, services, and traffic flows are structured

T2

I5

I4 I3 I2

I1

I0

T1 T0

Switch Switch

Switch

DNS FSPF

Zone RSCN DNS

FSPF Zone

RSCN

DNS

Zone

FSPF

RSCN


Network and Fabric

• “Edge-Core” or “Edge-Core-Edge”

Topology

• Servers connect to the edge switches

• Storage devices connect to one or more core switches

• HA achieved in two physically separate, but identical,

redundant SAN fabric

• Very low oversubscription in the fabric (1:1 to 12:1)

• FLOGI scaling considerations

Fabric ‘A’ Fabric ‘B’

HBA

FC

Core

Edge



• FC Specific

– Air gap redundant fabrics

– Multipathing software resident on host

– Co-location of hosts and storage wherever possible


Co-Location • What is it?

– Ideally, keeping hosts and storage on

same switch

– "Zero-hop" topologies

• Limits

– Scaling

– In reality, may not be possible

• Physical location can be a limitation

• ISLs for distance extension

• Keep in mind when you lose a link or a

switch, the other fabric has to be able to

pick up that slack

– Design that into the system



• FC specific - Scale – VE_Ports (ISLs) allows

for greater scalability

– Can still have mesh

fabric

– Can still maintain HA



• HA Elements common to

both FC and Ethernet

– Single Spoke (non-HA)



• HA Elements common to both

FC and Ethernet

– Redundant connectivity (HA)



• Ethernet-specific (not used for block I/O)

– NAS (NFS/CIFS)

– Dual attached to the same network segment

– Active/Active teaming

– Smaller L2 domains



• iSCSI specific (used for block I/O)

– Dual Attached to DIFFERENT Network

segments*

– No teaming

– Avoid routers if possible*

* Best Practice


Oh, by the way... Server virtualization

Server virtualization considerations: How do you manage virtual switches?

Where is the switching performed Virtual switch, VEB, VEPA / VN_TAG Extending the L2 Domain over L3 constructs

Network virtualization Open vSwitch, VXLAN, etc. Bottom Line

If you have a redundant network, your virtual networks will also be redundant

Server and network virtualization will probably not have a big impact on basic HA network design


Isolation and Redundancy

$

$

Less Sharing

More Sharing

Failure Tolerance


Isolation and Redundancy

LAN SAN


From Ethernet POV

• How can you maintain Ethernet best practices of HA . . .

– NIC teaming

– Multichassis trunking

Ethernet Fabric

FC

FC


From a Storage POV

• . . . while still maintaining FC best practices

of HA?

– Isolated fabrics

– Multipathing

Ethernet Fabric

FC

FC


High Availability Topologies


Application Tiers

• How would you categorize your applications tiers today?

– By application type

– By server-to-server and/or server-storage bandwidth

• What is the growth you see for application bandwidth?

• Would you want different topologies / deployment models based on application type?

• What is the effect of server virtualization to the above questions?

Separate LAN / SAN SAN A / B w/ Converged

No SAN A / B


Topology 1: No Convergence

• Same topologies as existing

networks, but using Ethernet

switches for SANs

• Physical and Logical separation of

LAN and SAN traffic

• Additional Physical and Logical

separation of SAN fabrics

Example Use Cases: - Outsourced networks - Compliance requirements

HBA/CNA

L2

L3

NIC or CNA

Fabric ‘A’ Fabric ‘B’

FCoE

Value

SAN can utilize higher performance, higher density, lower cost Ethernet switches

Native Ethernet LAN Fibre Channel/ Fibre Channel over Ethernet SAN

Core

Aggregation

Access

Core

Edge

Isolation Convergence


Topology 2 (Converged at ToR)

• Consolidated the LAN Access and the SAN

Edge by using FCoE

• Physical and Logical separation

– LAN and SAN traffic at Aggregation Layer

– Additional Physical and Logical separation of

SAN fabrics

• Higher I/O, HA, fast re-convergence for host

LAN traffic

• The Unified Edge supports multiple LAN and

SAN topology options

– Virtualized Data Center LAN designs

– Fibre Channel edge with direct attached

initiators and targets

– Fibre Channel edge-core and edge-core-edge

designs

Converged

FCoE link

Dedicated

FCoE link

FC

Ethernet

Fabric ‘B’

L2

L3

CNA

Fabric ‘A’

FC FCoE


FC/FCoE Switch


Trunking and Channeling

• Switches operating in N_Port Virtualization mode(End-

Host mode for UCS) or FC-SW (Switch Mode for

UCS) Mode

• F-Port Trunking and Channeling on the links between

an NPV device and upstream switch (NP port ->

F_Port)

• F_Port Trunking: Better multiplexing of traffic using

shared links (multiple VSANs on a common link)

• F_Port Channeling: Better resiliency between NPV

edge and Director Core

– No host re-login needed per link failure

– No FSPF recalculation due to link failure

• Simplifies FC topology (single uplink from NPV device

to FC director

Fabric ‘A’ Supporting VSAN 20 & 40

VSAN 20, 40

Fabric ‘B’ Supporting VSAN 30 & 50

VF

VN

TF

TNP

Server ‘1’ VSAN 20 & 30

Server ‘2’ VSAN 40 & 50

NPV TOR Switch

VLAN 10,50 VLAN 10,30


VSAN 30,50

VLAN 10,20

VLAN 10,40

FC/FCoE Switch

With Intel® Xeon® processor


Trunking/Channeling with UCS • More flexibility in engineering FC traffic vs. 1 VSAN per uplink

– Aggregate Uplinks transparent to host Multi-path drivers

– Requires EMC Connectrix MDS or N5K to Work (both

features)

• Provide isolation to SAN traffic over the same physical link

– Help consolidate Infrastructure

– vHBAs can be on different VSANs

• All VSANs will be trunked on every uplink FC/FCoE port

– Selecting a subset of VSANs for individual uplink ports not

supported

• Scalability: Max of 32 VSANs per UCS system

• VSAN trunking supported in NPV and FC Switch mode FI

operation

• VSAN Trunking is not available for direct connect FC/FCoE

Storage Port types

vFCs VSAN 100

VSAN 300

VSAN 200

VSAN 400

SAN A SAN B



Topology 3 (Fabric/Completely Converged)

• LAN and SAN traffic share physical

switches and traffic uses dedicated

links between switches

• All Access and Aggregation

switches are FCoE FCF switches

• Improved HA, load sharing and

scale for LAN vs. traditional STP

topologies

VE

Fabric ‘B’

LAN/SAN

Converged

FCoE link Dedicated

FCoE link

FC

Ethernet

Fabric ‘A’



UCS iSCSI and Appliance Port Redundancy

• Host Multi-pathing drivers are used in lieu

of link aggregation network technology

• MS does not support using s/w iSCSI and

port channels for iSCSI failover

• Best practice is to use MPIO drivers

• Failure semantics look like FC in this

regard

UCS B-Series

UCS FI UCS FI

Storage

FCoE iSCSI NFS CIFS

Unified Appliance Port



Summary

• Highly Available network topologies

continue to be a requirement for today’s

data center environments

• Converging all protocols onto a single set

of physical links while maintaining isolation

can be managed

• Several converged topologies are

available that allow for various degrees of

isolation

Visit Cisco Booth 401

• Twitter: @ciscoDC

• Facebook.com/CiscoDC

• Video: http://www.youtubecisco.com/datacenter

• Cisco blog: http://blogs.cisco.com/datacenter

• Slideshare: http://slideshare.com/CiscoDataCenter

In Collaboration with Intel®

Intel, the Intel logo, Xeon and Xeon inside are trademarks of Intel Corporation in the U.S. and other countries.

http://www.youtubecisco.com/datacenter

http://blogs.cisco.com/datacenter

http://slideshare.com/CiscoDataCenter

http://twitter.com/

Technology

High Availability