Embed Size (px)
Citation preview
Dell EqualLogic Best Practices Series
Dell EqualLogic PS Series Reference Architecture for Cisco Nexus 5020 Two-Switch SAN Architecture
Storage Infrastructure and Solutions Engineering Dell Product Group December 2011
SWRA1013 Dell EqualLogic PS Series Reference Architecture for Cisco Nexus 5020 i
THIS WHITE PAPER IS FOR INFORMATIONAL PURPOSES ONLY AND MAY CONTAIN TYPOGRAPHICAL
ERRORS AND TECHNICAL INACCURACIES. THE CONTENT IS PROVIDED AS IS, WITHOUT EXPRESS
OR IMPLIED WARRANTIES OF ANY KIND.
© 2011 Dell Inc. All rights reserved. Reproduction of this material in any manner whatsoever without
the express written permission of Dell Inc. is strictly forbidden. For more information, contact Dell.
Dell, the DELL logo, and the DELL badge, PowerConnect™, EqualLogic™, PowerEdge™ and
PowerVault™ are trademarks of Dell Inc. Broadcom® is a registered trademark of Broadcom
Corporation. Cisco® and Cisco Nexus® are registered trademarks of Cisco Systems, Inc. and/or its
affiliates in the U.S. and other countries. Intel® is a registered trademark of Intel Corporation in the
U.S. and other countries. Microsoft®, Windows®, Windows Server®, and Active Directory® are either
trademarks or registered trademarks of Microsoft Corporation in the United States and/or other
countries.
SWRA1013 Dell EqualLogic PS Series Reference Architecture for Cisco Nexus 5020 ii
Table of Contents
1 Introduction ....................................................................................................................................................... 2
2 Objective ............................................................................................................................................................. 3
3 Conclusions........................................................................................................................................................ 4
4 Reference architecture..................................................................................................................................... 5
4.1 Reference architecture overview ........................................................................................................... 5
4.2 Server configuration .................................................................................................................................. 7
4.3 Array configuration .................................................................................................................................... 7
4.4 Switch configuration .................................................................................................................................. 7
4.4.1 Overview of the switch configuration ............................................................................................ 7
4.4.2 Global Switch Setting: Configure no-drop class and jumbo frames ....................................... 8
4.4.3 Array and server port configuration ............................................................................................... 9
4.4.4 Link Aggregation Group configuration ........................................................................................ 10
Appendix A Switch port mapping ...................................................................................................................13
Appendix B TCP/IP configuration .................................................................................................................. 14
SWRA1013 Dell EqualLogic PS Series Reference Architecture for Cisco Nexus 5020 1
Acknowledgements
This whitepaper was produced by the PG Storage Infrastructure and Solutions team between October
2011 and December 2011 at the Dell Labs facility in Round Rock, Texas.
The team that created this whitepaper:
Ron Bellomio, Tony Ansley, and Margaret Boeneke
Feedback
We encourage readers of this publication to provide feedback on the quality and usefulness of this
information. You can submit feedback to this email address:
SWRA1013 Dell EqualLogic PS Series Reference Architecture for Cisco Nexus 5020 2
1 Introduction The Cisco Nexus® 5020 is a two-rack-unit (2RU), 10 Gigabit Ethernet, Fibre Channel over Ethernet
(FCoE), and Fibre Channel switch built to provide 1.04 terabits per second (Tbps) throughput with very
low latency. It has 40 fixed 10 Gigabit Ethernet and FCoE Small Form-Factor Pluggable Plus (SFP+)
ports. The hardware of the first 16 fixed ports supports both 10 Gigabit Ethernet and Gigabit Ethernet,
providing a smooth migration path to 10 Gigabit Ethernet. Two expansion module slots can be
configured to support up to 12 additional 10 Gigabit Ethernet and FCoE SFP+ ports, up to 16 Fibre
Channel switch ports, or a combination of both. The switch has a serial console port and an out-of-
band 10/100/1000-Mbps Ethernet management port. The switch is powered by 1+1 redundant, hot-
pluggable power supplies and 4+1 redundant, hot-pluggable fan modules to provide highly reliable
front-to-back cooling.
SWRA1013 Dell EqualLogic PS Series Reference Architecture for Cisco Nexus 5020 3
2 Objective This document provides details on configuring the Cisco Nexus 5020 switch for use with Dell™ EqualLogic™ PS Series storage arrays. The goal of this exercise is not to provide a comprehensive set of possible configurations, but to illustrate one possible solution that provides excellent performance and scalability as validated by testing in our labs. Note: The Cisco Nexus 5020 switch is targeted at converged DCB environments, but the switch does
not fully support the IEEE DCB standards for iSCSI over DCB. Because of this, this reference
architecture is focused on using the Cisco Nexus 5020 switch as a non-DCB Ethernet network
infrastructure.
The test objectives used while testing the Cisco Nexus 5020 switch configuration are defined below:
• Test the ability of the switch configuration to pass iSCSI traffic as defined by realistic
application workloads and server/storage configurations while meeting stringent networking
performance parameters.
• Determine the scalability behavior of the switch configuration for a standardized set of I/O
workloads and provide sizing guidance in terms of the number of storage arrays and servers
that can be supported by a SAN consisting of Cisco Nexus 5020 switches.
SWRA1013 Dell EqualLogic PS Series Reference Architecture for Cisco Nexus 5020 4
3 Conclusions Cisco Nexus 5020 switch passed switch validation objectives in a dual switch configuration with up to 12 arrays.
• With the Cisco Nexus 5020 switches, the SAN scaled easily to support 12 arrays and 12 hosts.
• Sequential Write performance scaled linearly to 97% of the theoretical baseline in terms of
throughput as measured at the host.
• Sequential Read Performance scaled linearly to 91% in terms of throughput as measured at the
host.
• The Random Read/Write performance in terms of IOs per second scaled to 88% of the
theoretical baseline on a per-server performance basis.
• TCP retransmissions from arrays, as polled periodically from array counters and SAN
Headquarters (SANHQ) were low (< 0.1%) across all test configurations. This is another
indicator that there are no bottlenecks or design issues within the switch that limited the ability
of the switch to support the reference architecture.
• Recommended number of arrays is up to 12+ arrays in a dual switch configuration due to port count limits.
In conclusion, our test results indicate that the only limitation this reference architecture has with
respect to scaling an EqualLogic SAN is in the number of available ports that two switches provide, not
with any limitations in the performance of the switch.
SWRA1013 Dell EqualLogic PS Series Reference Architecture for Cisco Nexus 5020 5
4 Reference architecture Because our goal is to help you deploy a switch-based SAN easily and quickly, we use a standard host
configuration and a standard EqualLogic Group configuration, accept default switch settings wherever
possible, and employ all accepted best-practice recommendations for EqualLogic SANs.
When developing this reference architecture, we used the following guidelines:
• The Cisco Nexus 5020 switch is targeted at converged DCB environments, but the switch
does not fully support the IEEE DCB standards for iSCSI over DCB. Because of this, this
reference architecture is focused on using the Cisco Nexus 5020 switch as a non-DCB
Ethernet network infrastructure.
• All hosts have two iSCSI Ethernet ports attached to the SAN.
• All NICs are configured based on the default, “out of the box” settings where possible.
o The exceptions are the use of Jumbo Frames and Flow Control, both of which are
enabled for all testing.
• Microsoft® Windows Server® is the operating system used for all hosts.
• We use EqualLogic Host Integration Toolkit for all hosts. In particular, we use the MPIO Device
Specific Module to provide EqualLogic-aware multi-pathing.
• Host connections to the SAN are equal to the number of active array ports connected to the
SAN.
o Since each PS6x10 series array has two active array ports, for each array in the test
configuration there is one host connected to the SAN.
• All SAN infrastructures are single subnet, non-routed designs.
• All SAN infrastructures are based on standard IPv4 addressing.
• All testing includes three pre-defined standardized workloads that reflect various types of real-
world SAN utilization.
Note: For more information on EqualLogic SAN design common practice recommendations, consult
the EqualLogic Configuration Guide that can be found
at www.delltechcenter.com/page/equallogic+configuration+guide
4.1 Reference architecture overview The reference architecture described in this paper consists of two Cisco Nexus 5020 switches.
Connected to these switches are a total of 12 PS6010XV arrays and 12 PowerEdge servers, which make
use of most of the fixed ports available on the two Cisco Nexus 5020 switches.
Figure 1 shows the two switch/12 array SAN test configuration.
SWRA1013 Dell EqualLogic PS Series Reference Architecture for Cisco Nexus 5020 6
Figure 1 Two switch and 12 array SAN test configuration
SWRA1013 Dell EqualLogic PS Series Reference Architecture for Cisco Nexus 5020 7
4.2 Server configuration This reference architecture uses 12 PowerEdge R610 rack servers. Each server is described in Table 1 below.
Table 1 Server configuration
Server specifications
Server Model PowerEdge™ R610
BIOS 2.1.9 Intel 5500-5520 Chipset A02 OS Windows Server 2008 R2 SP1
Host Converged Network Adapter (CNA) Model Intel x520 – Dual Port Advanced Network Services Driver Installed FCoE using Data Center Bridging Driver Not Installed iSCSI using Data Center Bridging Driver Installed iSCSI Initiator Microsoft Windows Server 2008R2
SP1 Intel x520 Provider: Intel
Version: 2.8.32.0 Date: 3/4/2011
MPIO Configuration Dell EqualLogic™ Host Integration Toolkit Version 3.5.1 Dell EqualLogic MPIO Device Specific Module Maximum Sessions per Slice: 2
Maximum Sessions per Volume: 6
4.3 Array configuration Twelve EqualLogic PS Series arrays were used in this test. Each server is described in Table 2 below.
Table 2 Array configuration information
EqualLogic storage Array Model EqualLogic PS6010XV Firmware 5.1.1 (R187334) (H1) Enabled performance load balancing in pools enabled
4.4 Switch configuration For the purpose of validating the switches for use with EqualLogic storage, the Cisco Nexus 5020
switch is configured using 36x 10 GbE ports for array and host connections and 4x10 GBe ports are
configured into a dynamic LAG using the IEEE Link Aggregation Control Protocol (LACP) standard for
inter-switch connectivity. Though this reference architecture does not use the included DCB features
of the Cisco Nexus 5020 switch, Cisco requires that flow control and jumbo frames must be
configured using the no-drop system class. There several steps to do this properly which are
described in detail in section 4.4.2.
Based on our analysis of the switch architecture and lab testing, the following subsections specify the
commands and settings recommended to configure the switches in this reference architecture.
4.4.1 Overview of the switch configuration Table 3 provides an overview of the settings used to configure the switches for this SAN when they are in a LAG.
SWRA1013 Dell EqualLogic PS Series Reference Architecture for Cisco Nexus 5020 8
Table 3 Switch settings used
Switch settings Switch Model Cisco Nexus 5020 Switch inter-connection Dynamic Link Aggregation Group (LACP – LAG)
Flow control enabled on each port channel group Enable jumbo frames policy
Global Switch Settings Enable spanning-tree on edge-port Individual Port Settings Enable jumbo frames policy
Set Layer 2 mode Enable Port Activate Flowcontrol
Switch Firmware 5.0(3)N2(2) Host-Switch Cable Type Cisco 10G Active SFP-H10GB-CU3M
3 meter cable Array-Switch Cable Type Array: SFP+ SR Optical Transceiver (DP/N 0N743D);
Switch: Cisco Nexus SFP SR Optical Transceiver (SFP-10G-SR, 10-2415-03) LC-LC Fiber Optic Cable
Switch-Switch Cable Type Switch: Cisco Nexus SFP SR Optical Transceiver (SFP-10G-SR, 10-2415-03) LC-LC Fiber Optic Cable
4.4.2 Global Switch Setting: Configure no-drop class and jumbo frames To get flow control to properly work on the individual ports, you must configure a “no-drop” class and also apply jumbo frames. The following CLI commands must be done on both switches.
1. Define qos class-map
sw1(config)# ip access-list all_traffic
sw1(config-acl)# permit ip any any
sw1(config)# class-map type qos class-nodrop
sw1(config-cmap-qos)# match access-group name all_traffic
2. Define qos policy-map
sw1(config)# policy-map type qos policy-qos
sw1(config-pmap-qos)# class type qos class-nodrop
sw1(config-pmap-c-qos)# set qos-group 2
3. Apply qos policy-map under system qos
sw1(config)# system qos
sw1(config-sys-qos)# service-policy type qos input policy-qos
4. Define network-qos Class-Map
sw1(config)# class-map type network-qos class-nodrop
SWRA1013 Dell EqualLogic PS Series Reference Architecture for Cisco Nexus 5020 9
sw1(config-cmap-nq)# match qos-group 2
5. Configure class-nodrop as no-drop class in network-qos policy-map and add jumbo frames
sw1(config)# policy-map type network-qos policy-nq
sw1(config-pmap-nq)# class type network-qos class-nodrop
sw1(config-pmap-nq-c)# mtu 9216
sw1(config-pmap-nq-c)# pause no-drop
6. Apply network-qos policy-map under system qos context
sw1(config-pmap-nq-c)# system qos
sw1(config-sys-qos)# service-policy type network-qos policy-nq
7. Configure bandwidth allocation using queuing policy-map
sw1(config)# class-map type queuing class-nodrop
sw1(config-cmap-qos)# match qos-group 2
sw1(config-cmap-qos)# policy-map type queuing policy-queuing
sw1(config-policy-c-que)# class type queuing class-default
sw1(config-policy-c-que# bandwidth percent 5
sw1(config-policy-c-que)# class type queuing class-fcoe
sw1(config-policy-c-que))# bandwidth percent 0
sw1(config-policy-c-que)# class type queuing class-nodrop
sw1(config-policy-c-que)# bandwidth percent 95
sw1(config-pmap-c-que)# system qos
sw1(config-sys-qos)# service-policy type queuing output policy-queuing
sw1(config-sys-qos)#
8. Enable PAUSE for all interfaces and PortChannel
sw1(config)interface e1/1
sw1(config-if)# flowcontrol send on
sw1(config-if)# flowcontrol receive om
4.4.3 Array and server port configuration All ports that connect to server or array interfaces are configured identically. For all of these ports,
flow control is enabled, Spanning-Tree Edge port is enabled, and the port is put in Layer 2 access
SWRA1013 Dell EqualLogic PS Series Reference Architecture for Cisco Nexus 5020 10
mode. For this reference architecture, switch ports 0 through port 36 on each switch were used for
either server or array connections.
The following commands were used to configure the array and server ports on each switch.
sw1# configure
Enter configuration commands one per line. To end, press <Ctrl> <z>.
sw1(config)# interface ethernet 1/1-36
sw1(config-if-range)# flowcontrol send on
sw1(config-if-range)# flowcontrol receive on
sw1(config-if-range)# spanning-tree port type edge default
sw1(config-if)# exit
sw1(config)# interface ethernet 1/1-36
sw1(config-if-range)# switchport mode access
sw1(config-if-range)# switchport access vlan 1
sw1(config-if-range)# do copy running-config startup-config
sw1(config-if-range)# exit
sw1(config)# exit
4.4.4 Link Aggregation Group configuration Inter-switch connections for this reference architecture leverage the Cisco proprietary Virtual Port
Channel (vPC) construct to create a peer link between the two switches. VPC is configured to use a
standard IEEE Link Aggregation Control Protocol (LACP) Link Aggregation Group (LAG). The LAG
consists of four 10 Gb links providing an aggregate bandwidth of 40 Gb between each switch.
The following commands configure the single LAG used to interconnect the two Cisco Nexus 5020
switches. The commands must be run on each switch.
1. Create a port channel
sw1# configure
Enter configuration commands one per line. To end, press <Ctrl> <z>.
sw1(config)# interface port-channel 1
sw1(config-if)# switchport
sw1(config-if)# exit
SWRA1013 Dell EqualLogic PS Series Reference Architecture for Cisco Nexus 5020 11
2. Add interfaces to port channel
sw1# configure
Enter configuration commands one per line. To end, press <Ctrl> <z>.
sw1(config)# feature lacp
sw1(config)# exit
sw1# configure
Enter configuration commands one per line. To end, press <Ctrl> <z>.
sw1(config)# interface ethernet 1/37-40
sw1(config-if-range)# channel-group 1 mode passive
sw1(config-if-range)# exit
sw1(config)# exit
3. Configure port channel
sw1# configure
Enter configuration commands one per line. To end, press <Ctrl> <z>.
sw1(config)# interface port-channel 1
sw1(config-if)# switchport mode trunk
sw1(config-if)# switchport trunk allowed vlan 1
sw1(config-if)# flowcontrol send on
sw1(config-if)# flowcontrol receive on
sw1(config-if)# exit
sw1# conf
4. CONFIGURE VPC (Virtual Port Channel)
sw1(config)# feature vpc
sw1(config)# vpc domain 1
sw1(config-vpc-domain)# peer-keepalive destination 192.168.7.252 (used 192.168.7.251 on sw2)
sw1(config-vpc-domain)# int ethernet 1/37-40
sw1(config-if-range)# channel-group 1 mode active
SWRA1013 Dell EqualLogic PS Series Reference Architecture for Cisco Nexus 5020 12
sw1(config-if-range)# int po1
sw1(config-if)# vpc peer-link
sw1(config-if)# switchport mode trunk
sw1(config-if)# switchport trunk allowed vlan 1
sw1(config-if)# exit
sw1(config)# exit
SWRA1013 Dell EqualLogic PS Series Reference Architecture for Cisco Nexus 5020 13
Appendix A Switch port mapping Figure 2 shows the switch port mappings for the Cisco Nexus 5020 switch.
Figure 2 Cisco Nexus 5020 switch port mapping
SWRA1013 Dell EqualLogic PS Series Reference Architecture for Cisco Nexus 5020 14
Appendix B TCP/IP configuration Table 4 shows the TCP/IP configuration for the arrays.
Table 4 TCP/IP configuration for the arrays
Array Port 0 Port 1
EQL01 192.168.1.11 192.168.1.21
EQL02 192.168.1.12 192.168.1.22
EQL03 192.168.1.13 192.168.1.23
EQL04 192.168.1.14 192.168.1.24
EQL05 192.168.1.15 192.168.1.25
EQL06 192.168.1.16 192.168.1.26
EQL07 192.168.1.17 192.168.1.27
EQL08 192.168.1.18 192.168.1.28
EQL09 192.168.1.19 192.168.1.29
EQL10 192.168.1.30 192.168.1.40
EQL11 192.168.1.31 192.168.1.41
EQL12 192.168.1.32 192.168.1.42
Table 5 shows the TCP/IP configuration for the hosts.
Table 5 TCP/IP configuration for the hosts
Server NIC 1 NIC 2
SVR01 192.168.1.210 192.168.1.211
SVR02 192.168.1.212 192.168.1.213
SVR03 192.168.1.214 192.168.1.215
SVR04 192.168.1.216 192.168.1.217
SVR05 192.168.1.218 192.168.1.219
SVR06 192.168.1.220 192.168.1.221
SVR07 192.168.1.222 192.168.1.223
SVR08 192.168.1.224 192.168.1.225
SVR09 192.168.1.226 192.168.1.227
SVR10 192.168.1.228 192.168.1.229
SVR11 192.168.1.230 192.168.1.231
SVR12 192.168.1.232 192.168.1.233
THIS WHITE PAPER IS FOR INFORMATIONAL PURPOSES ONLY, AND MAY CONTAIN TYPOGRAPHICAL ERRORS AND TECHNICAL INACCURACIES. THE CONTENT IS PROVIDED AS IS, WITHOUT EXPRESS OR IMPLIED WARRANTIES OF ANY KIND.
