Upload
domien
View
237
Download
5
Embed Size (px)
Citation preview
FlexFrame®
Version 5.3A/1.0A
Network Design and Configuration Guide
Edition September 2013 Document Version 1.0
Fujitsu Limited
© Copyright Fujitsu Technology Solutions 2013
FlexFrame® and PRIMERGY™ are trademarks or registered trademarks of Fujitsu Limited in
Japan and other countries.
SAP® and NetWeaver™ are trademarks or registered trademarks of SAP AG in Germany
and in several other countries
Linux® is a registered trademark of Linus Torvalds
SUSE® Linux is a registered trademark of Novell, Inc., in the United States and other coun-
tries
Oracle™ and Java™ are trademarks of ORACLE Corporation and/or its affiliates
Intel® and PXE® are registered trademarks of Intel Corporation in the United States and oth-
er countries
MaxDB® is a registered trademark of MySQL AB, Sweden
MySQL® is a registered trademark of MySQL AB, Sweden
NetApp® and the Network Appliance® logo are registered trademarks and Network Appli-
ance™ and Data ONTAP™ are trademarks of NetApp, Inc. in the U.S. and other countries.
EMC®, CLARiiON®, Symmetrix® and Celerra™ are trademarks or registered trademarks of
EMC Corporation in the United States and other countries
VMware®, ESX®, ESXi, VMware vCenter, VMware vSphere are registered trademarks or
trademarks of VMware, Inc. in the United States and/or other jurisdictions.
Ethernet® is a registered trademark of XEROX, Inc., Digital Equipment Corporation and Intel
Corporation
Windows® and Word® are registered trademarks of Microsoft Corporation
All other hardware and software names used are trademarks of their respective companies.
All rights, including rights of translation, reproduction by printing, copying or similar methods,
in part or in whole, are reserved.
Offenders will be liable for damages.
All rights, including rights created by patent grant or registration of a utility model or design,
are reserved.
Delivery subject to availability. Right of technical modification reserved.
Network Design and Configuration Guide
Contents
General product information and naming ...................................................................... 1
1 Introduction ..................................................................................................... 2 1.1 Purpose of this Document ................................................................................. 2 1.2 Notational Conventions ..................................................................................... 2 1.3 Document History .............................................................................................. 2 1.4 Related Documents ........................................................................................... 3
2 Concept and Design ........................................................................................ 5 2.1 Switch Groups ................................................................................................... 5 2.2 Network Speed .................................................................................................. 6 2.3 Network Connection .......................................................................................... 7 2.4 Linux Bonding .................................................................................................... 8 2.5 ESX NIC Teaming ............................................................................................. 9 2.6 NetApps Interface Grouping ............................................................................ 10 2.7 BX Link State Propagation .............................................................................. 10 2.8 Virtual LAN ...................................................................................................... 11 2.9 Uplink Connection ........................................................................................... 13 2.10 Client LAN Connection .................................................................................... 14
3 FlexFrame Network Versions ....................................................................... 17 3.1 Small Version .................................................................................................. 17 3.2 Directly Connected Versions ........................................................................... 17 3.3 Enterprise Version ........................................................................................... 21
4 Configuration Commands ............................................................................ 23 4.1 Linux Bonding Interface ................................................................................... 23 4.2 Linux VLAN Interface ...................................................................................... 24 4.3 NetApp Filer Configuration .............................................................................. 24 4.4 Network Switch Configuration ......................................................................... 25
5 Abbreviations ................................................................................................ 31
6 Glossary ......................................................................................................... 33
7 Index ............................................................................................................... 36
Network Design and Configuration Guide 1
General product information and naming
This document describes the network concepts of the solution FlexFrame®.
This solution is offered in two characteristics:
the FlexFrame® for SAP
® V5.3A solution
This is the successor of the well-known solution FlexFrame® for SAP
® which is
in the market for years with its latest version V5.2A.
the FlexFrame® Orchestrator V1.0A solution
This is a new approach to offer enhanced functionality and features (e.g. support
of SAP HANA, support of KVM as virtual machine) step by step and become
more and more independent from certain hardware and software components.
These additional new features compared to the traditional FlexFrame for SAP
features are available by special licenses only.
This documentation is for both characteristics:
all descriptions regarding FlexFrame Orchestrator have a corresponding notifica-
tion within their chapters
if there is no notification regarding FlexFrame Orchestrator the described func-
tionality is valid for both characteristics
The name FlexFrame® is a generic term for both "FlexFrame
® for SAP
®" (FF4S) and
"FlexFrame® Orchestrator" (FFO).
The trademark myAMC® is not being used any more. The brand myAMC
® will be re-
placed by the brand FlexFrame®, e.g. instead of myAMC Accounting we now speak
about FlexFrame Accounting.
The abbreviation FA for FlexFrame Autonomy now denotes the FlexFrame Agents
(Application Agents, Control Agents and the Frame Agent). The FA Suite now denotes
the FlexFrame Agents Suite.
Introduction Purpose of this Document
2 Network Design and Configuration Guide
1 Introduction
1.1 Purpose of this Document
This document describes the FlexFrame networking concept and design and possible
networking topologies.
The reader should be familiar with IP networking and needs basic knowledge about
Ethernet-based networking with switches and virtual LANs. Knowledge of network switch
configuration is recommended. The knowledge of the FlexFrame Installation Guide and
the FlexFrame Administration and Operation Guide would be an advantage.
1.2 Notational Conventions
The following conventions are used in this manual:
Additional information that should be observed.
Warning that must be observed.
fixed font Names of paths, files, commands, and system output.
<fixed font> Names of variables.
fixed font User input in command examples
(if applicable using <> with variables).
1.3 Document History
Document Version Changes Date
1.0 First Edition 2013-09-18
Related Documents Introduction
Network Design and Configuration Guide 3
1.4 Related Documents
FlexFrame® – Administration and Operation
FlexFrame® – HW Characteristics Quickguides
FlexFrame® – Installation and Configuration of LVM 1.0 Standard Edition
FlexFrame® – Installation Guide for SAP Solutions
FlexFrame® – Installation of a FlexFrame Environment
FlexFrame® – Management Tool
FlexFrame® – FlexFrame Agents Installation and Administration
FlexFrame® – FlexFrame Messenger Installation and Administration
FlexFrame® – FlexFrame LogAgent Installation and Administration
FlexFrame® – Network Design and Configuration Guide
FlexFrame® – Security Guide
FlexFrame® – Technical White Paper
FlexFrame® – Upgrading FlexFrame 5.1A or 5.2A to 5.3A/1.0A
ServerView Documentation
SUSE Linux Enterprise Server Documentation
Network Design and Configuration Guide 5
2 Concept and Design
The design goals for the FlexFrame network concept:
avoid single points of failure (SPOF)
avoid unnecessary complexity
are achieved by the following means:
use redundant network
use a lightweight design
use standard interfaces and configurations
keep interface configuration as identical as possible on all server systems
keep design flexible enough to meet the requirements from small to medium up
to enterprise sized installations
reduce count of needed interfaces per system
and results in the concept described above.
2.1 Switch Groups
Switch 2
Switch 1
SwitchGroup
LAN A
LAN B
Application Node
PORT a
Application Node
Blade Rack
PORT c PORT b
Control Node 2
PORT a
PORT c PORT b
LAN A
LAN B
Application Node
LAN A
LAN B
Control Node 1
LAN A
LAN B
NAS
LAN A
LAN B
The very simplified drawing above may visualize the general concept (only data connec-
tions are considered). The core of the design is the switch group which provides redun-
dant usable ports. If one member of the switch group fails the ports of the other members
Concept and Design
6 Network Design and Configuration Guide
are still working. Every end system and the uplink are using this redundancy as described
later. In this way a failure of a single member of the switch group is tolerated and allows
continuous operation.
FlexFrame distinguishes the following switch group types:
CAT3750-STACK
Two up to nine switches of the Cisco Catalyst 3750G, 3750E or 3750X switch family
are building a Cisco Catalyst switch stack. The switches are connected via Cisco
StackWise cabling and behave like a single switch. For more than 4 switches with
10GbE ports the StackWise cabling may be a bottleneck.
NEXUS5000-VPC
Exactly two switches (nexus50xx) of the Cisco Nexus 5000 Switch family are building
a Cisco Nexus vPC domain. For the vPC peer-link as a necessary special channel
between the switches FlexFrame configures two ports on each switch. For the vPC
peer-keepalive link as a necessary alternative Layer 3 connectivity between the
switches FlexFrame uses the mgmt0 interface IP addresses as recommended from
Cisco.
NEXUS5500-VPC
Exactly two switches (nexus55xx) of the Cisco Nexus 5000 Switch family are building
a Cisco Nexus vPC domain. For the vPC peer-link as a necessary special channel
between the switches FlexFrame configures two ports on each switch. For the vPC
peer-keepalive link as a necessary alternative Layer 3 connectivity between the
switches FlexFrame uses the mgmt0 interface IP addresses as recommended from
Cisco.
For further details about supported switches see the FlexFrame Support Matrix.
For further details about the switches see the Cisco manuals.
For further details about the purpose of the different switch group types see below.
2.2 Network Speed
FlexFrame supports network connections for data communication with the following net-
work speeds:
1Gbit/sec (1GbE)
10Gbit/sec (10GbE)
Pay attention to the fact that 10GbE connected servers may also request a 10GbE con-
nected NAS system and 10GbE uplinks between.
For further details about supported end systems see the FlexFrame Support Matrix.
Concept and Design
Network Design and Configuration Guide 7
2.3 Network Connection
Network connection is established by connecting ports of end systems to ports of network
devices which itself are connected to other network devices. The characteristics of con-
nected ports must be compatible and a proper connector must be used.
Cat5 or Cat5e cable are used to connect
RJ45 Service LAN ports of RX servers
RJ45 management blade ports of BX cabinets
to RJ45 10/100/1000 ethernet ports of switches of the Cisco Catalyst 3750 switch family.
Cat5e cable are used to connect
RJ45 1GbE LAN ports of RX servers
RJ45 1GbE LAN ports of NAS systems
RJ45 1GbE switch blade ports of BX cabinets
to RJ45 10/100/1000 ethernet ports of switches of the Cisco Catalyst 3750 switch family.
LC-LC fibre optic cable are used to connect
10GbE LC ports of RX servers
10GbE SFP+ modules plugged into RX servers
10GbE SFP+ modules plugged into switches of the Cisco Nexus 5000 switch family
10GbE SFP+ modules plugged into the Cisco Catalyst 3750x 10G module
to 10 GbE SFP+ modules plugged into switches of the Cisco Nexus 5000 switch family
or to 10 GbE SFP+ modules plugged into the Cisco Catalyst 3750x 10G module
or to connect
1GbE SFP modules plugged into switches of the Cisco Catalyst 3750 switch family
1GbE SFP modules plugged into dual speed ports of switches of the Cisco Nexus
5000 switch family
to 1GbE SFP modules plugged into switches of the Cisco Catalyst 3750 switch family.
LC-SC fibre optic cable are used to connect
10GbE LC ports of RX servers
10GbE SFP+ modules plugged into RX servers
10GbE SFP+ modules plugged into switches of the Cisco Nexus 5000 switch family
to 10 GbE ports of switches of the Cisco Catalyst 3750E switch family.
10GBASE-CU SFP+ cable (Twinax) are used to connect
10GbE ports of switches of the Cisco Nexus 5000 switch family used for peer link
to switches of the Cisco Nexus 5000 switch family.
Concept and Design
8 Network Design and Configuration Guide
2.4 Linux Bonding
Linux provides a mechanism called bonding which can aggregate several physical inter-
faces to a virtual interface (bond). A bond interface can be used like a usual interface and
is operational as long as at least one physical interface assigned to the bond is opera-
tional. Several bonding configurations are available.
Switch 2
Switch 1
active eth0
eth1
rack server or
server blade
bond0
active
When running Linux on a physical server (rack server or server blade) used as Applica-
tion Node FlexFrame aggregates 2 physical interfaces using adaptive load balancing
(mode 6) and failure detection relies solely on the link state provided by the adapter (MII
monitoring). This is an active/active method and has no requirements on the link partners
which can be switch blades or switch group switches. The requirement on the adapter to
be able to change the MAC address is fulfilled from all supported servers.
On the Control Nodes bonding mode 1 (active-backup) is used. No requirements have to
be fulfilled neither from the link partners nor from the adapter. Failure detection relies
solely on the link state provided by the adapter (MII monitoring).
Concept and Design
Network Design and Configuration Guide 9
2.5 ESX NIC Teaming
ESX provides a mechanism called NIC teaming which is used if multiple physical Ether-
net adapters are assigned to a single virtual switch (vSwitch). A vSwitch is operational as
long as at least one physical Ethernet adapter assigned to the vSwitch is operational.
Several NIC teaming policies are available and can be set on the port group level.
Switch 2
Switch 1
active vmnic0
vmnic1
rack server or
server blade
vSwitch active
FlexFrame assigns 2 physical Ethernet adapters in ESX default configuration to the
vSwitch and puts all these adapters in the active list for the FlexFrame generated port
groups. The adapters are used with load balancing based on the originating virtual Port
ID and failure detection relies solely on the link state provided by the adapter
This method has no requirements on the link partners which can be switch blades or
switch group switches.
Concept and Design
10 Network Design and Configuration Guide
2.6 NetApps Interface Grouping
NetApps Data ONTAP provides a mechanism called interface grouping which can aggre-
gate several physical interfaces to a virtual interface. An interface group can be used like
a usual interface and is operational as long as at least one physical interface assigned to
the group is operational. Several modes are available.
Switch 2
Switch 1
SwitchGroup
NetApp Filer
ifgrp
e0a
e0b
active
active
FlexFrame aggregates at least 2 physical interfaces into a dynamic multimode interface
group which is compliant with IEEE 802.3ad. All physical interfaces are active. The corre-
sponding ports on the switch group switches must build a channel also complaint with
IEEE 802.3ad (LACP).
2.7 BX Link State Propagation
BX switch blades provide a mechanism which allows associating several downstream
ports with an upstream port or channel for link state propagation. If the link of the up-
stream port or channel goes down the link of all associated downstream ports are also
forced to go down. In this way a bonding driver on a server blade will get aware of the
upstream problem and switches the traffic of concerned interfaces.
Concept and Design
Network Design and Configuration Guide 11
active
eth0
eth1
server blade
bond0 active
Switch 2
Switch 1
SwitchGroup BX chassis
Switchblade 1
Switchblade 2
FlexFrame associates all downstream ports of a switch blade and a single upstream
channel for link state propagation. The upstream channel is compliant with IEEE 802.3ad.
The corresponding ports on the switch group switches must also build a channel com-
plaint with IEEE 802.3ad (LACP). The channel consists of at least two ports with crosso-
ver cabling. More ports can be used for higher bandwidth.
2.8 Virtual LAN
Virtual LAN (VLAN) is a mechanism which allows having multiple separated networks on
the same physical network infrastructure. A widely used standard is tagged VLAN accord-
ing IEEE 802.1q. The protocol specifies an extension of the Ethernet header by a VLAN
identifier (tag). A packet belongs to the VLAN identified by the tag. Communication be-
tween VLANs is only possible by routing as is common for usual networks. For compati-
bility reason packets without a tag are also allowed. A switchport can be configured to
assign received packets without tag to a specific VLAN and remove the tag when a pack-
et of this VLAN should be send. The VLAN is then called native or untagged. End sys-
tems which access only one VLAN or cannot handle VLAN tags are connected this way.
End systems used within FlexFrame like NAS systems or servers running Linux are able
to handle VLAN traffic according IEEE 802.1q.
FlexFrame can handle different pools with independent networks. The traffic between
communication partner types (client – server, server – server and server – storage) is al-
so separated. Therefore VLANs are used as helpful mechanism to reduce the necessary
physical infrastructure and increase the flexibility with respect to the pooling concept.
According IEEE 802.1q the VLAN tag has a value range from 1 to 4094 but there are
switch specific restrictions.
Concept and Design
12 Network Design and Configuration Guide
Cisco Catalyst 3750 Switch Family
1005 VLANs are supported. Normal usable VLANs are in the range from 1 to 1001. VLAN
numbers 1002 through 1005 are reserved for Token Ring and FDDI VLANs. VLAN num-
bers 1006 through 4094 are extended-range VLANs and usable if not already used inter-
nally for routed ports.
Cisco Nexus 5000 Switch Family
507 VLANs are supported. Normal usable VLANs are in the range from 1 to 1005. VLAN numbers 1006 through 4094 are extended-range VLANs and usable with the following restrictions. The state is always active and VLANs 3968 to 4047 and 4094 are reserved for internal use.
Physical Network Layout & Virtual LANs per Pool
V1 V2 V4
eth 2 eth 1eth 0
IPMI
Onboard
LAN Port 3
Onboard
LAN Port 1
V1 V2 V4
Bond 0
eth 2 eth 1eth 0
IPMI
Control Node A (RX300S6) Control Node B (RX300S6)
Onboard
LAN Port 2Onboard
LAN Port 3
Onboard
LAN Port 1
Onboard
LAN Port 2
Cisco Switch BCisco Switch A Switchgroup
Control Nodes and Linux Application Nodes (RX300S6), both Storage Types
V3
V1
cge0 cge1
Data
Mover
trunk
EMC Celerra
Logical View: VLANs
VLANsV4: Client
V3: Control V2: Server
V1: Storage
Control
Station
V2 V4
eth 2 eth 1eth 0
IPMI
Application Node 1 (RX300S6)
Application Node n
Onboard
LAN Port 3
Onboard
LAN Port 1Onboard
LAN Port 2
Bond 0V1 V3
Bond: vif
eXa eXb
NetApp
Filer
Physical View:
Redundant Connections
Bond 0
The picture shows the physical and logical view of networking in FlexFrame with respect
to a single pool. That is:
An Application Node is assigned to a pool and connected to the pools client, server and
storage VLAN. When running on ESX server for every connection an interface is created
on the vSwitch, when running native the interfaces for client and server LAN are created
Concept and Design
Network Design and Configuration Guide 13
as VLAN interface on top of the bond. The storage LAN is also used for PXE boot and
assigned native because no tags can be handled in this early phase. Therefore the stor-
age LAN is accessible via the bond directly.
An ESX server is connected to all VLANs necessary for the Application Nodes and to the
control LAN which is assigned native.
IPMI interfaces of rack servers and management blades of BX cabinets are connected
native to the control LAN.
A Control Node is connected to all VLANs, the control LAN is native.
A NAS system is connected to all VLANs, all VLANs are tagged.
2.9 Uplink Connection
Switch 2
Switch 1
SwitchGroup
FlexFrame can assign uplinks to a switch group. An uplink consists of ports from different
switch group members building a channel and carries the traffic of all used VLANs. An
uplink is used to connect a switch group to another switch group or to the customer cor-
porate LAN. The supported topologies stick to the following rules:
A CAT3750-STACK has maximal one uplink and is connected to another
CAT3750-STACK or to a NEXUS5x00-VPC or to the customer corporate LAN.
A NEXUS5x00-VPC may have more than one uplink but maximal one of them is
connected to another NEXUS5x00-VPC or to the customer corporate LAN. Each
other uplink is connected to a CAT3750-STACK.
Concept and Design
14 Network Design and Configuration Guide
Following the above rules ensures a topology without loop. Nevertheless, the Cisco per
default enabled rapid per VLAN spanning tree (rapid-PVST) algorithm is used to avoid
loops which may appear e.g. in early phase when booting a switch or by misconfiguration
especially because FlexFrame does not check whether the rules are followed.
Examples for supported topologies are given in chapter 3.
When connected to customer corporate LAN the connected customer ports must also
build a channel. The channel must be compliant with IEEE 802.3ad (LACP) and all used
VLANs must be allowed tagged.
2.10 Client LAN Connection
For accessing a FlexFrame infrastructure solution connection to the client LAN must be
established. In general this means the client LAN must be available in the context of a
router and appropriate routing must be configured. This is a manual task and the custom-
er decides which network should have access to a special client LAN.
To bring a client LAN in the context of a router different ways are available:
a) If uplinks of switch groups are connected to the customer corporate LAN e.g. as
described in chapter 3.3 the client LANs are already available and no further ar-
rangements are necessary.
b) Special uplinks for client LAN connections are configured (e.g. see
ff_swgroup_adm.pl) and connected to the customer corporate LAN like normal
uplinks. The VLANs on these uplinks are restricted to client LAN VLANs.
c) Special ports for client LAN connections are configured. These CLAN ports are
thought being connected to a router and therefore configured as spanning tree
edge ports. The ports can be distributed over several switch groups and it is
possible to have particular ports for every client LAN. Only Ports of switches of a
CAT3750-STACK are allowed to be used as CLAN ports. See below an exam-
ple how CLAN ports configured for a single client LAN can be used.
Concept and Design
Network Design and Configuration Guide 15
Switch 2
Switch 1
SwitchGroup 1
Switch 2
Switch 1
SwitchGroup 2
uplink channel
Virtual Gateway Adress IP3
Layer 3 port IP1
Layer 3 port IP2
outside FlexFrame
The ports configured for client LAN connection are connected to Layer 3 ports of different
Layer 3 switches. An IP address is assigned to every Layer 3 port and the Layer 3
switches are configured to build a logical router e.g. via HSRP. The logical router sup-
ports a virtual IP address (IP3) which is used as standard gateway for the special client
LAN.
Network Design and Configuration Guide 17
3 FlexFrame Network Versions
The FlexFrame network design is flexible enough to meet requirements from small to en-
terprise size networks. In the following some examples are given.
3.1 Small Version
This version is based on one switch group as there is no need for
splitting FlexFrame into different data centers
providing more switch ports as one switch group can offer.
Switch 2
Switch 1
CAT3750-STACK
LAN A
LAN B Node
LAN A
LAN B Node
LAN A
LAN B Node
LAN A
LAN B Node
LAN A
LAN B Node
LAN A
LAN B Node
client LAN
All devices are connected to a single CAT3750-STACK. The connection of the devices is
shown simplified only with regard to the redundant connections for data communication.
For client LAN connection two ports are configured on different members of the switch
group. No uplink needs to be configured.
3.2 Directly Connected Versions
Topologies according to the rules mentioned in 2.9 with more than one switch group and
without any uplink connection to the customer corporate LAN are called directly connect-
ed versions.
The following version may meet the requirements if
FlexFrame has to be split into two data centers or
one switch group might have not enough ports for all devices
FlexFrame Network Versions
18 Network Design and Configuration Guide
Example1: Two CAT3750-STACKs
Switch 2
Switch 1
CAT3750- STACK
Switch 2
Switch 1
CAT3750- STACK
LAN A
LAN B Node
LAN A
LAN B Node
LAN A
LAN B Node
LAN A
LAN B Node
LAN A
LAN B Node
LAN A
LAN B Node
Uplink Channel
client LAN
All devices are connected to one of two CAT3750-STACKs. The connection of the devic-
es is shown simplified only with regard to the redundant connections for data communica-
tion. For client LAN connection one port is configured on every switch group.
The switch groups are connected via uplink channel a) directly without any other network Layer2 device between per copper or for
greater distance per fiber optic cable.
b) via network devices allowing a Layer2 connection e.g. DWDM for even greater
distance. In this case the network devices between must be transparent to the
switch groups, especially regarding link state. That means, if a port is shutdown
on one switch group the corresponding port on the other switch group must see
a link down event. Also if the connection between is broken both ports must see
a link down event.
FlexFrame Network Versions
Network Design and Configuration Guide 19
Example 2: single NEXUS5000-VPC with one CAT3750-STACK
Switch 2
Switch 1
NEXUS5000-VPC
Switch 2
Switch 1
CAT3750- STACK
LAN A
LAN B Node
LAN A
LAN B Node
LAN A
LAN B Node
Uplink Channel
client LAN
The CAT3750-STACK is directly connected to the NEXUS5000-VPC without any other
network Layer2 device between. The service LAN ports of rack servers and the control
nodes and the management ports of the nexus switches are connected to the CAT3750-
STACK. The end systems connected to NEXUS5000-VPC are 10GbE-connected. For
client LAN connection two ports are configured on different members of the CAT3750-
STACK.
FlexFrame Network Versions
20 Network Design and Configuration Guide
Example 3: two NEXUS5000-VPCs each with connection to one CAT3750-STACK
Switch 2
Switch 1
CAT3750-STACK
Switch 2
Switch 1
CAT3750-STACK
Switch 2
Switch 1
NEXUS5000-VPC
Switch 2
Switch 1
NEXUS5000-VPC LAN A
LAN B Node
LAN A
LAN B Node
LAN A
LAN B Node
LAN A
LAN B Node
LAN A
LAN B Node
LAN A
LAN B Node
uplink channel
client LAN
The service LAN ports of rack servers and the control nodes are connected to a
CAT3750-STACK which is connected to the NEXUS5000-VPC. The end systems con-
nected to NEXUS5000-VPC are 10GbE-connected.
The NEXUS5000-VPC switch groups are connected via uplink channel
directly without any other network Layer2 device between per fiber optic cable.
via network devices allowing a Layer2 connection e.g. DWDM for greater dis-
tance. In this case the network devices between must be transparent to the
switch groups, especially regarding link state. That means, if a port is shutdown
on one NEXUS5000-VPC switch group the corresponding port on the other
NEXUS5000-VPC switch group must see a link down event. Also if the connec-
tion between is broken both ports must see a link down event.
FlexFrame Network Versions
Network Design and Configuration Guide 21
3.3 Enterprise Version
The possibilities of directly connected versions are restricted. Greater configurations need
uplinks to customer corporate LAN e.g. the customer core switches.
Customer own Core
Switch
Customer own Core
Switch
Customer Corporate LAN
Node
Node
Node
Node
Node
Node
Node
CAT3750-STACK
Uplink Channel
NEXUS5000-VPC
NEXUS5000-VPC
Node
Uplink Channel Uplink Channel
CAT3750-STACK
CAT3750-STACK
Uplink Channel
Uplink Channel
Each NEXUS5000-VPC switch group is connected with its uplink ports to core switch
ports. CAT3750-STACK switch groups may be connected to core switches or to
NEXUS5000-VPC switch groups. The switch groups may be connected to different core
switches. Like the Uplink the corresponding ports on the core switch(es) must:
build a channel
carry all VLANs used within FlexFrame
transfer traffic without blocking
The core switches are not part of FlexFrame, neither of automated configuration nor sup-
port. Configuration changes at core switches have to be done manually. There are no
messages or instructions from FlexFrame to support this.
Network Design and Configuration Guide 23
4 Configuration Commands
As described in Chapter 2 FlexFrame uses high availability configurations concerning
network on many devices. This chapter describes how this is done in detail.
4.1 Linux Bonding Interface
On the control nodes the bonding configuration is performed via configuration files
/etc/sysconfig/network/ifcfg-bond0:
STARTMODE=onboot
BOOTPROTO=static
IPADDR=192.168.20.1
NETMASK=255.255.255.0
NETWORK=192.168.20.0
BROADCAST=192.168.20.255
BONDING_MASTER=yes
BONDING_SLAVE_0=eth0
BONDING_SLAVE_1=eth1
BONDING_MODULE_OPTS="miimon=100 mode=6"
/etc/sysconfig/network/ifcfg-eth0
/etc/sysconfig/network/ifcfg-eth1 (both are the same):
STARTMODE=auto
On application nodes running native the bonding configuration is performed during boot
via commands. The network parameters are obtained through DHCP and include the IP
address for the storage LAN.
Configuration Commands
24 Network Design and Configuration Guide
4.2 Linux VLAN Interface
VLAN configuration is performed via configuration files
/etc/sysconfig/network/ifcfg-vlan<VLAN ID>
e.g. for VLAN 10: /etc/sysconfig/network/ifcfg-vlan10:
STARTMODE='onboot'
BOOTPROTO='static'
IPADDR=192.168.200.11
NETMASK=255.255.255.0
NETWORK=192.168.200.0
BROADCAST=192.168.200.255
ETHERDEVICE='bond0'
WIRELESS='no'
4.3 NetApp Filer Configuration
The configuration has to be done manually e.g. via serial cable (see the corresponding
HW Characteristics Quickguide you find on the Service CD in doc/hwinfo).
Network Design and Configuration Guide 25
4.4 Network Switch Configuration
The network switch configuration is performed via commands. The following tables shows
Cisco IOS and Cisco NX-OS configuration commands for different purposes.
Cisco Catalyst 3750 Family Configuration (IOS commands)
Basic configuration service password encryption
enable password secret
clock timezone CET 2
clock summer-time cet recurring last Sun Mar 2:00 last
Sun Oct 3:00
vtp mode transparent
udld aggressive
no ip http server
snmp-server community public RO
spanning-tree mode rapid-pvst
spanning-tree loopguard default
no spanning-tree optimize bpdu transmission
spanning-tree extend system-id
VLAN configuration for a system
with a pool pool1
vlan 10
name client-pool1
vlan 11
name storage-pool1
vlan 12
name server-pool1
vlan 13
name control
port configuration for a data port
of a rack server used as applica-
tion node
interface GigabitEthernet1/0/1
description Node rx300s6 dataNic 1
switchport trunk encapsulation dot1q
switchport trunk native vlan 11
switchport trunk allowed vlan 10-12
switchport mode trunk
no ip address
speed auto
no mdix auto
udld port
storm-control broadcast level 20
storm-control multicast level 10
no shutdown
Configuration Commands
26 Network Design and Configuration Guide
Cisco Catalyst 3750 Family Configuration (IOS commands)
spanning-tree portfast trunk
port configuration for a data port
of a rack server used as esx-
server
interface GigabitEthernet1/0/1
description Node rx300s6
switchport trunk encapsulation dot1q
switchport trunk native vlan 13
switchport trunk allowed vlan 10-12,20-22
switchport mode trunk
no ip address
speed auto
no mdix auto
udld port
storm-control broadcast level 20
storm-control multicast level 10
no shutdown
spanning-tree portfast trunk
port configuration for a IPMI port
of a rack
interface GigabitEthernet1/0/1
description Node rx300s6 mgmtNic
switchport trunk encapsulation dot1q
switchport trunk native vlan 13
switchport trunk allowed vlan 13
switchport mode trunk
no ip address
speed auto
no mdix auto
udld port
storm-control broadcast level 20
storm-control multicast level 10
no shutdown
spanning-tree portfast trunk
channel configuration e.g. for an
uplink
interface PortChannel 1
description ISL Uplink
switchport trunk encapsulation dot1q
switchport trunk allowed vlan 10-13,20-22
switchport mode trunk
no shutdown
interface GigabitEthernet1/0/1
description ISL Uplink
switchport trunk encapsulation dot1q
switchport trunk allowed vlan 10-13,20-22
switchport mode trunk
no ip address
Configuration Commands
Network Design and Configuration Guide 27
Cisco Catalyst 3750 Family Configuration (IOS commands)
speed auto
mdix auto
udld port
storm-control broadcast level 20
storm-control multicast level 10
cdp enable
channel-group 1 mode active
no shutdown
interface GigabitEthernet2/0/1
description ISL Uplink
switchport trunk encapsulation dot1q
switchport trunk allowed vlan 10-13,20-22
switchport mode trunk
no ip address
speed auto
mdix auto
udld port
storm-control broadcast level 20
storm-control multicast level 10
cdp enable
channel-group 1 mode active
no shutdown
Save configuration copy running-config startup-config
For details see the “Catalyst 3750 Switch Software Configuration Guide” at
http://www.cisco.com.
Cisco Nexus 5000 Family Configuration (NX-OS commands)
Basic configuration no feature telnet
no telnet server enable
cfs eth distribute
feature udld
feature lacp
feature vpc
vpc domain 2
peer-keepalive destination 172.11.6.19
vrf context management
ssh key rsa 2048
udld aggressive
spanning-tree mode rapid-pvst
Configuration Commands
28 Network Design and Configuration Guide
Cisco Nexus 5000 Family Configuration (NX-OS commands)
spanning-tree loopguard default
VLAN configuration for a system
with a pool pool1
vlan 10
name client-pool1
vlan 11
name storage-pool1
vlan 12
name server-pool1
vlan 13
name control
port configuration for a data port
of a rack server used as applica-
tion node
interface Ethernet1/1
description Node rx300s6 dataNic 1
switchport trunk native vlan 11
switchport trunk allowed vlan 10-12
switchport mode trunk
spanning-tree port type edge trunk
udld aggressive
storm-control broadcast level 20
storm-control multicast level 10
no shutdown
port configuration for a data port
of a rack server used as esx-
server
interface Ethernet1/1
description Node rx300s6
switchport trunk native vlan 13
switchport trunk allowed vlan 10-13,20-22
switchport mode trunk
spanning-tree port type edge trunk
udld aggressive
storm-control broadcast level 20
storm-control multicast level 10
no shutdown
vpc configuration e.g. for an up-
link (same vpc number on both
nexus switches of the switch
group, per convention equal to
port-channel number)
interface port-channel2
description ISL Uplink
switchport trunk allowed vlan 1,10-13,20-22
switchport mode trunk
vpc 2
storm-control broadcast level 20
storm-control multicast level 10
no shutdown
interface Ethernet1/1
description ISL Uplink
switchport trunk allowed vlan 1,10-13,20-22
switchport mode trunk
udld aggressive
storm-control broadcast level 20
storm-control multicast level 10
Configuration Commands
Network Design and Configuration Guide 29
Cisco Nexus 5000 Family Configuration (NX-OS commands)
channel-group 2 mode active
no shutdown
Save configuration copy running-config startup-config
For details see the “Cisco Nexus 5000 Series NX-OS Layer 2 Switching Configuration
Guide” at http://www.cisco.com.
Network Design and Configuration Guide 31
5 Abbreviations
DART Data Access in Real Time
DHCP Dynamic Host Configuration Protocol
FF FlexFrame
FF4S FlexFrame for SAP
FFO FlexFrame Orchestrator
IP Internet Protocol
LAN Local Area Network
MAC Media Access Control
MII Media Independent Interface
NAS Network Attached Storage
NIC Network Interface Card
ONTAP Open Network Technology for Appliance Products
PXE Preboot Execution Environment
SPOF Single Point Of Failure
TFTP Trivial File Transfer Protocol
UDP User Datagram Protocol
VLAN Virtual Local Area Network
VPC Virtual Port Channel
Network Design and Configuration Guide 33
6 Glossary
Application Node
A host for applications (e.g. SAP instances db, ci, agate, wgate, app etc.). This
definition includes Application Servers as well as Database Servers.
Blade
A special form factor for computer nodes.
Celerra
NAS system of EMC.
Client LAN
Virtual network segment within FlexFrame, used for client-server traffic.
Computing Node
From the SAP ACI perspective: A host that is used for applications.
Control LAN
Virtual network segment within FlexFrame, used for system management traffic.
Control Node
A physical computer system, controlling and monitoring the entire FlexFrame land-
scape and running shared services in the rack (dhcp, tftp, ldap etc.).
Control Station
A Control Node in an SAP ACI environment.
Dynamic Host Configuration Protocol
DHCP is a protocol for assigning dynamic IP addresses to devices on a network.
Dynamic Host Configuration Protocol server
A DHCP server provides configuration parameters specific to the DHCP client host,
required by the host to participate on the Internet.
EMC NAS
Network attached storage for file systems of EMC.
Ethernet
A Local Area Network which supports data transfer rates of 10 megabits per second.
Filer
Network attached storage for file systems of NetApp.
FlexFrame®
The name FlexFrame® is a generic term for both „FlexFrame
® for SAP
®“ and „Flex-
Frame® Orchestrator“.
FlexFrame® Orchestrator
This is the advancement of the Fujitsu solution FlexFrame for SAP and means a new
Glossary
34 Network Design and Configuration Guide
approach to offer enhanced functionality and features step by step and become more
and more independent from certain hardware and software components.
FlexFrame® for SAP
®
FlexFrame® for SAP
® is a Fujitsu solution and means a radically new architecture for
SAP environments. It exploits the latest business-critical computing technology to de-
liver major cost savings for SAP customers. FlexFrame for SAP is a joint project in
which the main partners are SAP, Network Appliance, Intel and Fujitsu.
FlexFrame internal LAN Switch
Network switches which are integral part of the FlexFrame for SAP hardware configu-
ration and which are automatically configured by the FlexFrame for SAP software.
Gigabit Ethernet
A Local Area Network which supports data transfer rates of 1 gigabit (1,000 mega-
bits) per second.
Host name
The name of a node (assigned to an interface) that is resolved to a unique IP ad-
dress. One node can have multiple host names (cf. node name).
In SAP environments host names are currently limited to 13 alphanumeric characters
including the hyphen (“ - “). The first character must be a letter. In the SAP environ-
ment host names are case-sensitive.
Internet Protocol Address
A unique number used by computers to refer to each other when sending information
through networks using the Internet Protocol.
Local Area Network
A computer network that spans a relatively small area. Most LANs are confined to a
single building or group of buildings. However, one LAN can be connected to other
LANs over any distance via telephone lines and radio waves. A system of LANs con-
nected in this way is called a Wide Area Network (WAN).
Local host name
The name of the node (physical computer); it can be displayed and set using the
command /bin/hostname.
Media Access Control address
An identifier for network devices, usually unique. The MAC address is stored physi-
cally on the device.
NAS system
Network Attached Storage of any vendor (in our context: EMC NAS or NetApp Filer).
Network Attached Storage
A data storage device that is connected via a network to one or multiple computers.
Network Interface Card
A hardware device that allows computer communication via networks.
Glossary
Network Design and Configuration Guide 35
Node
A physical computer system controlled by an OS.
Node name
The name of a physical node as returned by the command uname -n. Each node
name within a FlexFrame environment must be unique.
Open Network Technology for Appliance Products
The operating system of Network Appliance Filers.
Physical host
Name of a physical computer system (node).
Preboot Execution Environment
An environment that allows a computer to boot from a network resource without hav-
ing a local operating system installed.
Server
A physical host (hardware), same as node.
Service
A software program providing functions to clients.
Storage LAN
A virtual LAN segment within a FlexFrame environment, carrying the traffic to NAS
systems.
Trivial File Transfer Protocol
A simple form of the File Transfer Protocol (FTP). TFTP uses the User Datagram
Protocol (UDP) and provides no security features. It is often used by servers to boot
diskless workstations, X-terminals, and routers.
TFTP server
A simple FTP implementation.
Virtual host
The name of the virtual host on which an application runs; it is assigned to a physical
node when an application is started.
Virtual Local Area Network
A VLAN is a logically segmented network mapped over physical hardware according
to the IEEE 802.1q standard.
Network Design and Configuration Guide 36
7 Index
C
concept and design 5
F
FlexFrame network versions 17
enterprise version 21
medium version 17
small version 17
L
link aggregation 5, 8, 9, 10, 11, 14, 15
Linux
bonding interface 23
VLAN interface 24
N
NetApp Filer configuration 24
network speed 6
network switch configuration 25
node configuration 23
R
related documents 3
V
virtual LAN 12