Configuring Virtual Load Balancing with Ubuntu 11.04 ... · Configuring a Linux Netvisor KVM 139 ... Creating and Implementing Access Control Lists (ACLs) 195 Using a Deny IP ACL

Pluribus Networks nvOS Version 2.3.2

i

nvOS Introduction 5Introduction to nvOS Fabric 13Adding Switches to the Fabric 13Displaying Fabric Statistics 17Displaying Information about Nodes in the Fabric 18Using the Fabric Transaction Commands 20Displaying Fabric Statistics 22Troubleshooting the Fabric 22Configuring Basic Server-Switch Functionality 25Using the Serial Console Port for Initial Configuration 25Changing Other Switch Setup Parameters 27Creating an Initial Fabric 29Adding License Keys to nvOS 29Modifying and Upgrading Software 31Updating nvOS on the Server-Switch 32Saving and Restoring Server-Switch Configurations 35Changing the IP Port for vManage 38Configuring Virtual Network Interface Cards (vNICs) 39Displaying Layer 2 Networking Details 41Rebooting, Powering Off, and Resetting the Server-Switch 43Installing the nvOS Linux API 44Configuring Rapid Spanning Tree Protocol (RSTP) 45Configuring Link Aggregation Control Protocol (LACP) 49Configuring Trunking for Link Aggregation (LAG) 50Configuring Layer 2 Multipathing for Virtual Chassis Link Aggregation 51Configuring Active-Active VLAG 54Configuring Tagged and Untagged VLANs 59Displaying VLAN Statistics 61Implementing Virtual Networks 63Overview 63Specifying the Type of VNET Interface 64Creating a Virtual Network (VNET) 65Related Tasks 65Creating a Virtual Network 67Adding DHCP Service to a VNET 68Verify Administrator User Creation 69Configuring Administration Login Using SSH 69Adding a Default Gateway to the VNET 72Adding Ports to the VNET 73Configuring Virtual Resource Groups 77About Virtual Resource Group (VRG) Bandwidth Enforcement 78Configuring Network Services - DHCP and DNS 79Overview of DHCP and DNS 79Configuring IP Pools 81Configuring DHCP Services 83Adding DHCP Interfaces 84Adding DHCP and DNS Records 84Removing DHCP and DNS Services 85

Pluribus Networks nvOS Version 2.3.2 ii

Configuring DNS Services 87Adding a DNS Server 87Overview of NAT and Hardware NAT 89Hardware NAT 89NAT and Hardware NAT Use Cases and Scenarios 90Configuring Network Address Translation Services 93Configuring Port Forwarding for NAT 94Configuring Static NAT 94Configuring Hardware-based Network Address Translation(NAT) 97nvOS System Logging and SNMP 99Configuring System Logging 101Sending Log Messages to Syslog Servers 102Viewing Log Events 105Sending Log Messages to Syslog Servers 108Configuring SNMP 111SNMP Communities 111Users and SNMPv3 111Supported MIBs 115High Availability 121Configuring a Cluster 121Configuring Fabric-based Physical Storage Pools 125Creating Virtual Storage for a Virtual Network (VNET) 128Managing Host Operating Systems 128Provisioning Bare Metal Servers 130External Disk Drive Installation Guide 135Configuring High Availability for Storage Folders 137Configuring a Linux Netvisor KVM 139Creating a Disk-based Netvisor KVM 141Creating a KVM by Importing an ISO Image 143Adding Virtual Machine (VM) Instances to the Server-Switch 143Managing Linux VM Images 147Configuring and Implementing NetZones 149Overview 149Configuring a NetZone 149Configuring vRouter Services 153Overview 153Configuring Prefix Lists for BGP and OSPF 153Configuring Packet Relay for DHCP Servers 154Configuring Hardware Routing for a vRouter 154Configuring BGP on a vRouter 157Additional BGP Parameters 161Configuring Open Shortest Path First (OSPF) 163Adding Areas and Prefix Lists to OSPF 165Configuring Routing Information Protocol (RIP) 167Configuring Static Routes 169Adding IGMP Static Joins to a vRouter 171Configuring Virtual Router Redundancy Protocol 173Configuring Virtual Load Balancing 177


iii

Configuring Virtual Load Balancing with Ubuntu 11.04 Servers and nvOS 181Adding Virtual Router Redundancy Protocol to VLB Interfaces 186Configuring Roles and Users 189Configuring TACACS+ 193About TACACS+ 193Configuring TACACS+ 194Creating and Implementing Access Control Lists (ACLs) 195Using a Deny IP ACL to Block Network Traffic 195Using IP ACLs to Allow Network Traffic 196Using MAC ACLs to Deny Network Traffic 198Using MAC ACLs to Allow Network Traffic 198Configuring IP ACLs 201Configuring an Internal Deny ACL 201Configuring an External Deny ACL 201Configuring an External Allow IP ACL 202Configuring a MAC ACL to Deny Network Traffic 202Configuring a MAC ACL to Allow Network Traffic 203Configuring vFlow for Analytics 205Using vFlows to Disable Communication 209Configuring Mirroring for vFlows and Ports 211Managing Traffic Classes 213Using Application Flows and Statistics 215Displaying Standard Statistics 215Understanding vFlow Statistics 217Example Use Cases for vFlows 221Configuring VXLANs and Tunnels 225Creating Tunnels 227Edge Virtual Bridging 229Understanding Edge Virtual Bridging 229Configuring Edge Virtual Bridging 230Implementing OpenFlow with FloodLight 231Configuring OpenFlow 233Enabling a Virtual Network for an OpenFlow Controller 233Creating OpenFlow Controllers with Multiple VLANs 235Configuring the OpenFlow Controller 236Configuring Open Virtual Switch (OVS) for OpenFlow 236About sFlow 237Overview 237Configuring sFlow 241Configuring the sFlow Collector 241Enabling sFlow on the Network 241Adding Additional Ports to sFlow 242

Pluribus Networks nvOS Version 2.3.2 iv


v

THE SPECIFICATIONS AND INFORMATION REGARDING THE PRODUCTS IN THIS MANUAL ARE SUBJECT TO CHANGE WITHOUT NOTICE. ALL STATEMENTS, INFORMATION, AND RECOMMENDATIONS IN THIS MANUAL ARE BELIEVED TO BE ACCURATE BUT ARE PRESENTED WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED. USERS MUST TAKE FULL RESPONSIBILITY FOR THEIR APPLICATION OF ANY PRODUCTS.

THE SOFTWARE LICENSE AND LIMITED WARRANTY FOR THE ACCOMPANYING PRODUCT ARE SET FORTH IN THE INFORMATION PACKET THAT SHIPPED WITH THE PRODUCT AND ARE INCORPORATED HEREIN BY THIS REFERENCE. IF YOU ARE UNABLE TO LOCATE THE SOFTWARE LICENSE OR LIMITED WARRANTY, CONTACT YOUR PLURIBUS NETWORKS REPRESENTATIVE FOR A COPY.

NOTWITHSTANDING ANY OTHER WARRANTY HEREIN, ALL DOCUMENT FILES AND SOFTWARE ARE PROVIDED “AS IS” WITH ALL FAULTS. PLURIBUS NETWORKS DISCLAIMS ALL WARRANTIES, EXPRESS OR IMPLIED, INCLUDING WITHOUT LIMITATION, THOSE OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT OR ARISING FROM A COURSE OF DEALING, USAGE, OR TRADE PRACTICE.

IN NO EVENT SHALL PLURIBUS NETWORKS BE LIABLE FOR ANY INDIRECT, SPECIAL, CONSEQUENTIAL, OR INCIDENTAL DAMAGES, INCLUDING, WITHOUT LIMITATION, LOST PROFITS OR LOSS OR DAMAGE TO DATA, ARISING OUT OF THE USE OR INABILITY TO USE THIS MANUAL, EVEN IF PLURIBUS NETWORKS HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES.

Any Internet Protocol (IP) addresses used in this document are not intended to be actual addresses. Any examples, command display output, and figures included in the document are shown for illustrative purposes only. Any use of actual IP addresses in illustrative content is unintentional and coincidental.

© 2016 PLURIBUS NETWORKS, INC. ALL RIGHTS RESERVED.

Pluribus Networks nvOS Version 2.3.2 vi


1

Preface

This preface includes the following sections:

Audience

Organization

Conventions

Related Documentation

Obtaining Documentation and Submitting a Service Request

This preface describes the audience, organization, and conventions of this publication, and provides information about obtaining related documentation.

AudienceThis publication is for experienced network administrators responsible for configuring and maintaining Pluribus Networks switches with some expertise in the following areas:

Network administration

Storage administration

Server administration

Application delivery administration

Network security administration

OrganizationThis publication is organized as follows:

Layer 2 and Layer 3 Services

VNETs, Network Services, Cluster (SDF)

vFlows, OpenStack, OpenFlow, and Netvisor features

ConventionsThis document uses the following conventions:Table 2: CLI Conventions

Convention Indication

Bold font Keywords, user interface elements, and user-entered text appear in bold font.

Italic font Document titles, new or emphasized terms, and variables that you supply values are in italic font.

[ ] Elements in square brackets are optional.

{x|y|z} Required elements are grouped in curly braces and are separated by vertical bars.

[x|y|z] Optional parameters are grouped in brackets and separated by vertical bars.

String A non-quoted set of characters. Do not use quotation marks around the string or the string includes the quotation marks.

Pluribus Networks nvOS Version 2.3.2 2

Related DocumentationThe Pluribus Networks switch nvOS documentation set includes the following publications:

Pluribus Networks Hardware Installation Guide Pluribus Networks vManage® Administrative Guide

courier font Command Line Interface (CLI) commands and samples appear in courier font.

< > Nonprinting characters such as passwords are indicated by angle brackets.

[ ] Default responses to system prompts are in angle brackets.

CLI network-admin@switch > Indicates that you enter the following text at the command prompt.

Informational Note:Indicates information of special interest.

Indicates a situation that could cause equipment failure or loss of data.

TIP!

TIP!Indicates information that can help you solve a problem.

Timesaver:

Indicates information that can help you save time.

Table 2: CLI Conventions

Convention Indication


3

Release Notes for Pluribus Networks nvOS Releases Pluribus Networks nvOS Configuration Guide Pluribus Networks Command Reference

For a complete list of all Pluribus Networks documentation, see the Pluribus Networks support site at www.plurisbusnetworks.com/support.

Additional documentation describing log messages and MIBs are also available for download at www.plurisbusnetworks.com/support.

Documentation Feedback

To provide technical feedback on this document, or to report an error or omission, please send your comments to [email protected]. We appreciate your feedback.

Obtaining Documentation and Submitting a Service Request

For information on obtaining documentation, submitting a service request, and gathering additional information, please visit www.pluribusnetworks.com/support.

http://www.pluribusnetworks.com/support

http://www.plurisbusnetworks.com/support

http://www.plurisbusnetworks.com/support



5

nvOS Introduction

This chapter provides information for understanding and using the Pluribus Networks nvOS command line interface (CLI) on a Pluribus Networks switch. Included in this chapter is the following information:

Before You Start

Important Terms

Entering Commands and Getting Help

Finding Command Options

Understanding Role-based Access Control

Specifying IP Address Netmasks

Specifying Capacity, Throughput, and Scale

Customizing Show Output Formats

Using the CLI String Search

Specifying a Switch or Fabric for Command Scope

Before You StartPluribus Netvisor uses the concept of Fabric-Cluster to describe the interconnectivity of devices into a single logical network. The concept of a virtual network (VNET) describes is a “slice” of resources that apply to a single entity with assigned resources within the fabric. The VNET contains services and resources that apply only to that VNET. Typically, VNETs are used to house different tenants within a single large network. For switches with nvOS, the only available VNET is a global VNET created when a fabric is created for the first time.

Since you are just getting started with your switches and Netvisor, you may decide to configure a single switch first or you may have purchased a single switch. It’s important to understand that a single switch can consist of a single fabric and a single VNET, or many switches and VNETs.

Important Terms

The following list of important terms and concepts and their definitions is important for understanding Pluribus Networks features and determine the best configuration to meet your needs.

Term Meaning

API Application Programming Interface to the Pluribus Networks switch. It has a similar scope as the CLI.

CLI Command Line Interface to the Pluribus Networks switch. Depending on the command, it can be executed for an individual switch, a cluster, or a fabric.

Cluster A pair of Pluribus Networks switches configured as a high availability group. You can configure many clusters in the fabric, but a switch can be a member of only one cluster.

Disk-library Virtual machine storage within a storage pool.

eth0...ethX Virtual network interface names associated with virtual services.

Fabric A set of Pluribus Networks switches configured as a single entity. Any switch can be a member of only one fabric. Up to 4096 switches can be configured a single fabric.


Entering Commands and Getting HelpCommands, options, and arguments are entered at the CLI prompt. A command name must be typed, but included command-completion and help features contribute to the command entry process.

To display a list of command that you can use within a command mode, enter a question mark (?), or use the tab key, or type help at the command prompt. You can also display keywords and arguments for each command with this context-sensitive help feature. You can also use complete commands and display keywords and arguments for each command using the tab key to assist with context-sensitive command help and completion.

Flow A communication from one device outside of the fabric to another device outside of the fabric and traveling through the fabric.

GUI Graphic User Interface to the Pluribus Networks switch. It has a similar scope as the CLI.

In-band Management Address

The IP address of the switch on a production or management network for administration and inter-switch communication.

ISO- library ISO (operating system) image storage within a storage pool

LACP Link Aggregation Control Protocol allows a non-Pluribus Networks device to have multiple connections to the same switch, for example, IEEE 802.3ad trunks.

Netvisor Zone, Netvisor KVM, Netvisor VMM

A virtual machine running within the Pluribus Networks switch. A NetZone runs natively on the Unix-compatible operating system. A NetVM allows the use of arbitrary x86 operating systems and applications.

Server-switch A Pluribus Networks hardware device with aspects of both a server and a switch.

Storage-device Disk or PCI-based storage connected to the switch.

Storage-pool Storage in a RAID set available for use by storage commands.

Storage-folder General purpose file sharing system available within a storage pool.

vFlow A logical, manageable connection within or through the fabric.

VLAG Virtual Link Aggregation Group is the Pluribus Networks method for multiple connecting hosts to multiple switches, switches to each other, and switches to other switches.

VNET A virtual network configured within the fabric. All traffic within one VNET segregated from the traffic of all other VNETs. A VNET is an administrative entity as well, limiting the effects of changes to a single VNET. Everything in a Pluribus Networks server is associated with a VNET.

Term Meaning


7

Table 3 lists the command that you can enter to get help specific to a command, keyword, or argument.

Where a text string is used, such as name-string, the following characters are allowed as part of the text string: a-z, A-Z, 0-9, _ (underscore), . (period), , (comma), : (colon), and - (dash).

Finding Command OptionsThe syntax can consist of optional or required keywords. To display keywords for a command, enter a question mark (?) at the command prompt or after entering part of a command followed by a space. nvOS® CLI displays a list of available keywords along with a brief description of the keywords. For example, if you want to see all of the keywords for the command user, enter user ?.

Table , “Getting Help” displays examples of using the question mark (?) to assist you with entering commands.

Table 3: Getting Help

abbreviated- command-entry? Displays a list of commands that begin with a specific character string. Do not leave a space between the string and question mark.

abbreviated- command-entry <tab> Completes a partial command name.

? Lists all commands.

command ? Lists all keywords for the command. Leave a space between the command and the question mark.

command keyword ? Lists all arguments for the keyword. Leave a space between the command and the question mark.

Informational Note: If you enter a command that is invalid, then using the ? and tab key have no effect and do not return any changes to the CLI.

Informational Note: The CLI has an editing ability similar to UNIX and Linux functionality using emacs keys. For example, ˄p steps backward through previous commands, ˄n moves to the next command in the history, ˄a moves to the first character in the command and ˄e moves to the end of the line, ˄u erases the current line, and ˄w erases the previous word.

Informational Note: Also you can use the up and down arrows on your keyboard to retrieve the last command entered at the CLI.


Additional Information on the Command Line Interface

For some commands, the parameter delete is used, and in other commands, the parameter remove is used. This may appear as inconsistent usage, but the explanation is quite simple.

delete is used for top level commands, such as acl-ip-delete, or vlan-delete. The following list is a sample of top level commands:

aaa-tacacs-delete

dhcp-delete

ip-pool-delete

nat-delete

remove is used for commands with additional options, such as iso-library-image-remove where the top level command is iso-library and the additional option image is added to the top level command. The following list is a sample of top level commands with additional parameters that use remove:

dhcp-host-remove

disk-library-image-remove

dns-interface-remove

sflow-port-remove

Table 4: Finding Command Options

CLI network-admin@switch > ?

All commands:acl-ip-createacl-ip-delete...

Displays a list of commands that begin with a specific character string. Do not leave a space between the string and question mark.

Switch> user authUser: <user>Password: <password>

Completes a partial command name.

? Lists all commands.

command ? Lists all keywords for the command. Leave a space between the command and the question mark.

command option ? Lists all arguments for the option. Leave a space between the command and the question mark.

Informational Note: Other useful options, especially for displaying statistics, include sort, interval, duration, and show diff interval.


9

The same logic also applies to the usage of create and add. create is used for top level commands and add is used with top level commands with additional options. For example, sflow-create and sflow-port-add are two instances where this usage occurs in the CLI.

Alternate Command Format

The CLI has an alternate command format in that the commands start with a verb instead of a noun. This format omits the hyphen in the command names. For example, connection-stats-show can also be entered as show connection-stats. The command formats have the same features and can be used interchangeably.

Understanding Role-based Access ControlPluribus Networks nvOS® supports flexibly defined roles so that data centers can use the same best practices for managing discrete servers, storage, and networks to operate a Pluribus Networks fabric. You can create user roles with privileges that reflect user responsibilities in the data center. For example, you can create the following types of roles:

Fabric administrator roles with control over all fabric-wide tasks

Cluster administrator roles with control over all cluster-wide tasks

Switch-server administrator roles with control over single switch configuration tasks

Virtual Network (VNET) administrator roles with control over one or multiple VNET configuration tasks

Virtual network services administrator with control over one or multiple network service(s) configuration tasks.

Specifying IP Address NetmasksSome commands call for the specification of an IP address netmask. Pluribus Networks nvOS supports both CIDR and subnet notations.

For example, the range of IP addresses from 192.168.0.0 to 192.168.0.255 can be specified by either entering 192.160.0.0 for the IP address input for a CLI command or either 24 or 255.255.255.0 for the netmask.

Specifying Capacity, Throughput, and ScaleMany commands include input and output of capacity and throughput. Network values are always in bits and storage values in bytes. Scale factors are allowed on input and displayed in output as well as shown in Table 5, “Scale Numbers”.Table 5: Scale Numbers

Scale Indicator Meaning (Networking) Meaning (Storage)

K or k Kilobits Kilobytes

M or m Megabits Megabytes

G or g Gigabits Gigabytes

T or t Terabits Terabytes


Customizing Show Output FormatsThe output generated by the show commands can be customized by using the optional arguments described in Table 6, “Show Output Formats”.

Using the CLI String SearchThe pattern in the command output is referred to as a string. The CLI string search feature allows you to search or filter any show or more command output and allows you to search and filter at --More-- prompts. This feature is useful when you need to sort through large amounts of output, or if you want to exclude output that you don’t want to see.

With the search function, you can begin unfiltered output at the first line that contains a specified regular expression. You can then specify a maximum of one filter per command or start a new search from the --More-- prompt.

You can perform three types of filtering:

Use the begin keyword to begin output with the line that contains a specified regular expression.

Use the include keyword to include output lines containing a specified regular expression.

Use the exclude keyword to exclude output lines containing a specified regular expression.

Table 6: Show Output Formats

format <column_name1>, <column_name2>, <column_nameX>

Displays only the columns matching the list of column header names.NOTE: The list of column names is comma-separated without spaces.

format all Displays all available column headers. This output is also called verbose mode. By default, show commands output a terse set of the most commonly useful column headers.

parsable-delim <separator> Displays the output of show command by separating columns by the specified <separator> character(s). For example, parsable-delim , produces a comma-separated output (CSV).NOTE: If the parsable-delim option is specified, the column header names (titles) are suppressed from the output.


11

You can then search this filtered output at the --More-- prompts.Most commands optionally preceded by integer argument k. Defaults in brackets. Star

(*) indicates argument becomes the new default. ----------------------------------------------------------------<space> Display next k lines of text [current screen size]z Display next k lines of text [current screen size]<return. Display next k lines of text [1]*d or ctrl-D Scroll k lines [current scroll size, initally 11]*q or Q or ,interrupt> Exit from mores Skip forward k lines of text [11]f Skip forward k screenfuls of text [1]b or ctrl-b Skip backwards k screenfuls of text [1]‘ Go to place where previous search started= Display current line number/<regular expression> Search for kth occurrence of regular expression [1]n Search for kth occurrence of last r.e. [1]h Display this messagectrl-l Redraw the screen:n Go to kth next file [1]:p Go to kth previous file [1]. Repeat previous command

For example, to only display output that includes the IP address, 10.9.9, type the following at the --More-- prompt:

/10.9.9<return>

Specifying a Switch or Fabric for Command Scope

While a switch is the building block of a fabric, the goal of the Pluribus Networks design is that a fabric of switches is easy to manage as a single switch. Because of this, the CLI can be used to run commands on the local switch, a cluster of switches, other switches in the fabric, or the entire fabric. You don’t have to log into each switch that you want to run commands.

By default, commands are run on the switch you’re logged into and for example, the command port-config-modify port 5 disable disables port 5 on the switch you’re logged into on the network.

To specify a different switch for a single command, use the switch prefix. For example, switch pleiades23 port-config-modify port 28 enable enables port 28 on pleiades23, even if the CLI is connected to a different switch in the fabric.

To specify a different switch for a series of commands, use the switch prefix with no command. For example, type switch pleiades24 <return>. The CLI prompt changes to indicate that pleiades24 is the switch you are executing commands. Additional commands are run on pleiades24 rather than the switch that you’re physically connected.

For most CLI show commands, the command displays results from all switches in the fabric by default. For example, when the CLI command port-show is entered on the switch, it shows the ports of all switches in the fabric.

To specify that a CLI show command should apply to a specific switch, use the switch prefix to the CLI command. For example, for the port-show command to only show the ports of the switch named pleiades24, type the command switch pleiades24 port-show.

Informational Note: The CLI search function does not allow you to search or filter backward through previous output.


Topic FeedbackWas this topic useful to you? Please provide feedback to improve the content.

mailto:[email protected]?Subject=Topic%20Feedback


13

Introduction to nvOS Fabric

Adding Switches to the Fabric

Directly Connected Switches in a Fabric

Fabric Over Management Interface

Displaying Fabric Statistics

Displaying Information about Nodes in the Fabric

Using the Fabric Transaction Commands

More Information About Undo Commands and Transactions

At Pluribus Networks, a fabric is defined as a distributed architecture based on a collection of compute clustering techniques to present an open, standard-based Ethernet fabric as one logical switch. Every node shares the same view of the fabric including MAC and IP addresses, connections, and application flows.

When you add switches to the fabric, all switches are under a single management domain which is highly available through multiple link aggregation and load balancing between network resources.

The fabric performs a classic database 3-phase commit for configuration changes. All members of the fabric must accept the configuration changes before the change is made in the fabric. Figure 1 Fabric Architecturedisplays the fabric architecture of nvOS.

Figure 1: Fabric Architecture

Adding Switches to the FabricFor this example, the switches are connected as in Figure 2:


Figure 2: Directly Connected Switches in a Fabric

When you have more than one switch, you must add it to the fabric to take advantage of the features offered by the fabric. To add the new switch, use the following command on one of the switches:

CLI network-admin@switch > fabric-join name pn-EBC4 fab1

You can join the fabric using either the fabric name or the switch IP address. If you use the Tab key to display the available options, all fabrics that the switch is aware of are displayed as options.

If you specify a password for the fabric, you must type it in twice. The password is used to encrypt communication between the nodes in the fabric. When you join the fabric from a node, you must type in the password to join it.

You can specify a specific VLAN for the fabric when you create a new one, or by default, the fabric uses VLAN1. However, you cannot change the fabric VLAN without recreating the fabric.

When the fabric is created, the switch begins sending multicast messages out on Layer 2 looking for other switches. These messages are not propagated to other networks. This is how Switch B in Figure 2 learns about the fabric.

Once Switch B joins the fabric, the fabric configuration (commands with scope fabric) is downloaded on Switch B and the switch reboots.

If you want to connect to a switch over Layer 3, you must specify the IP address for the switch in the fabric using the following command:

CLI network-admin@switch > fabric-join switch-ip 192.168.11.1

Fabric Over Management InterfaceYou can now configure fabric communication run over either the management interface or the in-band interface. Because fabric communication over the in-band interface can be disrupted due to STP, ports going up/down, and other factors, fabric communication over management provides a more consistent configuration.

Informational Note:Avoid creating fabrics with the same name.


15

If you create a fabric with the management interface, any nodes joining the fabric inherit this setting. All nodes in a single fabric all run on the same network type. You cannot run a mixed configuration of management and in-band interfaces. Fabrics advertised on an incompatible network are not available for when you issue the fabric-join command. This keeps a switch from joining an incompatible fabric.

If the fabric is configured on the management interface, all fabric-communication is on the management network, except for the following:

Cluster synchronization-related traffic such as VLAG synchronizations and forwarded STP packets.

Cluster keep-alive packets on the fabric

Fabric keep-alive packets and global-discovery packets because both run on mgmt and in-band interfaces.

Two options, network-type and control-network are added to the command, fabric-create:

CLI network-admin@switch > fabric-create

name name-string

any of the following options:

vlan 0..4095

password

fabric-network in-band|mgmt

control-network in-band|mgmt delete-conflicts|abort-on-conflict

If not specified, the network defaults to in-band. Note the commands, fabric-join and fabric-unjoin, remain unchanged.

Specifying the fabric-network parameter sets the data path for fabric administration, which includes configuration changes and show commands.

Specifying the control-network parameter sets the data path for control plane traffic, which includes status updates, vLAG syncs, cluster syncs, and other control plane traffic.

Two new states are added to the state field of fabric-node-show:

fabric-node-show ?

[state offline|online|in-band-only-online|mgmt-only-online| fabric-joined|eula-required|setup-required|fabric-required| fresh-install]

Because there are now two networks for nvOS to monitor for connectivity, online means both management and in-band are reachable; in-band-only-online means the switch is only reachable through the in-band network; mgmt-only-online means it is only reachable through the management network; and offline means the switch is not reachable on either network.

Monitoring and reporting are reported on both the management and in-band network connectivity.


Displaying Fabric InformationYou can display information about the fabric using the fabric-info command:

CLI network-admin@switch > fabric-info format all layout vertical

name: pn-EBC4switch-ip: ::id: a0000c5:53ab701emcast-ip: 239.4.10.111tid: 327

tid is the fabric transcation ID assigned by nvOS


17

Displaying Fabric StatisticsYou can also display statistical information about fabric and node activity.

CLI network-admin@switch > fabric-stats-show format all layout vertical

switch: corp-sw1id: 0servers: 0storage: 0VM: 0vxlan: 0tcp-syn: 3tcp-est: 1tcp-completed: 17tcp-bytes: 3.56Mudp-bytes: 0arp: 0vlan: 0switch: corp-Leaf-1id: 0servers: 0storage: 0VM: 0vxlan: 0tcp-syn: 42.5Ktcp-est: 7.20Ktcp-completed: 1.99Mtcp-bytes: 4.63Tudp-bytes: 0arp: 0vlan: 0switch: corp-Spine1id: 0servers: 0storage: 0VM: 0vxlan: 0tcp-syn: 115Ktcp-est: 50.2Ktcp-completed: 106Mtcp-bytes: 222Tudp-bytes: 0arp: 0vlan: 0


Displaying Information about Nodes in the FabricYou can also display information about the nodes in the fabric. It is important to take note of the fab-tid value. If the fab-tid values do not match for each node, you can use the commands transaction-rollback-to or transaction-rollforward-toto resynchronize the fabric.

id: 167772619name: Leaf2fab-name: fab1fab-id: a0001c8:53e2601bcluster-id: 0:0fab-mcast-ip: 239.4.10.94local-mac: 64:0e:94:28:06:f2mgmt-nic: mgmt-ip: 192.168.1.14/24...in-band-ip: 192.168.254.14/24...fab-tid: 9out-port: 0version: 2.1.201015836,pn-nvOS-2.0.2-2000212196state: onlinefirmware_upgrade: not-requireddevice_state: okports: 72id: 201326827name: Leaf1fab-name: fab1fab-id: a0001c8:53e2601bcluster-id: 0:0fab-mcast-ip: 239.4.10.94local-mac: 64:0e:94:30:03:97mgmt-nic: mgmt-ip: 192.168.1.11/24...in-band-ip: 192.168.254.11/24...fab-tid: 9out-port: 129version: 2.1.201015836,pn-nvOS-2.0.2-2000212196state: onlinefirmware_upgrade: not-requireddevice_state: okports: 72id: 167772618name: Spine2fab-name: fab1fab-id: a0001c8:53e2601bcluster-id: 0:0fab-mcast-ip: 239.4.10.94local-mac: 64:0e:94:28:06:eemgmt-nic: mgmt-ip: 192.168.1.13/24


19

An example of a fabric that is out of sync for two nodes in the fabric:

CLI network-admin@switch > fabric-node-show format all layout vertical

id: 100663365name: CBF-switchfab-name: pn-CBF4fab-id: a0000c5:53ab701ecluster-id: 0:0fab-mcast-ip: 239.4.10.111local-mac: 64:0e:94:18:01:03mgmt-nic: mgmt-ip: 192.168.1.61/24...in-band-ip: 192.168.77.61/24...fab-tid: 328out-port: 128version: 2.1.201005800,pn-nvOS-2.0.2-2000212196state: onlinefirmware_upgrade: not-requireddevice_state: okports: 68id: 201326771name: CBF-Leaf-1fab-name: corp-CBF4fab-id: a0000c5:53ab701ecluster-id: 0:0fab-mcast-ip: 239.4.10.111local-mac: 64:0e:94:30:02:4dmgmt-nic: mgmt-ip: 192.168.1.53/24...in-band-ip: 192.168.77.53/24...fab-tid: 329out-port: 128version: 2.1.201005800,pn-nvOS-2.0.2-2000212196state: onlinefirmware_upgrade: not-requireddevice_state: okports: 72id: 167772357name: CBF-Spine1fab-name: pn-CBF4fab-id: a0000c5:53ab701ecluster-id: 0:0fab-mcast-ip: 239.4.10.111local-mac: 64:0e:94:28:02:demgmt-nic: mgmt-ip: 192.168.1.51/24...in-band-ip: 192.168.77.51/24

f you apply a configuration to the fabric, and a node does not respond to it, you can evict the node from the fabric, and then troubleshoot the problem. To evict a node, use the following command:

CLI network-admin@switch > fabric-node-evict name pleiades25


or

CLI network-admin@switch > fabric-node-evict id b000021:52a1b620

Using the Fabric Transaction CommandsYou can roll back the fabric to a specific fabric transaction number. If a failure occurs on the fabric, transactions on nodes in the fabric can go out of synch. Once transactions are out of synch, no further transactions can be executed across the scope of local, fabric, or cluster. Unjoining and rejoining the fabric causes the node to lose its configuration.

As part of a single node transaction recovery, you can roll back the transaction number to a previous one. If multiple nodes are out of synch, you must recover each node separately.

You can also roll the fabric transaction ID forward on a node if it is out of synch with the rest of the fabric.

In the previous example, the switch, CBF-Switch2, is out of synch with the rest of the fabric. The fabric transaction ID is 327 and the rest of the nodes have a transaction ID of 328. In this case, you can roll the node, CBF-Switch2, forward to transaction ID 328. Enter the following command on node CBF-Switch2:

CLI network-admin@switch > transaction-forward-to scope fabric tid 328

This command produces output when an error occurs during the transaction. If there is no output, the transaction is successful.

To display transaction information for CBF-Switch2,use the transaction-show command:

CLI network-admin@switch > transaction-show format all layout vertical

start-time: 03-19,13:46:42end-time: 03-19,13:46:43scope: fabrictid: 33state: remote-commitcommand: --unrecoverable-- vlan-delete id 22undo-command: --unrecoverable-- vlan-create id 22 nvid a000030:16 scope

fabric name vlan-22 active yes stats vrg 0:0 ports 1-72,128-129,255 untagged-ports none send-ports 31,41,47-48,51,65-66 active-edge-ports none ports-specified false flags

----------------------------------------start-time: 09:36:09end-time: 09:36:09scope: fabrictid: 34state: remote-commitcommand: vlan-create id 35 scope fabric stats ports-specified true

The scope parameter indicates which set of transactions to display as each scope has an independent set of transactions associated with it. The default scope is fabric unless another scope is specified.

You cannot copy and paste commands and undo-commands because they include information that cannot apply to new commands. These fields are informational-only and allow you to see exactly what happens to the configuration when you roll forward or roll back the transaction ID.

Once you decide which node you want to modify and the transaction that you want to roll forward or roll back, you use the transaction-rollforward-to or transaction-rollback-to commands to re-run the command (roll forward) or undo the command (rollback) on the node. This applies only to the local node.


21

More Information About Undo Commands and TransactionsYou may see output similar to this output:

start-time: 21:54:53end-time: 21:54:53scope: localtid: 3state: commitcommand: port-config-modify port 9 enableundo-command: port-config-modify port 9 enable

This output is actually correct. The undo information is taken from the current state on the fabric. So if the port is currently enabled, and you try to enable it again, you see the undo-command in the output, since the previous state is also enabled. If you actually disable the port first, and then enable it, you see the expected undo information in the transaction log.

start-time: 10:05:22end-time: 10:05:22scope: localtid: 20state: commitcommand: port-config-modify port 12 disableundo-command: port-config-modify port 12 enable----------------------------------------start-time: 10:05:48end-time: 10:05:48scope: localtid: 21state: commitcommand: port-config-modify port 12 enableundo-command: port-config-modify port 12 disable

So undo is not necessarily the opposite of the current command, but allows you to go back to the state before the command was issued. This may be the exact same state as before.




Displaying Fabric StatisticsTo display fabric statistics, use the following command:

CLI network-admin@switch > fabric-stats-show

switch: pleiades23id: 0servers: 0storage: 0VM: 0vxlan: 0tcp-syn: 229Ktcp-est: 171tcp-completed: 7.19Ktcp-bytes: 3.53Gudp-bytes: 0arp: 0vlan: 0switch: pleiades24id: 0servers: 0storage: 0VM: 0vxlan: 0tcp-syn: 85.6Ktcp-est: 125tcp-completed: 11.6Ktcp-bytes: 3.95Gudp-bytes: 0arp: 0vlan: 0switch: pleiades25id: 0servers: 0storage: 0VM: 0vxlan: 0tcp-syn: 179Ktcp-est: 20.9Ktcp-completed: 1.60Mtcp-bytes: 485Gudp-bytes: 0arp: 0vlan: 0

Troubleshooting the Fabric

There may be instances when you need to troubleshoot the fabric. The following is a list of helpful port numbers, multicast information, and communication on the fabric.

Internal Keepalive

Multicast IP: 239.4.9.7

UDP Destination Port: 23399


23

This packet is sent from the CPU to the internal port to ensure that the CPU path to the switch is working and the internal port is up.

Fabric Keepalive


Point to point UDP fabric keepalive

If these messages don't get through, the fabric node may go to offline state.

Global Discovery


UDP destination port: 23399

Each node periodically multicasts a message about the fabric. This enables fabric-show on L2-connected nodes to show available packets and also enables fabric-join name name. It also enables you to join a fabric over Layer 3 connectivity by specify an IP address.

Proxy commands

TCP Destination Port: 23397 SSL

Used for nvOSd-to-nvOSd commands. Used for internal purposes and also to implement commands executed on other switches from a local switch.

Status propagation


Port changes and vport changes propagated to other nodes in the fabric.

TCP API clients


C API clients connect to this port. Can be disabled using admin-service-modify if <mgmt/data> no-net-api command.

File System replication


For ZFS send and ZFS receive messages when replicating file systems across the fabric.

L2 ARP/DMAC miss/Broadcast encapsulation


These are VXLAN-encapsulated packets sent from CPU to CPU between two L2 connected switches.

L3 ARP/DMAC miss/Broadcast encapsulation


These are VXLAN-encapsulated packets sent from CPU to CPU between two L3 connected switches.

vPORT status



vPort updates from hypervisors or hosts in the fabric.

vFlow CPU packets


These packets are sent point-to-point for vflow-snoop of a fabric-scoped vFlow.


All of these messages need to be able to get through in order to keep an L2 fabric healthy. The multicast messages don't propagate through routers so they aren't used for L3 fabrics.

fabric-node-show displays information about nvOS internal data structures for each node in the fabric. If no keepalive or other messages are received from a fabric node for about 20 seconds, the node is marked as offline.

Anything that prevents keepalive or other kinds of messages from flowing freely between fabric nodes can cause problems for fabric connectivity.

If the fabric transaction IDs become unsynchronized, use the transaction commands to either roll forward or back the transaction IDs. See Using the Fabric Transaction Commands.

Configuring Transaction Settings

Transactions are allowed to proceed if at least one node in the cluster is reachable. If a cluster node is offline when a configuration change is requested the transaction proceeds even though one of the cluster members is offline. Nodes that were ignored for transactions automatically try to recover the transactions. Auto-recovery is enabled by default but may be disabled. You can also configure the length of time between retry attempts between the nodes.

This feature is enabled by default, but may be disabled.

The following is a sample CLI output with one cluster node offline:

CLI (network-admin@switch1) > vlan-create id 24 scope fabricWarning: cluster node switch2 not reachable, continuing anyway

The following is a sample of CLI output with both cluster nodes offline:

CLI (network-admin@switch2) > vlan-create id 33 scope fabricWarning: cluster node switch1 not reachable, continuing anywayvlan-create: fabric error: switch1 unreachable, both cluster nodes offline

To configure transaction settings, use the transaction-settings-modify command and configure the following options:

allow-offline-cluster-nodes — select this option to allow transactions to proceed on cluster configurations even if the cluster is offline.

auto-recover — select this option to automatically recover missed transactions.

auto-recover-retry-time — specify the duration of the retry time in days, hours, minutes, or seconds.




25

Configuring Basic Server-Switch Functionality

Using the Serial Console Port for Initial Configuration

Aggregation for Management Network Interface Card (NIC)

Creating an Initial Fabric

Changing Other Switch Setup Parameters

Confirming Connectivity on the Network

Updating nvOS on the Server-Switch

Implementing a Fabric Upgrade or a “Rolling” Fabric Upgrade

Saving and Restoring Server-Switch Configurations

Copying and Importing Configuration Files

Configuring Virtual Network Interface Cards (vNICs)

Displaying Physical Port Details

Displaying Layer 2 Networking Details

Rebooting, Powering Off, and Resetting the Server-Switch

Topic Feedback

Using the Serial Console Port for Initial Configuration This procedure assumes that you have installed the server-switch in the desired location and it is powered on.

If you are going to cable host computers to the switch, there is an option to enable or disable host ports by default.

1. Connect the console port on the rear or front (depending on the model) of the server-switch to your laptop or ter-minal concentrator using a serial cable.

2. From the terminal emulator application on your computer, log into the switch with the username network-admin and the default password admin.

3. . You can begin initial configuration using the setup questions displayed:switch console login: network-adminPassword: adminLast login: Fri Oct 3 12:23:04 on consolePluribus Command Line Interface v1.2.2System setup required:

System Name (switch): pleaides01 <return>network-admin Password: password <return>Re-enter Password:****** <return>Enable mgmt link aggregation (no): yesThis might reset SSH connections after thesetup.Are you Sure? (no): yesLACP mode of the mgmt LAG interface[active|passive|off](passive): invalidPlease answer "active", "passive", or "off"LACP mode of the mgmt LAG interface[active|passive|off](passive): active

CAUTION! Do not connect any ports to the network until the server-switch is configured. You can accidentally create loops or cause IP address conflicts on the network.


Mgmt IP/Netmask (10.9.19.107Mgmt IP/Netmask: ip-address/netmask <return>In-band IP/Netmask: ip-address/netmaskGateway IP (0.0.0.0): 192.168.100.254 <return> or ip-addressPrimary DNS IP (0.0.0.0): 192.168.100.253 <return> or ip-addressSecondary DNS IP (0.0.0.0): 192.168.200.253 <return> or ip-addressDomain name (pluribusnetworks.com): domain-name <return>Automatically Upload Diagnostics (yes): <return>Enable host ports by default (yes): no

nvOS system info:serial number: 1245LC8500018hostid: a000044

user auth cookie val = 152895552Switch Setup:

Switch Name: pleaides01Switch Mgmt IP: 192.168.100.1/24Switch In-band IP: 192.168.200.1/24Switch Gateway: 192.168.100.254Switch DNS Server: 192.168.100.254Switch DNS2 Server: 192.168.100.253Switch Domain Name: pluribusnetworks.comSwitch NTP Server: 0.us.pool.ntp.orgSwitch Timezone: US/PacificSwitch Date: 2013-10-03, 13:02:39Upload Crash Reports: yes

Fabric required. Please use fabric-create/join/showConnected to Switch pluribus; nvOS Identifier:0x000044; Ver: 0.19.3398

Aggregation for Management Network Interface Card (NIC)

Out of band management interfaces areaggregated to provide high availability (HA) and failover capabilities in nvOS in the presence of two management NICs. You can configure nvOS to pool two management NICs into a single logical management interface to increase bandwidth of the management link and add redundancy to the out of band connection. By default, management link aggregation is disabled. When you configure link aggregation, a new interface is created on the platform and a trunk link is also created. Physical management interfaces, MGMT0 and MGMT1, are added to it. The IPv4 and IPv6 addresses are copied from MGMT0 if configured.

LACP is disabled by default, but can be enabled using the switch-setup-modify mgmt-lacp-mode command. The default aggregation mode is active-active, and after configuring the link aggregation interface, nvOS waits for a short interval to ensure that the interface is receiving packets. If no packets are seen on the second physical interface configuration reverts back to the single management interface, and the appropriate error message is generated.

You are now ready to begin the rest of the configuration on the switch.

Changing the Default TimezoneThe default timezone is US/Pacific Standard Time (PST). To change the timezone, use the switch-setup-modify command:

CLI network-admin@switch > switch-setup-modify timezone timezone

Informational Note: In order to use the “phone home” feature, you must open ports 8084 and 8443 on your firewall.


27

Changing Other Switch Setup ParametersYou can also modify other switch parameters including the following:

Switch name

Management IPv4 and IPv6 addresses

Management IPv4 and IPv6 netmasks

Management IPv4 and IPv6 address assignments

In-band IP address

In-band netmask

Gateway IPv4 address

Gateway IPv6 address

Primary and secondary IPv4 addresses for DNS services

Domain name

NTP server

End User License Agreement (EULA) acceptance and timestamp

Password

Date

Phone home for software updates

Analytics store (storage type)

Message of the Day (MOTD)

Banner

CLI network-admin@switch > switch-setup-modify mgmt-ip6 2001::2/64 gateway-ip 10.10.10.1 gateway-ip6 2001::35 dns-ip 10.10.10.11 dns-secondary-ip 10.10.10.1 domain-name corpinfo.com ntp-server 0.us.pool.ntp.org timezone US/Pacific <return>

To display the configured settings, use the switch-setup-show command:

CLI network-admin@switch > switch-setup-show

name: pleiades01mgmt-ip: 10.10.10.79/16mgmt-ip6: 2001::2/64in-band-ip: 192.168.21.1/24gateway-ip: 10.10.10.1gateway-ip6: 2001::35dns-ip: 10.10.9.1dns-secondary-ip: 10.10.10.1domain-name: corpinfo.comntp-server: 0.us.pool.ntp.orgtimezone: US/Pacificdate: 2013-10-31, 16:00:00phone-home: yesanalytics-store: optimized


The analytics-store parameter refers to the storage location of nvOS analytics. The parameter, optimized, indicates that a Fusion IO card is installed on the switch. You can now store statistics for connections, hosts, client servers, and CPU package logs on the Fusion IO card. When you specify optimized, the statistics are stored on the IO card with the highest amount of free space. If you select default, the statistics are stored on the nvOShard drive.

You can also configure a “Message of the Day” for users to see when logging into the switch. You may enter up to 511 characters including spaces. If you use spaces, enclose the MOTD in quotes. The MOTD can be used as a temporary or short term message to display downtime or other activity. To add the message, “switch down 2-4pm 3/31/15” use the following syntax:

CLI network-admin@switch > switch-setup-modify motd “switch down 2-4pm 3/31/15”

When you log into the switch, the MOTD is displayed after the software version:admin@pubdev03:~$ cliNetvisor OS Command Line Interface 2.2Please enter username and password: Username (network-admin): Password: Connected to Switch pubdev03; nvOS Identifier:0xa0000e3; Ver: 2.2.202036795pubdev03 down 2-4pm 3/31/15

You can also configure static banners to display switch information such as server identity.


Confirming Connectivity on the NetworkAfter you’ve connected your server-switch, you may want to take the time to ensure that you have connectivity by pinging an external IP address, and pinging a domain to ensure that you can resolve a domain name.

To ping the external network from the server-switch, use the ping command:

CLI network-admin@switch > ping 98.138.253.109

98.138.253.109 is alive.

To ping a domain, use the ping command again:

CLI network-admin@switch > ping yahoo.com

yahoo.com is alive.


Informational Note: Fusion IO cards are only available as an additional upgrade or when you purchase the F68-F1LT model.




29

Creating an Initial Fabric

After you complete the initial setup on the switch, you must create a new fabric for the switch or join an existing fabric. When switches form a fabric, the fabric becomes one logical switch, and shares state information as well as communicates commands so that any scope of a fabric- command is executed on each switch in the fabric. A switch must be in a fabric in order to keep track of the fabric state. However, a switch can be a member of fabric, and consist of a single switch. A switch leaving one fabric and joining another loses the fabric state of the first fabric and learns the fabric state of the second fabric.

1. To create a new fabric over Layer 2, use the following command:

CLI network-admin@switch > fabric-create name name-string

2. Create a name for the new fabric. To require a password before joining the fabric, use the password option. Press the return key after typing the password parameter:

CLI network-admin@switch > fabric-create name name-string <return>

password:*******Re-enter password:*******By default, the fabric is created on VLAN1. You can specify a different VLAN, but if you change the VLAN, you must recreate the fabric.

To create a fabric over Layer 3, use the fabric-join command and the switch IP address. For example,

CLI network-admin@switch > fabric-join switch-ip 192.168.2.2 vlan 20

3. To show fabric details, use the fabric-show command:

CLI network-admin@switch > fabric-show

name id vlan fabric-network control-network tid ---------------- ---------------- ---- -------------- --------------- ---- info-dev a000030:5537b46c 3 in-band in-band 365 ursa-lyon 6000210:566621ee 0 mgmt in-band 4928

You can also specify to send network traffic over the fabric network or the control plane network. To specify the fabric network, use the fabric-network parameter, specify the in-band or management IP address.

Specifying the fabric-network parameter sends traffic over the data path for fabric administration, which includes configuration changes and show commands.

To specify the control plane network, use the control-network parameter, and specify the in-band or management IP address.

Using the control-network parameter specifies the data path for control plane traffic, which includes status updates, vlag syncs, cluster syncs, and similar traffic.

Adding License Keys to nvOSThe license key for nvOS is bound to the serial number of the Pluribus Network switch and ships with the switch.

To install the license key, use the following syntax:

CLI network-admin@switch > software-license-install key license-key


The license key has the format of four words separated by commas. For example.

License Key: rental,deer,sonic,solace

Once the license key is installed, you can display information about the key using the following command:

CLI network-admin@switch > software-license-show format all layout vertical

switch: Pleaides01license-id: F-ASDF-NVOS2.0description: Freedom F-Line Advanced Software Defined Fabric License for

Netvisor 2.xkey: rental,deer,sonic,solacefeature: allupgrade-from:

To display the status of the server-switch, use the switch-status-show command:

CLI (switch)>switch-status-show

switch name value units state -------- --------------- ----- --------- ----- pluribus Switch Temp 41 degrees-C ok pluribus CPU1 Temp 57 degrees-C ok pluribus CPU2 Temp 49 degrees-C ok pluribus System Temp 46 degrees-C ok pluribus Peripheral Temp 30 degrees-C ok pluribus PCH Temp 43 degrees-C ok pluribus VTT volts ok pluribus CPU1 Vcore volts ok pluribus CPU2 Vcore volts ok pluribus VDIMM AB volts ok pluribus VDIMM CD volts ok pluribus VDIMM EF volts ok pluribus VDIMM GH volts ok pluribus +1.1 V volts ok pluribus +1.5 V volts ok pluribus 3.3V volts ok pluribus +3.3VSB volts ok pluribus 5V volts ok pluribus +5VSB volts ok pluribus 12V volts ok pluribus VBAT volts okpluribus switch-3.3v volts ok pluribus switch-1.1v volts ok pluribus switch-vcore volts ok pluribus switch-5.0v volts ok pluribus switch-2.5v volts ok pluribus switch-0.95v volts ok pluribus switch-1.8v volts ok pluribus switch-1.2v volts ok pluribus fan-1 3525 rpm ok pluribus fan-2 3760 rpm ok pluribus fan-3 3525 rpm ok pluribus fan-4 3760 rpm ok

This command displays the physical status of the switch including fan speed, electrical voltage, temperature.


31

To display additional physical information about the switch, use the switch-info-show command:

CLI (switch)>switch-info-show

switch: pluribusmodel: F64-HWENTchassis-serial: 1243PN8500014cpu1-type: Intel(R) Xeon(R) CPU E5-2620 0 @ 2.00GHzcpu2-type: Intel(R) Xeon(R) CPU E5-2620 0 @ 2.00GHzsystem-mem: 64.0Gswitch-device: okswitch-version: b2polaris-device: okgandalf-version: caff0044fan1-status: okfan2-status: okfan3-status: okfan4-status: okps1-status: okps2-status: n/a

To display information about a specific switch, specify the name of the switch in the command:

CLI network-admin@switch > switch-info-show name name-string

If you don’t specify the name of the switch, all switches in the fabric are displayed.


Modifying and Upgrading Software

A switch can contact an upgrade server, either directly or through a proxy, to download and upgrade to a newer version of nvOS. You can modify the upgrade process for the switch and add a proxy host.

What are Software Tracks?Software tracks are a method for Pluribus Networks to manage different software releases available to customers. The software track, release, is the default standard track, but other tracks, such as Beta, may be available for download.

CLI network-admin@switch > software-modify phone-home

Informational Note: This upgrade procedure applies to only one switch. To upgrade switches on the fabric or to create a “rolling upgrade” on the fabric, see



Updating nvOS on the Server-SwitchPluribus Networks switches can send “phone home” messages to the Pluribus Networks update servers to determine if a new release of software is available for download.

1. To view the current version of nvOS on the switch, use the following command:

CLI network-admin@switch > software-show

version: 2.2.1-202016524track: 2.2-releaseupgrade-status: availableversion-available: 2.2.0-202006524 -> 2.2.1-202016554auto-upgrade: disableuse-proxy: no

2. If the upgrade status indicates that a newer version of nvOS is available, request an update from the server:

CLI network-admin@switch > software-upgrade

upgrade successful. rebooting...

To check the status while the switch is upgrading, use the software-upgrade-status-show command.

3. To check the status of the switch after upgrading, reconnect to the switch, and enter the following command:

CLI network-admin@switch > software-show

version: 2.2.1-202016554track: 2.2-releaseupgrade-status: up-to-dateauto-upgrade: disableuse-proxy: no

To upgrade the current nvOS to a later release, use the software-upgrade command.

CLI network-admin@switch > software-upgrade package nvos-2.3.1-203018600.tgz

The parameter package allows you to specify the name of the upgrade file.

Informational Note: Allow plenty of time for the switch to download and install the new version of software. Do not interrupt the operation while the upgrade is in progress. When the upgrade is complete, the switch reboots and loads the latest version of the software. If you encounter any problems with the new version of the software, a previous version can be selected as the boot software. See “Topic Feedback” on page 1–33

Informational Note: Upgrading without an Internet connection - If the switch does not have direct access to the Internet but can use a proxy server, enter the software-modify use-proxy command to configure the proxy and then check for software upgrade availability. If there is no access to the Internet from the switch, contact Pluribus Technical Support for instructions on upgrading a switch offline.


33

To display information about the software upgrade path, you can use the software-track-show command.


Implementing a Fabric Upgrade or a “Rolling” Fabric UpgradeYou can now implement a fabric-wide upgrade and reboot the switches at the same time or in a sequential order. A fabric upgrade requires downloading the new nvOS software package to each switch, and rolling upgrade downloads the software packages from the update server and then copies the software to each switch as the upgrade proceeds.

The upgrade controller is the switch where the fabric-upgrade-start command is issued. All upgrade commands should be executed from the upgrade controller.

The fabric upgrade feature has two phases:

Upgrade — start the upgrade which creates and updates nvOS to new boot environments but does not reboot the fabric.

Reboot — reboots the entire fabric after all server-switches are upgraded to new boot environments. It is also possible during this phase to abort the process and discard the new boot environments.

The fabric is locked during the entire process and you cannot change any configurations during the process.

Before You Begin the Fabric Upgrade

Before you begin, you may want to consider the following options for the fabric-upgrade-start command:

auto-finish — you can specify to automatically reboot the entire fabric after the upgrade is complete.

rolling — specify if you want to perform a rolling fabric upgrade. A rolling fabric upgrade performs the upgrade procedure on a switch-by-switch basis and copies the software package from the controller to other switches in the fabric. If you specify no-rolling, all switches are booted after the upgrade.

abort-on-failure — specify if you want the upgrade to stop if there is a failure during the process.

manual-reboot — specify if you want to manually reboot individual switches after the upgrade process. If you specify no-manual-reboot, all switches reboot automatically after the upgrade is complete.

prepare — specify if you want to perform setup steps prior to performing the upgrade. This step copies the offline software package and then extracts and prepares it for the final upgrade process. Once you begin the prepare process, you cannot add new switches to the fabric.

reboot-parallel — specify to reboot switches in parallel if the switches are in a cluster configuration. Or, you can reboot them one at time using the reboot-single option.

reboot-group — specify the number of switches to reboot as a group in parallel mode. The default is the maximum number of switches in the fabric up to 100 switches.

Starting the Fabric Upgrade

1. Download the latest nvOS software from the update server onto a switch in the fabric. 2. Copy the nvOS software package to each switch in the fabric. 3. Select a switch in the fabric to act as the upgrade controller switch, and use the fabric-upgrade-start

command to begin the upgrade. 4. Depending on the options selected, the upgrade completes by reboot the fabric or rebooting all of the switches.



Starting the Rolling Fabric Upgrade

If you opted for a rolling fabric upgrade, then the upgrade controller switch begins copying to software packages to other switches in the fabric. Other than this step, the rolling fabric upgrade has the same behavior as a fabric upgrade depending on the selected options.

You can check the status of the upgrade using the fabric-upgrade-status-show command:

CLI (network-admin@sw1) > fabric-upgrade-status-showlog switch state----------------------------------------------- -------- ------------------(0:00:36)Upgrading software upgrade framework sw3 Running(0:00:08)Computing package update requirements. sw2 Running(0:00:12)Agent needs restart sw1* Agent restart wait

The first entry in the log is the duration of the upgrade process. It does not include waiting time. The switch with the asterisk (*) is the controller server-switch where the fabric-upgrade-start command was issued.

Additional commands for the fabric upgrade feature:

fabric-upgrade-finish — you can issue this command at any time during the fabric upgrade to reboot all nodes in the fabric and complete the upgrade. Once the upgrade phase is complete, all server-switches display the “Upgrade complete” message in the log field. You can then safely reboot the fabric.

fabric-upgrade-abort — aborts the software upgrade process. All changes to the server-switches are cleaned up and the server-switches do not reboot. The configuration lock on the fabric is also released.

If you issue the fabric-upgrade-abort command during the upgrade process, it may take some time before the process stops because the upgrade has to reach a logical completion point before the changes are rolled back on the fabric. This allows the proper cleanup of the changes.

fabric-upgrade-prepare-cancel — cancels a fabric upgrade that was prepared earlier.

fabric-upgrade-prepare-resume — resume a fabric upgrade that was prepared earlier.

fabric-upgrade-prepare-show — displays the status of prepared upgrades on the fabric nodes.

Enabling Administrative ServicesThere are many features of the Pluribus Networks fabric that require or can be enhanced using remote access. For example, when packets are written to a log file, you may want to transfer that file from a switch to a different system for analysis. Also, if you are creating a NetVM environment, an IOS image of the guest OS must be loaded on the switch.

There are two file transfer methods:

Secure File Transfer Protocol (SFTP)

Network File System (NFS)

Both methods must be enabled before you can use them. Because SFTP relies on Secure Shell (SSH), you must enable SSH before enabling SFTP.

1. To check the status of SFTP, use the following command:

CLI network-admin@switch > admin-service-show

switch nic ssh nfs web web-port snmp net-api icmp------- --- --- --- --- -------- ---- ------- ----

pleiades24 mgmt off on off 80 off off off


35

2. To enable SSH, use the following command:

CLI network-admin@switch > admin-service-modify nic mgmt ssh

admin-sftp-modify enablesftp password: <password>confirm sftp password: <password>

The default SFTP username is sftp and the password can be change using the admin-sftp-modify command:

CLI network-admin@switch > admin-sftp-modify

sftp password: <password>confirm sftp password: <password>


switch nic ssh nfs web web-port snmp net-api icmp------ --- --- --- --- -------- ---- ------- ----pleiades24 mgmt on on off 80 off off off

CLI network-admin@switch > admin-sftp-show

switch: pleiades24sftp-user: sftpenable: yes

Use SFTP from a host to the switch, and login with the username sftp and the password that you configured for SFTP. Then you can download the available files or upload files to the switch.

3. You can check the status of NFS service and enable it using the following command:


switch nic ssh nfs web web-port snmp net-api icmp

------ --- --- --- --- -------- ---- ------- ----

pleiades01 mgmt on off on 80 off on on

To enable NFS, use the following command:

CLI network-admin@switch > admin-service-modify nic mgmt nfs

After you enable NFS, the directory /nvOS is mountable using NFS through the management IP addresses for access to the files in that directory.

Saving and Restoring Server-Switch ConfigurationsA switch contains local configuration information such as port settings as well as fabric configuration information. Fabric configurations are stored on every switch in the fabric and does not require that you save and restore before replacing a switch. When a switch is replaced, removed, or otherwise disrupted, you can save and restore the local configuration information.

The information that is saved and restored on the local switch includes the following:

VNETs with VNET manager running on the switch


Port VLAN associations

Netvisor Zone configuration details, but not any modifications to NetZones such as installed applications

Netvisor VMM configuration details, but not ISO images or disk images

Netvisor KVM configuration details, but not ISO images or disk images

Network services running on the switch

To display a full list of the current configuration details for a switch, use the running-config-show command.

SFTP and NFS can be used to transfer the configuration file, but you must enable the two features before using them.

1. To save the switch configuration to a file, use the following command:

CLI network-admin@switch > switch-config-export export-file pleiades24

Exported configuration to /nvOS/export/pleiades24.2013-11-04T22.33.31.tar.gz

2. To display the files available for import and export, use the following command:

CLI network-admin@switch > switch-config-show

switch export-filepleiades24 pleiades24.2013-11-04T22.33.31.tar.gz

You can now copy the configuration file to a different host using SFTP or NFS. For example, you can SFTP to the switch-ip-address, and login using the SFTP password. Then use cd/nvOS/import, and use get to download the configuration file.

The switch-config-export command is used to export the configuration of the local switch. The file that is created is a tar file that includes a number of configuration files for the switch. The file is created under /nvOS/export. This is the command used to export the current configuration on the local switch. vAlso, each time you reset the switch using the command, switch-config-reset, a backup of the configuration is made and places a file in the same location.

Once the switch configuration is exported, it becomes available to import on the same switch, by using the switch-config-copy-to-import command. nvOS copies the configuration tar file from the /nvOS/export to the /nvOS/import directory. Once in the /nvOS/import directory, it is possible to use the switch-config-import command to import the switch configuration.

The switch-config-import command is used to import a configuration on the local switch. When using that command, the intention is to import a switch configuration t previously exported by the same switch.

The switch-config-import command has a few parameters to it. The ignore-system-config and the apply-system-config parameters are 2 parameters that allow the imported configuration of the switch to override or not override the currently configured information found under the switch-setup-show command. When you select the ignore-system-config parameter, the local configuration is saved to an archive. If you select apply-system-config, the settings in the tar file are applied to the local switch.

Caution! There is a potential for data loss when restoring a configuration. The configuration on the switch is replaced by the configuration stored in the import file. Although ISO images and disk-library images are not likely to disappear, you should only perform switch-config-import on a switch that doesn’t have important data stored on it. As a precaution, consider using the command switch-config-export to save the data on the switch that you are importing the configuration file. Also, copy the ISO images and disk images from the switch using the iso-image-library and disk-library-image-export commands and copying the files from the switch.


37

When you import a configuration using the switch-config-import command, the current configuration on the switch is overwritten by the imported configuration file.

The skip-fabric-join option imports the fabric configuration from the tar file. However, this information may be out of date with respect to the fabric if transactions have occurred on the fabric since the file was exported which causes the imported configuration to be out-of-sync with the current fabric. The alternative is to specify do-fabric-join, which extracts the fabric name from the tar file, and attempts to join the fabric and download the current fabric configuration, so that it is in sync with the rest of the fabric. The fabric configuration in the tar file is ignored, but cluster and local configurations are imported from the tar file.

When a switch that was part of a cluster is replaced, the fabric-join repeer-to-cluster-node command is used for the new switch to receive all required switch configuration, including the local configuration.

To upload a configuration file to a switch and set the configuration for the switch using the configuration file, you must transfer the configuration file to the target switch using the following sequence of commands:

sftp sftp@<switch-ip-address>Connecting to switch-ip-addressPassword: <password>sftp> cd nvOS/importsftp> put pleiades24.2013-11-04T22.33.31.tar.gz

Now load the configuration file which replaces the current configuration on the switch with the information in the file.

CLI network-admin@switch > switch-config-import import-file pleiades24.2013-11-04T22.33.31.tar.gz

New configuration imported. Restarting nvOS...Connected to Switch pleiades24; nvOS Identifier:0xb000011; Ver: 0.19.3747

There are many options available that allow you to control how the switch-config-import modifies the switch, including the following:

ignore-system-config - ignore the current system configuration. The settings in the *.tar file are not applied to the local switch.

apply-system-config — apply the system configuration in the imported file. The settings in the *.tar file are applied to the local switch. You typically do not want to use this option as it changes the in-band IP address and other settings.

skip-fabric-join — opt out of joining the fabric. This setting imports the fabric configuration from the *.tar file, but this information may be out of date with respect to the fabric if additional transactions occur on the fabric since the file was exported.

Informational Note: The configuration file must use the *.tar.gz extension to be recognized by nvOS.

CAUTION! Loading the configuration file causes nvOS to restart which results in a brief interruption to switch traffic flow.


do-fabric-join — join the current fabric. This setting extracts the fabric name from the *.tar file and attempts to join the fabric. Then the switch contacts the current fabric to download the configuration so that the switch is in sync with the rest of the fabric. Cluster and local configurations are imported from the *.tar file.

no-replace-switch — do not replace the current switch.

replace-switch — replace the current switch. This setting is used to replace a faulty switch and after importing the file, has the same configuration as the replaced switch. This replaces all of the local, cluster, and fabric configuration by downloading the configurations from peer switches. No configuration is necessary or advised before running this command. However, you need to run the initial quickstart to obtain an in-band IP address.

By default, the initial switch system configuration, management IP addresses and other parameters, are not applied if there is another switch in the fabric with the same settings. To apply the initial settings, use the apply-system-config option. Also, by default, the imported configuration attempts to join the same fabric that the original switch was a member. If that join fails, then the import fails. You can avoid this issue by using the skip-fabric-join option. Finally, if the original switch is still on the network and you want to copy the configuration to a new switch, but you want to prevent the new switch from taking ownership of any objects specific to the original switch, such as VNET services, or VLAN port settings, you must use the no-replace-switch option.

Copying and Importing Configuration Files You can create a configuration file to import to another switch by using the switch-config-copy-to-import command. To create a configuration file with the name config-092613 to import on another switch, use the following syntax:

CLI network-admin@switch > switch-config-copy-to-import export-file config-092613

After you create the configuration file, you can export it to /nvOS/export/ directory, and SFTP to it from the target switch.

To review the available files for import and export, use the following syntax:

CLI network-admin@switch > switch-config-show

switch export-filepbg-nvos config-092613.tar.gz

Depending on the available remote access services, you can now copy the configuration file to a different switch. For example, you can SFTP to another switch using the IP address of the switch, login as SFTP with the password that you previously set, cd /nvOS/import and get the configuration file.

To upload the configuration file to the target switch and set the configuration from the configuration file, transfer the configuration file to the target switch with the IP address, 192.168.3.35.


Changing the IP Port for vManagevManage is a Web-based service and it listens on an IP port to accept communications. By default, vManage listens on port 80 on the management IP address that you set during the initial configuration, and can be reached using a supported Web browser such as Safari, Firefox, or Chrome using the URL http://mgmt-ip. In some cases, you may want to configure vManage to listen on a different port as in the case of a virtual load balancer sending traffic arriving on port 80 of the management IP address to other systems. In this case, vManage cannot listen on port 80.



39

Use the admin-service command to change the listening port. Changing the port disrupts any current connections to vManage.

1. To change the listening port to 8080 for vManage, use the following syntax:

CLI network-admin@switch > admin-service-modify nic mgmt web-port 8080

2. To check the status of admin services, use the following command:


switch nic ssh sftp nfs web web-port snmp net-api icmppleiades24 mgmt on on on on 8080 off on on

After this change, you use the URL http://mgmt-ip:8080.

Configuring Virtual Network Interface Cards (vNICs)You can create vNICs on the switch to provide connectivity for some virtual services. You can use the vNICs for data or management purposes. To create a vNIC with the IP address 172.16.21.33/24 on VLAN 301 for data traffic and the type of vNIC is e1000, use the following command:

CLI network-admin@switch > switch-vnic-create ip 172.16.21.33/24 assignment none vlan 301 if data vm-nic-type e1000

To modify the configuration, use the switch-vnic-modify command, and to delete the vNIC, use the switch-vnic-delete command. To display information about vNICs, use the switch-vnic-show command.

CLI network-admin@switch > switch-vnic-show layout vertical

nic: global.eth0ip: 10.12.111.103/24assignment: staticmac: 66:0e:94:21:c8:a2vlan: 10vxlan: 0if: datato_vnic_flow_name:

Informational Note: There are three types of interfaces in nvOS:

• Physical• vNIC applies to virtual interfaces created for the server-switch.• VNET interfaces created for virtual services.


Displaying System Statistics on a Server-SwitchYou display system statistics on a server-switch using the system-stats-show command:

CLI network-admin@switch > system-stats-show layout vertical

switch: pleiades24uptime: 1h22m26sused-mem: 27%used-swap: 0%swap-scan: 0cpu-user: 0%cpu-sys: 1%cpu-idle: 98%

The swap-scan output displays the number of scans performed on the swap. A nonzero number indicates that memory is paged from the physical memory (RAM) to virtual memory (disk or swap). A consistently high value indicates that all memory, both physical and virtual, is exhausted and the system may stop responding.




41

Displaying Layer 2 Networking DetailsTo display fabric-wide Layer 2 (L2) networking table, use the l2-table-show command. This table displays the MAC addresses associated with IP addresses and the ports that the MAC addresses appeared.

CLI network-admin@switch > l2-table-show

switch: pleiades24mac: 00:04:f2:41:cb:d4ip: 10.10.11.210vlan: 1vxlan: 0state: activecreate_time: 007-10,10:00:13last-seen: 2013-05-23,12:41:51hit: 1983migrate: 0drops: 0switch: pleiades24mac: 00:25:90:62:12:3aip: 10.10.10.115vlan: 1vxlan: 0last_time 2013-05-23,12:53:51hit: 89803migrate: 2398022drops: 1863639switch: pleiades24mac: 64:0e:94:28:00:faip: 10.13.3.23vlan: 1vxlan: 0last_time 2013-05-23,12:57:53hit: 13989migrate: 2drops: 177


Using the command options can help quickly determine fabric activity. For example, using the l2-table-show sort-desc reveals the MAC address that appears most frequently and the associated port:

CLI network-admin@switch > l2-table-show sort-desc hit

switch: pubdev02mac: 06:a0:00:0e:30:81vlan: 38intf: 128ports: 47-48state: activecreate-time: 04-13,15:03:08last-seen: 09:29:57hit: 112switch: pubdev02mac: 06:a0:00:0e:30:81vlan: 22create-time: 01-21,11:02:30last-seen: 01-30,11:22:16hit: 60migrate: 1119drops: 74

This information may lead to further investigation of the events by using the connection-stats-show commands:

CLI network-admin@switch > connection-stats-show ip 10.10.11.3

switch: pleiades24mac: 66:0e:94:21:0e:7bvlan: 14ip: 172.16.23.1port: 65iconns: 13oconns: 0ibytes: 132Kobytes: 375Mtotal-bytes: 375Mfirst-seen: 06-16,08:15:24last-seen: 06-16,08:19:11last-seen-ago: 31d30m19sswitch: pleiades24mac: 66:0e:94:21:f3:34vlan: 14ip: 172.16.23.1port: 65iconns: 14oconns: 0ibytes: 132Kobytes: 375Mtotal-bytes: 375Mfirst-seen: 06-16,11:54:12last-seen: 06-16,11:58:25last-seen-ago: 30d20h51m5sswitch: pleiades24mac: 66:0e:94:21:67:e1vlan: 11


43

ip: 172.16.23.1port: 65iconns: 57oconns: 0ibytes: 398Kobytes: 1.10Gtotal-bytes: 1.10Gfirst-seen: 06-20,15:05:39last-seen: 07-02,09:44:05last-seen-ago: 14d23h5m25sswitch: pleiades24mac: 66:0e:94:21:78:2evlan: 14ip: 172.16.23.1port: 65iconns: 69oconns: 1ibytes: 662Kobytes: 1.83Gtotal-bytes: 1.83Gfirst-seen: 06-16,14:58:42last-seen: 06-17,11:12:48last-seen-ago: 29d21h36m42s

Checking and Fixing Layer 2 Table Issues

You can use the command, l2-check-show, to display any discrepancies with the Layer 2 entries:

CLI network-admin@switch > l2-check-show

pubdev01: Matched: 12To repair any issues with the Layer 2 table, use the l2-check-fix command:

CLI network-admin@switch > l2-check-fix

OK:12

Rebooting, Powering Off, and Resetting the Server-SwitchThere are two recommended ways to reboot a switch:

CLI command switch-reboot

Power button

To reboot the switch using the CLI, use the following command:

CLI network-admin@switch > switch-reboot

Alternatively, you can use the power button located on the front of the switch to power off.

Informational Note: The switch-reboot command applies only to the switch where the command is executed. You cannot reboot a remote switch using this command.


To power off the switch, press and hold the front power button for approximately ten seconds until the power button light changes from a rapid blink to a slow flashing cycle. The power button light tuns off and now the switch is powered off.

You can also use the command, switch-poweroff, to turn off a switch.

To complete the process, switch the power toggle on the rear of the switch from 1 to 0. The system is now completely powered off.


Installing the nvOS Linux API nvOS is bundled with a Linux API that allows installation of nvOS on any Linux-based server. The API installs libraries under /lib64, documents under /usr/share/java/doc/libnvOS/index.html, and sample code under /usr/share/src/nvOS/samples.

1. Modify the SFTP permissions on the switch using the admin-sftp-modify enable command. To install the API on a Linux platform, use the following command:

CLI network-admin@switch > api-install linux-host name linux-host-string user user-string

To run nvOS on the Linux host, use the following command:

cli --host switch-name ip

Informational Note: You must physically connect the Linux host to the switch.



45

Configuring Rapid Spanning Tree Protocol (RSTP)

Spanning Tree Protocol (STP) is a standard inter-switch protocol to ensure that an ad hoc network topology is loop-free at Layer 2, on a per-VLAN basis. If your network connections form loops and STP is disabled, packets re-circulate between the switches, causing a degradation of network performance. If you are certain that your network connections are loop-free, you do not need to enable STP.

To build a loop-free topology, switches (“bridges”) have to determine the root bridge and compute the port roles, root, designated, or blocked. To do this, the bridges use special data frames called Bridge Protocol Data Units (BPDUs) to exchange information about bridge IDs and root path costs. BPDUs are exchanged regularly, typically at two second intervals, and enable switches to keep track of network topology changes and to start and stop forwarding on ports as required. Hosts should not send BPDUs to their switch ports and to avoid malfunctioning or malicious hosts from doing so, the switch can filter or block BPDUs. If you enable BPDU filtering on a port, BPDUs received on that port are dropped but other traffic is forwarded as usual. If you enable BPDU blocking on a port, BPDUs received on that port are dropped and the port is shut down.Pluribus Networks switches support the Per VLAN Spanning Tree (PVST) variation of STP, and if a PVST BPDU is detected on a port, PVST is used on that port.

Rapid Spanning Tree Protocol is also supported by modifying an STP port and configuring it as an edge port.

Before you begin, view the status of STP on the switch by using the following command:

CLI network-admin@switch > stp-show

switch: pleiades24enable: yesbridge-priority: 32768hello-time: 2forwarding-delay: 15max-age: 20switch: pleiades23enable: yesbridge-priority: 32768hello-time: 2forwarding-delay: 15max-age: 20

1. To disable STP, use the following command:

CLI network-admin@switch > stp-modify disable

Informational Note: RSTP is enabled on the switch by default.


2. To display the STP state, use the following command:

CLI network-admin@switch > stp-state-show

switch: techpubs-aquila2vlan: 1name: stg-default-stgbridge-id: 64:0e:94:18:00:8fbridge-priority: 32769root-id: 64:0e:94:18:00:8froot-priority: 32769root-port: 128hello-time: 2forwarding-delay: 15max-age: 20disabled: nonelearning: noneforwarding: 65-66,255discarding: 128edge: 65-66,255designated: 65-66,255alternate: nonebackup: nonevlag-mirror: none


47

To display information about STP on ports, use the stp-port-show command:

CLI network-admin@switch > stp-port-show

switch port block filter guard -------- ---- ----- ------ ----- pubdev03 65 off off no pubdev03 66 off off no pubdev03 67 off off no pubdev03 68 off off no pubdev03 69 off off no pubdev03 70 off off no pubdev03 71 off off no pubdev03 72 off off no pubdev03 255 off off no pubdev02 65 off off no pubdev02 66 off off no pubdev02 67 off off no pubdev02 68 off off no pubdev02 69 off off no pubdev02 70 off off no pubdev02 71 off off no pubdev02 72 off off no pubdev01 65 off off no pubdev01 66 off off no pubdev01 67 off off no pubdev01 68 off off no pubdev01 69 off off no pubdev01 70 off off no pubdev01 71 off off no pubdev01 72 off off no pubdev01 255 off off no

3. To filter BPDUs on port 17, use the following command:

CLI network-admin@switch > stp-port-modify port 17 filter

4. To block BPDUs on port 17 and shut down the port if BPDUs are received on the port, use the following command:

CLI network-admin@switch > stp-port-modify port 17 block

5. To stop blocking BPDUs on port 17, use the following command:

CLI network-admin@switch > stp-port-modify port 17 no-block

6. You can disable STP on a port or a group of ports. If the devices connected to the switch ports are hosts and not downstream switches, or you know that a loop is not possible, then disable STP and the port is enabled much faster when the switch restarts.

7. To enable RSTP on port 35, use the following command:

CLI network-admin@switch > stp-port-modify port 35 edge

8. To enable STP, use the following command:

CLI network-admin@switch > stp-modify enable





49

Configuring Link Aggregation Control Protocol (LACP)

Configuring Trunking for Link Aggregation (LAG)

Configuring Layer 2 Multipathing for Virtual Chassis Link Aggregation

Configuring Active-Active VLAG

Active-Active VLAG over a Trunk with a Server-Switch and Host

Link Aggregation Control Protocol (LACP) is part of the IEEE specification 802.3ad that allows you to bundle several physical ports to form a single logical channel. When you change the number of active bundled ports on a port channel, traffic patterns reflect the rebalanced state of the port channel.

LACP supports the automatic creation of Gigabit Ethernet port trunks by exchanging LACP packets between ports. It learns the capabilities of port groups and informs the other ports. Once LACP identifies correctly matched Ethernet links, it facilitates grouping the links into a Gigabit Ethernet port trunks.

LACP packets are exchanged between ports in these modes:

Active — Places a port into an active negotiating state, and the port initiates negotiations by sending LACP packets.

Passive — Places a port into a passive negotiating state where the port responds to LACP packets it receives but does not initiate LACP negotiation. In this mode, the port channel group attaches the interface to the bundle.

Off — LACP is not enabled on the switch port or trunk.

Active and passive modes allow LACP to negotiate between ports to determine if they can form a port channel based on criteria such as port speed and trunking state.

To enable or disable LACP, or change the system priority, use the following command:

CLI network-admin@switch > lacp-modify enable system-priority 35000

The default system priority value is 32768 with a range from 0 to 65535.

LACP system priority can be configured on each switch running LACP. The configuration uses the default value or you can use another value. LACP uses the system priority with the MAC address to form the system ID and also during negotiation with other systems.

To create a trunk with LACP, use the following command:

CLI network-admin@switch > trunk-create name trunk23 port 20-36 lacp-mode active

To modify a trunk with LACP, use the following command:

CLI network-admin@switch > trunk-modify name trunk23 lacp-mode passive

To modify a port configuration and add LACP priority to the port, use the following command:

CLI network-admin@switch > port-config-modify port 33 lacp-priority 34

LACP port priority is configured on each port using LACP. You can use the default value, 32768, or configure a specific value from 0 to 65535. LACP uses the port priority with the port number to form the port identifier. The port priority determines which ports should be in standby mode when there is a hardware limitation that prevents all compatible ports from aggregating.


Configuring Trunking for Link Aggregation (LAG)

To configure a trunk for aggregating the links connected to ports 1, 2, 3, use the following steps:

1. Create a trunk called trunk-1 on ports 1, 2, 3, enter the following command:

CLI network-admin@switch > trunk-create name trunk-1 port 1,2,3

2. To verify the configuration, use the trunk-show command:

CLI network-admin@switch > trunk-show

name port speed autoneg jumbotrunk-1 1-3 10g off off

3. Modify the trunk configuration by removing port 2:

CLI network-admin@switch > trunk-modify name trunk-1 port 1,3

4. Verify the updated trunk configuration.

CLI network-admin@switch > trunk-show

name port speed autoneg jumbotrunk-1 1,3 10g off off

Notice that the ports have changed from 1-3 to 1,3 indicating that port 2 is no longer a member of the trunk configuration.

5. Delete the trunk configuration from the switch:

CLI network-admin@switch > trunk-delete name trunk-1

Verify that the trunk configuration is removed by using the trunk-show command.

LACP Control Changes

This feature enables ports in a static LACP trunk to operate as individual ports in the absence of proper LACP negotiation with network peer. Once any port member hears a LACP PDU from the peer, all port members of the trunk are bundled to operate as a trunk. This feature is useful for servers with multiple network interfaces that would use PXE boot.

Informational Note: You must create unique names for each VLAG.

Informational Note: This feature is not supported on virtual link aggregation (vLAG) configurations.


51

With this configuration, nvOS creates the trunk in the switch, but does not add any of the port to the trunk. The ports continue to operate individually until LACP PDUs are heard on any of the ports that constitute the trunk. Once LACP PDUs are heard from the peer, then all ports of the trunk cease to operate individually and are added to the trunk.

If no LACP PDUs are received for the number of seconds configured as the fallback timeout, nvOS LACP checks if LACP negotiation has expired if LACP negotiation has expired, the ports return to individual mode. If LACP negotiation has not expired, another fallback timer is scheduled at a value equal to the fallback timeout.

Notes

LACP fallback timeout is set to 50 seconds and LACP negotiation is set to default 90 seconds.

After 50 seconds, fallback timer is rescheduled because LACP negotiation has not expired.

After an additional 40 seconds (90 total) LACP negotiation expires and become inactive. Another 10 seconds passes (100 seconds total) when the fallback timer expires and the ports fallback to individual.

Configuring Layer 2 Multipathing for Virtual Chassis Link AggregationYou can aggregate links between two switches by configuring Layer 2 multipathing and virtual chassis Link Aggregation.

A virtual chassis Link Aggregation Group (VLAG) allows links that are physically connected to two different switches to appear as a single Ethernet trunk to a third device. The third device can be a server, switch, or any other networking device. A VLAG can create Layer 2 multipathing which allows you to create redundancy, enabling multiple parallel paths between nodes.

A VLAG requires that a least one cross connection between the two switches, also called peers, where the VLAG links terminate. The specific ports that connect the different switches, do not require explicit configuration before creating a VLAG.

VLAGs can provide the following benefits:

Allows a single device to use an Ethernet trunk across two access layer (top of rack) switches.

Eliminates Spanning Tree Protocol (STP) blocked ports.

Provides a loop-free topology

Provides fast convergence if a link or device fails.

Provides link-level resiliency.

Helps ensure high availability.


VLAG Topology ExamplesFigure 1:L2 Design - Leaf and Spine with Active-Passive VLAG

Figure 2:L2 Design - Leaf and Spine with Active-Active VLAG


53

Figure 3:L2 Design - Leaf and Third Party Spine without Multichassis LAG or VPC Mode

Figure 4:L2 Design - Leaf and Third Party Spine with Multichassis LAG, vPC and MLAG

To create a VLAG for aggregating links connected to ports 70 on the local switch and the peer called, eng-switch-b, you must first create a cluster configuration between the two switches. Pluribus Networks switches must be members of a cluster configuration before you can add VLAGs to them.

Third Party Interoperability with nvOS

Operating System Host PN Switch

SmartOS, OpenSolaris, Illuminos, Oracle Solaris

Create aggr with lacp-mode passive.

Create lacp-mode active ad lacp-timeout fast.

Red Hat, Linux Create bond with mode 3.

Create lacp-mode off.


Configuring Active-Active VLAG Using the sample topology in Figure 5 Active-Active VLAG over a Trunk with a Server-Switch and Host, use the following steps to configure Active-Active VLAG:

Figure 5:Active-Active VLAG over a Trunk with a Server-Switch and Host

Three Pluribus Networks switches in a common fabric with the Spine switch as the RSTP root. It is important to note that ports 19-22 on PN-0 and PN-1 are ports connected to PN-2 (Spine). Port 26 connects PN-0 to PN-1 for the cluster configuration required for VLAG.

1. On PN-2, use the following command:

CLI network-admin@switch > stp-modify bridge-priority 4096

2. Create the fabric and add the switches:

CentOS Create bond with mode 4.

Create lacp-mode on.

Informational Note: There must be a physical connection between PN-0 and PN-1 before you can configure VLAG.

Operating System Host PN Switch


55

On PN-2, use the fabric-create command:

CLI network-admin@switch > fabric-create name fab-vlag

On PN-1, join the fabric:

CLI network-admin@switch > fabric-join name fab-vlag

On PN-0, join the fabric:

CLI network-admin@switch > fabric-join name fab-vlag

3. Create VLAN connectivity from the top switch to the bottom:

On PN-2, create the VLAN with scope fabric:

CLI network-admin@switch > vlan-create id 25 scope fabric

On PN-0, add the VLAN and untag the port connected to the host.

CLI network-admin@switch > vlan-port-add vlan-id 25 untagged ports 9

On PN-1, add the VLAN and untag the port connected to the host.

CLI network-admin@switch > vlan-port-add vlan-id 25 untagged ports 9

On PN-0, modify the host STP port to be an edge port.


On PN-1, modify the host STP port to be an edge port.


4. Create a cluster configuration between PN-1 and PN-0. This creates the cluster across port 26.

On PN-0, enter the cluster-create command:

CLI network-admin@switch > cluster-create name vlag cluster-node-1 PN-0 cluster-node-2 PN-1

5. You must disable ports between PN-2 and PN-0, and then create a static trunk between them:

On PN-0, modify the ports facing PN-2:

CLI network-admin@switch > port-config-modify port 19,20 disable


Then create the trunk on PN-0:

CLI network-admin@switch > trunk-create name pn0-to-pn2 port 19,20 lacp-mode off

CLI network-admin@switch > trunk-show format all layout vertical

switch: PN-0intf: 128name: pn0-to-pn2port: 19-20speed: 10gautoneg: offjumbo: offenable: offlacp-mode: offlacp-priority: 32768lacp-timeout: slowreflect: offedge-switch: nopause: nodescription: loopback: offmirror-only: offunknown-ucast-level: 100%unknown-mcast-level: 100%broadcast-level: 100%lport: 0rswitch-default-vlan: 0port-mac-address: 06:60:00:02:10:80status: config: send-port: 0

From the above output, you can find the name of the trunk configuration, pn0-to-pn2. You need this information to create the VLAG.

Then, on PN-1, repeat the same commands to create a trunk between PN-1 and PN-2.

6. You must disable ports between PN-2 and PN-1, and then create a static trunk between them:

On PN-1, modify the ports facing PN-2:


57

port-config-modify port 21,22 disable

CLI network-admin@switch > trunk-create name pn1-to-pn2 port 21,22 lacp-mode off

CLI network-admin@switch > trunk-show format all layout vertical

switch: PN-0intf: 129name: pn1-to-pn2port: 21-22speed: 10gautoneg: offjumbo: offenable: offlacp-mode: offlacp-priority: 32768lacp-timeout: slowreflect: offedge-switch: nopause: nodescription: loopback: offmirror-only: offlport: 0rswitch-default-vlan: 0port-mac-address: 06:60:00:02:10:80status: config: send-port: 0

7. Now create the VLAG from the bottom switches going upward and static trunk from the top down. Keep one side of the VLAG disabled while you configure this step.

On PN-0, use the vlag-create command:

CLI network-admin@switch > vlag-create name to-spine port 128 peer-port 129 peer-switch PN-1 lacp-mode off mode active-active

On PN-2, create a trunk with the name trunk-pn:

CLI network-admin@switch > trunk-create name trunk-pn port 19,20,21,22 lacp-mode off

8. Now, you can enable ports on all switches:

On PN-2, enter the port-config-modify command:

CLI network-admin@switch > port-config-modify port 19,20,21,22 enable


CLI network-admin@switch > port-config-modify port 19,20 enable


CLI network-admin@switch > port-config-modify port 21,22 enable


9. Create the server-facing VLAG:

On PN-0, enter the vlag-create command:

CLI network-admin@switch > vlag-create name to-spine port 9 peer-port 9 peer-switch PN-1 lacp-mode active mode active-active

Display the VLAG configuration information:

CLI network-admin@switch > vlag-show format all layout vertical

id: a000024:0name: to-spinecluster: vlagmode: active-activeswitch: pubdev02port: trunk2peer-switch: pubdev01peer-port: 129failover-move-L2: nostatus: normallocal-state: enabled,uplacp-mode: offlacp-timeout: slowlacp-key: 26460lacp-system-id: 110013777969246




59

Configuring Tagged and Untagged VLANs

Creating untagged VLANs is useful for connecting the switch to devices that do not support IEEE 802.1Q VLAN tags. You can configure ports to map untagged packets to a VLAN.

Reserved VLANs and VLAN 0 and 1

The VLAN identifier is a 12-bit field in the header of each packet. Therefore, the maximum number of VLANs you can define is 4096. Pluribus Networks switches reserve VLANs 0, 1, 4093, 4094, and 4095 for internal use.VLAN 0 is not a standard VLAN in nvOS. It is used to represent all untagged or non-VLAN traffic. VLAN 1 is the default untagged traffic VLAN. Untagged traffic can be mapped to any VLAN, but by default, it is mapped to VLAN 1.

1. To create a VLAN on the current switch, with the identifier 595, use the following command:

CLI network-admin@switch > vlan-create name VLAN595 id 595 scope local

By default, all ports are trunked on the new VLAN. If you want to specify ports that are trunked, use the optional parameter, ports, with a comma separated list of ports, or specify a range of ports.

In some cases, you may not want the VLAN created on all ports. You can specify none to apply the VLAN to internal ports only.

CLI network-admin@switch > vlan-create id 35 scope fabric ports none

CLI network-admin@switch > vlan-show

switch: pubdev01id: 35nvid: a000030:23scope: fabricname: vlan-35active: yesstats: yesvrg: 0:0ports: 65-72,255untagged-ports: noneactive-edge-ports: noneswitch: pubdev02

To map ports on different switches into the scope fabric VLAN, use the following command:

CLI network-admin@switch > vlan-port-add switch switch-name ports

It’s important to note that if you create a VLAN with scope fabric and untag all ports, you can cause problems with the fabric communication.

Informational Note: The untagged VLAN feature is not the same as the default VLAN using the IEEE 802.1Q tag 1.


To modify a VLAN name, use the vlan-modify command to modify VLAN 25 name from blue to red:

CLI network-admin@switch > vlan-modify id 25 name blue

To modify the port list, use the vlan-port-add and the vlan-port-remove commands.

2. To display the VLANs configured on the switch, use the vlan-show command.

CLI network-admin@switch > vlan-show format all layout vertical

switch: pubdev01id: 1nvid: a000030:1scope: localname: default-1active: yesstats: yesvrg: 0:0ports: 1-72,128,255untagged-ports: 1-72,128,255active-edge-ports: 31,45-46,66,128active-edge-ports: 65,128-129 switch: pubdev02id: 1nvid: a000024:1scope: localname: default-1active: yesstats: yesvrg: 0:0ports: 1-72,128-129,255untagged-ports: 1-72,128-129,255

3. To configure ports 17 and 18 to accept untagged packets and map them to VLAN 595, use the following command:

CLI network-admin@switch > vlan-port-add vlan-id 595 ports 17,18 untagged


61

Displaying VLAN StatisticsYou can display network traffic statistics per VLAN using the vlan-stats-show command. This may be useful when troubleshooting network issues.

CLI network-admin@switch > vlan-stats-show format all layout vertical

switch: pubdev03time: 10:51:02vlan: 1ibytes: 36.2Tipkts: 89.0Gidrops-bytes: 119Midrops-pkts: 313Kobytes: 0opkts: 0odrops-bytes: 0odrops-pkts: 0switch: pubdev03time: 10:51:02vlan: 35ibytes: 10.8Kipkts: 154idrops-bytes: 0idrops-pkts: 0obytes: 0opkts: 0odrops-bytes: 0odrops-pkts: 0switch: pubdev02time: 10:51:02vlan: 1ibytes: 34.9Tipkts: 84.6Gidrops-bytes: 3.03Midrops-pkts: 5.69Kobytes: 0opkts: 0odrops-bytes: 0odrops-pkts: 0

The output displays the following information:

switch

time

VLAN ID

incoming and outgoing bytes

incoming and outgoing packets

incoming and outgoing dropped bytes

incoming and outgoing dropped packetsTopic FeedbackWas this topic useful to you? Please provide feedback to improve the content.




63

Implementing Virtual Networks

Overview

Using VNETs with nvOS

Creating a Virtual Network

Adding DHCP Service to a VNET

Verify Administrator User Creation

Configuring Administration Login Using SSH

Adding a Default Gateway to the VNET

Adding Ports to the VNET

Configuring Virtual Resource Groups

OverviewA Virtual Network (VNET) is an abstract network resource realized across a fabric of Pluribus Networks switches. Using VNETs, you can segregate a physical fabric into many logical networks, each with its own resources, network services, and Quality of Service (QoS) guarantees. A VNET allows you to completely separate all traffic in one VNET from the traffic of other VNETs.

Figure 1:Using VNETs with nvOS


Each VNET has a single point of management. As the fabric administrator, you can create VNETs and assign ownership of each VNET to individuals with responsibility for managing those resources. You can create separate usernames and passwords for each VNET manager. Using the separate VNET administration credentials, the VNET admin can use Secure Shell (SSH) to connect to the VNET manager and access a subset of the nvOS® CLI commands to manage that VNET. This way, multiple tenants can share a fabric with each managing a VNET with security, traffic, and resource protection from other VNETs.

VNETs are very flexible and can be used to create complex network architectures. For example, a Pluribus Networks switch, or a fabric of switches, can be used to create multiple tenant environments in an OpenStack deployment. In Figure 1 Using VNETs with nvOS, there are three VNETs, each with a management interface and a data interface. Each VNET is assigned an IP address pool used for DHCP assignment of IP addresses to each node, server, or OS component.

Underlying each VNET is the VNET manager. Each VNET manager runs in an OpenSolaris zone. When services are created for a VNET they occupy the same zone on a server-switch. This is called a shared service and it is the default when creating services. However, each zone can only support a single instance of a service. If a second service instance is needed for a VNET, then it needs to occupy a separate zone. This is called a dedicated service. In most cases, you can create services as shared unless you specifically want to create a dedicated service.

When a fabric is created, a VNET is automatically created and named fabric-name-global. This VNET owns all resources within the fabric, and as new VNETs are created, resources are moved from the default VNET to the new VNETs. Global services remain in the default VNET unless assigned specifically to a VNET. The software license for IPS allows only the global VNET, but you can use it to create DHCP servers and other services for the entire switch.

Specifying the Type of VNET InterfaceThe mgmt, data, and span keywords used in different commands specify the path used to connect to the network service. For example, to specify an out-of-band connection to a management interface of a VNET, the interface is specified using the mgmt keyword. If in-band access to that management interface of the VNET is required, then the data or span keywords are used in the specific command. The keywords, data and span, are essentially equivalent but apply to two separate paths. To maximize throughput between the server and the switch components, it is recommended to use both. The data keyword applies to port 65, and the span keyword applies to port 66.

Each VNET can have one or more isolating zones and network services are applied to each zone. Network services have their own zone or share the zone with the VNET manager which is the zone that the VNET user logs into to manage the VNET. In shared zones, the network interfaces are available to all network services in the shared zones, regardless of the service that created the network interface.

Informational Note: This is an important concept as you can use service commands such as vlb-interface-add to add an interface or you can use vnet-manager-interface-add to add interfaces to a VNET. If you want the service to be specific to a VNET as a dedicated service, then add the interfaces using the service-interface-add commands.


65

Creating a Virtual Network (VNET)To separate resources, including switch ports, IP addresses, VLANs, and VXLANs, into separate management spaces, create a VNET and place the resources in the VNET. Then configure a separate VNET admin to manage the network.

There is no performance impact when you send network traffic through a VNET. Packets are switched in the hardware with full line-rate bandwidth and the same latency even if the packets are on a VNET or not. But, the VNET allows you to provide different Service Level Agreements (SLAs) to each VNET when there are multiple VNETs on a physical switch and there is resource contention based on traffic loads.

Related Tasks Creating a Virtual Network



Informational Note: You cannot create another VNET inside of a VNET.




67

Creating a Virtual Network

To separate resources, including switch ports, IP addresses, VLANs, and VXLANs, into separate management spaces, create a VNET and add those resources to the VNET. Then configure a separate administrator for that VNET.

To create a VNET named vnet1 with VLANs, 125 to 130, and a scope of fabric, use the following command:

CLI network-admin@switch > vnet-create name vnet1 scope fabric vlans 123-130

Vnet created.To confirm that the VLAN is created, use the vnet-show command:

CLI network-admin@switch > vnet-show layout vertical

switch: antares10name: vnet1scope: fabricvlans: 125-130managed-ports: noneadmin: vnet1-adminvnet-mgr-name: vnet1-mgrswitch: antares15name: vnet2scope: fabricvlans: 131-135managed ports: noneadmin: vnet2-adminvnet-mgr-name: vnet2-mgr

When you add VLANs to a VNET, you can either assign a range of VLANs, such as 100-199, or a number of VLANs, such as 5, which then assigns 5 VLANs from nvOS, starting with the lowest number of the available VLANs. You can see the difference by using the num-vlans parameter to assign VLANs:

CLI network-admin@switch > vnet-create name tester-1 scope fabric num-vlans 3

CLI network-admin@switch > vnet-show name tester-1 layout vertical

switch: antares10name: vnet1scope: fabricvrg: vnet1-vrgnum-vlans: 3vlans: 5-7managed-ports: noneadmin: vnet1-adminvnet-mgr-name: vnet1-mgrswitch: antares15name: vnet2scope: fabricvlans: 123-130managed ports: noneadmin: vnet2-adminvnet-mgr-name: vnet2-mgr

All switches in the fabric are now in this VNET.


Each VNET is associated with a VNET manager (VNM). The default VNM appends the suffix “mgr” to the name created for the VNET. If you want to create a different name, use the vnet-mgr-option when creating a VNET.

The VNM represents the management interface to the VNET. You can log into the VNM in the same way you can log into the management plane of the overall logical switch. In multi-tenant environments, access to the VNM is typically provided to individual VNET administrators such as cloud tenants or application managers. This way the VNET administrators can manage the configurations and properties of their VNETs. .

Adding Untagged VLANs to a VNETTo add untagged VLANs to a VNET, use the vlan-port-add command:

CLI network-admin@switch > vlan-port-add vlan-id 311 ports 15-20 untagged

Adding DHCP Service to a VNETTo add a pool of IP addresses used by a DHCP service, create the IP pool first. For example, you can create the IP Pool, dhcp-pool, and addresses in the 172.16.23.0/24 network:

CLI network-admin@switch > ip-pool-create name dhcp-pool vnet vnet1 start-ip 172.16.23.0 end-ip 172.16.23.254 netmask 24

Then create the DHCP service:

CLI network-admin@switch > dhcp-create name dhcp-vnet1 vnet vnet1 initial-ip-pool dhcp-pool

The final step is creating the gateway for the DHCP service:

CLI network-admin@switch > dhcp-pool-modify dhcp-name dhcp-vnet1 dhcp-pool-name dhcp-pool gateway-ip 172.16.23.1

Now when you add Virtual Machines (VMs) such as Ubuntu 11.04 or CentOS 6.5, the interfaces receive IP addresses from the DHCP service assigned to the VNET.

Informational Note: Command Execution Time

Some commands may take a few seconds to complete since there are multiple steps in the commands.

Informational Note: Storage Pool Use

Use the vnet-create command option vnet-mgr-storage-pool to place the VNET into a storage pool other than the default storage pool.

Informational Note: You can only run one instance of a DHCP service per VNET.


69

Verify Administrator User CreationWhen a VNET is created, an administrator for that VNET is automatically created in addition to the VNET manager. In this example, the VNET, vnet1, is created, and the user vnet1-admin is created. The keyword, admin, is appended to the name of the VNET. This is the default value, so if you want to create an administrator with a different name, use the vnet-create admin option. vnet1-admin and the superuser, network-admin can log into the VNET and manage it.

To confirm that the user was created, use the user-show command:

CLI network-admin@switch > user-show

name scope uidvnet1-admin fabric 20001

Use the user-modify command to change the password for the VNET administrator. The default password is the same as the account name, vnet1-admin, in this example.

CLI network-admin@switch > user-modify name vnet1-admin

password:********confirm password:*********

Configuring Administration Login Using SSHIn order for the vnet1-admin to login and administer the VNET using SSH, you must add an IP address on either the switch data port or the mgmt interface. You cannot access the VNET through the management IP address of the switch. To add the IP address, use the following command:

CLI network-admin@switch > vnet-manager-interface-add vnet-manager-name vnet1-mgr if data ip 10.100.1.1/24

If you do not specify a VLAN, the interface is added, by default, to the lowest numbered VLAN in the VNET. To verify that the interface was added, use the vnet-manager-interface-show command:

CLI network-admin@switch > vnet-manager-interface-show vnet-manager-name vnet1-mgr layout vertical

vnet-manager-name: vnet1-mgrnic: vnet1.mgr.eth0ip: 10.100.1.1/24assignment: staticmac: 66:0e:94:4b:68:96vlan: 123vxlan: 0if: datato_vnic_flow_name:

CAUTION!It is not recommended to change the initial role for a VNET administrator. User roles have different implications and allow access to the entire switch instead of just the VNET.


Now you can SSH to the VNET, using the following syntax:

ssh [email protected]


71

Once you log into the VNET, you are placed directly into the CLI for nvOS. The following commands are available to a VNET administrator:

acl-ipacl-macclient-server-statsconnectionconnection-latencyconnection-statsdhcpdhcp-leasedisk-librarydnsfabricfabric-nodefabric-statsigmpigmp-static-groupigmp-static-sourceinterface-statsip-pooliso-libraryl2-historyl2-tablelldplog-auditlog-eventlog-system-counterslog-systemmcastnatnetvisor-kvmnetvisor-vmmnetvisor-zoneopenflowopenstackopenstack-pluginpingport-configportport-statsport-vlanrolerunning-config-showsflowsoftware-licensesoftwaresshssh-known-hosts-deletestorage-folderstorage-poolstp-port-eventstp-statetech-support-showuservflowvflow-share


vflow-statsvlanvlan-statsvlbvnet-managervnet-servicevnet vrouter vrouter-cached-routespager switch help quit exit

Once you are logged into the VNET, you can add VMs or other features to it. For instance, you can install CentOS and run applications on it or add Ubuntu servers to the VNET.

To remove an interface from the VNET manager, use the vnet-manager-interface-remove command.

Adding a Default Gateway to the VNETUse the vnet-manager-modify command to add the gateway, 10.100.1.254 to the configuration.

CLI network-admin@switch > vnet-manager-modify name vnet1-mgr gateway 10.100.1.254

To verify the configuration, use the vnet-manager-show command:

CLI network-admin@switch > vnet-manager-show name vnet1-mgr layout vertical

name: vnet1mgrtype: vnet-mgrscope: fabricvnet: vnet1vnet-service: sharedstate: enabledgateway: 10.100.1.254

Modifying and Displaying VNET Manager ServicesYou can modify the services on the VNET manager using the vnet-manager-service-modify command. If, for example, you want to disable Web access to the interface, use the following syntax:

CLI network-admin@switch > vnet-manager-services-modify name pn-lab-vnet-mgr if pn.lab.vnet.mgr.eth0 no-web


73

To display information about the VNET services, use the vnet-services-show command:

CLI (server-switch)>vnet-service-show layout vertical

name: pn-dhcp-dnstype: dhcpscope: fabricvnet: pn-fab-globalvnet-service: sharedstate: enabledgateway: 10.9.9.1name: lab-dhcptype: dhcpscope: fabricvnet: pn-lab-vnetvnet-service: sharedstate: enabledgateway: ::

To display information about VNET Manager services, use the vnet-manager-service-show command:

CLI network-admin@switch > vnet-manager-service-show layout vertical

vnet-manager-name: pn-lab-vnet-mgrif: pn.lab.vnet.mgr.eth0ssh: onweb: onweb-ssl: offweb-ssl-port: 443web-port: 80icmp: onvnet-manager-name: pn-lab-vnet-mgrif: pn.lab.vnet.mgr.eth1ssh: onweb: onweb-ssl: offweb-ssl-port: 443web-port: 80icmp: on

Adding Ports to the VNETPorts can be managed by the VNET, but the VNET does not have absolute control over the port. Untagged traffic on the port can be tagged to a VLAN that is assigned to the VNET. In most cases, it is not necessary to add a port to the VNET.

Now, add ports, 5-8, 20-30, to the VNET on the local switch and a remote switch.

CLI network-admin@switch > vnet-port-add vnet-name vnet1 ports 5-8,20-30

CLI network-admin@switch > switch antares15 vnet-port-add vnet-name vnet1 ports 20-50 ports added.


To verify the ports, use the vnet-show command:

CLI network-admin@switch > vnet-show name vnet1 layout vertical

switch: antares15name: vnet1scope: fabricvlans: 123-130managed-ports: 5-8,20-30admin: vnet1-adminvnet-mgr-name: vnet1-mgrswitch: pleiades15name: vnet1scope: fabricvlans: 123-130managed-ports: 5-8,20-30admin: vnet1-admin

Adding a vRouter to the VNETIf you have a VLAN 10 with a subnet 192.168.10.0/24 and a VLAN 12 with a subnet 192.168.12.0/24 on the same VNET, net-resources, and you want to route traffic between the two VLANs, use the following steps:

1. Create the VNET.

CLI network-admin@switch > create-vnet name net-resources scope local vlans 10,12

2. Create VLAN 10.

CLI network-admin@switch > vlan-create id 10 scope local ports 10 untagged-ports 10

3. Create VLAN 12.

CLI network-admin@switch > vlan-create id 12 scope local ports 12 untagged-ports 12

4. Create the vRouter, subnets.

CLI network-admin@switch > vrouter-create name subnets vnet net-resources enable

5. Add a vRouter interface for VLAN 10.

CLI network-admin@switch > vrouter-interface-add vrouter-name subnets ip 192.168.10.254 netmask 255.255.255.0 vlan 10

6. Add a vRouter interface for VLAN 12.

CLI network-admin@switch > vrouter-interface-add vrouter-name subnets ip 192.168.12.254 netmask 255.255.255.0 vlan 12


75

To view the configuration, use the vrouter-interface-show command:

CLI network-admin@switch > vrouter-interface-show layout vertical

switch: pleiades24vrouter-name: subnetsnic: net-resources.mgr.eth1ip: 192.168.10.254/24assignment: staticmac: 66:0e:94:24:34:31vlan: 10vxlan: 0if: dataswitch: pleiades24vrouter-name: subnetsnic: net-resources.mgr.eth2ip: 192.168.12.254/24assignment: staticmac: 66:0e:94:24:f8:s9vlan: 12vxlan: 0if: data

To complete the VNET configuration, you can assign a Virtual Resource Group (VRGs) to the VNET. VRGs allow you allocate resources to each VNET so that a single VNET does not consume all of the resources on a switch. See Configuring Virtual Resource Groups.


Informational Note: Network Services Locations and Migration

All network services, such as VNET managers, DHCP servers, and virtual load balancers, consume disk space, CPU, and memory on one of the switches in a fabric. There may be instances when you need to move a service, for example, when a disk space shortage occurs, or you replace a switch. The migrate commands, such as vnet-manager-migrate, provide the ability to move the service to a different disk pool if you specify the storage-pool option, or to a different switch within the fabric, if the location option is specified.

You cannot migrate NetVMs and NetZones. Instead, you export and import them from the configuration using the commands iso-image-library-export and disk-library-image-export.




77


After creating a VNET, a corresponding Virtual Resource Group (VRG) is created. You can configure VRGs to limit the resources assigned to a VNET so that a single VNET cannot monopolize all of the resources of the fabric. The VRG can be modified to limit the specific resources allocated to a VNET.

To create a VRG, use the following command:

CLI network-admin@switch > vrg-create name vnet1-vrg scope fabric num-vlans 8 vlans 123-150

To check the status of a VRG, use the vrg-show command:

CLI network-admin@switch > vrg-show name vnet1-vrg layout vertical

switch: antares15name: vnet1-vrgscope: fabricnum-vlans: 8vlans: 123-130ports:num-flows: 0rack-bw-limit (Mbps): 0rack-bw(Mbps): 0storage-bw(Mbps): 0dc-bw(Mbps): 0wan-bw(Mbps): 0traffic-class: 0priority: 0restricted resources:

If you want to limit the data bandwidth to 400 Mbps for the VNET, you can modify the VRG:

CLI network-admin@switch > vrg-modify name vnet1-vrg data-bw 400m

CLI network-admin@switch > vrg-show name vnet1-vrg layout vertical

switch: antares15name: vnet1-vrgscope: fabricnum-vlans: 8vlans: 123-130ports: Nonenum-flows: 0data-bw: 400storage-bw: 0service-bs: 0restricted resources: data-bw


And finally, you want to assign the VRG to a VNET so the resource limitations apply to the VNET:

CLI network-admin@switch > vnet-modify name vnet1 vrg vnet1-vrg

vnet-show name vnet1 format all layout verticalswitch: antares15id: a1634:0name: vnet1scope: fabricvrg: vnet1-vrgnum-vlans: 1vlans: 150managed-ports:admin: vnet1-adminvnet-mgr-name: vnet1-mgrswitch: antares16id: a1635:0name: vnet1scope: fabricvrg: vnet1-vrgnum-vlans: 1vlans: 150managed-ports:admin: vnet1-adminvnet-mgr-name: vnet1-mgr

About Virtual Resource Group (VRG) Bandwidth EnforcementThe resources available in a fabric of nvOS devices can be managed by allocating them to Virtual Resource Groups (VRGs). Each VRG can include an allocation of VLANs and a guarantee of a minimum network bandwidth. VNETs are then assigned to a VRG. The VNET can also include VLANs as well as other services and resources.

In this implementation, each VRG is assigned a Guaranteed Bandwidth (GBW) parameter specified in Mbps. To enforce the GBW allocation, all network traffic associated with the VRG is sent to the Networking Processor Unit (NPU). Flows running on VLANs associated with a VRG is assigned a portion of the GBW assigned to the VRG.

This version has the following limitations:

Bandwidth guarantees for services and data are supported.

Storage bandwidth guarantees are not supported.

Available bandwidth is not enforced per VNET when there are multiple VNETs assigned to the same VRG. Only VRGs and vFlows are allowed specified guaranteed bandwidth.


Timesaver: If the VRG is created before you assign it to a VNET, you can save a step by specifying the VRG when the VNET is created.



79

Configuring Network Services - DHCP and DNS

Overview of DHCP and DNS

Configuring IP Pools

Configuring DHCP Services

Adding DHCP Interfaces

Adding DHCP and DNS Records

Removing DHCP and DNS Services

Configuring DNS Services

Creating a DNS Server

Configuring Network Address Translation Services

Configuring Hardware-based Network Address Translation(NAT)

Overview of DHCP and DNSIn general, network services are associated with a VNET. When a fabric is created, a global VNET is also created and should be used if the network service is available to all Server-Switches and all nodes on the network. Select a specific VNET if the network service applies to a single VNET, limited to the VNETs resources, and is managed by the VNET manager. Another option is to decide if the network service is applicable to the same logical zone as the VNET (shared) or applicable to another separate zone (dedicated). For example, the zone on the VNET may already have a service running, and another instance of the service is needed to avoid a conflict on the network. In the dedicated instance, the VNET and the dedicated zone must be configured to see the same network traffic, for example, on the same VLAN.

This topic describes configuring two virtual services, DNS and DHCP.

Figure 1:VNETs Configured for DHCP and DNS


Related Tasks Configuring IP Pools


Adding DHCP and DNS Records

Removing DHCP and DNS Services




81

Configuring IP PoolsIP addresses are resources managed as pools. An IP address pool must be associated with a VNET, because a service associated with the IP address pool, and the supported service must reside in a VNET. The VNET can be the default fabric VNET created when the fabric is first created, and if this is the case, the IP address pool or pools are available fabric-wide and have no resource limitations. If you want to assign restrictions to the IP pool, for example, assign it to a VLAN or set of VLANs. Create a VNET, and then assign the IP address pool to the VNET.

A private IP address pool consists of private IPv4 addresses, which means that the addresses are not routable on the Internet. However, you can later create and associate a virtual network address translation (vNAT) service between the external network IP addresses and internal private IP addresses.

Create an IP address pool with the name dhcp-pool on VNET vnet1 using the IP address pool of 192.168.18.2 through 192.168.18.255 and specifying the optional VLAN group 124.

CLI network-admin@switch > ip-pool-create name dhcp-pool vnet vnet1 start-ip 192.168.18.2 end-ip 192.168.18.255 netmask 24 vlan 124

Pool created successfully.

CLI network-admin@switch > ip-pool-show layout vertical

name: dhcp-poolvnet: vnet1scope: fabricvlan: 124start-ip: 192.168.18.2end-ip: 192.168.18.254network: 192.168.18.0/24

The IP address, 192.168.18.1, is excluded from this configuration because you need to configure it as the gateway IP address of the DNS and DHCP services.

To modify an IP pool, use the ip-pool-modify command. You cannot modify the assigned VNET. If you decide that you want to use the IP address pool on another VNET, you must delete the IP pool, and create a new one for the new VNET.

To delete an IP pool, use the ip-pool-delete command.





83


In this configuration, you use the IP address reserved from the IP address pool to create the DCHP service.

Before you begin, see Configuring DNS Services to configure the DNS service shared by the DHCP.

1. Use the following command to create the DHCP service for VNET, vnet1. The DHCP server uses the assigned IP address pool to allocate IP addresses to clients on the VNET.

CLI network-admin@switch > dhcp-create name vnet1-dhcp vnet vnet1 initial-ip-pool dhcp-pool

dhcp-show layout verticalname: vnet1-dhcptype: dhcpscope: fabricvnet: vnet1vnet-service: sharedstate: enabledpxe-boot: disabled

2. Create the DHCP server for the VNET. Assign the IP pool configured earlier to the DHCP server which is used to dis-tribute IP addresses.

CLI network-admin@switch > dhcp-create name vnet1-dhcp vnet vnet1 initial-ip-pool dhcp-pool

3. To display the configuration, use the dhcp-show command:

CLI network-admin@switch > dhcp-show layout vertical

dhcp-showname: vnet1-dhcptype: dhcpscope: fabricvnet: vnet1vnet-service: sharedstate: enabledpxe-boot: disabled

It is not necessary to add a network interface for the DHCP server since it is sharing the DNS service. In this case, the vNIC is shared between DHCP and DNS.

Informational Note: Once you assign an IP address pool to a DHCP service that allocates dynamic IP addresses, you cannot assign the same addresses as static IP addresses by other virtual network services.


4. To display the vNIC information, use the dhcp-interface-show command:

CLI network-admin@switch > dhcp-interface-show

dhcp-name nic ip mac vlan if

--------- --- -- --- ---- --vnet1-dhcp vnet1.mgr.eth0 10.100.1.1/24 66:0e:94:4b:a3:e8 123 mgmtvnet1-dhcp vnet1.mgr.eth1 192.168.18.1/24 66:0e:94:4b:af:75 124 data

5. Configure the options that the DHCP provides to DHCP clients. You can add the default route using the gateway IP address, DNS domain name, and the IP address of the DNS server.

CLI network-admin@switch > dhcp-pool-modify dhcp-name vnet-dhcp name dhcp-pool gateway-ip 192.168.18.1 ddns-domain pluribusnetworks.com dns-ip 192.168.18.1

Adding DHCP InterfacesYou can add DHCP services to an interface on the switch. To add DHCP to interface, dhcp-eng, with the IP address, 172.21.16.25, use the following command:

CLI network-admin@switch > dhcp-interface-create name dhcp-eng ip 172.21.16.25 netmask 32 assignment dhcp vlan 25

To modify the DHCP interface, use the dhcp-interface-modify command.

To remove the interface, use the dhcp-interface-remove command.

To display information about the DHCP interfaces, use the dhcp-interface-show command:

CLI network-admin@switch > dhcp-interface-show layout vertical

dhcp-name: ext-50-dhcpnic: ext.50.mgr.eth0ip: 10.111.1.1/24assignment: staticmac: 66:0e:94:23:c4:7evlan: 50vxlan: 0if: mgmtto_vnic_flow_name: dhcp-name: www-51-dhcpnic: www.51.mgr.eth0ip: 10.222.1.1/24assignment: staticmac: 66:0e:94:23:bd:f6vlan: 51vxlan: 0if: data

Adding DHCP and DNS RecordsThe DHCP service adds hostname and IP address records dynamically to the DNS service if the DHCP client specifies a hostname or if there is a static DHCP record for the client. You can also add hostname and IP address records manually to the DHCP and DNS services.


85

To manually add a static DHCP record, use the dhcp-host-add command:

CLI network-admin@switch > dhcp-host-add dhcp-name vnet1-dhcp hostname host1 fixed-ip 192.168.18.20 mac 10:0a:dd:ee:ff

When this DHCP client obtains a DHCP lease, the hostname and IP address pair are automatically added to the DNS service.

To manually add a DNS record, use the dns-record-add command:

CLI network-admin@switch > dns-record-add dns-name vnet1-dns domain pluribusnetworks.com host host2 ip 192.168.18.1

CLI network-admin@switch > dns-record-show

dns-name ip hostvnet1-dns 192.168.18.1 vnet-dns.pluribusnetworks.comvnet1-dns 192.168.18.21 host2.plurisbusnetworks.com

Removing DHCP and DNS ServicesTo remove the configured DHCP and DNS services and the IP address pool, use the following commands:

CLI network-admin@switch > dhcp-delete name vnet1-dhcp

Deleted vnet1-dhcp

CLI network-admin@switch > dns-delete name vnet1-dns

Deleted vnet1-dns

CLI network-admin@switch > ip-pool-delete name dhcp-pool

Pool dhcp-pool deleted





87

Configuring DNS Services

In this topic, the necessary tasks required to configure DNS as a service to provide name translations for the IP addresses assigned to the DHCP service.

Adding a DNS ServerAdd a DNS server for the fabric-wide VNET, vnet1. The DNS and DHCP services are going to share the service zone with the VNET manager.

1. To add the DNS server, use the following command:

CLI network-admin@switch > dns-create name vnet1-dns vnet vnet1 shared-vnet-service

2. The DNS service must communicate to hosts on the switch ports, so you must create a virtual NIC (vNIC) and add an IP address. You have to specify the netmask and VLAN for the vNIC.

CLI network-admin@switch > dns-interface-add dns-name vnet1-dns if data ip 192.168.18.1/24 vlan 24

3. To display the configuration, use the dns-interface-show command:

CLI network-admin@switch > dns-interface-show layout vertical

dns-name: vnet1-dnsnic: vnet1.mgr.eth0ip: 10.100.1.1/24assignment: staticmac: 66:0e:94:4b:a3:e8vlan: 123if: datadns-name: vnet1-dnsnic: vnet1.mgr.eth1ip: 192.168.18.1/24assignment: staticmac: 66:0e:94:4b:af:75vlan: 124if: data

This is a shared service, so in addition to the interface you just configured, the interface for the VNET manager is also present.

Multiple domain names can be associated with an IP address. A reverse lookup is a query of the DNS for a domain names when the IP address is known. This configuration requires that you define a reverse lookup pool IP addresses.


4. Configure the DNS server for the domain and the reverse lookup pool for the DNS.

CLI network-admin@switch > dns-domain-add dns-name vnet1-dns domain pluribusnetworks.com reverse-lookup-ip-pool dhcp-pool dns-ip 192.168.18.1

dns-domain-show layout verticaldns-name: vnet1-dnsdomain: pluribusnetworks.comtype: masterdns-ip: 192.168.18.1reverse-lookup-ip-pool: dpoolreverse-lookup-network: 192.168.10.0/24forwarding: noneforwarder: ::




89

Overview of NAT and Hardware NAT

Hardware NAT

NAT and Hardware NAT Use Cases and Scenarios

Static Mapping of Individual Private IP Addresses to Public IP Addresses



Network Address Translation (NAT) substitutes the real address in a packet with a mapped address that is routable on the destination network. NAT uses two steps: 1) translating a real address into mapped address, and 2) reversing the process for returning traffic.

Just as you can assign DHCP and DNS services to a VNET, you can assign NAT services to a VNET. When you create the NAT service, you can optionally configure it as a dedicated service, in a separate zone, or shared, in the same logical zone, on a VNET, and assign a storage pool to it. You can also disable and enable the NAT service on the VNET.

Hardware NATPreviously, NAT services were available only in ONVL software.Hardware-based NAT has the following functionality:

HW-NAT only translates traffic that travels between different IP address realms and is configured for HW-NAT.

The IP addresses inside of an internal domain can be re-used by other internal domains such as a VNET.

A HW-NAT-enabled router, a vRouter, has an IP address translation table to translate addresses between realms.

A HW-NAT-enabled router translates IP addresses in packets before forwarding the packets according to the translation table lookup result.

Endpoints are unaware of the NAT translation.

If there is more than one exit point, for example, from internal to external realms, each NAT-enabled router must have the same IP address translation table.

nvOS supports the following types of hardware-based NAT:

Static basic NAT (Outbound NAT)

Static basic NAT with subnet mask

Dynamic NAT

NAT-Protocol Translation (PT)

1K bi-directional NAT sessions or subnets

Only traditional NAT (outbound NAT) is supported. Two way NAT, bi-directional NAT and Twice NAT are not supported.

Applications with IP addresses in the payload, for example FTP, are supported with software NAT.


NAT and Hardware NAT Use Cases and ScenariosFigure 1: Static Mapping of Individual Private IP Addresses to Public IP Addresses

In Figure 1, a simple NAT diagram of mapping two internal IP addresses to a single external IP addresses.


91

Figure 2: Dynamic NAT and NAT-PT

Figure 3: Static NAT


Figure 4: NAT with Port Forwarding

Figure 5: NAT with Dynamic Mapping




93


To create a NAT service, vnet-nat1, on VNET, vnet-customer, as a dedicated service and enable it, use the following command:

CLI network-admin@switch > nat-create name vnet-nat1 vnet vnet-customer dedicated-vnet-service enable

Since this is a dedicated service, or if you have not created any network interfaces, then use the nat-interface-add command to create the vNICs.

CLI network-admin@switch > nat-interface-add vnet-nat1 ip 10.100.1.1/24 assignment none vlan 123 if data

CLI network-admin@switch > nat-interface-add vnet-nat1 ip 192.168.18.1/24 assignment none vlan 124 if data

To modify the configuration, use the nat-interface-modify command. For instance, to change the VLAN from 124 to 201, use the following syntax:

CLI network-admin@switch > nat-interface-modify vnet-nat1 ip 192.168.18.1/24 vlan 201

To display the configuration, use the nat-interface-show command:

CLI network-admin@switch > nat-interface-show nat-name vnet1-nat layout vertical

nat-name: vnet1-nat nic: vnet1.mgr.eth0ip: 10.100.1.1/24assignment: staticmac: 66:0e:94:4b:b8:0cvlan: 123vxlan: 0if: data

nat-name: vnet1-natnic: vnet1.mgr.eth1ip: 192.168.18.1/24assignment: staticmac: 66:0e:94:4b:9d:ccvlan: 201vxlan: 0if: data

To remove the NAT interfaces, use the nat-interface-remove command.

To delete the NAT service, use the nat-delete command. This command removes the entire NAT configuration including the associated interfaces.

To modify the NAT service, use the nat-modify command.


To enable dynamic NAT for internal IP addresses within the VNET, use the nat-map-add command. Traffic from the interface is sent to the external IP address of the VNET.

CLI network-admin@switch > nat-map-add nat-name vnet1-nat name to-internal ext-interface vnet1.mgr.eth0 network 192.168.18.2/24

To display the configuration, use the nat-map-show command:

CLI network-admin@switch > nat-map-show

nat-name name ext-interface network-------- ---- ------------- -------vnet-1-nat to-internal vnet1.mgr.eth0 192.168.18.2/24

The hosts on the VNET must have a default router with the internal IP address of the VNET manager. In this example, the IP address is 192.168.18.1.

To remove the NAT mapping, use the nat-map-remove command.

Configuring Port Forwarding for NATPort forwarding or port mapping consists of configuring a gateway to send all packets received on a particular port to a specific device on the internal network. For example, if the external network requires access a Web server with port 80 and IP address 192.168.1.2, it is necessary to define a port forwarding rule on the gateway. The rule redirects all TCP packets received on port 80 to machine 192.168.1.2.

To configure port forwarding from IP address 10.100.1.1:8888 to the internal IP address 192.168.18.4 and port 22, use the following command:

CLI network-admin@switch > nat-port-forward-add nat-name vnet1-nat name vm1_ssh ext-port 8888 int-ip 192.168.18.4 int-port 22

The NAT service now forwards from external address 10.100.1.1 port 8888 to the internal address 192.168.18.4 port 22 and permit Secure Shell connections on the well-known SSH port 22.

To remove the NAT port forwarding configuration, use the nat-port-forward-remove command.

To display NAT port forwarding information, use the nat-port-forward-show command.

Configuring Static NATStatic NAT maps an unregistered IP address to a registered IP address on a one-to-one basis. This is useful when a device needs to be accessible from outside the network. To configure a one-to-one mapping of the internal address 192.168.18.4 to the external IP address 10.100.1.1, use the following command:

CLI network-admin@switch > nat-static-nat-add nat-name gateway external-ip 10.100.1.1 internal-ip 192.168.18.4

To display the static NAT configuration, use the nat-static-nat-show command.

To remove the static NAT configuration, use the following syntax:

CLI network-admin@switch > nat-static-nat-remove nat-name gateway external-ip 10.100.1.1




95



97


Before you can add the hardware-based NAT router, you must configure a fabric, VLAN, and vRouter interface. In this example, we have the following configuration information:

fabric-name — corp-fabric

VLANs — VLAN 2 and VLAN 3

ports — 53 and 55

IP addresses — 2.2.2.1/24, 20.20.20.1/24, and 20.20.20.2/24

1. Create the fabric:

CLI network-admin@switch > fabric-create name corp-fabric

2. Create the vRouter:

CLI network-admin@switch > vrouter-create name hw-nat vnet global-default router-type hardware

3. Add the VLANs to the configuration:

CLI network-admin@switch > vlan-create id 2 scope local ports all untagged-ports 53

CLI network-admin@switch > vlan-create id 3 scope local ports all untagged-ports 55

4. Add the vRouter interfaces:

CLI network-admin@switch > vrouter-interface-add vrouter-name hw-nat ip 2.2.2.1/24 vlan 2 if data

CLI network-admin@switch > vrouter-interface-add vrouter-name hw-nat ip 20.20.20.1/24 vlan 3 if data

CLI network-admin@switch > vrouter-interface-add vrouter-name hw-nat ip 20.20.20.2/24 alias-on hw.nat.eth1

5. Add the hardware-based NAT configuration:

CLI network-admin@switch > hw-nat-create name nat1 vrouter-name hw-router

Configuring Static NATTo add a static NAT configuration to the hardware-NAT vRouter, add the following commands, and use the IP address 20.20.20.2 for an additional interface:

CLI network-admin@switch > hw-nat-static-nat-add hw-nat-name nat1 name static-nat1 internal-ip 2.2.2.10 external-ip 20.20.20.1

CLI network-admin@switch > hw-nat-static-nat-add hw-nat-name nat1 name static-nat2 internal-ip 2.2.2.20 external-ip 20.20.20.2


Configuring NAT with Port ForwardingTo add port forwarding from Host 1 using ports 1122 and 3344 to Host 2, add the following statements to the configuration:

1. Remove the static NAT configuration from the previous example:

CLI network-admin@switch > hw-nat-static-nat-remove hw-nat-name nat1

CLI network-admin@switch > hw-nat-static-nat-remove hw-nat-name nat2

2. Add the port forwarding configuration:

CLI network-admin@switch > hw-nat-port-forward-add hw-nat-name nat1 name pf1 ext-ip 20.20.20.1 ext-port 80 int-ip 2.2.2.10 int-port 1122

CLI network-admin@switch > hw-nat-port-forward-add hw-nat-name nat1 name pf1 ext-ip 20.20.20.1 ext-port 80 int-ip 2.2.2.10 int-port 3344

Configuring Dynamic Mapping for NATTo add dynamic mapping for hardware NAT, remove the port forwarding configuration and add the dynamic mapping statements:

CLI network-admin@switch > hw-nat-port-forward-remove hw-nat-name nat1 name pf1

CLI network-admin@switch > hw-nat-port-forward-remove hw-nat-name nat1 name pf2

CLI network-admin@switch > hw-nat-map-add hw-nat-name nat1 name map1 network 2.2.2.1/24 ext-ip 20.20.20.1

To display the dynamic mapping, use the hw-nat-session-show:

CLI network-admin@switch > hw-nat-session-show


99

nvOS System Logging and SNMP

Configuring System Logging

Displaying Log Counters Information

Sending Log Messages to Syslog Servers

Sending Log Messages to Syslog Servers

Viewing Log Events

Modifying and Displaying Log Event Settings

Configuring SNMP

SNMP Communities

Users and SNMPv3

Supported MIBs

OverviewnvOS logs all important activities that occur on the switch and fabrics created on them. Logging is enabled by default and is viewable using the CLI. You can also configure system logging to send syslog-formatted messages to other servers configured to receive them as part of centralized logging and monitoring.

Figure 1: nvOS Switch with Syslog Server

nvOS Switch


There are three types of activities logged by nvOS:

Each log message includes the following information:

Category - event, audit, or system

Timestamp within a microsecond

Process name and process ID of the process producing the message

Unique message name

Unique five digit numerical message code

Message: additional message-specific parameters and explanation

A log message may include optional parameters, including associated VLAN, VXLAN, or switch port.An audit log message includes additional information:

User

Process ID

Client IP of the remote computer issuing the command

An event log also includes the event type.


Table 7: Log Events

Type Description

Event Records action observed or performed by switches. Each Event type can be enabled or disabled. Events are collected on a best effort basis. If events occur too rapidly to be recorded, the event log is annotated with the number of events lost. The following are examples of event types:• Port state changes• TCP connections• STP port changes• PTP time corrections

Audit When an administrative change to the configuration is made, an audit log is recorded. An audit log consists of the command and parameters along with the success or failure indication. When a command fails, an error message is also recorded.

System The system log records error conditions and conditions of interest. There are four levels in the system log:• critical• error• warn• note

Perror The perror log records messages on standard error output, describing the last error encountered.



101

Configuring System Logging

To view event logs using the CLI, enter the following command:

CLI network-admin@switch > log-event-show

category time name code event-type port messageevent 2013-06-04,13:12:18.304740 port_up 62 port 62 upevent 2013-06-04,13:12:18.304740 port_up 62 port 50 upevent 2013-06-04,13:12:18.304740 port_up 62 port 10 up...

To view audit log entries, enter the following command:

CLI network-admin@switch > log-audit-show

category time name code user messageaudit 2013-06-04,13:12:18.304740 command 1101 network-admin Command

create vnet id=b000011:! name=vnet1 scope=fabric vrg=b000011:0 vlans=100 vnet_mgr_id=b00001

audit 2013-06-04,13:12:18.304740 command 1101 network-admin Command create vrouter id=b000011:! name=vnet1 scope=fabric vrg=b000011:0 vlans=100 vnet_mgr_id=b00001

To view system log entries, use the following command:

CLI network-admin@switch > log-system-show

time: 2015-09-17, 06:28:09.351514-07:00name: 11006level: warntime: 2015-09-17, 11:28:09.351514-07:00name: 11006level: warntime: 2015-09-17, 13:28:09.351514-07:00name: 11006level: warn

Modifying and Displaying Log Event SettingsBy default, only system and port events are logged. Other logging is possible, and you can add other events using the log-event-settings-modify command. You can modify the way nvOS logs events by using the log-event-settings-modify command to remove or add log events. For instance to remove logging of PTP events, use the following command:

CLI network-admin@switch > log-event-settings-modify no-ptp

To display log event settings information, use the log-event-settings-show command.


Displaying Log Counters InformationYou can display information about the number of events that have occurred on the network by using the log-system-counters-show command:

CLI network-admin@switch > log-system-counters-show layout vertical

switch: pleiades24critical: 0error: 0warn: 1061note: 9

To reset the log counters, use the log-system-counters-reset command.

Formatting and Filtering of Logging MessagesThere are many options for filtering and formatting of log messages returned by these commands. Use the <tab> completion method and ? to explore them.

The log files are also available using SFTP, switch-ip:/sftp/nvOS/logs and NFS, /net/switch-name/nvOS/logs if you have enabled the services.

Many systems support a syslog facility for sending or receiving log messages. Pluribus Networks infrastructure can send messages to syslog servers using either RFC 5424 (Structure) or RFC 3164 (legacy) formats.

Sending Log Messages to Syslog ServersTo configure the switch to send all log messages to a syslog server with an IP address of 172.16.21.67, use the following command:

CLI network-admin@switch > admin-syslog-create name log-all scope fabric host 172.16.21.76

To display the configuration use the admin-syslog-show command:

CLI network-admin@switch > admin-syslog-show

name scope host port message-formatlog-all fabric 172.16.21.67 514 legacy

To specify sending the syslog messages in structured format, per RFC5424, add the message-format option to the configuration.

CLI network-admin@switch > admin-syslog-modify name log-all message-format structured

You can also modify the port that the service listens on to another port. More than one syslog listening service can be configured and appropriate syslog messages are sent to each one.

By default, all log messages are forwarded to syslog servers. To filter the log messages, use the msg-level option to specify the severity or other options:

CLI network-admin@switch > admin-syslog-match-add syslog-name log-all name critical-msgs msg-level critical


103

You can modify syslog matching using the admin-syslog-match-modify command, or remove matching criteria using the admin-syslog-match-remove command.

To display the configuration, use the show command:

CLI network-admin@switch > admin-syslog-match-show

syslog-name msg-level namelog-all critical critical-msgs

Using Facility Codes with Log MessagesLog messages are labeled with a facility code indicating the area of the software that generated the log message. ONVLuses the following facility codes by default:

Log_Daemon for events and system messages

Log_AUDIT for audit messages

The following severities are used by default:

Log_INFO for events and audit messages

Log_Critical = critical

Log_ERROR = error

Log_WARNING = warn

Log_NOTICE = note

You can override the default values by configuring matches for each syslog configuration which allows ONVLto translate log messages into fields that the syslog servers understand.





105

Viewing Log Events

For information about specific log events and their meaning, see the Pluribus Networks Log Message Reference Guide.

A log message consists of common parameters separated by spaces and a colon (:), and optional parameters such as key and value pairs, another colon, and then the log-specific message.

To view event logs using the CLI, enter the following command:

CLI network-admin@switch > log-event-show

category: eventtime: 2014-07-17,07:37:17.466173-07:00switch: pleiades24program: nvOSdpid: 6344name: mac_ip_changedcode: 11023event-type: portvnet: global-defaultport: 65vlan: 200message: ip address change: mac=50:33:a5:e0:7f:fd ip=172.16.23.7category: eventtime: 2014-07-17,07:37:50.109133-07:00switch: pleiades24program: nvOSdpid: 6344name: mac_ip_changedcode: 11023event-type: portvnet: vlb-web-svrport: 65vlan: 200message: ip address change: mac=50:33:a5:e0:7f:fd ip=172.16.23.1category: eventtime: 2014-07-17,07:42:17.418349-07:00...


To view audit log entries, enter the following command:

CLI network-admin@switch > log-audit-show layout vertical

category: audittime: 2014-04-01,14:56:40.763626-07:00name: user_commandcode: 11001user: network-adminmessage: Command "vlan-create id 25category: audittime: 2014-04-01,14:56:40.765839-07:00name: logoutcode: 11100user: network-adminmessage: logoutcategory: audittime: 2014-04-01,14:56:40.847912-07:00name: logincode: 11099user: network-adminmessage: logincategory: audittime: 2014-04-01,14:56:40.888363-07:00name: logoutcode: 11100

...

To view system log entries, use the following command:

CLI network-admin@switch > log-system-show

time: 2013-09-17, 06:28:09.351514-07:00name: 11006level: warntime: 2013-09-17, 11:28:09.351514-07:00name: 11006level: warntime: 2013-09-17, 13:28:09.351514-07:00name: 11006level: warn

Modifying and Displaying Log Event SettingsBy default, only system and port events are logged. Other logging is possible, and you can add other events using the log-event-settings-modify command. You can modify the way nvOS logs events by using the log-event-settings-modify command to remove or add log events. For instance to remove logging of PTP events, use the following command:

CLI network-admin@switch > log-event-settings-modify no-ptp


107

To display log event settings information, use the log-event-settings-show command.

CLI network-admin@switch > log-event-settings-show

switch: pleiades24system: onport: ontcp: offstp: offigmp: offlldp: offlacp: offvdp: offecp: offevb: offptp: offopenflow: offstorage: ontacacs: on

You can modify the log event settings using the log-event-settings-modify command. For example, if you want to turn on TCP events, use the following command:

CLI network-admin@switch > log-event-settings-modify tcp

CLI network-admin@switch > log-event-settings-show

switch: pleiades24system: onport: ontcp: onstp: offigmp: offlldp: offlacp: offvdp: offecp: offevb: offptp: offopenflow: offstorage: ontacacs: onopenstack:on

Displaying Log Counters InformationYou can display information about the number of events that have occurred on the network by using the log-system-counters-show command:

CLI network-admin@switch > log-system-counters-show layout vertical

switch: pleiades24critical: 0error: 0warn: 1061note: 9

TCP is now turned on.


To reset the log counters, use the log-system-counters-reset command.

Formatting and Filtering of Logging MessagesThere are many options for filtering and formatting of log messages returned by these commands. Use the <tab> completion method and ? to explore them.

The log files are also available using SFTP, switch-ip:/sftp/nvOS/logs and NFS, /net/switch-name/nvOS/logs if you have enabled the services.

Many systems support a syslog facility for sending or receiving log messages. Pluribus Networks infrastructure can send messages to syslog servers using either RFC 5424 (Structure) or RFC 3164 (legacy) formats.

Sending Log Messages to Syslog ServersTo configure the switch to send all log messages to a syslog server with an IP address of 172.21.16.144, use the following command:

CLI network-admin@switch > admin-syslog-create name log-all scope fabric host 172.21.16.144

To display the configuration use the admin-syslog-show command:

CLI network-admin@switch > admin-syslog-show

name scope host port message-formatlog-all fabric 172.21.16.144 514 legacy

To specify sending the syslog messages in structured format, per RFC5424, add the message-format option to the configuration.

CLI network-admin@switch > admin-syslog-modify name log-all message-format structured

You can also modify the port that the service listens on to another port. More than one syslog listening service can be configured and appropriate syslog messages are sent to each one.

By default, all log messages are forwarded to syslog servers. To filter the log messages, use the msg-level option to specify the severity or other options:

CLI network-admin@switch > admin-syslog-match-add syslog-name log-all name critical-msgs msg-level critical

You can modify syslog matching using the admin-syslog-match-modify command, or remove matching criteria using the admin-syslog-match-remove command.

To display the configuration, use the show command:

CLI network-admin@switch > admin-syslog-match-show

syslog-name msg-level namelog-all critical critical-msgs

The parameters to match include msg-start, msg-end, msg-duration, msg-starting-point, msg-length, and msg-reverse.


109

Using Facility Codes with Log MessagesLog messages are labeled with a facility code indicating the area of the software that generated the log message. ONVLuses the following facility codes by default:

Log_Daemon for events and system messages

Log_AUDIT for audit messages

The following severities are used by default:

Log_INFO for events and audit messages

Log_Critical = critical

Log_ERROR = error

Log_WARNING = warn

Log_NOTICE = note

You can override the default values by configuring matches for each syslog configuration which allows ONVLto translate log messages into fields that the syslog servers understand.





111

Configuring SNMP

Simple Network Management Protocol (SNMP) is a widely used protocol for monitoring the health and welfare of network equipment such as routers, computer equipment and even devices like UPSs. ONVL has implemented SNMP using Net-SNMP version 5.7.2.

SNMP generally works the same in most implementations and this document does not provide indepth information about SNMP overall. You can locate many resources on SNMP functionality on the Internet.

SNMP v1, v2, and v3 are now supported in nvOS. The SNMP daemon runs as a service and is launched by using the following command:

CLI network-admin@switch > admin-service-modify if mgmt snmp

This command launches the daemon, subagents, and opens a port so that remote queries can reach the daemon.

SNMP CommunitiesCommunities are used in SNMPv1 as a method of controlling access to information. You can create a community using the following command:

CLI network-admin@switch > snmp-community-create community-string name-string community-type read-only|write-only

To create a SNMP community string named, snmp-group, with read-only privileges, use the following command:

CLI network-admin@switch > snmp-community-create community-string snmp-group community-type read-only

To modify the SNMP community, snmp-group, to write-only, use the following command:

CLI network-admin@switch > snmp-community-modify community-string snmp-group community-type write-only

To display information about the SNMP community, snmp-group, use the following command:

CLI network-admin@switch > snmp-community-show community-string snmp-group

switch community-string community-type------ ---------------- --------------pleiades24 snmp-group read-only

To delete the SNMP community, snmp-group, use the following command:

CLI network-admin@switch > snmp-community-delete community-string snmp-group

Users and SNMPv3SNMPv3 creates users as access control mechanisms, and creating users is more complex but also more secure and more flexible. You can also require that users must authenticate and use encryption. Use the following command to create a user:

CLI network-admin@switch > snmp-user-create user-name name-string auth-password [auth|no-auth] priv-password [priv|no-priv]


To create the user, snmp-admin, with authentication, password m0nk3ys, use the following command:

CLI network-admin@switch > snmp-user-create user-name snmp-admin auth-password auth

auth password: ********confirm password: ********

To modify the SNMP user and add private with the password, b33hiv3, use the following command:

CLI network-admin@switch > snmp-user-modify user-name snmp-admin auth-password auth

priv-password privauth password: ********confirm password: ********priv password: ******confirm password: ******

To display information about the SNMP user, use the following command:

CLI network-admin@switch > snmp-user-show user-name snmp-user

switch user-name auth priv-------- --------- ---- ----pleiades24 snmp-user yes yes

To delete the SNMP user, use the snmp-user-delete command.

After you create the user, you must grant permission, using View Access Control Model (VACM) to view SNMP objects:

CLI network-admin@switch > snmp-vacm-create user-name name-string user-type [rouser|rwuser] oid-restrict string [auth|no-auth] [priv|no-priv]

The parameter, oid-restrict, is an optional argument that specifies a MIB sub-tree that the view is restricted. In other words, if you specify an OID, only that OID and the descendants in the tree are visible in this view.

To continue with the previous example, snmp-user is a read-only user restricted only to sysContact OID:

CLI network-admin@switch > snmp-vacm-create user-name snmp-user user-type rouser oid-restrict sysContact no-auth no-priv

To modify the VACM configuration and change no authentication to authentication, use the following command:

CLI network-admin@switch > snmp-vacm-modify user-name snmp-user user-type rouser auth

To display information about the VACM configuration, use the snmp-vacm-show command:

switch user-type user-name oid-restrict view auth priv------ --------- --------- ------------ ---- ---- ----pleiades24 rouser snmp-user sysContact no no

To delete the VACM user from the SNMP configuration, use the snmp-vacm-delete command:

CLI network-admin@switch > snmp-vacm-delete user-name snmp-user


113

Supported MIBsnvOS customized MIBs:

IfTable

IfXTable

EntPhySensorTable

OpenSolaris-supported MIBs:

SNMPv2

DISMAN-EVENT — monitors disks, processes and execs

IF — monitors interfaces

IP — monitors IP addresses and related information such as ipForwarding, ipForwarding, ipDefaultTTL, ipInReceives, ipInHdrErrors, ipInAddrErrors, ipForwDatagrams, ipInUnknownProtos, ipInDiscards, ipInDelivers, ipOutRequests, ipOutDiscards

ipOutNoRoutes

ipReasmTimeout

ipReasmReqds

ipReasmOKs

ipReasmFails

ipFragOKs

ipFragFails

ipFragCreates

ipAddrTable

ipRouteTable

ipNetToMediaTable

ipRoutingDiscards

Last bit mask

TCP — monitors TCP packet information such as tcpRtoAlgorithm, tcpRtoMin, tcpRtoMax, tcpMaxConn, tcpActiveOpens, tcpPassiveOpens, tcpAttemptFails, tcpEstabResets, tcpCurrEstab, tcpInSegs, tcpOutSegs, tcpRetransSegs, tcpConnTable, tcpInErrs, tcpOutRsts,

UDP — monitors UDP packet information

HOST-RESOURCES

NOTIFICATION-LOG

SNMPv2-SMI

IF-EXT

ENTITY-SENSOR

See additional supported MIBs in Table , “”.

Additional commands that support SNMPv1, SNMPv2, and SNMPv3:

snmp-engineid-show — The SNMP engine ID is a unique string of 24 characters that identifies the device for administrative purposes. This command displays the identification of the local SNMP engine and all remove engines configured on the switch.

snmp-trap-enable-modify — Used to enable notifications about link conditions and common system errors. This is used with the snmp-monitor commands.


snmp-trap-enable-show — Display enabled SNMP traps.

snmp-trap-sink-create — Used to specify a SNMPv1 trap receiver.

snmp-trap-sink-delete — Remove SNMP sink traps.

snmp-trap-sink-modify — Modify SNMP sink traps.

snmp-trap-sink-show — Display SNMP sink traps.

snmp-v3-trap-sink-create - Used to specify a SNMPv3 trap receiver.

snmp-v3-trap-sink-delete — Used to delete a SNMPv3 trap receiver.

snmp-v3-trap-sink-modify — Used to modify a SNMPv3 trap receiver.

snmp-v3-trap-sink-show — Used to display a SNMPv3 trap receiver.




115

Supported MIBs

Table 8: Supported MIBs

MIB Description

AgentX This is the MIB module for the SNMP Agent Extensibility Protocol (AgentX). This MIB module is implemented by the master agent.

Bridge The Bridge MIB module for managing devices that support IEEE 802.1D.

Disman-Event The MIB module for defining event triggers and actions for network management.

Disman-Schedule This MIB module defines a MIB which provides mechanisms to schedule SNMP set operations periodically or at specific points in time.

Disman-Script This MIB module defines a set of objects that allow you to delegate management scripts to distributed managers.

Entity The MIB module for representing multiple logical entities supported by a single SNMP agent.

Entity-Sensor This module defines Entity MIB extensions for physical sensors.

Ether-Like The MIB module that describes generic objects for Ethernet-like network interfaces.

HCNUM-TC A MIB module containing textual conventions for high capacity data types. This module addresses an immediate need for data types not directly supported in the SMIv2. This short-term solution is meant to be deprecated as a long-term solution is deployed.

Host-Resources This MIB is for use in managing host systems. The term `host' is construed to mean any computer that communicates with other similar computers attached to the Internet and that is directly used by one or more human beings. Although this MIB does not necessarily apply to devices whose primary function is communications services (e.g., terminal servers, routers, bridges, monitoring equipment), such relevance is not explicitly precluded. This MIB instruments attributes common to all Internet hosts including, for example, both personal computers and systems that run variants of Unix.

Host-Resources-Types This MIB module registers type definitions for storage types, device types, and file system types.

IANA-Address-Family-Numbers

The MIB module defines the AddressFamilyNumbers textual convention.


IANA-Language The MIB module registers object identifier values for well-known programming and scripting languages. Every language registration MUST describe the format used when transferring scripts written in this language.Any additions or changes to the contents of this MIB module require Designated Expert Review as defined in the Guidelines for Writing IANA Considerations Section document. The Designated Expert will be selected by the IESG Area Director of the OPS Area.Note, this module does not have to register all possible languages since languages are identified by object identifier values. It is therefore possible to registered languages in private OID trees. The references given below are not normative with regard to the language version. Other references might be better suited to describe some newer versions of this language. The references are only provided as `a pointer into the right direction'.

IANA-RTPROTO This MIB module defines the IANAipRouteProtocol and IANAipMRouteProtocol textual conventions for use in MIBs which need to identify unicast or multicast routing mechanisms.

IANAifType This MIB module defines the IANAifType Textual Convention, and thus the enumerated values of the ifType object defined in MIB-II's ifTable.

IF-Inverted-Stack The MIB module which provides the Inverted Stack Table for interface sub-layers.

IF The MIB module to describe generic objects for network interface sub-layers. This MIB is an updated version of the ifTable for MIB-II, and incorporates the extensions defined in RFC 1229.

INET-Address This MIB module defines textual conventions for representing Internet addresses. An Internet address can be an IPv4 address, an IPv6 address, or a DNS domain name. This module also defines textual conventions for Internet port numbers, autonomous system numbers, and the length of an Internet address prefix.

IP-Forward The MIB module for the management of CIDR multipath IP Routes.

IP The MIB module for managing IP and ICMP implementations, but excluding their management of IP routes.

IPv6-Flow-Label This MIB module provides commonly used textual conventions for IPv6 Flow Labels.

IPv6-ICMP The MIB module for entities implementing the ICMPv6.

IPv6 The MIB module for entities implementing the IPv6 protocol.

IPv6-TC Imports Integer32 From SNMPv2-SMI

IPv6-TCP The MIB module for entities implementing TCP over IPv6.

IPv6-UDP The MIB module for entities implementing UDP over IPv6.


MIB Description


117

NET-SNMP-AGENT Defines control and monitoring structures for the Net-SNMP agent.

NET-SNMP-EXAMPLES Example MIB objects for agent module example implementations

NET-SNMP-EXTEND Defines a framework for scripted extensions

NET-SNMP Top-level infrastructure of the Net-SNMP project enterprise MIB tree

NET-SNMP-PASS Example MIB objects for "pass" and "pass-persist" extension script

NET-SNMP-TC Textual conventions and enumerations for the Net-SNMP project

NET-SNMP-VACM Defines Net-SNMP extensions to the standard VACM view table.

NOTIFICATION-Log The MIB module for logging SNMP Notifications, that is, Traps and Informs.

RFC-1215 This module is a empty module. It has been created solely for the purpose of allowing other modules to correctly import the TRAP-TYPE clause from RFC-1215 where it should be imported from. It's a built in type in the UCD-SNMP code, and in fact RFC-1215 doesn't actually define a mib at all; it only defines macros. However, importing the TRAP-TYPE is conventionally done from an import clause pointing to RFC-1215.

RFC-1155-SMI Exports everything including Lnternet, directory, mgmt, experimental, private, enterprises, OBJECT-TYPE, ObjectName, ObjectSyntax, SimpleSyntax, ApplicationSyntax, NetworkAddress, IpAddress, Counter, Gauge, TimeTicks, Opaque;

RFC-1213 Imports mgmt, NetworkAddress, IpAddress, Counter, Gauge, TimeTicks

RMON Imports MODULE-IDENTITY, OBJECT-TYPE, OBJECT-IDENTITY, NOTIFICATION-TYPE, mib-2, Counter32, Integer32, TimeTicks FROM SNMPv2-SMI, and TEXTUAL-CONVENTION, DisplayString FROM SNMPv2-TC, and MODULE-COMPLIANCE, OBJECT-GROUP, NOTIFICATION-GROUP FROM SNMPv2-CONF

SCTP The MIB module for managing SCTP implementations.

SMUX Imports enterprises FROM RFC1155-SMI DisplayString FROM SNMPv2-TC OBJECT-TYPE FROM RFC-1212;

SNMP-Community This MIB module defines objects to help support coexistence between SNMPv1, SNMPv2c, and SNMPv3.

SNMP-Framework The SNMP Management Architecture MIB

SNMP-MPD The MIB for Message Processing and Dispatching


MIB Description


SNMP-Notification This MIB module defines MIB objects which provide mechanisms to remotely configure the parameters used by an SNMP entity for the generation of notifications.

SNMP-Proxy This MIB module defines MIB objects which provide mechanisms to remotely configure the parameters used by a proxy forwarding application.

SNMP-Target This MIB module defines MIB objects which provide mechanisms to remotely configure the parameters used by an SNMP entity for the generation of SNMP messages.

SNMP-User-Based-SM The management information definitions for the SNMP User-based Security Model.

SNMP-USM-AES Definitions of Object Identities needed for the use of AES by SNMP's User-based Security Model.

SNMP-USM-DH-Objects The management information definitions for providing forward secrecy for key changes for the usmUserTable, and for providing a method for 'kickstarting' access to the agent via a Diffie-Helman key agreement.

SNMP-View-Based-ACM The management information definitions for the View-based Access Control Model for SNMP.

SNMPv2-Conf Imports ObjectName, NotificationName, ObjectSyntax from SNMPv2-SMI

SNMPv2 The MIB module for SNMP entities.

SNMP-SMI The MIB module that provides the notation for writing SNMP MIBs.

SNMP-TC Imports TimeTicks from SNMPv2-SMI

SNMP-TM The MIB module for SNMP transport mappings.

TCP The MIB module for managing TCP implementations.

Transport-Address This MIB module provides commonly used transport address definitions.

Tunnel The MIB module for management of IP Tunnels, independent of the specific encapsulation scheme in use.

UCD-Demo SMIv2 version converted from older MIB definitions.

UCD-DISKIO This MIB module defines objects for disk IO statistics.

UCD-DLMOD This file defines the MIB objects for dynamic loadable MIB modules.


MIB Description


119


UCD-IPFWACC This module defines MIB components for reading information from the accounting rules IP Firewall. This would typically let you read the rules and the counters. I did not include some flags and fields that I considered irrelevant for the accounting rules. Resetting the counters of the rules by SNMP would be simple, but I don't consider it so useful. I gave no consideration to implementing write access for allowing modification of the accounting rules.

UCD-SNMP This file defines the private UCD SNMP MIB extensions.

UDP The MIB module for managing UDP implementations.


MIB Description




121

High Availability

Pluribus Networks switches automatically perform functions that ease your administrative burden. In the case of high availability, switches in a fabric automatically detect other switches in the fabric. If multiple connections exist between two switches, they automatically create an 801.3ad Link Aggregation Group (LAG) between the two switches for resiliency and load balancing. Other features require configuration such as connecting one device to two switches, or if LAGs are desired between Pluribus switches and other manufacturers’ equipment.

Configuring a Cluster

If you have two Pluribus switches, and want them to work together to provide networking services in the event one of the switches fails, the switches must be members of the same fabric, and you must configure them as a cluster.


To set up a cluster of two switches, pleiades4 and pleiades6, you must verify that they are members of the existing fabric:

CLI network-admin@switch > fabric-node-show layout vertical

name: pleiades4fab-name: corp-fabmgmt-ip: 10.9.9.141/16mgmt-vlan: 0fab-tid: 29out-port: 0version: 0.18.2789,pn-nvOS-b144astate: onlinename: pleiades6fab-name: corp-fabmgmt-ip: 10.9.9.139/0mgmt-vlan: 0fab-tid: 29out-port: 60version: 0.18.2789,pn-nvOS-b144astate: online

To create a cluster configuration, use the following command:

CLI network-admin@switch > cluster-create name cluster1 cluster-node-1 pleiades4 cluster-node-2 pleiades6

To verify the status of the cluster, use the cluster-show command:

CLI network-admin@switch > cluster-show

name state cluster-node-1 cluster-node-2cluster1 online pleiades4 pleiades6

To replace a failed cluster node, use the cluster-repeer command. However, you must evict the failed node from the fabric, and then run the cluster-repeer command on an active node after replacing the failed node.

To display information about the cluster, use the cluster-info command:

CLI network-admin@switch > cluster-info format all layout vertical

name: vlagid: a000030:1state: onlinecluster-node-1: 167772208cluster-node-2: 167772196tid: 1ports: 26validate: yes


123

If you want to connect the cluster nodes to an uplink switch, you must configure a VLAG between the ports on the cluster nodes and the uplink switch.

For example, if pleiades6 has port 53 connected to the uplink switch and pleiades4 has port 19 connected to the uplink switch, create a VLAG by executing the vlag-create command on either of the switches:

CLI network-admin@switch > vlag-create name vlag-uplink local-port 53 peer-switch pleiades4 peer-port 19

This example assumes that you’ve entered the command on pleiades6.

To verify the configuration, use the following command:

CLI network-admin@switch > vlag-show

name local-port peer-switch peer-port statusvlag-uplink 53 pleiades4 19 online


Informational Note: Before you can create a VLAG, you must configure the two switches in a cluster.




125

Configuring Fabric-based Physical Storage Pools

You can create storage pools on the disks shipped with your switch and create physical storage resources. These resources can be virtualized and allocated to individual virtual networks. Physical storage consists of hard disk drives (HDD), solid-state disk drives (SSD), or high-IOPS Fusion-IO Flash-based storage.

When the switch is booted up, it performs checks for uninitialized storage devices. If found, the devices are automatically formatted and a storage pool is created on each one.

Before you start, display information about the storage set up on the switch:

CLI network-admin@switch > storage-pool-show

switch name raid-type used avail status state ------------- -------- --------- ----- ----- ------ ------ pleiades01 datapool no_raid 213G 1.58T ok ONLINE pleiades01 rpool no_raid 87.5G 21.7G ok ONLINE

You can also display the physical storage media installed on the switch that is available to create a new storage pool:

CLI network-admin@switch > storage-device-show

switch name label disk type capacity in-use data-set ------------- ----- ---------- ------ ---- -------- ------ -------- pleiades01 disk0 internal-0 c6t0d0 disk 112G yes rpool pleiades01 disk1 internal-1 c6t1d0 disk 112G yes pleiades01 disk4 back-0 c6t4d0 disk 932G yes datapool pleiades01 disk5 back-1 c6t5d0 disk 932G yes datapoolpleiades01 disk6 internal c1d0p0 flash1.35T yes pooldisk1

The column, data-set, refers to the ZFS root pool parameter which identifies the location for storage.

The column, type, identifies the type of storage media as disk or flash.

Informational Note: Additional storage is not available on the E68 series. For the F64 series, additional storage is available and must be ordered as an additional component to the switch.

Informational Note: If you prefer other pool layouts, such as a RAID 1 mirror created from two disks, then delete the pools on the disks you want to use and add the now-free disks to other pools.


To create a new physical storage pool, with no RAID protection, using available disk disk3, enter the following command at the command prompt:

CLI network-admin@switch > storage-pool-create name store-new device1 disk3 raid-type no_raid

storage-pool-showswitch name raid-type used avail------ ---------- --------- ---- -----pleiades01 rpool no_raid 62.7G 10.2Gpleiades01 store-new no_raid 92.5K 457G

By default, the storage-pool-create command creates a disk library and image library within the new storage pool, and exports the libraries to the network by using NFS sharing. Since disk and image library storage is limited to storage pools other than rpool, optional disk storage is needed to implement those features.

To verify that the disk library is created, use the following command:

CLI network-admin@switch > disk-library-show storage-pool store-new layout vertical

switch: pleiades01name: disk-lib-pluribusstorage-pool: store-newsharing: nfsimport-share: pleiades01:/disk-lib/newpool/importexport-share: pleiades01:/disk-lib/newpool/exportswitch: pleiades01name: disk-lib-pool-disk1storage-pool: pool-disk1sharing: nfsimport-share: pleiades01:/disk-lib/pool-disk1/importexport-share: pleiades01:/disk-lib/pool-disk1/export

To display the ISO image library, use the following command:

CLI network-admin@switch > iso-library-show storage-pool store-new layout vertical

switch: pleiades01name: iso-lib-store-newstorage-pool: store-newsharing: nfsimport-share: pleiades24:/iso-lib/store-new/importexport-share: pleiades24:/iso-lib/store-new/exportdedup: no

To delete the physical storage pool, store-new, use the following command:

CLI network-admin@switch > storage-pool-delete name store-new

To verify that the storage pool is deleted, use the storage-pool-show command:


switch name raid-type used avail status statepleiades01 rpool no-raid 62.7G 10.2G ok ONLINE


127

To verify that the disk space is now free, use the storage-device-show command:

storage-device-showswitch name label disk type capacity in-use data-set------ ---- ----- ---- ---- -------- ------ --------pleiades01 disk0 internal-0 c6t0d0 disk 74.5G yes rpoolpleiades01 disk1 internal-1 c6t1d0 disk 74.5G yes rpoolpleiades01 disk3 back-0 c6t3d0 disk 466G no

Displaying and Downloading Storage ImagesYou can use the storage-image commands to view downloaded image files, refresh the list, and download files.

1. Refresh the image list:

CLI network-admin@switch > storage-image-refresh

2. Display the available images:

CLI network-admin@switch > storage-image-show

switch name size status ------------- --------------------------------- ----- ----------- mitch-aquila2 CentOS-6.4-x86_64-bin-DVD1.iso.gz 3.94G downloaded mitch-aquila2 CentOS-6.5-x86_64-bin-DVD1.iso.gz 4.04G downloaded mitch-aquila2 openstack-centos-neutron.vhd.gz 2.81G downloaded mitch-aquila2 openstack-centos.vhd.gz 4.31G server-only

3. The status, downloaded, means that the images are already downloaded from the server, and the status, server-only, means that the image is available for downloading.

4. To download the openstack-centos.vhd.gz image, use the following syntax:

CLI network-admin@switch > storage-image-download name openstack-centos.vhd.gz

Periodically run the storage-image-show command to check the status of the download. Once the status changes to downloaded, you can use the image to create VMs on the switch.




Creating Virtual Storage for a Virtual Network (VNET)

Virtual storage is useful to store virtual machine (VM) images for a elastic compute pool and as a data share for a virtual network. Elasticity, in this case, means that you can shift and pool resources across your infrastructure without over provisioning the network. Virtual storage is available to hosts on the VNET through the NFS protocol.

1. Create an IP pool and VNET to host the servers in the elastic compute pool.

CLI network-admin@switch > vnet-create name elas-com-pool scope local mgr-eth1-vlan 10 vnet-mgr-name ecp1_vmgr mgr-eth0-ip 10.11.37.4 mgr-eth0-netmask 16

Vnet created.

CLI network-admin@switch > ip-pool-create name vpool vnet elas-com-pool start-ip 192.168.1.1 end-ip 192.168.1.254 netmask 24

2. Create the virtual storage for VMs with the maximum size of 80GB and set the performance optimization to latency:

CLI network-admin@switch > storage-folder-create elas-com-pool storage-pool store-new max-space 80g optimization latency sharing nfs

3. Use the storage-folder-show command to display the storage folder configuration:

name storage-pool vnet max-space backup sharing dedup optimization

ec1_vstor store-new 0:0 80 no nfs no latency

To delete the storage folder, ec1_vstor, use the storage-folder-delete command.


Managing Host Operating SystemsYou can set up host operating system ISO images and disk images on your switch. Host OS images are useful to automatically provision servers assigned to a virtual network in a stateless computing environment, and create local Netvisor VMs.

With stateless computing, the underlying compute resources, server hardware, are completely transparent to the OS or applications using it. This allows an OS or application to move from one server to another very easily.

In this example, the VM image is an ISO file named ubuntu-12.10-desktop-i386.iso that you copy and then install on the switch.



129

Using the storage pool, store-new, verify that you have enough disk space and that an ISO library is created:


switch name raid-type used availpleiades24 store-new no_raid 92.5K 457G

CLI network-admin@switch > iso-library-show layout vertical

switch: pleiades24name: iso-lib-pool-store-newstorage: store-newsharing: nfsimport-share: pleiades24:/iso-lib/pool/store-new/importexport-share: pleiades24:/iso-lib/pool/store-new/exportdedup: no

1. Copy the VM image to your switch from another computer using the ISO library NFS share that was added when the storage pool was created. Copying the image depends on your computer’s OS, but on a Mac OS platform, use the $ showmount -e ip-address using the IP address of your switch and the Terminal application. $showmount -e 10.10.20.147Exports list on 10.10.20.147:/disk-lib/store-new/export Everyone/nvOS/log Everyone/mnt/vmiso/ubuntu-11.04-amd64 Everyone/disk-lib/new-store/import Everyone/mnt/vmiso/centOS-6.5-x86_64 Everyone/mnt/vmiso/centOS-6.4-x86_64 Everyone/nvOS/vlb-web-svr-mgr/kickstarts Everyone

$cd /net/10.10.20.147/disk-lib/store-new/import

$cp ubuntu-12.10-desktop-i386.iso

2. Add the new VM image to your switch using the iso-library-image-import command:

CLI network-admin@switch > iso-library-image-import iso-library-name iso-lib-pool-disk1 image-label ubuntu-12 image-file ubuntu-12.10-desktop-i386.iso

Your VM image is now transferred to the virtual store and available for installation on bare metal or virtualized servers.


3. To display a list of VM images on your switch, use the following command:

CLI network-admin@switch > iso-library-image-show iso-library-name iso-lib-disk1

switch iso-library-name label ------------- --------------------- ------------------ pleiades24 iso-lib-pool-datapool ubuntu-13.iso pleiades24 iso-lib-pool-datapool vmware-setup.iso pleiades24 iso-lib-pool-datapool ubuntu-12.iso pleiades24 iso-lib-pool-datapool ubuntu-13.1 pleiades24 pluribus ubuntu-11.04-amd64 pleiades24 pluribus centOS-6.4-x86_64 pleiades24 pluribus centOS-6.5-x86_64 pleiades24 pluribus Netvisor-b144b-kvm


Provisioning Bare Metal ServersA bare metal environment is a computer system or a network in which a virtual machine is installed directly on hardware rather than within a host operating system (OS). The term, bare metal, refers to the hard disk where a computer’s OS is typically installed.

Preboot Execution Environment (PXE - pronounced “pixie”) is an industry standard client and server interface that allows networked computers without an OS to be configured and booted remotely. PXE provides three things:

DHCP which allows the client to receive an IP address and gain access to the network servers.

A set of Application Programming Interfaces (API) used by the client’s Basic Input/Output System (BIOS) or a Network Bootstrap Program (NBP) that automates the booting of the OS.

A standard method of initializing the PXE code in the PXE ROM chip or boot disk.

How does PXE work? The process consists of the following steps:

1. The client notifies the switch that it uses PXE.2. Since the switch is configured for PXE, it sends the client a list of boot servers that contain the available OS. 3. The client finds the boot server that it can use and receives the name of the file to download. 4. The client downloads the file and executes it.

Before You BeginBefore you start the PXE process and provisioning a bare metal server, be sure that you have the following parameters configured:

The switch is configured as part of a fabric.

You have at least one VNET configured.

Create an IP address pool for the DHCP server.

CLI network-admin@switch > ip-pool-create name dhcppool vnet pxevnet network 172.24.100.0 netmask 24



131

The DHCP server provides IP addresses to clients that are PXE booting, and using the parameter pxe-boot all-hosts allows any host to receive an IP address from the IP address pool.

CLI network-admin@switch > dhcp-create name pxedhcp vnet pxevnet initial-ip-pool dhcppool pxe-boot all hosts

If you specify the parameter, pxe-boot by-host-mac, only PXE-booting systems with registered MAC addresses are allowed to PXE boot and get an IP address.

1. Rack your bare metal server hardware and connect it to your switch. If you are not using the option pxe-boot all-hosts, write down the MAC address of the network adapter.

2. To boot a specific MAC address with hostname r5-d4 using PXE boot, use the following command:

CLI network-admin@switch > dhcp-host-add dhcp-name pxedhcp hostname r5-d4 mac 00:25:90:63:8c:26 pxe-boot

3. Power on the bare metal server. 4. After the server has PXE booted, it obtains an IP address from the DHCP server and downloads pxelinux.0

bootloader code. 5. The PXE Boot Menu is displayed on the bare metal server. 6. Select an installation type from the list to install on the bare metal server and complete the installation.


Customizing PXE Boot OptionsTo create a custom PXE boot image, copy the desired file to the switch, and be sure that an ISO library is created, and NFS automounting is configured:

cp CentOS-6.2-x86_64-bin-DVD1.iso /net/server-ip-address/iso-lib/pool-name/import

The IP address is the IP address of the switch, and the pool-name is the storage pool created in the ISO library. Be sure to import a CD/DVD image that includes the PXE boot files.

Configure the ISO image as an available image for the switch to use in PXE boot environments using the following syntax:

CLI network-admin@switch > iso-library-image-import iso-library-name store-new image-label centOS-6.2-x86_64dvd image-file Centos-6.2-x86_64-bin-DVD1.iso image-library store-new



You can use the dhcp-pxe-menu-show to display the default values for the menu:

CLI network-admin@switch > dhcp-pxe-menu-show dhcp-name pxedhcp

dhcp-name: pn-dhcp-dnsname: centOS-6.2-amd64-installiso-library: pluribusiso-label: centOS-6.2-x86_64dvdmenu-label: CentOS 6.2 amd64 Installkernel-iso-path: images/pxeboot/vmlinuzinitrd-iso-path: images/pxeboot/initrd.imgappend: initrd=<initrd-path> ks=http://<dhcp-server-ip>:80/kickstarts/centos.ks ksdevice=eth0 interface=eth0iso-url: http://::/vmiso/centOS-6.2-x86_64

The server-ip is the IP address of the switch, and the initrd-path is the path to the copied file on the TFTP server and is replaced when the PXE menu is generated. You are likely to find any append arguments on the Linux DVD in the pxelinux.cfg/default file.

Some arguments depend on your switch configuration. The first argument is the DHCP server IP address. The second argument is the path to the copied initrd file. This file is shared on the TFTP server and is replaced when the PXE boot menu is generated. Connect using TFTP and download the file to inspect it.

Creating a Custom PXE Boot MenuYou can create your own PXE boot menu based on the details of the ISO image:

CLI network-admin@switch > dhcp-pxe-menu-add dhcp-name pxedhcp name centos-6.5 iso-library iso-lib-pool-disk1 iso-label centOS-5.5-x86_64 kernel-iso-path /image/pxeboot/vmlinuz initrd-iso-path images/pxeboot/initrd.img append “initrd=10.10.20.147” menu-label CentOS-6.5

name The name of the PXE boot menu item.

iso-label The name chosen when the ISO image was added.

menu-label The label for the file as it appears in the PXE boot menu.

kernel-iso-path The path to the kernel on the ISO image.

initrd-iso-path The path to initrd on the ISO image

append Any arguments to pass to the kernel at boot time.

iso-url The location of the ISO image


133

Use the dhcp-pxe-menu-show command to display the menu:

CLI network-admin@switch > dhcp-pxe-menu-show

name: centOS-6.5iso-library: pluribusiso-label: centOS-6.5-x86_64menu-label: CentOS 6.5kernel-iso-path: images/pxeboot/vmlinuzinitrd-iso-path: images/pxeboot/initrd.imgappend: initrd=<initrd-path>

ks=http://<server-ip>:<web-port>/kickstarts/centos-6_5.ks ksdevice=eth0 interface=eth0

iso-url: http://172.16.23.1/vmiso/centOS-6.5-x86_64dhcp-interface: dhcp-name: vlb-dhcp



135

External Disk Drive Installation Guide

For Pluribus Networks hardware models F64 and E28Q, you can install external hard drive disks for additional storage. You can install either SSD or Fusion I/O disk types.

Locating the Disk Drive CarrierThe disk drive carrier is located on the rear of the F64 and E28Q models.

Before adding or removing disks from the switch, power down the switch.

To remove the disk drive from the switch, use the following steps:

1. Locate the small slot in the drive button, and using a small slot screwdriver or a small coin, turn the slot to align with the Unlock icon.

2. Press the button to release the drive carrier from the drive slot and release the front latch. 3. Use the latch to carefully pull the drive carrier from the slot.4. Place the external memory drive into the drive carrier. 5. Line up the holes on the memory drive with the holes on the carrier.

Be sure to follow all appropriate precautions to prevent Electrostatic Discharge on the new hard drive disk. Take care when removing the disk from the ESD bag, and installing it in the hard drive carrier.

External Drive Location

Button slot aligned with Unlock icon.


6. Insert the screws on each side and using a Phillips head screwdriver, hand tighten the screws into the disk.7. Return the carrier to the empty slot on the switch, and push the drive into the slot. 8. Close the latch of the drive carrier and be sure that it clicks into place.9. With a slot screwdriver or small coin, turn the slot in the round button to a vertical position. This locks the drive

into the switch.10. Power on the switch and the new disk is initialized during the boot process.




137

Configuring High Availability for Storage Folders

Storage folders can be replicated between two switches by configuring a vFolder on one switch. This creates a similar folder on the second switch which is replicated from the active switch to the peer switch at the configured backup interval.

You can also configure an IP address for the vFolder that allows you to share the folder using NFS or SFTP.

In this example, there are two switches in the fabric, pleiades24 and pleiades25. You configured a storage folder, iso-images, on Pleiades24. The VLAN 110 has the scope fabric, and has a IP pool of 192.168.11.0/24. To backup the vFolder every 30 minutes, configure the backup interval to 30 minutes. Pleiades25 has a storage pool, datapool, configured on it.

1. Create a vFolder on pleaides24 and add pleaides25 as the peer switch:

CLI network-admin@switch > storage-vfolder-create name my-backup folder iso-files local-switch pleiades24 peer-switch pleiades25 peer-pool datapool backup-interval 30 ha-ip 192.168.11.17 ha-netmask 24 ha-vlan 110 ha-if data

2. Display the configuration using the storage-vfolder-show command:

CLI network-admin@switch > storage-vfolder-show format all layout vertical

name: my-backupfolder: iso-fileslocal-switch: pleiades24local_pool: pool-disk4peer-switch: pleiades25peer-pool: datapoolbackup-interval: 1800last-backup: 10:23:51active-sw: pleiades24ha-nic: eth2.110ha-ip: 192.168.11.17/24ha-vlan: 110ha-vxlan: 0ha-if: mgmtfailover_controller: 0failover_action: stop-oldforce: false

The show output displays the failover controller as 0, the failover-action as stop-old, and force as false by default.

Currently, failover to the peer switch does not occur automatically. When you issue the storage-vfolder-failover command, you failover to the peer switch to become the active switch.

CLI network-admin@switch > storage-vfolder-failover name my-backup active-sw pleiades25

Informational Note: Before you begin configuring this feature, there are two pre-requisites for it:

• You must create a storage folder using the storage-folder-create command. • You must have the name of the peer storage pools to add to the configuration.


When you issue this command, the following actions occur on the local switch:

The folder, my-backup on the current active switch is deactivated. It is unshared, and unmounted on the local switch.

The folder, my-backup, on the peer switch is activated.

If a HA IP address is configured, it is added to the new primary switch.

If the local folder is shared over NFS or SFTP, the sharing is activated on the new primary folder.

The local switch begins replicating the folder, my-backup, onto the peer switch.

Using the Force Option for vFolder FailoverDuring vFolder failover, if the primary switch is not available, the failover operation fails and returns an error message. If the force option is specified, the failover operation continues by enabling the folder on the peer switch. The vFolder on the primary switch is not deactivated.

To use the force option, use the following syntax:

CLI network-admin@switch > storage-vfolder-failover name mybackup active-sw pleiades25 force




139

Configuring a Linux Netvisor KVM

There are three ways to create a Netvisor KVM:

From a bootable ISO image that runs in memory and is not persistent.

From a bootable ISO image used to install the Linux distribution onto a disk-image within the switch.

From an already created disk image imported onto the switch from another switch.

1. Your developer virtual machine requires a disk volume to install and store the operating system. Verify that your switch has sufficient physical storage capacity (GB):


switch name raid-type used avail------ ---- --------- ---- -----pbg-nvos pool-disk1 no_raid 422K 5.88Gpbg-nvos rpool no_raid 21.2G 10G

Using the storage-pool-show command also displays any problems with storage pools, such as failed disks or degraded RAID states.

Creating a storage pool also creates a disk library. After you create a storage pool, verify that a disk library was created:

CLI network-admin@switch > disk-library-show layout vertical

switch: pbg-nvosname: disk-lib-pool-disk1sharing: nfsimport-share: pbg-nvos:/disk-lib/pool-disk1/importexport-share: pbg-nvos:/disk-lib/pool-disk1/export

Look for available ISO images on the switch:

CLI network-admin@switch > iso-library-image-show

switch label library------ ----- -------pbg-nvos ubuntu-12 iso-lib-pool-disk1

By default, creating a Netvisor KVM occurs on a non-rpool storage pool randomly chosen when you use the netvisor-kvm-create command. To specify the storage pool for the Netvisor KVM, use the parameter storage-pool pool-name when creating the Netvisor KVM.

Informational Note: You cannot store disk images and ISO libraries in the root storage pool, rpool. Storage outside of rpool must be configured using storage-pool commands before you can store images and ISOs.


2. To create a Netvisor KVM from a bootable ISO image for temporary use, you can use the CentOS-6.5 ISO image on the switch and add 2 GB of memory for it.

CLI network-admin@switch > netvisor-kvm-create name test vnet VNET33 iso-label centOS-6.5-x86_64 enable storage-pool p1-testpool memory 2g cpus 2 hda-size 10g boot-order hdisk,cdrom hda-lib disk-lib-vnet1 hda-if ide

Netvm created. Please use netvm-interface-add to add interfaces and netvm-start to boot.

3. Add a network interface to the Netvisor KVM:

CLI network-admin@switch > netvisor-kvm-interface-add netvm-name vm-temp if mgmt

4. Verify the interface is added:

CLI network-admin@switch > netvisor-kvm-interface-show

netvisor-kvm-name nic ip assignment mac vlan vxlan if----------------- --- --- -------------- ---------- --

vm-temp vm-temp.eth0::/0 none 66:0e:94:11:ae:cc 0 0 mgmt

5. Now, you can start the NetVM, using the netvisor-kvm-start command:

CLI network-admin@switch > netvisor-kvm-start name vm-temp

VM running. From outside switch, connect to vnc port :1.Ex: vncviewer 172.17.245.201:1

The IP address for the VNC is the same as the IP address of the KVM interface.

6. To display the status of the Netvisor KVM, use the netvisor-kvm-show command:

CLI network-admin@switch > netvisor-kvm-show layout vertical

name: vm-temptype: netvmscope: fabricvnet: corp-fabricvnet-service: dedicatedgateway: ::memory(MB): 2000cpus: 1vm-state: runningboot-order: cdrom,hdiskiso-label: centOS-6.5hda-label:hdb-label:hdc-label:hdd-label:vnc-port: 1


141

7. To access the Netvisor KVM virtual console, use a compatible VNC viewer.

vncviewer 172.17.245.201:1

TigerVNC Viewer for X version 1.0.0...

8. The installation interface for the Ubuntu image is displayed.


Creating a Disk-based Netvisor KVMTo create a disk-based Netvisor KVM, use the Ubuntu ISO image, 2GB of memory, and create a virtual disk for the Netvisor KVM. You can use the Netvisor KVM disk library created when you create the Netvisor KVM.

Informational Note: The KVM exists until the switch is reset by a reboot or power loss. In this case, you need to recreate the KVM.



1. Create the Netvisor KVM and disk library:

CLI network-admin@switch > netvisor-kvm-create name disk-vm vnet corp-fabric iso-label unbuntu-12 memory 2g hda-size 5g hda-lib disk-lib-pool-disk1

Netvm created. Please use netvm-interface-add to add interfaces, and then netvm-start to boot

2. Add a network interface to the Netvisor KVM, and then start the Netvisor KVM.

CLI network-admin@switch > netvisor-kvm-interface-add netvm-name disk-vm if mgmt

CLI network-admin@switch > netvisor-kvm-start name disk-vm


3. Display the Netvisor KVM information:

CLI network-admin@switch > netvisor-kvm-show layout vertical

name: disk-vmtype: netvmscope: fabricvnet: corp-fabricvnet-service: dedicatedgateway: ::memory: 2GBcpus: 1vm-state: runningboot-order: cdrom,hdiskiso-label: ubuntu-12hda-label: netvm-disk-vm-hdahdb-label:hdc-label:hdd-label:vnc-port: 2




143

Creating a KVM by Importing an ISO ImageTo create a NetVM from an imported ISO image, you must copy the image to the disk-library where you install the NetVM.

1. Copy the ISO image to the disk library:

% cp vm-disk2.img /mnt/tmp/disk-lib/newpool/import

2. Verify that the image is available:

CLI network-admin@switch > disk-library-imports-showname disk-lib-newpool

name----vm-disk2.img

3. Import the ISO image into the disk library:

CLI network-admin@switch > disk-library-image-import disk-library-name disk-lib-newpool image-label vm-disk2 image-file vm-disk2.img

4. Create the NetVM that uses the disk image:

CLI network-admin@switch > netvisor-kvm-create name vm-disk2 vnet corp-fabric hda-lable vm-disk2 memory 2g cpus 2

Netvm created. Please use netvm-insterface-add interfaces, and then netvm-start to boot.

Adding Virtual Machine (VM) Instances to the Server-SwitchBhyve images (VMM) provides support for virtual machines but provides better throughput than KVM.

Kernel-based Virtual Machine (KVM) is a Linux kernel virtualization hypervisor that can host different guest operating systems. VMM is used in a similar manner as KVM, but does not support a graphical user interface (GUI).

To create a VM for CentOS 6.5 with a 20G disk space, and 4G memory on VNET, centos, use the following steps:

Informational Note: nvOS does not have VM-compatible images in the ISO library. You must import compatible images onto the switch.

You cannot run KVM and VM on the same switch. You must shut down any KVM instances before you can start VM instances.

Informational Note: VM supports only 1 CPU per virtual machine and does not support a graphical user interface (GUI).


1. Create the VMM disk and storage:

CLI network-admin@switch > netvisor-vm-create name centos6.5 vnet centos scope fabric iso-label centos-6.5-86_64 memory 4g hda-size 20g boot-at-console-connect true

Netvisor vm created. Please use interface-add to add interfaces and then start to boot.

2. Add the interface to the VM:

CLI network-admin@switch > netvisor-vm-interface-add name centos6.5 vlan 100 if mgmt

3. Start the VMM image:

CLI network-admin@switch > netvisor-vm-start name centos6.5

VM running. Use vmm-console to connect to VM

4. Log into the VM:

CLI network-admin@switch > netvisor-vm-console-login

5. Complete the VM configuration using the CLI interface for CentOS 6.5.

To display a list of VMs on the switch, use the following command:

CLI network-admin@switch > netvisor-vm-show format all layout vertical

id: a0000dd:10name: centos-6.5type: netvmmscope: fabricvnet: test-bvnet-service: dedicatedstate: enabledlocation: techpubs-aquila1storage-pool: rpoolgateway: ::template: nomemory: 4Gcpus: 1vm-state: runningiso-label: centOS-6.5-x86_64hda-label: netvisor-vm-centos6.5-hdavmm-hda-if: ahci-hdhdb-label: vmm-hdb-if: ahci-hdhdc-label: vmm-hdc-if: ahci-hdhdd-label: vmm-hdd-if: ahci-hdboot-at-console-connect: truedelete-hda: false


145

To view a list of VMM interfaces, use the netvisor-vm-interface-show command:

CLI network-admin@switch > netvisor-vm-interface-show format all layout vertical

netvisor-vmm-name: b33h1v3nic: eth0.106ip: ::/0assignment: nonemac: 66:0e:94:dd:69:dfvlan: 106vxlan: 0if: mgmtalias-on: exclusive: nonic-config: enablenic-state: downnetvisor-vmm-name: test-beenic: eth1.110ip: ::/0assignment: nonemac: 66:0e:94:dd:16:42vlan: 110vxlan: 0if: mgmtalias-on: exclusive: nonic-config: enablenic-state: downnetvisor-vmm-name: ubuntu-11nic: eth0.13ip: ::/0assignment: nonemac: 66:0e:94:dd:dd:02vlan: 13vxlan: 0if: mgmtalias-on: exclusive: nonic-config: enablenic-state: downnetvisor-vmm-name: centos65nic: eth1.101ip: ::/0assignment: nonemac: 66:0e:94:dd:1f:78vlan: 101vxlan: 0if: mgmtalias-on: exclusive: nonic-config: enablenic-state: down





147

Managing Linux VM Images

Linux NetVMs enable you to write software that runs directly on the switch with Linux OS. If the NetVM is configured on a VNET with the scope fabric, then software that runs on the VMs has access to the complete set of Pluribus Networks nvOS® APIs which provide an open, programmatic interface to the network.

1. To display the list of all VMs on the switch, use the netvisor-kvm-show command. 2. To start the NetVM named vm-disk, use the netvisor-kvm-start command.

3. To modify the NetVM, use the netvisor-kvm-modify command.

CLI network-admin@switch > netvisor-kvm-modify name vm-disk [disable|enable] memory cpus hda-size hda-lib boot-order iso-label hda-label hdb-label hdc-label hdd-label

4. To reset a NetVM, use the netvisor-kvm-reset command.

5. To shutdown the NetVM, use the netvisor-kvm-shutdown command.

6. To immediately halt the NetVM, use the netvisor-kvm-kill command.

7. To permanently delete the NetVM, use the netvisor-kvm-delete command.

The disk library images with NetVM content are not automatically deleted when the NetVM is deleted. The images remain available if you want to reinstall them. To delete the disk library image and free space in the disk library, use the disk-library-image-remove command.

Changing the State of a NetVMThe command, netvisor-kvm-kill, is similar to pressing the power button for an extended period on the virtual system with the NetVM. The command, netvisor-kvm-shutdown, sends an ACPL shutdown signal to the NetVM and may display a dialog box with a message asking if you want to shutdown the NetVM. The command, netvisor-kvm-reset sends an ACPI reset signal to the NetVM.

Since netvisor-kvm-shutdown and netvisor-kvm-reset send an ACPI signal to the NetVM, the NetVM is running until the guest OS shuts it down. The command, netvisor-kvm-show may display a status of running even after a state change command is issued.





149

Configuring and Implementing NetZones

OverviewNetZones allow you to execute code within the switches, and allows you to execute x86 Solaris code, either custom programs or pre-compiled applications. NetVMs allow you to install x86 Linux distributions and execute x86 Linux code, either custom programs or pre-compiled applications. Software installed in a NetZone or a NetVM can access the nvOS®APIs which provide an open, programmatic interface to the network.

A NetZone or NetVM can implement one or more standard network interfaces which allows the NetZone or NetVM to send and receive data on networks. The network interfaces can access the span and data network ports, and vflow commands can send specific data to the network ports so applications can access the data.

Configuring a NetZoneThe following tasks assist you with creating an OpenSolaris NetZone.

1. Create a NetZone on the switch using the following command:

CLI network-admin@switch > netvisor-zone-create name netzone-solaris vnet corp-fabric user admin

netzone admin password:*******confirm netzone admin password:*******

CLI network-admin@switch > netvisor-zone-show layout vertical

name: netzone-solaristype: netzonescope: fabricvnet: corp-fabricvnet-service: dedicatedstate: enabledgateway: ::user: adminpassword:floodlight-enable: no

The output specifies the name of the NetZone as netzone-solaris with the scope of fabric. The scope of the NetZone is the same as the VNET where you created the NetZone. In this case, the default VNET has the scope of fabric and the NetZone has access to all switches in the fabric.

Informational Note: The nvOS® APIs are declared in the following C header files:

• /usr/include/nvc_client.h• /usr/include/nvOS.hThe Java bindings are documented in /usr/java/doc/libnvos/index.html

Only C and Java APIs are supported by nvOS®.

Informational Note: When you create a Netvisor zone, the zone is created in the rpool storage pool unless you specify a datapool location to create the zone. Use the storage-pool parameter to specify a storage pool.


2. To allow traffic to flow through the NetZone, you create an interface and add an IP address:

CLI network-admin@switch > netvisor-zone-interface-add netzone-name netzone-solaris if data ip 172.17.176.11/16

CLI network-admin@switch > netvisor-zone-interface-show layout vertical

netzone-name: netzone-solarisip: 172.17.176.11/16assignment: staticmac: 66:0e:94:11:26:5cvlan: 0vxlan: 0if: data

The NetZone is assigned the IP address 172.17.176.11 on the switch interface for data. If you want access to the NetZone through the management ports, then you should create another interface and add the parameter, mgmt, instead of data.

3. To access the NetZone, use SSH and any terminal application:

% ssh 172.17.176.11 -t adminPassword:********Last login: Tue Jan 31 22:07:31 2012 from 172.17.176.100Pluribus Networks, Inc. SunOS 5.11 pn-snv137 January 2012

4. Display the sample code installed in the admin home directory:

-bash-4.0$ ls -lr.:total 3drwxr-xr-x 6 pbg staff 6 May 30 19:03 samples

./samples:total 12drwxr-xr-x 2pbg staff 5 May 30 19:03 Eventsdrwxr-xr-x 2pbg staff 5 May 30 19:03 Snoopdrwxr-xr-x 2pbg staff 5 May 30 19:03 eventsdrwxr-xr-x 2pbg staff 5 May 30 19:03 nvsnoop...

-bash-4.0$ cd samples/nvsnoop/-bash-4.0$ lsMakefile README nvsnoop.c

5. gcc and gmake are preinstalled in the developer zone. Use gmake to build the sample code:

-bash-4.0$ gmakegcc -pthreads -c nvsnoop.cgcc -pthreads -o nvsnoop nvsnoop.o -lnvOS -lsocket -lnsl


151

6. You can now run the nvsnoop sample program. Use the admin password that you configured when you installed the switch.

-bash-4.0$ nvsnoop --vnet myfabric-global --vlan 5 --user network-admin \--pass <password>Displaying captured packets. Press Ctrl-C to stop.switch: b000038, flow: b000038:25, port: 15, size: 102src-mac: 02:08:20:23:a4:da, dst-mac: 02:08:20:67:ca:2f, vlan: 5, etype: ipsrc-ip: 192.168.3.125, dst-ip: 192.168.3.115, proto: icmpswitch: b000038, flow: b000038:25, port: 54, size: 102src-mac: 02:08:20:67:ca:2f, dst-mac: 02:08:20:23:a4:da, vlan: 5, etype: ipsrc-ip: 192.168.3.115, dst-ip: 192.168.3.125, proto: icmp

To delete the NetZone, use the netzone-delete command.

The NetZone is configured with the created user, in this case, admin, as a sudo-er which means that the user can be the root and install software packages or configure the NetZone to facilitate the creation of the correct environment for your application.

If the NetZone is configured as part of the global VNET, you can use privileged nvOS® CLI commands and call privilege nvOS® API library routines.





153

Configuring vRouter Services

Configuring BGP on a vRouter

Configuring Open Shortest Path First (OSPF)

Configuring Routing Information Protocol (RIP)

Configuring Static Routes

Adding IGMP Static Joins to a vRouter

Configuring Virtual Router Redundancy Protocol

OverviewVirtual Routers (vRouters) are an important part of fabric functionality. For example, for a VNET to communicate with other VNETs, or networks external to the fabric, it may need a vRouter that spans the VNET and the external network. vRouter commands can only be executed at the fabric level by the fabric administrator, so there is no network disruption by VNET administrators. You cannot use the vRouter commands as a VNET administrator.Routing protocols essentially work the same way on virtual routers as physical routers. Detailed information about routing protocols is not covered in this overview.

The vRouter feature supports common routing protocols such as BGP, OSPF, RIP, and static routes.

To create a vRouter on the global VNET, and create a gateway between two networks that connect to the switch ports, use the following command:

CLI network-admin@switch > vrouter-create name default-gateway vnet fabricname-global

CLI network-admin@switch > vrouter-interface-add vrouter-name default-gateway ip 172.16.23.33/24 if data

CLI network-admin@switch > vrouter-interface-add vrouter-name default-gateway ip 10.9.18.147/16 if data

You just created an interface for the external network (10.9.18.147) and the internal network (172.16.23.33). By default a static route is created between interfaces added to a vRouter.


Configuring Prefix Lists for BGP and OSPF Prefix lists allow you to permit or deny host IP addresses from route distribution in BGP and OSPF configurations. To configure prefix lists for BGP, this example assumes that you have a vRouter configured for BGP, vrouter-bgp, and you want to deny the IP address, 172.26.0.0 with the netmask 255.255.0.0, sequence number 5, and minimum prefix length 17 bits:

CLI network-admin@switch > vrouter-prefix-list-add vrouter-name vrouter-bgp name deny-bits action deny prefix 172.26.0.0 netmask 255.255.0.0 seq 5 min-prefix-len 17

This prefix list rejects any subnets of 172.26.0.0/16 with prefixes 17 bits or longer. For example, the subnets 172.26.16.9/30 and 172.26.101.0/24 are rejected from route distribution.



The sequence number allows you to insert or remove new lines in a prefix list as well as at the beginning or end. It is recommended that you increment the sequence numbers by 10 so you can easily add or subtract lists from the configuration.

Configuring Packet Relay for DHCP ServersYou can configure a vRouter to relay DHCP requests from local clients to a centralized DHCP server. Because the initial DHCP request arrives from a client that typically does not have an IP address, the client must find the DHCP server using a Layer 2 broadcast.

The DHCP server must know the subnet and the MAC address of the client before the server can allocate an IP address to the client. The DHCP server needs the subnet information to ensure that the IP address that the client receives can work on the client’s subnet. The MAC address is necessary so that the DHCP server can find any information that is unique to the client.

When you configure the vRouter as a DHCP proxy, the vRouter converts the local broadcast packet from the client to a unicast packet and forward it to the server.

Because the DHCP client does not have an IP address when it sends the DHCP request packet, the client uses the IP address, 0.0.0.0, as the source IP address and the general broadcast address 255.255.255.255 for the destination.

The vRouter replaces the source address with the IP address assigned to the interface where the request is received, and replaces the destination IP address with the address you specify in the vRouter packet-relay command.

To configure packet-relay for a DHCP server with the IP address 172.16.21.34 and vRouter interface eth11.100, use the following syntax:

CLI network-admin@switch > vrouter-packet-relay add vrouter-name vrouter-dhcp forward-proto dhcp forward-ip 172.16.21.34 nic eth11.100

Once you’ve added the configuration, you cannot modify it. If you made a mistake or want to add a new configuration, you must use the vrouter-packet-relay-remove command.


Configuring Hardware Routing for a vRouterHardware routing implements the same mechanisms as software routing for the control plane. You create interfaces on hardware routers and map them to VNICs in the vRouter zone. You can configure up to seven (7) hardware routers on a platform.

The supported protocols are as follows:

OSPF — OSPF does not use a TCP/IP transport protocol such as UDP or TCP, but is encapsulated in the IP datagram with protocol number 89. OSPF uses multicast addressing for route flooding on a broadcast domain. For nonbroadcast network, special provisions in the configuration facilitate neighbor discovery. OSPF reserves the multicast addresses 224.0.0.5/6 for IPv4 or FF02::5/6 for IPv6.

BGP — BGP uses TCP and port number 179.

RIP — uses the following parameters:

• RIPv1 — IPv4 uses UDP and port 520, and advertise address - broadcasting

• RIPv2 — IPv4 uses UDP and port 520, and advertise address - 224.0.0.9

• RIPng — IPv6 uses UDP and port 521, and advertise address - FF02::9

PIM — IPv4 uses protocol 103 with multicast address 224.0.0.13



155

To create a hardware routing on a vRouter, hwtest, on VNET, fabricname-global, use the following command:

CLI network-admin@switch > vrouter-create hwtest vnet fabricname-global router-type hardware

Use the same commands as software routing to add protocols and interfaces.





157

Configuring BGP on a vRouter

Border Gateway Protocol (BGP) is a path-vector protocol and is the most commonly used routing protocol on the Internet. It advertises the paths required to reach a certain destination. BGP is also a Layer 4 protocol that sits on top of TCP, and is simpler than Open Shortest Path First (OSPF). In Figure 1 Configuring BGP for Two VLANs, you want network traffic from the source host to reach the destination host. But when different VLANs are configured, the source host traffic is not aware of the route between the source host and the destination host. However, there is a VLAN that spans VLAN 33 and VLAN 55. You solve this problem by configuring BGP in the same Autonomous System (AS) 100 that sends traffic over VLAN 35. This allows the source host to learn the route to the destination host.

Using a loopback address for peering is useful when there are multiple paths between the BGP peers which would otherwise tear down the BGP session if the physical interface us ed for establishing goes down. It also allows the vRouters running BGP with multiple links between them to load balance over the available paths.

Figure 1: Configuring BGP for Two VLANs

This example assumes that you have two VLANs, VLAN33 and VLAN55. Also, that you have added ports to the configuration.

Begin by configuring vRouter1, a software vRouter, on VLAN 33 with the BGP information:

CLI network-admin@switch > vrouter-create name vrouter1 vnet fabricname-global router-type software bgp-as 100 bgp-redist-connected-metric none

Additional BGP parameters include the following:

bgp-redist-static-metric — redistribute static BGP route metric number

bgp-redist-connected-metric — redistribute connected BGP route metric


bgp-redist-rip-metric — redistribute BGP into RIP process metric

bgp-redist-ospf-metric — redistribute BGP into OSPF process metric

bgp-cluster-id — the ID assigned to the BGP cluster.

bgp-max-paths — maximum number of BGP paths

bgp-ibgp-multipath — allow the BGP vRouter to select multiple paths for load sharing.

bgp-bestpath-as-path — allow BGP to use the best path for traffic forwarding.

bgp-dampening|no-bgp-dampening — suppress flapping routes so they are not advertised.

bgp-graceful-restart|no-bgp-graceful-restart — mechanism for BGP that helps minimize the negative effects on routing caused by BGP restart.

bgp-stalepath-time — how long a router waits before deleting stale routes after an end of record (EOR) message is received from the restarting router.

Add the IP addresses and VLANs:

CLI network-admin@switch > vrouter-interface-add vrouter-name vrouter1 ip 10.16.35.33/24 vlan 35


Add the BGP information:

CLI network-admin@switch > vrouter-bgp-add vrouter-name vrouter1 neighbor 10.16.35.55 remote-as 100

CLI network-admin@switch > vrouter-bgp-add vrouter-name vrouter1 network 10.16.33.0/24


159

Display the interface information for vrouter33:

CLI network-admin@switch > vrouter-interface-show format all layout vertical

vrouter-name: vrouter33 nic: eth1.33ip: 10.9.100.100/16assignment: staticmac: 66:0e:94:30:c6:92vlan: 33vxlan: 0if: dataalias-on: exclusive: nonic-config: enablenic-state: upsecondary-macs: vrouter-name: vrouter33nic: eth2.33ip: 192.168.42.11/24assignment: staticmac: 66:0e:94:30:25:5evlan: 33vxlan: 0if: dataalias-on: exclusive: nonic-config: enablenic-state: upsecondary-macs:

If you want to filter IP hosts, you can add prefix lists to the BGP configuration. See Configuring Prefix Lists for BGP and OSPF.

Then, configure vRouter2 on VNET 55:

CLI network-admin@switch > vrouter-create name vrouter2 vnet fabricname-global router-type software bgp-as 100 bgp-redist-connected-metric none

Add the IP addresses and VLANs:



Then add the BGP information:

CLI network-admin@switch > vrouter-bgp-add vrouter-name vrouter2 neighbor 10.16.35.33 remote-as 100

CLI network-admin@switch > vrouter-bgp-add vrouter-name vrouter2 network 10.16.55.0/24


And finally, add the loopback address:

CLI network-admin@switch > vrouter-loopback-interface-add vrouter-name vrouter1 index 5 ip 1.1.1.1

The index value is a number that uniquely identifies the vRouter in the AS.

Display the vRouter BGP configuration:

CLI network-admin@switch > vrouter-bgp-show format all layout vertical

vrouter-name: vrouter33ip: 10.16.35.55neighbor: 10.16.35.55remote-as: 100next-hop-self: noroute-reflector-client: nooverride-capability: nosoft-reconfig-inbound: nomax-prefix-warn-only: novrouter-name: vrouter33ip: 10.16.33.0network: 10.16.33.0/24vrouter-name: vrouter55ip: 10.16.35.33neighbor: 10.16.35.33remote-as: 100next-hop-self: noroute-reflector-client: nooverride-capability: nosoft-reconfig-inbound: nomax-prefix-warn-only: novrouter-name: vrouter55ip: 10.16.55.0network: 10.16.55.0/24

To reset BGP neighbors, use the vrouter-bgp-neighbor-reset command.


161

To display BGP neighbors, use the vrouter-bgp-neighbor-show command.

CLI network-admin@switch > vrouter-bgp-neighbor-show

vrouter-name: vrouter1 neighbor: 10.9.100.201ver: 4remote-as: 100msg_rcvd: 11msg_sent: 19tblver: 0inQ: 0outQ: 0up/down: 00:54:04state/pfxrcd: Connectvrouter-name: vrouter2neighbor: 10.9.100.101ver: 4remote-as: 100msg_rcvd: 12msg_sent: 18tblver: 0inQ: 0outQ: 0up/down: 00:53:37state/pfxrcd: Connect

Additional BGP ParametersThere are additional BGP parameters that you can use to optimize your BGP network. Add any of the following parameters:

ebgp-multihop — a value for external BGP to accept or attempt BGP connections to external peers, not directly connected, on the network. This is a value between 1 and 255.

update-source vrouter — the source IP address of BGP packets sent by the router. This parameter is required if you want BGP to perform peering over a loopback interface.

prefix-list-in — specify a list of incoming prefixes for route redistribution.

prefix-list-out — specify a list of outgoing prefixes for route redistribution.

override-capability — override the result of capability negotiation with the local configuration. This parameter allows you to ignore a remote peer’s capability value.

soft-reconfig-inbound — defines the route refresh capability by allowing the local device to reset inbound routing tables dynamically by exchanging route refresh requests to supporting peers.

max-prefix — allows you to specify the maximum number of IP prefixes to filter.

max-prefix-warn — add a parameter to warn when the maximum number of prefixes is reached.





163

Configuring Open Shortest Path First (OSPF)

Open Shortest Path First (OSPF) is a robust link-state interior gateway protocol (IGP). You can use it when Router Internet Protocol (RIP) is not enough for your network or when you need fast convergence on the network. It uses Autonomous Systems (AS) and the concept of Areas which allows further segmentation on the network.

OSPF uses link-state information to make routing decisions, and make route calculations using the shortest path first (SPF) algorithm. Each vRouter configured for OSPF floods link-state advertisements throughout the AS or area that contains information about the router’s attached interfaces and routing metrics.

You can add more configuration options, such as hello intervals, for OSPF using the vrouter-interface-config commands. In addition, you can add stub or not-so-stubby areas to the OSPF configuration.

You can also manually change the OSPF cost for the configuration. Cost is the metric used by OSPF to judge the feasibility of a path. If you specify 0 as the cost, the vRouter automatically calculates the cost based on the bandwidth of the interface.

In this example, you configure OSPF for two vRouters with an area of 5. The network has the following configuration:

VLAN 35 with IP addresses 10.16.35.0/24

VLAN 45 with IP addresses 10.16.55.0/24

Figure 1: OSPF

1. First, create the vRouter for VNET33, vrouter1.

CLI network-admin@switch > vrouter-create name vrouter1 vnet fabricname-global

Informational Note: For switches with ONVL, the only available VNET is a global VNET created when a fabric is created for the first time. Use tab complete in the CLI to display the VNET and continue the configuration.


2. Add vRouter interfaces to the vRouter:

CLI network-admin@switch > vrouter-interface-add vrouter-name vrouter1 ip 10.16.35.1 netmask 24 vlan 35 if data nic-enable

CLI network-admin@switch > vrouter-interface-add vrouter-name vrouter1 ip 10.16.55.1 netmask 24 vlan 55 if data nic-enable

3. Add the subnets, 10.16.35.0/24 and 10.16.45.0/24, to VLAN33 with the area 0:

CLI network-admin@switch > vrouter-ospf-add vrouter-name vrouter1 network 10.16.35.0/24 ospf-area 0

4. Add the second IP address with the area 0.

CLI network-admin@switch > vrouter-ospf-add vrouter-name vrouter1 network 10.16.55.0/24 ospf-area 0

5. Add interfaces for OSPF hello intervals of 30 seconds:

CLI network-admin@switch > vrouter-interface-config-add name vrouter1 nic eth0.35 ospf-hello-interval 30 ospf-cost 0

CLI network-admin@switch > vrouter-interface-config-add name vrouter1 nic eth0.55 ospf-hello-interval 30 ospf-cost 0

If you specify 0 as the cost value, the vRouter calculates the OSPF cost automatically based on the bandwidth of the interface.

When you modify the OSPF hello interval, the ospf-dead-interval is automatically reset to 4 times the hello interval.

6. Display the configuration by using the vrouter-ospf-show command:

CLI network-admin@switch > vrouter-ospf-show layout vertical

vrouter-name: vrouter1network: 10.16.35.0netmask: 24ospf-area: 0vrouter-name: vrouter1network: 10.16.55.0netmask: 24ospf-area: 0stub-area: 11stub-type: stubospf-hello-interval: 30metric: 34

The metric value can reflect the cost of routes advertised as OSPF routes. It may also reflect the cost of routes advertised with other protocols.


165

Adding Areas and Prefix Lists to OSPFYou can now configure OSPF areas as a stub area, stub-no-summary area, or a not so stubby area (NSSA). Stub areas see detailed routing information from other areas, but only summary information about networks outside of the AS. Stub-no-summary areas summarize external routes and routes from other areas. Routers in this type of area only see routing information local to their area. Not so stubby areas (NSSA) connects to the external network by introducing a Link State Advertisement (LSA) used within the area to carry external routes originating with boundary routers connected to this area.

To add a stub area to vRouter, vrouter-ospf, with area 100, use the following command:

CLI network-admin@switch > vrouter-ospf-area-add vrouter-name vrouter-ospf area 100 stub-type stub

The parameter, stub-type, is a required parameter.

In addition, you can add prefix lists to filter host IP addresses. To add prefix lists to OSPF areas, see Configuring Prefix Lists for BGP and OSPF.



167

Configuring Routing Information Protocol (RIP)

Routing Information Protocol (RIP) is the oldest routing protocol and provides networking information to routers. Routers need to know what networks are available and how the distance required to reach it.

RIP is a distance vector protocol, and uses hop counts to determine distance and destination. Every 30 seconds, RIP sends routing information to UDP port 50. If the router is default gateway, it advertises itself by sending 0.0.0.0 with a metric of 1.

Figure 1:I RIP

1. Create vRouter1 on VNET33:

CLI network-admin@switch > vrouter-create name vrouter1 vnet fabricname-global

You can also specify how RIP routes are distributed using the parameter, rip-redistribute

static|connected|ospf|bgp.

2. Add network 10.16.33.0/24 to vrouter1:

CLI network-admin@switch > vrouter-rip-add vrouter-name vrouter1 network 10.16.33.0/24 metric 2

3. Add network 10.16.35.0/24 to vrouter1:

CLI network-admin@switch > vrouter-rip-add vrouter-name vrouter1 network 10.16.55.0/24 metric 2

4. To view the configuration, use the vrouter-rip-show command. This displays all RIP routes configured using the vrouter-rip-add command.

To view RIP routes not configured using the vrouter-rip-add command, use the vrouter-rip-routes-show command.





169

Configuring Static Routes

vRouters forward packets using either routing information from route tables manually configured or routing information calculated using dynamic routing algorithms.

Static routes define explicit paths between two vRouters and are not automatically updated. When network changes occur, you have to reconfigure static routes. However, static routes use less bandwidth than dynamic routes.

Figure 1: Configuring a Static Route

In this example, you configure a static route on vRouter1 for the network, 172.16.10.10/24 with a gateway IP address, 172.16.20.1:

CLI network-admin@switch > vrouter-static-route-add vrouter-name vrouter1 network 172.16.10.10/24 gateway-ip 172.16.20.1





171

Adding IGMP Static Joins to a vRouter

Internet Group Membership Protocol (IGMP) is used to inform vRouters about multicast groups that hosts want to join on the network, and vRouters use IGMP to verify that a host is interested in listening to a multicast group.

You can add IGMP static group membership to a vRouter in a VNET. When you enable static group membership, data is forwarded to an interface without the interface receiving membership reports from downstream hosts. This allows fast switching for multicast traffic.

You must create IGMP static groups before configuring IGMP static joins. To configure IGMP static groups, use the following command:

CLI network-admin@switch > igmp-static-group-create group-ip 239.4.9.3 vlan 33 ports 5-7

To configure an IGMP static join for group 239.4.9.3, and source IP address 192.0.2.3, use the following command:

CLI network-admin@switch > vrouter-igmp-static-join-add vrouter-name vrouter1 name igmp-vrouter-group group-ip 239.4.9.3 source-ip 192.0.2.3 interface vrouter33





173

Configuring Virtual Router Redundancy Protocol

Virtual Router Redundancy Protocol (VRRP) is an election protocol that enables virtual routing functions for a master or standby routing infrastructure for a given IP address. A virtual router is defined by a virtual router identifier (VRID) and a virtual router IP address (VIP). The scope of the virtual routers is restricted to a single VLAN.

VRRP provides information on the state of a virtual router, not the routes processed and exchanged by the router. It increases the availability and reliability of routing paths by automatic gateway selections on an IP subnetwork.

VRRP provides rapid transition from master to standby and from standby to master. The master router sends advertisements every second. If the master VRRP advertisements are not received within a window of time, three (3) seconds, then the standby virtual router becomes the master virtual router and begins performing routing for the virtual router. If the master router becomes active again, it can become the master again or allow the standby to continue as the master router. The role depends on the value assigned to VRRP priority.

Configuring VRRP PriorityThe Priority is a value used by the VRRP router for master election. The valid priority range for a virtual router is from 1 to 254. 1 is the lowest priority and 254 is the highest priority. The default value for standby routers is 100. Higher values indicate higher priority for the virtual router.

Configuring the VRRP IDThe Virtual Router Identifier is a configurable value between 1 and 255. There is no default value.

Example ConfigurationIn this example, you have the following configurations on two switches (SW1 and SW2) on the network:

VLAN 100 with IP address range 192.168.11.0/24

VNET with the name vrrp-router and scope fabric

1. On SW1, configure a vRouter:

CLI network-admin@switch > vrouter-create name vrrp-rtr1 vnet vrrp-router router-type software enable

VRRP is supported on hardware and software routers, but for this example, software is the router type on both switches.

2. Add the first vRouter interface:

CLI network-admin@switch > vrouter-interface-add vrouter-name vrrp-rtr1 ip 192.168.11.3 netmask 24 vlan 100 if data

Informational Note: You can configure up to seven hardware routers for VRRP, and only one VLAN for VRRP.


3. Use the vrouter-interface-show command to see the name of the interface:


vrouter-name: vrrp-rtr1nic: eth0.100ip: 192.168.11.3/24assignment: staticmac: 66:0e:94:dd:18:c4vlan: 100vxlan: 0if: dataalias-on: exclusive: nonic-config: enablenic-state: up

4. Now create the VRRP interface:

CLI (switch)>vrouter-interface-add vrouter-name vrrp-rtr1 ip 192.168.11.2 netmask 24 vlan 100 if data vrrp-id 10 vrrp-primary eth0.100 vrrp-priority 100

5. Now, create the vRouter and interfaces on SW2:

CLI network-admin@switch > vrouter-create name vrrp-rtr2 vnet vrrp-router router-type software dedicated-vnet-service

Note that the second vRouter is created as a dedicated VNET service because a VNET supports only one shared vRouter service.

6. Add the vRouter interface:

CLI network-admin@switch > vrouter-interface-add vrouter-name vrrp-rtr2 ip 192.168.11.4 netmask 24 vlan 100 if data

7. Use the vrouter-interface-show command to see the name of the interface:


vrouter-name: vrrp-router2nic: eth2.100ip: 192.168.11.3/24assignment: staticmac: 66:0e:94:21:a9:6cvlan: 100vxlan: 0if: dataalias-on: exclusive: nonic-config: enablenic-state: up

8. Now create the VRRP interface:

CLI network-admin@switch > vrouter-interface-add vrouter-name vrrp-rtr2 ip 192.168.11.2 netmask 24 vlan 100 if data vrrp-id 10 vrrp-primary eth0.100 vrrp-priority 50


175

9. Display the information about the VRRP setup:


vrouter-name: vrrp-router1nic: eth0.100ip: 192.168.11.3/24assignment: staticmac: 66:0e:94:dd:18:c4vlan: 100vxlan: 0if: dataalias-on: exclusive: nonic-config: enablenic-state: upvrouter-name: vrrp-router1nic: eth1.100ip: 192.168.11.2/24assignment: staticmac: 00:00:5e:00:01:0avlan: 100vxlan: 0if: dataalias-on: exclusive: nonic-config: enablenic-state: upvrrp-id: 10vrrp-primary: eth1.100vrrp-priority: 100vrrp-state: mastervrouter-name: vrrp-router2nic: eth3.100ip: 192.168.11.4/24assignment: staticmac: 66:0e:94:21:54:07vlan: 100vxlan: 0if: dataalias-on: exclusive: nonic-config: enablenic-state: upvrouter-name: vrrp-router2nic: eth3.100ip: 192.168.11.2/24assignment: staticmac: 00:00:5e:00:01:0avlan: 100vxlan: 0if: dataalias-on: exclusive: nonic-config: enablenic-state: downvrrp-id: 10


vrrp-primary: eth3.100vrrp-priority: 50vrrp-state: slave

When you intentionally disable the VRRP interface, the slave interface becomes the master interface:

vrouter-name: vrrp-router2nic: eth3.100ip: 192.168.11.1/24assignment: staticmac: 00:00:5e:00:01:0avlan: 100vxlan: 0if: dataalias-on: exclusive: nonic-config: enablenic-state: upvrrp-id: 10vrrp-primary: eth3.100vrrp-priority: 50vrrp-state: master

When you re-enable the VRRP interface, it becomes the master again, and the second interface returns to the slave:

vrouter-name: vrrp-router2nic: eth3.100ip: 192.168.11.2/24assignment: staticmac: 00:00:5e:00:01:0avlan: 100vxlan: 0if: dataalias-on: exclusive: nonic-config: enablenic-state: downvrrp-id: 10vrrp-primary: eth3.100vrrp-priority: 50

vrrp-state: slave




177

Configuring Virtual Load Balancing

Virtual load balancing (vLB) uses virtual servers instead of physical servers to balance traffic across the network. Each virtual server points to a cluster of services that reside on one or more physical hosts.

VLB uses the following transactions:

1. The client attempts to connect to the service on the load balancer.2. The load balancer accepts the connection and then decides which host receives the connection. The port and des-tination IP address are changed to match the service of the selected host.

3. The host accepts the connection and responds to the original source, the client, through the default route which is the load balancer.

4. The load balancer intercepts the return packet from the host and changes the source IP and port to match the vir-tual server IP and port, and forwards the packet back to the client.

5. The client receives the return packet and continues the process.

VLB uses four different algorithms to control and distribute traffic as well as load distribution and server selection.

roundrobin — In a round-robin algorithm, the load balancer assigns requests to a list of servers on a rotating basis. Once a server is assigned a request, the server moves to the bottom of the list.

hash-ip — In the source IP hash method, the load balancer selects a server based on the hash value of the source IP address of the incoming request.

hash-ip-port — In the source virtual IP, port hash method, the load balancer selects a server based on the hash value of the source IP address, and the source port of the incoming request.

hash-ip-vip — In the source IP, VIP hash method, the load balancer selects a server based on the hash value of the source IP address, and the destination IP address of the incoming requests.

If you already have servers that you want to use for VLB, you can following the instructions below. If you want to install Ubuntu servers as virtual machines on the switch, see Configuring Virtual Load Balancing with Ubuntu 11.04 Servers and nvOS.

If you are configuring VLB as a dedicated service on a VNET or you have not defined network interfaces for the VNET, use the vlb-interface-add command to create the vNICs.

CLI network-admin@switch > vlb-create name vlb-vnet1 vnet vnet1 dedicated-vnet-service

You need two interfaces to configure VLB: one for the external address and one for the internal address. To create the interfaces, use the following commands:

CLI network-admin@switch > vlb-interface-add vlb-name vlb-vnet1 ip 192.168.100.27 netmask 24 assignment none vlan 57 if data

CLI network-admin@switch > vlb-interface-add vlb-name vlb-vnet1 ip 10.10.10.113 netmask 24 assignment none vlan 58 if data


Display the configuration information:

CLI network-admin@switch > vlb-interface-show vlb-name vnet1-vlb layout vertical

vlb-name: vnet1-vlbnic: vnet1.mgr.eth0ip: 10.10.10.113/24assignment: staticmac: 66:0e:94:4b:b8:0cvlan: 123vxlan: 0if: datavlb-name: vnet1-vlbnic: vnet1.mgr.eth1ip: 192.168.100.27/24assignment: staticmac: 66:0e:94:4b:9d:ccvlan: 124vxlan: 0if: data

Create a VLB to balance TCP port 80 (HTTP) requests in full NAT mode between the external and internal interfaces. Full NAT mode sends all traffic to and from the servers and route through the load balancer.

CLI network-admin@switch > vlb-group-add vlb-name vnet1-vlb name vnet1-vlb-http topology full-nat proto tcp start-port 80 ext-interface vnet.mgr.eth0 int-interface vnet1.mgr.eth1

When you create a vLB group, you can also add the following parameters:

vip — the destination IP address for incoming requests

proxy-src-ip — the proxy host source IP address

proxy-src-netmask — the proxy host source netmask

start-port — the starting port of the vLB group

end-port — the ending port of the vLB group

healthcheck — the name of a healthcheck configuration

CLI network-admin@switch > vlb-group-show layout vertical

vlb-name: vnet1-vlbname: vnet1-vlb-httptopology: full-natproto: tcpext-interface: vnet1.mgr.eth0int-interface: vnet1.mgr.eth1start-port: 80end-port: 80group-enable: group-enable


179

Configure the VLB service to load balance incoming requests on group vnet-vlb-http to a pod of five Web servers:

CLI network-admin@switch > vlb-server-add vlb-name vnet1-vlb ip 192.168.18.3 group vnet1-vlb-http





Display the server information:

CLI network-admin@switch > vlb-server-show

vlb-name group ip server-enable id-------- ----- -- ------------- --vnet1-vlb vnet1-vlb-http 192.168.18.3 server-enable _vnet1-vlb-http.0vnet1-vlb vnet1-vlb-http 192.168.18.4 server-enable _vnet1-vlb-http.1vnet1-vlb vnet1-vlb-http 192.168.18.5 server-enable _vnet1-vlb-http.2vnet1-vlb vnet1-vlb-http 192.168.18.6 server-enable _vnet1-vlb-http.3vnet1-vlb vnet1-vlb-http 192.168.18.7 server-enable _vnet1-vlb-http.4

CLI network-admin@switch > vlb-show

name type scope vnet vnet-service state gateway ------------- ---- ------ --------- ------------ ------- --------- vlb-web vlb fabric vlb-web shared enabled 10.12.1.1


Monitoring the Health of VLBYou can configure health monitoring for your VLBs so that network traffic can determine if the server is available before attempting to send connections to it. Basic monitoring is simply pinging the host and determining if the host is active. Or you can send service pings ranging from simple TCP connections or using scripting interaction.

To create a VLB health monitor for vlb-vnet1 using ping, timeout 10 seconds, attempts 5, and 120 seconds interval between checks:

CLI network-admin@switch > vlb-health-config-add vlb-name vlb-vnet1 name vlb-health type ping timeout 10 attempts 5 interval 120

To remove the VLB health configuration, use the vlb-health-config-remove command.

To display the VLB health configuration, use the vlb-health-config-show command.



To display the status of the VLB health configuration, use the vlb-health-status-show command:

CLI network-admin@switch > vlb-health-status-show layout vertical

vlb-name: vlb-vnet1name: vlb-healthid: _vlbgroupstatus: alivefail: 0last: 13:47:16next: 13:47:30rtt: 1836

Viewing vLB Group StatisticsYou can view vLB Group statistics using the vlb-group-stats-show command:

CLI network-admin@switch > CLI vlb-group-stats-show format all layout vertical

switch: pubdev01name: vlb-1group: vlb-groupprocessed-bytes: 0processed-pkts: 0dropped-bytes: 0dropped-pkts: 0switch: pubdev03name: vlb-1group: vlb-groupprocessed-bytes: 0processed-pkts: 0dropped-bytes: 0dropped-pkts: 0switch: pubdev02name: vlb-1group: vlb-groupprocessed-bytes: 0processed-pkts: 0dropped-bytes: 0dropped-pkts: 0


181

Configuring Virtual Load Balancing with Ubuntu 11.04 Servers and nvOSIn this example, you configure the following features:

VNET

IP Pool

DHCP Server

Ubuntu 11.04 Servers (2)

Apache Services

VLB

VLB Health

Configuring the VLB VNET1. Using the name, vlb-web, scope fabric, and vlans 200, configure the VNET:

CLI network-admin@switch > vnet-create name vlb-web scope fabric vlans 200

2. Create the IP pool, web-ip-pool, with the IP address range of 172.16.23.0, netmask 24:

CLI network-admin@switch > ip-pool-create name web-ip-pool vnet vlb-web start-ip 172.16.23.0 end-ip 172.16.23.254 netmask 24 vlan 200


3. Create the DHCP server, web-dhcp, and add the gateway:

CLI network-admin@switch > dhcp-create name web-dhcp vnet vlb-web initial-ip-pool web-ip-pool

CLI network-admin@switch > dhcp-pool-modify dhcp-name web-dhcp dhcp-pool-name web-ip-pool gateway-ip 172.16.23.1

4. Add connectivity to your network. You’ll need this to download Apache2.

CLI network-admin@switch > vnet-manager-interface-add vnet-manager-name vlb-web-mgr ip 10.0.0.0 netmask 16 if mgmt vlan 0

CLI network-admin@switch > vnet-manager-modify name vlb-web-mgr gateway 10.0.0.1 enable

5. Create the Ubuntu servers using KVMs on the switch:

CLI network-admin@switch > netvisor-kvm-create name vlb-web-svr1 vnet vlb-web iso-label ubuntu-11.04-amd64 memory 4g cpus 2 hda-size 20g storage-pool pool-disk4

Netvisor vm created. Please use interface-add to add interfaces and then start to boot

CLI network-admin@switch > netvisor-kvm-interface-add netvisor-kvm-name vlb-web-svr1 if mgmt vlan 0

CLI network-admin@switch > netvisor-kvm-interface-add netvisor-kvm-name vlb-web-svr1 if data vlan 200

CLI network-admin@switch > netvisor-kvm-start name vlb-web-svr1


Informational Note: This step varies depending on the setup of your corporate network. In this example, the corporate network is a 10.0.0.0/16 network.

Informational Note: There is no requirement that the Ubuntu servers reside on the same switch. For this purpose, the servers are on the same switch.


183

The Ubuntu server installation takes 20-30 minutes. In the meantime, configure the KVM for vlb-web-svr2:






VM running. From outside switch, connect to vnc port :2.Ex: vncviewer 10.9.11.147:3The Ubuntu server installation takes 20-30 minutes. In the meantime, configure the KVM for vlb-web-svr3:






VM running. From outside switch, connect to vnc port :3.Ex: vncviewer 10.9.11.147:36. Next, you install Apache2 on each Ubuntu server by executing the following commands on each one. Open your VNC application and connect to an Ubuntu server:

sudo apt-get install apache2

sudo vi/var/www/index.html

7. Create the virtual load balancer:

CLI network-admin@switch > vlb-create name vlb-web vnet vlb-web shared-vnet-service enable


name type scope vnet vnet-service state gateway----------- ---- ------ ----------- ------------ ------- ------- vlb-web vlb fabric vlb-web shared enabled ::


8. Create the health check for the VLB service:

CLI network-admin@switch > vlb-health-config-add vlb-name vlb-web switch pleiades24 name web-http type http timeout 3 attempt 3 interval 11

This configuration means that the health check is performed every 11 seconds, and it verifies the service 3 times and times out after 3 seconds.

9. Create the virtual load balancing group. Note that the group name must be less than 14 characters:

CLI network-admin@switch > vlb-group-add vlb-name vlb-web name web-svc-grp proto tcp algorithm roundrobin vip 172.16.23.20 topology full-nat proxy-src-ip 172.16.23.20 proxy-src-netmask 24 start-port 80 healthcheck web-http group-enable

10. Add the Ubuntu Apache servers to the VLB group:

CLI network-admin@switch > vlb-server-add vlb-name vlb-web ip 172.16.23.3 port 80 group vlb-web-group



11. Display the configuration:


12. Display the VLB servers:

CLI network-admin@switch > vlb-server-show

vlb-name group ip port server-enable id ----------- ----------- ----------- ---- ------------- -------------- vlb-web web-svc-grp 172.16.23.2 80 server-enable _web-svc-grp.0 vlb-web web-svc-grp 172.16.23.3 80 server-enable _web-svc-grp.1vlb-web web-svc-grp 172.16.23.4 80 server-enable _web-svc-grp.2

13. Display the VLB group:

CLI network-admin@switch > vlb-group-show layout vertical

vlb-name: vlb-webname: web-svc-grptopology: full-natproto: tcpalgorithm: roundrobinvip: 172.16.23.7proxy-src-ip: 172.16.23.7/24start-port: 80end-port: 80group-enable: group-enablehealthcheck: http-service


185

14. Display the VLB health status:

vlb-health-status-show layout verticalswitch: mitch-aquila2vlb-name: vlb-webname: http-serviceid: _web-svc-grp.0status: alivefail: 0last: 09:53:01next: 09:53:17rtt: 507switch: mitch-aquila2vlb-name: vlb-webname: http-serviceid: _web-svc-grp.1status: alivefail: 0last: 09:53:14next: 09:53:28rtt: 572switch: mitch-aquila2vlb-name: vlb-webname: http-serviceid: _web-svc-grp.2status: alivefail: 0last: 09:53:14next: 09:53:28rtt: 578

15. Stop the Apache2 service on one of the Ubuntu servers by connecting with VNC and executing the command:

sudo etc/init.d/apache2 stop


16. Display the VLB health status again to verify that the server is in a failed state:

CLI network-admin@switch > vlb-health-status-show

CLI (network-admin@mitch-aquila2) > vlb-health-status-show layout verticalswitch: mitch-aquila2vlb-name: vlb-webname: http-serviceid: _web-svc-grp.0status: alivefail: 0last: 09:54:42next: 09:54:57rtt: 568switch: mitch-aquila2vlb-name: vlb-webname: http-serviceid: _web-svc-grp.1status: deadfail: 3last: 09:54:42next: 09:54:57rtt: 565switch: mitch-aquila2vlb-name: vlb-webname: http-serviceid: _web-svc-grp.2status: alivefail: 0last: 09:54:42next: 09:54:57rtt: 572


Adding Virtual Router Redundancy Protocol to VLB InterfacesYou can add VRRP to the VLB configuration so that if one interface becomes unavailable, then the second interface becomes the virtual router. Add interfaces to the VLB configuration with VRRP parameters. To configure Web server 1 as the master, use the following commands:

CLI network-admin@switch > vlb-interface-add vlb-name vlb-web if data vlan 200

CLI network-admin@switch > vlb-interface-modify vlb-name vlb-web-svr1 nic eth1.200 vrrp-id 10 vrrp-primary vlb-web-svr1 vrrp-priority 100

Informational Note: You must use the same VRRP ID for both interfaces. Otherwise, the configuration is invalid. You must also create a VRRP priority with a higher value for the primary interface and a lower VRRP priority for the secondary interface.

After stopping the Web service on server 1, the status changes to dead.



187

To add Web server 2 as the secondary virtual router, use the following command:

CLI network-admin@switch > vlb-interface-add vlb-name vlb-web if data vlan 200

CLI network-admin@switch > vlb-interface-modify vlb-name vlb-web-svr2 nic eth2.200 vrrp-id 10 vrrp-primary vlb-web-svr1 vrrp-priority 50





189

Configuring Roles and Users

Role-Based Access Control (RBAC) is a secure method of restricting access to authorized users. This method enables the network administrator to add users and assign each user to specific roles. Each role has specific permissions and allows users to perform various actions based on the scope of their role.

In this context, users are personnel that can log into the switch, and perform certain functions.

A role defines the level of access for a user account. By assigning roles to users, you can allow multiple users to complete their tasks. RBAC limits risk by ensuring that users do not have access beyond their training or level of control.

nvOS allows you to create roles and assign them to users. You can create the following types of roles:

Scope — A role can apply to the scope of local, or fabric.

Access — You allow read-only access or read-write access.

Configuration — A role can apply to the running configuration or not.

Once you create a user with a scope of local or fabric, you cannot modify the user scope. If you decide that your user needs local scope rather than fabric scope, you must delete the user and create a new one.

There are three types of roles configured for user access:

network-admin — this is a super user role and can perform all functions on the switch.

read-only-network-admin — this is a read only role and the user can only execute show commands from the CLI.

fabric-admin — this role can perform fabric-wide functions only.

Configuring Custom RolesYou can create custom roles in addition to the preconfigured ones in nvOS. When you create a role, you configure the following parameters:

name — create a name for the role

scope — specify fabric or local. Once you’ve configured the role as local or fabric, you can’t modify it. To change the scope, you must delete the role and create a new one.

access — specify the type of access for the user. You can specify any of the following types of access:

• read-write — the role can display information and make changes to the configuration. You can modify this role to read-only if you decide that the role can only use show commands at the CLI.

• running-config — the role has access to the running configuration on the switch.

• no-running-config — the role cannot access the running configuration on the switch.

For example, create the role, local-admin, with scope local, read-write access to the running configuration:

CLI network-admin@switch > role-create name fabric-admin scope local access read-write running-config

To modify the role parameter, access to read-only, use the following command:

CLI network-admin@switch > user-role-modify name fabric-admin scope fabric access read-only

When you modify the role, you can also specify to remove the role from users with the delete-from-users parameter.


To delete the role, local-admin, use the user-role-delete command:

CLI network-admin@switch > user-role-delete name fabric-admin

To display the role configuration, use the role-show command.

CLI network-admin@switch > role-show

role-show format all layout verticalid: 6000021:402name: web-svr-adminscope: fabricaccess: read-writerunning-config: denyid: 6000021:404name: test-vnet-adminscope: fabricaccess: read-writerunning-config: denyid: 6000021:405name: test-adminscope: fabricaccess: read-writerunning-config: denyid: 6000021:406name: vlan-test-adminscope: fabricaccess: read-writerunning-config: denyswitch: pleiades24id: 0:0name: network-adminscope: localaccess: read-writerunning-config: permitswitch: pleiades24id: 0:1name: read-only-network-adminscope: localaccess: read-onlyrunning-config: deny

This user has read-write access but not to the running configuration.


191

Creating and Managing UsersYou can create users and apply roles to them to manage access to the switch or network. To create a user, jdoe, scope local, password p1zz@, and initial role, local-admin, use the following syntax:

CLI network-admin@switch > user-create name jdoe scope local password p1zz@ initial-role local-admin

password:Confirm password:

To modify the initial role from local-admin to network-admin, use the following command:

CLI network-admin@switch > user-modify name jdoe initial-role network-admin

To delete the user, use the user-delete command.

To add roles to a user, jdoe, role name fabric-admin, use the following syntax:

CLI network-admin@switch > user-role-add name jdoe role fabric-admin

You can assign multiple roles to a user. For instance, if jdoe is a fabric-admin, and you also want to assign the role, local-admin, use the following command:

CLI network-admin@switch > user-role-add user-name jdoe role local-admin

CLI (network-admin@mitch-aquila2) > user-role-showswitch user-name role ------------- ----------------- ----------------------- network-admin network-admin vlb-web-svr-admin vlb-web-svr-admin test-admin test-admin test-admin test-admin-admin vlan-test-admin vlan-test-admin jdoe network-admin jdoe local-admin

ops-test1-admin fabric-admin pleiades01 java-api-admin java-api-admin

To remove a role from the user, jdoe, use the following command:

CLI network-admin@switch > user-role-remove name jdoe role fabric-admin

Informational Note: Once you configure the scope for a user, you cannot modify it. To change the scope, delete the user, and create a new one with the intended scope.

jdoe now has two roles assigned.


To display user roles, use the user-role-show command.

CLI (network-admin@pleiades24)> user-role-showswitch user-name role ------------- ----------------- ----------------------- network-admin network-admin vlb-web-svr-admin vlb-web-svr-admin test-admin test-admin test-admin test-admin-admin vlan-test-admin vlan-test-admin laurap read-only-network-admin

ops-test1-admin fabric-admin pleiades01 java-api-admin java-api-admin

To display information about all users configured in nvOS, use the user-show command:

CLI network-admin@switch > user-show

name scope uid network-admin fabric 39999 ops-mgmt-admin fabric 40000 ext-50-admin fabric 40001 www-51-admin fabric 40002 jdoe fabric 40003

The User ID (UID) is assigned by nvOS and is not configurable. You need the UID to configure user passwords for TACACS+ authentication.

To configure user, jdoe, on a TACACS+ server, use the following command:

CLI network-admin@switch > user-set-password name jdoe scope fabric uid 4003 server aaa-tacacs

See Configuring TACACS+.




193

Configuring TACACS+

About TACACS+Terminal Access Controller Access Control System (TACACS+) is an Authentication, Authorization, and Accounting (AAA) protocol that was introduced in the early 2000s. The main goal of TACACS+ is to provide a centralized database to use for authentication. It uses a client server approach by which the client queries a server and the server replies with a pass or fail for authentication. The communication between the client and server uses TCP as the connection protocol, and requires a secret key.

nvOS can be configured to use external TACACS+ servers for authentication, authorization, and accounting. You can configure any number of TACACS+ servers, and each server may be configured to handle any combination of authentication, session authorization, command authorization, session accounting, and command accounting.

It is important to note that the default “network-admin” account is exempt from all TACACS+ integration, as a fail-safe account for sites without TACACS+ and to allow access to Pluribus Networks facilities if TACACS+ is unavailable or unreachable.

TACACS+ is configured using the aaa-tacacs-create command, and using options to specify the IP address, port, password, priority, authentication methods, and accounting options. Once set up, a user can login to the switch and get CLI access using an account configured on the specified TACACS+ server.

The TACACS+ server determines what role the user has by returning a “role” attribute. The roles include “networkadmin” for full access and "read-only-network-admin" users who can only run show commandsPAP, CHAP, and MS-CHAP authentication protocols are supported.

Figure 1 illustrates a simple TACACS+ implementation.

Figure 1: TACACS+ AAA with a nvOS switch


Configuring TACACS+Using Figure 1 as an example, you can configure TACACS+ access to the switch with the following command:

CLI network-admin@switch > aaa-tacacs-create name tacacs-server scope fabric port 34 m0nk3y6 priority 3 authen authen-method ms-chap sess-acct

This command configures basic access from a user on the network to the switch. You can add the following optional parameters to the configuration:

Session accounting

Command accounting

Session Authorization

Command Authorization

To add optional parameters or to modify the current configuration, use the aaa-tacacs-modify command.

To display the status of the TACACS server, use the aaa-tacacs-status command.

To delete the configuration, use the aaa-tacacs-delete command.


195

Creating and Implementing Access Control Lists (ACLs)

Access Control Lists (ACLs) allow you to configure basic traffic filtering for IP addresses and MAC addresses. The ACL controls if routed packets are forwarded or blocked on the network. The packet is examined by the switch and then determines if the packet is forwarded or dropped based on the criteria configured in the ACLs. ONVL supports Layer 2 (MAC) or Layer 3 (IP) ACLs.

ACL criteria can be based on source or destination addresses or the protocol type. nvOS supports UDP, TCP, IGMP, and IP protocols.

You can use ACLs to restrict contents of routing updates or provide traffic flow control. ACLs can allow one host to access part of your network and prevent another host from accessing the same area. You can also use ACLs to decide what types of traffic are forwarded or blocked.

If you need more background on ACLs and using them on your network, refer to the many networking resources available.

Using a Deny IP ACL to Block Network Traffic In this example, a network is shown with a Finance server on one part of the network, and an Engineering server on another part. You want to block the Engineering server from the Finance server in order to protect company sensitive information. See Configuring an Internal Deny ACL to review the configuration sample.

Figure 1: Network Example - IP ACL for Internal Servers

Or you may discover that an external source is attempting to access your network, and ping your servers for IP addresses. You can use an ACL to block the specific source using an IP ACL.


Figure 2:IP ACL Blocking External Access

See Configuring an External Deny ACL to review the configuration example.

Using IP ACLs to Allow Network TrafficIn the same manner, you can allow specific traffic to a destination such as the external server in Figure 2 IP ACL Blocking External Access. To allow HTTP traffic to 209.225.113.24, see Configuring an External Allow IP ACL to review the configuration example.


197

Figure 3:IP ACL Allowing HTTP Traffic


Using MAC ACLs to Deny Network TrafficYou can create ACLs based on MAC addresses to deny network traffic from a specific source. MAC addresses are Layer 2 protocols and most often assigned by the hardware manufacturer. Figure 4 MAC ACL Blocking Access shows an example of a MAC address and Ethernet type that you want to block from the network.

Figure 4: MAC ACL Blocking Access

See Configuring a MAC ACL to Deny Network Traffic to review the example configuration.

Using MAC ACLs to Allow Network TrafficSo now that you’ve blocked the MAC address, let’s reverse the scenario and allow IPv4 network traffic from the MAC address to the network.


199

Figure 5:MAC ACL Allowing Access

See Configuring a MAC ACL to Allow Network Traffic to review the example configuration.




Pluribus Networks ONVL Version 2.3 201

Configuring IP ACLs

From Figure 1 Network Example - IP ACL for Internal Servers, the following information is available:

Source IP address

Source netmask

Destination IP address

Destination netmask

Type of protocol to deny - IP

Ports

VLAN

Configuring an Internal Deny ACLConfigure the ACL for denying traffic from the Engineering server to the HR server and name the ACL, deny-hr:

CLI network-admin@switch > acl-ip-create name deny-hr action deny scope local src-ip 192.168.10.2 src-ip-mask 24 dst-ip 192.168.200.3 dst-ip-netmask 24 proto ip src-port 55 dst-port 33 vlan 1505

To review the configuration, use the acl-ip-show command:

CLI network-admin@switch > acl-ip-show name deny-hr layout vertical

name: deny-ipid: b00011:20action: denyproto: ipsrc-ip: 192.168.10.2/24src-port: 55dst-ip: 192.168.200.3/24dst-port: 33vlan: 1505scope: localport: 0

Now, when you attempt to access the Finance server from the Engineering server, the packets are dropped.

Configuring an External Deny ACLFrom Figure 2 IP ACL Blocking External Access, the following information is available:

IP Address

Port Number

To configure an ACL to deny traffic from the external server, use the acl-ip-create command to create an ACL named deny-external:

CLI network-admin@switch > >acl-ip-create name deny-external scope fabric src-ip 209.255.113.24/28



CLI network-admin@switch > acl-ip-show name deny-external layout vertical

name: deny-externalid: b000022:20action: denyproto: tcpsrc-ip: 209.225.113.24/28src-port: 0dst-ip: ::/0dst-port: 0vlan: 0scope: fabricport: 0

Configuring an External Allow IP ACLTo allow HTTP traffic to the external server, 209.225.113.24 with a netmask of 255.255.255.240 and a scope of fabric, you can create an IP ACL called allow-http using the following syntax:

CLI network-admin@switch > acl-ip-create name allow-http permit scope fabric src-ip 0.0.0.0. src-mask 255.255.255.255 dst-ip 209.225.113.24 dst-ip-mask 255.255.255.240 protocol tcp dst-port 57


CLI network-admin@switch > >acl-ip-show name allow-http layout vertical

name: allow-httpid: b000025:20action: allowproto: tcpsrc-ip: 0.0.0.0/255.255.255.255src-port: 0dst-ip: 209.225.113.24/28dst-port: 57vlan: 0scope: fabricport: 0

To delete the ACL configuration, use the acl-ip-delete command.

To modify the ACL configuration, use the acl-ip-modify command.

Configuring a MAC ACL to Deny Network TrafficTo deny IPv4 network traffic from MAC address, 01:80:c2:00:00:0X, for the scope fabric, create the MAC ACL, deny-MAC, using the following syntax:

CLI network-admin@switch > acl-mac-create name deny-mac action deny src-mac 01:80:c2:00:00:0X ether-type ipv4 scope fabric


To review the configuration, use the acl-mac-show command:

CLI network-admin@switch > acl-mac-show name deny-mac layout vertical

name: deny-macid: b000015:12action: denysrc-mac: 01:80:c2:00:00:0Xdst-mac: 00:00:00:00:00:00dst-mac-mask: aa:aa:aa:aa:aa:aaether-type: ipv4vlan: 0scope: fabricport: 0

Configuring a MAC ACL to Allow Network TrafficTo allow IPv4 network traffic from MAC address, 01:80:c2:00:00:0X, for the scope fabric, create the MAC ACL, allow-MAC, using the following syntax:

CLI network-admin@switch > acl-mac-create name allow-mac action permit src-mac 01:80:c2:00:00:0X ether-type ipv4 scope fabric

To review the configuration, use the acl-mac-show command:

CLI network-admin@switch > acl-mac-show name deny-mac layout vertical

name: deny-macid: b000015:12action: denysrc-mac: 01:80:c2:00:00:0Xdst-mac: 00:00:00:00:00:00dst-mac-mask: aa:aa:aa:aa:aa:aaether-type: ipv4vlan: 0scope: fabricport: 0

To delete the ACL configuration, use the acl-mac-delete command.

To modify the ACL configuration, use the acl-mac-modify command.





205

Configuring vFlow for Analytics

A vFlow can be used to capture packets for analysis, and you can determine if the vFlow captures packets across the fabric or on a single switch. Packets are captured by forwarding them from the data plane of the switch to the control plane.

A vFlow that directs packets to the switch CPU can be configured to save packets to a file by enabling the log-packets parameter. The file is written using a libcap compatible format so that programs like TCPdump and Wireshark can be used to read the file. The file is exported to clients using NFS or SFTP.

Packet capture data is available with switch or fabric scope. The pcap files are stored over NFS in the following locations:

/net/<ServerSw_Name>/nvOS/global/flow/<Flow_Name>/switch/<Switch_Name>/pcap

/net/<ServerSw_Name>/nvOS/vnet/<VNET_Name>/flow/<Flow_Name>/switch/<Switch_Name>/pcap

/net/<ServerSw_Name>/nvOS/global/flow/<Flow_Name>/fabric/pcap

/net/<ServerSw_Name>/nvOS/vnet/<VNET_Name>/flow/<Flow_Name>/fabric/pcap

Snooping only works if you use the parameters, copy-to-cpu or to-cpu. The copy-to-cpu parameter ensures that the data plane forwards the packets and sends a copy to the CPU. Use this parameter if you want traffic to flow through the switch. The to-cpu parameter doesn’t forward packets and interrupts traffic on the switch. To snoop all application flow packets of protocol type TCP, enter the following CLI commands at the prompt:

CLI network-admin@switch > vflow-create name snoop_all scope local proto tcp action copy-to-cpu

Then use the following command to display the output:

CLI network-admin@switch > vflow-snoop

switch: pleiades24, flow: snoop_all, port: 65, size: 66, time: 20:07:15.03867188

smac: 64:0e:94:28:00:fa, dmac: 64:0e:94:2c:00:7a, etype: ipsip: 192.168.2.51, dip: 192.168.2.31, proto: tcpsport: 42120, dport: 33399


smac: 64:0e:94:28:00:fa, dmac: 64:0e:94:2c:00:7a, etype: ipsip: 192.168.2.51, dip: 192.168.2.31, proto: tcpsport: 42120, dport: 33399


smac: 64:0e:94:2c:00:7a, dmac: 64:0e:94:28:00:fa, etype: ipsip: 192.168.2.31, dip: 192.168.2.51, proto: tcpsport: 33399, dport: 42120


To restrict the flows captured to TCP port 22, SSH traffic, create the following vFlow:

CLI network-admin@switch > vflow-create name snoop_ssh scope local action copy-to-cpu src-port 22 proto tcp vflow-add-filter name snoop_ssh

Then use the vflow-snoop command to display the results:

switch: pleiades24, flow: snoop_ssh, port: 41, size: 230, time: 10:56:57.05785917 src-mac: 00:15:17:ea:f8:70, dst-mac: f4:6d:04:0e:77:60, etype: ip src-ip: 10.9.11.18, dst-ip: 10.9.10.65, proto: tcp src-port: 22, dst-port: 62356

switch: pleiades24, flow: snoop_ssh, port: 41, size: 118, time: 10:56:57.05922560 src-mac: 00:15:17:ea:f8:70, dst-mac: f4:6d:04:0e:77:60, etype: ip src-ip: 10.9.11.18, dst-ip: 10.9.10.65, proto: tcp src-port: 22, dst-port: 62356

The optional parameter vflow-add-filter restricts the output of the vflow-snoop command to the packets matching the snoop_ssh flow definition.

To capture traffic packets for a flow across the entire fabric, you create a flow with the scope of fabric. To copy the packets to a pcap file, add the log-packets option:

CLI network-admin@switch > vflow-create name fab_snoop_all scope fabric action copy-to-cpu port 22 log-packets yes

If you enable log-packets, the separate pcap files for all switches are available on any switch. In addition a consolidated pcap file is available that aggregates the packets from all switches in the entire fabric.


Analyzing Live Traffic Using WiresharkWireshark is a well known network protocol analyzer and one of many applications used for network protocol analysis. Wireshark can interactively browse packet data from a live network or from a previously save pcap file.

To use Wireshark to decode a previously saved packet flow capture file, export the file from the switch and analyze it with Wireshark.

Informational Note:You can download Wireshark from http://www.wireshark.org

Informational Note:The path to a Pluribus Networks switch pcap file has the format: /net/<ServerSw_Name>/nvOS/global/flow/<Flow_Name>/<Switch_Name>/pcap

http://www.wireshark.org



207

To use Wireshark to interactively analyze packets in real time, you need to capture a packet traffic flow, either on a specific switch or across the entire fabric using the scope option. Include the log-packets option to send packets to the associated pcap files, for example

CLI network-admin@switch > vflow-snoop scope fabric src-ip 112.168.3.105 action copy-to-cpu log-packets

Next, create a fifo on the host running Wireshark.

mkfifo /tmp/pcap

Start Wireshark, and select Options from the Capture menu.

Enter the fifo path that you created in the Interface field: /tmp/pcap

Use tail to copy the pcap file to the FIFO:

tail +0f \/net/ServerSw_Name/nvOS/global/flow/Flow_Name/switch/Switch_Name/

pcap/tmp/pcap

You need to substitute ServerSw_Name, Flow_Name and Switch_Name to match your environment. Live capture continues until the packet capture file is rotated. By default, the maximum packet capture file size is 10MB but it is configurable with the packet-log-max option of the vflow-create and vflow-modify commands.



TIP! The mkfifo command used in this task is a standard feature of UNIX-like operating systems, including MacOS. For Windows platforms, you may need to install the GNU CoreUtils package available at http://gnuwin32.sourceforge.net/packages/coreutils.htm.



209

Using vFlows to Disable Communication

vFlows can be used to specify communications that are not allowed with a switch or a fabric. Use the following steps to create a vFlow as a firewall:

1. Define a VLAN and destination IP-based flow and specify that the flow is dropped by the switch, with statistics monitoring enabled:

CLI network-admin@switch > vflow-create name flow3 scope local vlan 99 dst-ip 172.168.24.1 action drop stats enable

Display the statistics for the new flow above as the traffic is dropped:

CLI network-admin@switch > vflow-stats-show name flow3 show-diff-interval 5

switch name packets bytes cpu-packets cpu-bytesaquila02 flow3 864 116K 0 0switch name packets bytes cpu-packets cpu-bytesaquila02 flow3 5 936K 0 0

There are many options available for creating vFlows, and vFlows can be used to shape traffic, capture statistics, capture flow metadata, capture packets, or manage communications. The options include:

vlan

vnet

in-port

out-port

ether-type

src-mac

src-mac-mask

dst-mac

dst-mac-mask

src-ip

src-ip-mask

dst-ip

dst-ip-mask

src-port

dst-port

dscp

tos

proto

flow-class

uplink-ports

bw-min

bw-max


precedence

action

action-value

no-mirror

mirror

no-process-mirror

process-mirror

no-log-packets

log-packets

packet-log-max

stats

stats-interval

duration

no-transient

transient

vxlan

vxlan-ether-type

vxlan-proto


211

Use Case ScenarioIn a real use case, the command connection-show server-ip 10.9.10.117 was used to analyze a suspicious connections to server 10.9.10.117:

Switch vlan client-ip server-ip service dur(s) latency(us) out-bytes in-bytes active

------ ---- --------- --------- ------- ------ ----------- --------- -------- ------

switch: switch02vlan: 1client-ip: 10.9.9.33server-ip: 10.9.9.107service: httpdur(s): 0latency(us): 65out-bytes: 0in-bytes: 0active: yesswitch: switch02vlan: 1client-ip: 10.9.9.33server-ip: 10.9.9.107service: httpdur(s): 210latency(us): 7out-bytes: 48804in-bytes: 6120active: yesswitch: switch02vlan: 1client-ip: 10.9.9.33server-ip: 10.9.9.107service: httpdur(s): 328latency(us): 30out-bytes: 48720in-bytes: 612620active: yes

Configuring Mirroring for vFlows and PortsA Pluribus Networks fabric administrator can run services and applications within the switch. Consider the use case of an application that needs access to data that is flowing through the switch, but does not want to impede that flow. The port-mirroring feature provides this functionality.

The system predefines a mirror configuration, but does not insert any traffic into that mirror. Use the following steps to setup mirroring to send from all of the data ports to the span port (port 66). In this version of nvOS, the port-mirror command is deprecated and replaced with the command mirror-modify to allow support for vFlow-based and port-based mirroring. The command syntax for mirror-modify is as follows:

CLI network-admin@switch > mirror-modify out-port port-list in-port port-list [policy port|vflow] mirroring|no-mirroring

CLI network-admin@switch > mirror-show [format fields-to-display] [parsable-delim character] [sort-asc] [sort-desc] [show dups] [layout vertical|horizontal] [show-interval seconds-interval]


View the status of mirroring by entering the following at the CLI command prompt:

CLI network-admin@switch > mirror-show

switch: aquila19direction: bidirectionout-port:in-port:mirroring: disable

The parameter out-port is not configured and mirroring is disabled therefore, no data mirroring can occur.

To modify the mirroring configuration, use the following steps:

1. Use the mirror-modify command to set the output to the span port. However, if there is more than 10Gb of traffic on ports 1-64, do not execute this command.

CLI network-admin@switch > mirror-modify in-port 1-64 out-put 66 mirroring

mirror-showswitch: pleiades24direction: bidirectionout-put: 66in-port: 1-64mirroring: enable

To disable the configuration, use the following command:

CLI network-admin@switch > mirror-modify no-mirroring

mirror-showswitch: aquila19direction: bidirectionout-port: 66in-port: 1-64mirroring: disable




213

Managing Traffic Classes

nvOS provides a full set of traffic class features, including the ability to view and create traffic classes, as well as assign traffic classes to flows to manage the quality of service of the flow traffic and shape the traffic passing through an nvOS fabric.

To display the currently defined traffic classes:

CLI network-admin@switch > vflow-class-show

name scope type priority ------------- ------ ------ -------- meter fabric system 0 guaranteed_bw fabric system 9 lossless fabric system 10 control fabric system 11

The higher the priority number, the higher the priority of the class. To add a vflow class, use the vflow-class-create command:

CLI network-admin@switch > vflow-class-create name traffic-1 scope fabric priority 5

This creates a traffic class with a scope of fabric and medium priority.

To add a traffic class to a vFlow, create a vFlow and assign a traffic class. In this case the flow is for a single IP address:

CLI network-admin@switch > vflow-create name losslessflow scope local src-ip 10.11.1.10 src-ip-mask 255.255.255.255 action none flow-class lossless

CLI network-admin@switch > vflow-show name losslessflow layout vertical

switch: aquila12name: losslessflowscope: localtype: vflowvlan: 0vnet:in-port:out-port:ether-type: 0src-ip: 10.11.1.10dst-ip:::src-port: 0dst-port: 0proto: ipflow-class: losslessbw-max: 0pri: 0action: noneaction-value: 0transient: no

Traffic from IP address 10.11.1.10 now has a very high priority throughout the switch. For a similar high priority throughout the fabric use scope fabric rather than scope local.


When a TCP session goes through the NPU, and capacity is exceeded, the return traffic with TCP ACK packets can get dropped from the session. To avoid this, create a flow that matches the TCP ACK packets and set a higher precedence for it.




215

Using Application Flows and Statistics

Displaying Standard StatisticsYou can display standard statistics that consist of flow-based information collected and tracked continuously by the switch.

To modify statistics logging, use the stats-log-modify command and disable or enable statistical logging as well as change the interval, in seconds, between statistical events.

To display statistical logging information, use the stats-log-show command:

CLI network-admin@switch > stats-log-show

switch: pleiades24enable: yesinterval: 60

To show connection-level statistics, traffic flows between a pair of hosts for an application service, including current connections and all connections since the creation of the fabric, enter the following CLI command at the prompt:

CLI network-admin@switch > connection-stats-show

switch: pleiades24mac: 00:e0:81:e4:02:12vlan: 200ip: 100.200.1.3port: 53iconns: 80oconns: 0ibytes: 0obytes: 0total-bytes: 0last-seen-ago: 4d19h32m23sswitch: pleiades24mac: 00:12:c0:80:1e:85vlan: 200ip: 100.200.1.4port: 16iconns: 0oconns: 70684ibytes: 578Mobytes: 890Mtotal-bytes: 1.43Glast-seen-ago: 46s

From the information displayed in the output, you can see statistics for each switch, VLANs, client and server IP addresses, as well as the services on each connection. Latency and other information is also displayed.

The latency(us) column displays the running latency measurement for the TCP connection in microseconds. It indicates end-to-end latency and includes the protocol stack processing for the connected hosts and all intermediary network hops.

This is not the same latency measurement experience by a packet transiting the switch port-to-port. The port-to-port latency is platform-dependent and you should refer to the datasheet for your switch model.


To display specific types of connections, use the additional parameters with the command. For instance to display active connections,

CLI network-admin@switch > connection-stats-show active

switch vlan vxlan vnet client-ip server-ip service active ageswitch12 1 0 10.9.10.152 96.17.77.96 http yes 35m27sswitch12 5 0 10.12.1.47 10.9.10.204 445 yes 7m56sswitch12 1 0 10.9.9.21 23.62.97.88 http yes 3m41sswitch12 1 0 10.9.9.21 23.60.129.224http yes 3m44sswitch12 1 0 10.9.10.72 10.9.99.23 http yes 7s. . .

To display a summary of traffic statistics for each application service, use the service-stats-show command.

CLI network-admin@switch > service-stats-show

switch service bytespleiades24 53495 584pleiades24 8084 845Mpleiades24 59475 33.9Kpleiades24 imap 1.83Mpleiades24 35356 106pleiades24 54341 584

From the information displayed in the output, you can review each switch, service, and the number of bytes used by each service.

To display storage traffic statistics, use the storage-stats-show command:

CLI network-admin@switch > storage-stats-show

switch server-ip port read-bytes write-bytes------ --------- ---- ---------- -----------switch12 10.9.9.9 65 3.63T 302Kswitch12 10.9.10.113 nfs 0 0switch12 10.9.9.33 nfs 284G 6.15Kswitch12 10.9.11.18 65 137G 6.02Kswitch12 10.9.10.69 nfs 46.0G 402K. . .

From the information displayed in the output, you can review the storage data for each server, the port, and the number of read-write bytes.


217

To display interface statistics, use the interface-stats-show command:

CLI network-admin@switch > interface-stats-show

switch: pleiades24time: 09:20:27nic: dataibytes: 100Mipkts: 302Kierrs: 0obytes: 126Mopkts: 453Koerrs: 0switch: pleiades24time: 09:20:27nic: spanibytes: 11.7Mipkts: 396Kierrs: 0obytes: 0opkts: 0oerrs: 0switch: pleiades24time: 09:20:27nic: ops.mgmt.mgr.eth1ibytes: 64.2Mipkts: 774Kierrs: 0obytes: 46.2Kopkts: 1.10Koerrs: 0switch: pleiades24time: 09:20:27nic: ext.50.mgr.eth0ibytes: 2.41Mipkts: 34.2Kierrs: 0obytes: 679Kopkts: 11.9Koerrs: 0

From the information displayed in the output, you can review the inbound and outbound traffic for each NIC on the switch. You can also check for errors in the inbound and outbound traffic.


Understanding vFlow StatisticsVirtual network-based flows, vflows, display statistics for packet traffic flows on a switch and across the fabric. vFlows are very powerful and provide many features such as quality of service (QoS), traffic shaping, packet redirect, drop actions, mirror, and capture.



A vFLow can be configured to store log statistics to a file accessible to clients using NFS and SFTP. If statistics logging is enabled, ONVL periodically polls the switch for the most recent statistics for each flow and saves the statistics to an exported file. ONVL also saves individual statistics received from other switches in the fabric and combines the statistics from all switches to record aggregate statistics for the entire fabric.

The switch consists of two components, the switch and the server. vFlows with operations like drop are executed within the switch component. Some vFlows operations for QoS take place in the switch component, while others operate within the coprocessor by directing pertinent traffic to the coprocessor. There, the traffic is managed and then sent back to the switch component.

Other actions such as copy-to-cpu sends the match traffic to the server component where the traffic is managed and then forwards packets for delivery. In general, the details are managed by nvOS including fabric scope commands that cause all switches within a fabric to participate in an operation and then sends the compiled results to the CLI or to log files.

Before you can access the files, you must enable NFS or SFTP access to the log files by using the admin-service-modify command.

.

CLI network-admin@switch > vflow-share-show

switch vnet enable share-path pleiades24 fab1-global no pleiades24:/nvOS/vnet/fab1-globalpleiades24 ops-mgmt no pleiades24:/nvOS/vnet/ops-mgmt pleiades24 ext-50 no pleiades24:/nvOS/vnet/ext-50 pleiades24 www-51 no pleiades24:/nvOS/vnet/www-51 pleiades24 folsom no pleiades24:/nvOS/vnet/folsom

CLI network-admin@switch > vflow-share-modify vnet fab1-global enable

vflow-share-showswitch vnet enable share-path pleiades24 fab1-global yes pleiades24:/nvOS/vnet/fab1-global pleiades24 ops-mgmt no pleiades24:/nvOS/vnet/ops-mgmt pleiades24 ext-50 no pleiades24:/nvOS/vnet/ext-50 pleiades24 www-51 no pleiades24:/nvOS/vnet/www-51 pleiades24 folsom no pleiades24:/nvOS/vnet/folsom

You can then access the statistics log files using NFS in the following locations:

For the switch scope, the files are located in

/net/switch-name/nvos/vnet/vnet-name/flow/flow-name/switch/switch-name/stats

For the fabric scope, the files are located in

/net/switch-name/nvos/vnet/vnet-name/flow/flow-name/fabric/stats

To create a vFLow for example, Host-Agent-Discover, and measure statistics, enter the following command:

CLI network-admin@switch > vflow-create name Host-Agent-Discover scope local system


219

To view all vFlows currently tracked by the switch or fabric, use the vflow-show command:

CLI network-admin@switch > vflow-show

switch: pleiades24name: Host-Agent-Discoverscope: localtype: systemdst-ip: 224.4.9.6precedence: 2action: copy-to-cpuswitch: pleiades24name: DHCP-clientscope: localtype: systemin-port: 1-68src-port: 68proto: udpprecedence: 2action: copy-to-cpuswitch: pleiades24name: Host-Agent-Discoverscope: localtype: systemdst-ip: 224.4.9.6precedence: 2action: copy-to-cpuswitch: pleiades24name: DHCP-clientscope: localtype: systemin-port: 1-68src-port: 68proto: udpprecedence: 2action: copy-to-cpu

From the information displayed in the output, you can review the switch, the name of the vFlow, scope, type of vFlow, destination IP address, precedence, and action for the vFlow.

To display statistics for all vFlows, use the vflow-stats-show command:

CLI network-admin@switch > vflow-stats-show

switch name packets bytes cpu-packets cpu-bytes------ ---- ------- ----- ----------- ---------pleiades24IGMP-Flow 368K 23.0M 392K 23.0Mpleiades24 LLDP-Flow 82.9K 26.3M 82.9K 26.0Mpleiades24 Host-Agent 17.8K 1.11M 0 0pleiades24 ECP 0 0 0 0

To monitor statistics of a vFlow and update every 10 seconds, use the following syntax:

CLI network-admin@switch > vflow-stats-show name flow1 show-diff-interval 10


To log persistent records of flow statistics, use the logging parameter and collect statistics every 10 seconds:

CLI network-admin@switch > vflow-create name monitor-flow scope local ether-type arp stats log stats-interval 5

You can display the statistics logs for the new flow using the vflow-stats-show command.


Creating vFlows with the Scope FabricTo create vFlows across the entire fabric, configure the vFlow with the scope fabric and stats enable option. Using these parameters enables statistics for the flow on all switches that are members of the fabric and you can display the statistics for any switch in the fabric.

To create a vFlow for VLAN1 with the scope fabric, use the following syntax:

CLI network-admin@switch > vflow-create name fab_flow1 scope fabric stats enable vlan 1

To display the statistics for the new vFlow for any switch in the fabric, use the following syntax:

CLI network-admin@switch > switch switch-name vflow-stats-show name fab_flow1

name packets bytes cpu-packets cpu-bytes---- ------- ----- ----------- ---------fab_flow1 51.4K 13.8M 50.1K 13.1M

If you omit the switch name, all vFlow statistics for the fabric are displayed.

switch name packets bytes cpu-packets cpu-bytes------ ---- ------- ----- ----------- ---------pleiades1 fab_flow1 1.32K 305K 1.29K 291Kpleiades2 fab_flow1 910 256K 884 243K


Informational Note: Conflicting vFlows

Multiple vFlows can be active at once, but nvOS cannot apply them at the same time. You can use the precedence parameter is used to set the order of the vFlows. If you set the precedence to a higher value (0 - 10 with 0 as the lowest precedence), the vFlow has a higher precedence than those with lower values. If you’re seeing error messages about vFlow conflicts, try adding a precedence value to new or existing vFlows.




221

Example Use Cases for vFlows

The following examples illustrate how to use vFlows to impact traffic on the switch. You can regulate bandwidth, create multiple vFlows, or share bandwidth.

Regulating Bandwidth for a VNETTo regulate bandwidth for all hosts in a VNET, create a vFlow and associate it with the appropriate flow class:

1. Create a VNET, bwvnet, using the vnet-create command:

CLI network-admin@switch > vnet-create name bwvnet scope fabric

2. All traffic associated with this VNET has a bandwidth of 5 Gbps. Create a vFlow:

CLI network-admin@switch > vflow-create name bwflow scope fabric vnet bwvnet flow-class guaranteed-bw bw-min 5g

vflow-create:In order to use bw-min, please use vrg-modify to specify a min bandwidth for vrg bwvnet-vrg

Creating the vFlow failed because a flow can only use the minimum bandwidth parameter if the associated VRG (Virtual Resource Group) has minimum bandwidth allocated to it. You need to modify the VRG associated with the VNET before assigning a minimum bandwidth to the vFlow.

3. Modify the VRG:

CLI network-admin@switch > vrg-modify name bwvnet-vrg data-bw-min 5g

4. Now create the vFlow for regulating bandwidth:

CLI network-admin@switch > vflow-create name bwflow scope fabric vnet bwvnet flow-class guarantee-bw bw-min 5g

You can also regulate bandwidth to a certain speed using vFlows.

5. Modify the VRG associated with the VNET:

CLI network-admin@switch > vrg-modify name bwvnet-vrg data-bw-max 5g

6. And then create the vFlow:

CLI network-admin@switch > vflow-create name bw-reg scope fabric vnet bwvnet flow-class meter bw-max 5g

This creates a vFlow that allows bandwidth of up to 5 Gbps for all traffic on the VNET, bwvnet.

Informational Note: Before you assign minimum bandwidth to a vFlow, the associated VRG must have the same bandwidth value or higher allocated to it.


Suppose you want to offer guaranteed bandwidth on a VNET, and cap the bandwidth to a fixed value. Add another vFlow to perform this service:

CLI network-admin@switch > vflow-create name gw-bw scope fabtic vnet bwvnet flow-class guaranteed-bw bw-min 5g bw-max 8g


Creating Multiple vFlows for the Same VNETYou can create multiple vFlows for the same VNET and add precedence values to the vFlows. The packet is matched to the vFlow with the highest precedence. For example,

1. Create the first vFlow:

CLI network-admin@switch > vflow-create name client-flow1 scope fabric vnet bwvnet flow-class meter bw-max 2g

2. Create the second vFlow:

CLI network-admin@switch > vflow-create name client-flow2 scope fabric vnet bwvnet flow-class meter bw-max 5g src-ip 192.168.20.1

vflow-create: Flow conflicts with Flow client-flow1, ID68: specify fields to make flows mutually exclusive or change the flow precedence

The error message is generated because the vFlow configurations conflict with each other. To differentiate between the two flows, assign a different precedence to client-flow2:

CLI network-admin@switch > vflow-create name client-flow2 scope fabric vnet bwvnet flow-class meter bw-max 5g src-ip 192.168.20.1 precedence 5

Configuring Bandwidth Sharing for a Single VLAN with Different IP Addresses or SubnetsIn some instances, you want to allow different subnets to share a guaranteed bandwidth on the same VNET. To do this, you must create a VRG with the required bandwidth:

CLI network-admin@switch > vrg-create name admin-vrg vlans 100 data-bw-min 1g data-bw-max 2g scope fabric

Informational Note: You cannot create a new vFlow if a packet matches an existing flow.



223

You have now created a VRG with the guaranteed bandwidth of 1 Gbps and limited to a maximum of 2 Gbps. Now, create a vFLow for each IP address:

CLI network-admin@switch > vflow-create name vfl-1 scope fabric vlan 100 src-ip 1.1.1.1




In this example, the specified IP addresses each have a guaranteed bandwidth between 1 Gbps and 2 Gbps.

If you want to specify a subnet, 100.100.100.0/28, and VLAN 53 with maximum bandwidth of 50 Mbps, use the following syntax:

CLI network-admin@switch > vrg-create name vrg-custom scope fabric data-bw-min 50M data-bw-max 50M vlan 53

CLI network-admin@switch > vflow-create name vfl-cust scope fabric src-ip 100.100.100.0 src-ip-mask 255.255.255.240 vlan 53

But later on, you found that sixteen IP addresses were not enough and you needed an additional 8 with the subnet, 101.101.101.8/29 that require the same bandwidth as the previous subnet. Use the following syntax:

CLI network-admin@switch > vflow-create name vfl-cust-2 scope fabric src-ip 101.101.101.8 src-ip-mask 255.255.255.248 vlan 53

You now have two vFlows on VLAN 53.

Then, you discover that 50 Mbps is not sufficient to support the network traffic affected by the vFlow, and you want to upgrade to 80 Mbps:

CLI network-admin@switch > vrg-modify name vrg-custom data-bw-min 80M data-bw-max 80M





225

Configuring VXLANs and Tunnels

Configuring a VXLAN with nvOS

Configuration Example

Creating Tunnels

In today’s virtualized environments, there is increasing demand on MAC address tables of switches that connect to servers. Instead of learning one MAC address per server link, the switch now has to learn the MAC addresses of individual VMs, and if the MAC address table overflows, the switch may stop learning new MAC addresses until idle entries age out.

Virtual Extensible LAN (VXLAN) is essentially a Layer 2 overlay scheme over a Layer 3 network, and each overlay is called a VXLAN segment. Only VMs within the same VXLAN segment can communicate with each other. Each VXLAN segment is identified by a 24 bit segment ID called the VXLAN Network Identifier (VNI).

VXLANs increase the scalability of your network up to 16 million logical networks and is used to contain broadcast, multicast, and unknown unicast traffic.

Because of this encapsulation, VXLAN could also be called a tunneling scheme to overlay Layer 2 networks over top of Layer 3 networks. However, the tunnel does not terminate on the switch, and the switch sits in the middle of the tunnel and sees packets as L3 tunneled packets. These packets are then forwarded using L2 or L3 forwarding.

Pluribus Networks supports two scenarios for VXLAN:

1. The tunnel does not terminate on the switch and VTEP is not supported. Though the switch does not participate in the creation of a tunnel, the following tasks are still performed.

a. Analytics Collection — All TCP control packets are captured as well as ARP packets traversing the tunnel. These packets are used to build connection statistics and provide visibility as to which VXLAN nodes are on specific ports.

b. ARP Optimization — An ARP request is captured and if an L2 entry exists in the switch L2 table, a response is sent back to the sender of the ARP request over the tunnel. Otherwise, the ARP request is re-injected into the tunnel without any modification to continue crossing the tunnel.

2. The tunnels are terminated at a switch and the switch performs the role of a VTEP. In this scenario, the switch is responsible for encapsulating packets that arrive from non-VXLAN nodes on a L2 network and transmitting them over the tunnel. Similarly, the packets arriving through the tunnel are decapsulated and the inner packet is for-warded over the L2 network. The switch also collects statistics and optimizes ARP requests as in the first scenario.

Configuring a VXLAN with nvOSFor the first scenario, no additional configuration is required. The second scenario requires the following steps, in order:

1. Create a hardware vRouter. 2. Add interfaces to the vRouter, one per tunnel. The tunnel endpoint IP address should be routable.

3. Create one or more tunnels.

4. Create the VXLAN with the VNI, and add the tunnels created in the previous steps.

Informational Note: There is a one to one mapping of VXLAN to VLAN. Multicast traffic is not supported. VXLAN has the scope local on all switches, and must be in the same subnet.


To create a VXLAN, vx-seg1, with the VNID 25, scope fabric, and turn off deep inspection, use the following syntax:

CLI network-admin@switch > vxlan-create name vx-seg1 vnid 25 scope fabric deep-inspection no

To delete a VXLAN, use the vxlan-delete command.

To display information about VXLANs, use the vxlan-show command.

If you added a port to the VXLAN configuration, use the vxlan-port-remove command.


Configuration ExampleThe following example assumes that one VTEP is on the generic switch and the other VTEP is on a Pluribus Networks switch. Also, the nodes are connected on a L3 IP network, and the tunnel is formed between the generic switch and the Pluribus Networks switch.

The example also includes VLAN 10 and port 47 on Host2 as well as the VNET fab-global.

1. Create the vRouter using the vrouter-create command:

CLI (server-switch)> vrouter-create name vx-vrouter vnet fab-global router-type hardware

2. Add the vRouter interface:

CLI (server-switch)>vrouter-interface-add vrouter-name vx-vrouter ip 192.168.0.1 netmask 255.255.255.0 vlan 10

3. Create the tunnel:

CLI (server-switch)>tunnel-create name vx-tunnel scope local local-ip 192.168.0.1 remote-ip 192.168.5.1 next-hop 192.168.0.2 next-hop-mac 00:01:02:03:04:05 router-if vx-router.eth0

4. Create the VXLAN:

CLI (server-switch)>vxlan-create vnid 14593470 scope local name vxlan1 vlan 10

If VLAN 10 does not exist, then the vxlan-create command creates it on the switch, but you may need to add local ports to the VLAN.

5. Add port 47 to the VXLAN:

CLI (server-switch)>vxlan-port-add vxlan-name vxlan1 ports 47

This associates all packets from port 47 on VLAN 10 with the VXLAN ID, 14593470.



227

6. Add the tunnel to the VXLAN:

CLI (server-switch)>vxlan-tunnel-add vxlan-name vxlan1 tunnel-name vx-tunnel

To display the configuration, use the vxlan-show command.

You cannot configure different VLANs for the tunnel and the local hosts, and you cannot associate different VLANs on different ports for the same VXLAN.


Creating TunnelsYou can create tunnels to encapsulate protocols on the network. You can create tunnels for IP-in-IP, VXLAN, and NVGRE network traffic. However, tunnels are supported on the local scope only and do not use any discovery mechanism.

IP-in-IP protocol encapsulates an IP header with an outer IP header for tunneling. The outer IP header source and destination identifies the endpoints of a tunnel. The inner IP header source and destination identify the original sender and recipient of the datagram.

In addition to the IP header and the VXLAN header, the VTEP also inserts a UDP header. During ECMP, the switch includes this UDP header to perform the hash function. The VTEP calculates the source port by performing the hash of the inner Ethernet frame's header. The Destination UDP port is the VXLAN port.

The outer IP header contains the Source IP address of the VTEP performing the encapsulation. The destination IP address is the remote VTEP IP address or the IP Multicast group address.

Network Virtualization using Generic Routing Encapsulation (NVGRE) uses GRE to tunnel Layer 2 packets over Layer 3 networks. NVGRE is similar to VXLAN but it doesn’t rely on IP multicast for address learning.

To create a tunnel for IP-in-IP traffic, local IP address 192.168.100.35, and the router, tunnel-network, use the following syntax:

CLI network-admin@switch > tunnel-create scope local name ipinip type ip-in-ip local-ip 192.168.100.35 router-if vrouter-hw-if eth0.0

To remove a tunnel, use the tunnel-delete command.

To modify a tunnel, use the tunnel-modify command.






229

Edge Virtual Bridging

Understanding Edge Virtual BridgingEdge Virtual Bridging (EVB) is a software capability on a switch running Pluribus Networks nvOS® that allows multiple VMs to communicate with each other and with external hosts in the Ethernet network.

Virtual Ethernet Port Aggregator (VEPA) is a software capability on a server that collaborates with an adjacent, external switch to provide bridging support between multiple VMs and external networks. The VEPA collaborates with the adjacent switch by forwarding all VM-originated frames to the adjacent switch for frame processing and frame relay, including hairpin forwarding, and by steering and replicating frames received from the VEPA uplink to the appropriate destinations.

Why Use VEPA instead of Virtual Ethernet Bridging (VEB)?

Even though VMs are capable of sending packets directly to one another with a technology called Virtual Ethernet Bridging (VEB), physical switches are used for L2/L3 forwarding because VEB uses server hardware to accomplish the task. Instead of using VEB, you can install VEPA on a server to offload switching functions to an adjacent physical switch that offers less expensive L2/L3 forwarding.

Additional advantages of using VEPA include the following:

VEPA reduces complexity and allows higher performance on the server

VEPA takes advantage of the physical switch security and tracking features.

VEPA provides visibility of inter-VM traffic to management tools designed for network switches.

VEPA reduces the amount of network configuration required by server administrators, and as a consequence, reduces workload for a network administrator.

How Does EVB Work?

EVB uses two protocols to work: Virtual Station Interface (VSI) Discovery and Configuration Protocol (VDP) and Edge Control Protocol (ECP), to program policies for each individual virtual switch instance.

EVB maintains the following information for each VSI instance:

VLAN ID

VSI type

VSI type version

MAC address of the server

VDP is used by the VEPA server to propagate VSI information to the switch. This allows the switch to program policies on individual VSIs and supports VM migration by implementing logic to pre-associate a VSI with a particular interface.

ECP is an LLDP (Link Layer Discovery Protocol)-like transport layer that allows multiple upper layer protocols to send and receive protocol data units (PDUs). ECP improves upon LLDP by implementing sequencing, retransmission and an ACK mechanism. ECP is implemented in an EVB configuration when you configure LLDP on ports that you have configured for EVB. In other words, you configure LLDP, not ECP.

You can configure EVB on a switch when that switch is adjacent to a server that includes VEPA technology. In general, this is how to implement EVB:

A network administrator creates a set of VSI types. Each VSI type is represented by a VSI type ID and a VSI version. You can deploy one or several VSI versions at any time.


The VM administrator configures VSI which is a virtual station interface for a VM represented by a MAC address and VLAN ID pair. The VM administrator queries available VSI type IDs (VTIDs) and creates a VSI instance consisting of a VSI Instance ID and the chosen VTID. This instance is known as VTDB and contains a VSI manager ID, a VSI type ID, a VSI version, and a VSI instance ID.


Configuring Edge Virtual Bridging

Remember, EVB does not convert packets, but it ensures that packets from one VM destined to another VM on the same server are switched. When the source and destination of a packet are on the same port, EVB delivers the packet, reflective relay, which otherwise would not happen because standard switching never forwards a packet to the port from which it received the packet.

Before You BeginBe sure that you have performed the following:

Configured packet aggregation on the server connected to the port on the switch used for EVB.

Configured the EVB port for all VLANs located on the VMs.

1. To enable VDP processing on all ports, enter the following CLI command at the prompt:

CLI network-admin@switch > vdp-modify enable

You can verify if VDP is enabled on a switch by using the vdp-show command.

2. To display the VSI instances and their state, use the vsi-state-show command:

CLI network-admin@switch > vsi-state-show

port mgrid vsiid_format vsiid linkspeed bw_limit traffic_class state keepalive

49 :: mac 02:08:20:a8:13:67 10Gbps 10% 0 ASSOC

10949 :: mac 02:08:20:b0:25:39

10Gbps 20% 0 ASSOC 109

3. To display ECP protocol statistics, use the following command:

CLI network-admin@switch > ecp-port-show

port ipkts opkts timeouts retransmits tx_errors last_rx_seqno last_ack_seqno49 987 987 27 27 0 481 481





231

Implementing OpenFlow with FloodLight

Floodlight Open Software Defined Network (SDN) Controller is an enterprise-class, Apache-licensed, Java-based OpenFlow controller. It works with both physical and virtual switches that can interpret the OpenFlow protocol. Since it is Apache licensed, you can use Floodlight for almost any purpose.

In this example, you create a NetZone to enable Floodlight, and use the VNET, vnet-engr, with the username admin-opf, and the IP address 10.13.0.203/24:

CLI network-admin@switch > netzone-create name floodlight1 vnet vnet-engr user admin-opf

netzone user password: passwordconfirm netzone user password: password

CLI network-admin@switch > netzone-interface-add netzone-name floodlight1 ip 10.13.0.203 netmask 24

CLI network-admin@switch > netzone-modify name floodlight1 floodlight-enable

By default, Floodlight OpenFlow Controller listens for OpenFlow protocol messages on port 6633 and exposes the REST API to applications on port 8080.

Now, you can configure the OpenFlow daemon for the VNET, vnet-engr:

CLI network-admin@switch > openflow-connection-add name floodlight1 vlan 10 controller-ip 10.13.0.203 failmode standalone(open) control-port 6633

To begin using the Floodlight OpenFlow Controller within the NetZone, you can SSH to the NetZone using the IP address that you configured in the previous example.

For additional documentation on using Floodlight, go to http://docs.projectfloodlight.org/display/floodlightcontroller/Floodlight+Documentation

Informational Note: For more information about Floodlight Controller, go to http://www.floodlight.org.

http://www.floodlight.org

http://www.floodlight.org

http://docs.projectfloodlight.org/display/floodlightcontroller/Floodlight+Documentation



233

Configuring OpenFlow

Enabling a Virtual Network for an OpenFlow Controller

Creating OpenFlow Controllers with Multiple VLANs

Configuring the OpenFlow Controller

Configuring Open Virtual Switch (OVS) for OpenFlow

OpenFlow is the first standard communications interface defined between the control and forwarding layers of an SDN architecture. OpenFlow allows direct access to the forwarding plane and allows you to manipulate the forwarding plane of network devices such as switches and routers, both physical and virtual. Because current networking devices lack an open interface, it has led to the characterization of the devices as monolithic, closed, and mainframe-like. There is no other standard protocol like OpenFlow and an OpenFlow is needed to move network control out of the networking switches to logically centralized control software.

The OpenFlow protocol is a key enabler for software-defined networks and is currently the only standardized SDN protocol that allows direct access and manipulation of the forwarding plane on network devices.

For more information about OpenFlow, go to http://www.opennetworking.org.

Enabling a Virtual Network for an OpenFlow ControllerYou can enable OpenFlow for a virtual network (VNET) with one or more VLANs and connecting the VLANs to an OpenFlow controller.

If the VNET assigned to OpenFlow has the scope, local, the switch ports configured for the VNET appears to the OpenFlow controller as a traditional, standalone OpenFlow switch with those ports.

If the VNET assigned to OpenFlow has the scope, fabric, the OpenFlow controller is presented with the abstraction of a single logical big switch containing the ports from each switch in the fabric configured for the VNET. The Pluribus Networks Netvisor (nvOS®) ensures that the state is distributed and rules are programmed into the individual physical switch tables as necessary to present the abstraction of a single big switch.

A switch or fabric can virtualize the physical network for one or more OpenFlow networks. Use the following steps to create a VNET:

1. Create a virtual network and assign it to a VLAN, for example, VLAN10.

CLI network-admin@switch > vnet-create name openflow-1 scope fabric vlans 10

vnet created.

You can apply the standard VNET parameters such as bandwidth guarantee by configuring a virtual resource group (VRG).

Informational Note: The switch supports OpenFlow version 1.0 protocol. For more information about the OpenFlow 1.0 protocol, go to http://www.opennetworking.org/index.php.

http://www.opennetworking.org

http://www.opennetworking.org/index.php.


2. Create an OpenFlow service for the VNET:

CLI network-admin@switch > openflow-create name openflow-1 vnet openflow-vnet

3. Create an OpenFlow daemon for the VNET, openflow-1 with the IP address of 192.168.1.11 on port 6633. Port 6633 is the well-known port for OpenFlow.

CLI network-admin@switch > openflow-connection-add name openflow-1 vlan 10 controller-ip 192.168.1.11 control-port 6633 failmode standalone(open | secure (timeout)

The failure mode dictates the policy to follow if OpenFlow controllers configured for the VNET are unresponsive.

In standalone(open) failure mode, the VNET performs as a legacy Layer 2 switch. When connected to a controller again, the existing flow entries remain. The controller can then delete all flow entries.

In secure(timeout) failure mode, packets and messages sent to the OpenFlow controllers are dropped from the network. Flows expire according to the configured timeouts.

The default failure mode is standalone(open) mode.

4. Repeat the previous step for each OpenFlow controller on the VNET. For example, you may want to configure a primary OpenFlow controller and a secondary OpenFlow controller as a backup option.

There may be certain times that you want to reset the connection from the VNET, openflow-1, to the OpenFlow controller. You can use the openflow-restart command to perform this action.

To remove an OpenFlow controller from a VNET, specify the IP address associated with the OpenFlow controller. For example,

CLI network-admin@switch > openflow-connection-remove name openflow-1 vlan 10 controller-ip 192.168.1.11

To remove all OpenFlow controllers from the VNET, omit the IP address from the command.

CLI network-admin@switch > openflow-connection-remove name openflow-1 vlan 10

To check the status of an OpenFlow connections, use the openflow-connection-show command.


235

Creating OpenFlow Controllers with Multiple VLANsIf a VNET contains multiple VLANS, then each VLAN is controlled by a separate OpenFlow controller. In this example, you have VLANs 0, 595, and 222, IP address 10.9.21.72/16, and you are creating a fabric named corp-fabric.

CLI network-admin@switch > fabric-create name corp-fabric

CLI network-admin@switch > vnet-create name vnet-engr scope fabric vlans 595,222

CLI network-admin@switch > vnet-manager-interface-add vnet-manager-name vnet-engr-mgr ip 10.9.21.72/16 vlan 0 if mgmt

CLI network-admin@switch > vnet-manager-interface-add vnet-manager-name vnet-engr assignment none vlan 595

CLI network-admin@switch > vnet-manager-interface-add vnet-manager-name vnet-engr assignment none vlan 222

CLI network-admin@switch > openflow-create name engr-openflow vnet vnet-engr

CLI network-admin@switch > openflow-connection-add name engr-openflow controller ip 10.9.21.17 failmode secure(timeout) vlan 595

CLI network-admin@switch > openflow-connection-add name engr-openflow controller ip 10.9.21.17 failmode secure(timeout) vlan 222

CLI network-admin@switch > vlan-port-add vlan-id 595 untagged ports 46,49

CLI network-admin@switch > vlan-port-add vlan-id 222 untagged ports 45,50

After executing these commands on the switch, the fabric is in the following state:

OpenFlow service, engr-openflow, is created on the VNET, vnet-engr.

OpenFlow connection, engr-openflow, is added to VLAN 595 and VLAN 222.

Ports 46 and 49 are added to VLAN 595.

Ports 45 and 50 are added to VLAN 222.


Configuring the OpenFlow Controller

nvOS has a built-in OpenFlow controller, Floodlight, that you can enable and then explore switch information using the OpenFlow protocol. nvOS provides commands that allows you to send and receive data from the OpenFlow controller.

For more information about the Floodlight controller, go to http://www.projectfloodlight.org/floodlight/

1. To enable the built-in OpenFlow controller, use the following commands:

CLI network-admin@switch > netvisor-zone-create name floodlight vnet openflow-1 user admin

netzone user password: <password>confirm netzone user password: <password>

CLI network-admin@switch > netvisor-zone-interface-add netvisor-zone floodlight ip 192.168.11.13 netmask 24

CLI network-admin@switch > netvisor-zone-modify name floodlight floodlight-enable

Use an IP address on your network that allows you to access the Floodlight OpenFlow controller.

2. Now add the OpenFlow daemon to the virtual network:

CLI network-admin@switch > openflow-connection-add name floodlight vlan 10 controller-ip 192.168.11.13 failmode standalone(open)control-port 6633

The failure mode dictates the policy that is followed if all OpenFlow controllers configured for the virtual network are unresponsive.

You can now begin using your built-in Floodlight OpenFlow controller with the Netvisor Zone that you just created. For documentation on the configuration and management steps for Floodlight, go to http://www.projectfloodlight.org/documentation/

Configuring Open Virtual Switch (OVS) for OpenFlow Open Virtual Switch (OVS) is a production quality, multilayer virtual switch licensed under the open source Apache 2.0 license. It is designed to enable massive network automation through programmatic extension, while still supporting standard management interfaces and protocols, for example, NetFlow, sFlow, IPFIX, RSPAN, CLI, LACP, and 802.1ag.

After you create OpenFlow version 1.3 on your switch, you can add OVS as your OpenFlow controller by creating a zone in the same manner as Floodlight.

CLI network-admin@switch > openvswitch-create name openflow13 vnet openflow dedicated-vnet-service storage-pool diskpool1 gateway 192.168.11.13 db-conn-type default db-ip 192.168.11.15 db-port 6633

And then start the OVS using the openvswitch-start command.

http://www.projectfloodlight.org/floodlight/

http://www.projectfloodlight.org/documentation/


237

About sFlow

OverviewBecause businesses rely on network services for mission critical applications, small changes in network usage can impact network performance and reliability. As a result, these changes can also impact a business’ ability to conduct key business functions and increase the cost of maintaining network services.

Figure 1: Overview of sFlow

sFlow provides the visibility into network usage and active routes on the network by providing the data required to effectively control and manage network usage. This ensures that network services provide a competitive edge to the business.

A few examples of sFlow applications include the following:

Detecting, diagnosing, and fixing network problems

Real-time congestion management

Understanding application mixes such as P2P, Web, DNS

Usage accounting for billing

Audit trail analysis to identify unauthorized network activity and trace sources of Denial of Service (DoS) attacks

Route profiling and optimizing peers

Trending and capacity planning

sFlow is an open source sampling tool providing constant traffic flow information on all enabled interfaces simultaneously. sFlow data is sent to a collector that formats the data into charts and graphs while recording and identifying trends on the network. You can use this information for troubleshooting a network, perform diagnostics, and analysis of data.


The sFlow agent on the switch samples packets from data flows and forwards headers of the sample packet to a collector at regular intervals. You can specify the number of packets to sample from the total packets which is called the sample rate. The packets are stored and sent to the collector at an interval that you can configure on the switch. This is called the polling interval. You can sample different types of packets such as frames sent to the CPU or interfaces of the switch, routed packets, flooded packets, and multicast packets. However, the following packet types are not sampled by sFlow:

LACP frames

LLDP frames

STP RPDUs

IGMP packets

Ethernet PAUSE frames

Frames with CRC errors

PIM_HELLO packets

Packets dropped by ACLs

Packets dropped as a result of VLAN violations

Routed packets with IP options or MTU violations

Counter SamplingFor counter sampling, also called polling, the sFlow agent periodically polls the hardware interface statistics registers, counters, in the switch chip for per port statistics, and stores them in RAM until it is time to send the next message to the sFlow collector. Overall port statistics such as the number of broadcasts, errors, are collected by the sFlow agent.

The agent then includes the statistics in the sFlow datagrams sent to the sFlow collector along with the packet sampling information. From these statistics, the sFlow obtains information about the actual utilization of each port. For instance, information about broadcast to multicast to unicast rations is captured.

When you configure the agent for counter sampling, it sends an sFlow datagram at intervals of a second, at most. The datagram contains a snapshot of the counters cached in RAM from the most recent polling of interface counters.

Packet SamplingPacket sampling is used to characterize network traffic. If the sFlow agent is configured for packet sampling, the agent takes copies of random samples of packets forwarded within the switch CPU and sends them to the switch for processing. The CPU sends a configured portion of the sampled packet, containing a number of protocol headers and possibly some of the payload data to the sFlow collector. Random sampling prevents the synchronization of periodic traffic patterns. On the average, 1 in every N packets is captured analyzed. The sampling can apply to ingress and egress frames independently. The rate that the agent sends datagrams depends on the sampling rate, the traffic rate, and the configured maximum datagram size. Typically, several samples are included in the datagram.

Agent to Collector DatagramsAfter gathering packet and counter samples, each sFlow agent creates a packet of the data and sends it to an sFlow collector in UCP datagrams. The datagrams contain the IP address of the sFlow collector and the standard UDP destination port number of 6343. Using a standardized port helps avoid configuration between sFlow agents and collectors. If the sFlow agent is configured for counter sampling or packet sampling, or both, an sFlow datagram can contain either interface counters, packet samples, or a mixture of both.


239

The following table provides information about the contents of sFlow datagrams:

Packet Header Information

Version The sFlow version used on the network.

IP Address Type An IPv4 or IPv6 address

Source IP Address The IP address of the sFlow agent

Sequence Number The sequence number of the datagram

System Uptime The length of time that the system is operational.

Sample Count The number of samples in the datagram

Ingress Interfaces The ifindex of the switch port where the packets entered the agent.

Egress Interfaces The ifindex of the switch port where the packets exited the agent.

Sample dataset sFlow-specific parameters:• Sequence Numbers• Sampling Rate• Total Packets available for sampling• Number of sampled packets dropped

because there was no processing resource for them.

Packet Samples Packet sample information and may contain several samples.

Packet data The sampled data that may include the packet payload data and the number on length of protocol headers. This information depends on the size of the size, up to 200 bytes.

Counter Sample Counter statistical information - fitted in where space permits.

If index The ifindex of the interface related to the counters.

Physical Interface Parameters • Speed• Duplex mode• Admin status• Operational status of the interface

In Counters • ifInOctets• ifInUnicastPkts• ifInMultiPkts• ifInBroadcastPkts• ifInDiscards• ifInErrors• ifInUnknownProbs



Out Counters • ifOutOctets• ifOutUcastPkts• ifOutDiscards• ifOutErrors

Promiscuous Mode The private VLAN promiscuous mode of the interface

Ethernet Statistics • Alignment Errors• FCS Errors• SQE Errors• Deferred Transmission• Internal MAC errors• Carrier sense errors• Overlength frame errors• Symbol errors

Packet Header Information



241

Configuring sFlow

From the following network diagram, let’s configure sFlow and sFlow agents.

Figure 1: sFlow Network with IP Addresses

Configuring the sFlow CollectorBefore configuring the sFlow agents, you must configure the sFlow collector. The sFlow collector receives sFlow datagrams from the sFlow agents. In this example, the sFlow collector has an IP address of 10.1.1.243, and a default port of 6343. The collector name is net-man-all, and the scope is fabric. If the scope is fabric, then additional switches that join the fabric receive the sFlow collector configuration. If the scope is local, then the sFlow collector is configured only on one switch.

CLI network-admin@switch > sflow-collector-create collector-ip 10.1.1.243 collector-port 6343 name net-man-all scope fabric

You can add as many collectors as needed for your configuration.


Enabling sFlow on the NetworkYou must configure and enable sFlow on each switch that you want to use for monitoring network traffic. You can only configure one sFlow per switch.



On each switch in the example diagram, use the following command to enable sFlow, net-monitor, on ingress ports 57-59, sample type raw, sample-rate 4096, sample interval 5 seconds, trunc-length 160 bytes, on VLAN 200:

CLI network-admin@switch > sflow-create name net-monitor sample-type raw ports 57-59 sample-rate 4096 trunc-length 160 vlan 200

Adding Additional Ports to sFlowTo add the ports, 61-62, to the sFlow configuration, you must use the following command on each switch:

CLI network-admin@switch > sflow-port-add sflow-name net-monitor switch 10.1.1.23 ports 61-62

In this example, the IP address of the switch is used as the name of the switch.

Removing Ports from the sFlow ConfigurationYou can remove ports from the sFlow configuration by using the sflow-port-remove command:

CLI network-admin@switch > sflow-port-remove sflow-name net-monitor switch 10.1.1.23 ports 61-62



Documents

Configuring Virtual Load Balancing with Ubuntu 11.04 ... · Configuring a Linux Netvisor KVM 139 ... Creating and Implementing Access Control Lists (ACLs) 195 Using a Deny IP ACL