STO1193BU A Closer Look at vSAN Networking Design and or … · 2017-10-12 · 3 NIC Teaming and Load Balancing 4 Network Topologies (incl. Stretched and 2-node) 5 Network Performance

Cormac HoganAndreas Scherr

STO1193BU

#VMworld #STO1193BU

A Closer Look at vSAN Networking Design and Configuration Considerations

VMworld 2017 Content: Not fo

r publication or distri

bution

• This presentation may contain product features that are currently under development.

• This overview of new technology represents no commitment from VMware to deliver these features in any generally available product.

• Features are subject to change, and must not be included in contracts, purchase orders, or sales agreements of any kind.

• Technical feasibility and market demand will affect final delivery.

• Pricing and packaging for any new technologies or features discussed or presented have not been determined.

Disclaimer

2#STO1193BU CONFIDENTIAL



bution

Agenda

1 vSAN Networking Overview

2 Multicast and Unicast

3 NIC Teaming and Load Balancing

4 Network Topologies (incl. Stretched and 2-node)

5 Network Performance Considerations




bution

Where Should I Begin? StorageHub!

https://storagehub.vmware.com/#!/vmware-vsan/plan-and-design




bution

https://storagehub.vmware.com/#!/vmware-vsan/plan-and-design

vSAN Networking Overview

5



bution

vSAN Networking – Major Software Components

• CMMDS (Cluster Monitoring, Membership, and Directory Service)

• Inter cluster communications and metadata exchange

– Multicast with <= vSAN 6.5

– Unicast with >= vSAN 6.6

– Heartbeat sent from master to all hosts every second

• Traffic light in steady state

• RDT (Reliable Datagram Transport)

• Bulk of vSAN traffic

– Virtual Disk data distributed across cluster

– Replication /Resynch Traffic

#STO1193BU CONFIDENTIAL 6



bution

vSAN Networking – Ports and Firewalls

• ESXi Firewall considerations

– On enablement of vSAN on a given cluster, all required ports are enabled/disabled automatically; no admin action

• Ports

– CMMDS (UDP 12345, 23451, 12321)

– RDT (TCP 2233)

– VSANVP (TCP 8080)

– Witness Host (TCP port 2233 and UDP Port 12321)

– vSAN Encryption / KMS Server

• Communication between vCenter and KMS to obtain keys

• vSAN Encryption has special dynamic firewall rule opened on demand on ESXi hosts




bution

Network Connectivity – IPv6

• vSAN can operate in IPv6-only mode

– Available since vSAN 6.2

– All network communications are through IPv6 network

• vSAN supports mixed IPv4 & IPv6 during upgrade only

– Do not run mixed mode in production




bution

Minimum NIC Requirements for vSAN Networking

9

+10Gb

support1Gb

support Comments

Hybrid Cluster Y Y10Gb min. recommended, but 1Gb supported,

<1ms RTT

All-Flash Cluster Y NAll Flash requires 10Gb min. 1Gb not supported,

<1ms RTT

Stretched Cluster - Data to Data Y N10Gb required between data sites*,

<5ms RTT

Stretched Cluster - Witness to Data Y Y100Mbps connectivity required from data sites to witness.

<200ms RTT

2-node Data to Data Y Y10Gb min. required for All-Flash. 1Gb supported for

hybrid, but 10Gb recommended

2-node Witness to Data Y Y1.5Mbps bandwidth required.

<500ms RTT

#STO1193BU CONFIDENTIAL



bution

• vSphere Standard Switch

• No management dependence on vCenter

• Recovery is simple

• Prone to misconfiguration in larger setups

• vSphere Distributed Switch

• Consistency

Avoids configuration skew

• Teaming and Failover

LACP/LAG/ether-channel

• Network I/O Control

Manage/allocate network bandwidth for

different vSphere traffic types

Distributed or Standard Switches?


vSphere Distributed Switch is Free with vSAN



bution

Network I/O Control (NIOC) Configuration Sample

• Single 10-GbE physical adapters for simplicity

• NICs handles traffic for vSAN, vMotion, and virtual machines and management traffic

• If adapter becomes saturated, Network I/O Control controls bandwidth allocation

• Sample configuration:

11

Traffic Type Custom Shares Value Bandwidth

vSAN 100 5Gbps

vMotion 50 2.5Gbps

Virtual Machine 30 1.5Gbp

Management 20 1Gbps




bution

NIC Teaming and Failover Options

All Virtual Switches Support (vSS + vDS)

• Routed based on IP Hash / Virtual Port ID

Distributed Switch Only (vDS)

• Route based on Physical NIC Load (LBT)

Distributed Switch + Physical Switch Only

• Physical switches that support LACP/LAG/ether-channel provide additional load balancing algorithms


Keep it simple folks!

Multi chassis link aggregation capable switches



bution

vSAN Multicast & Unicast



bution

What Is Multicast?

• vSAN 6.5 (and earlier) used multicast traffic as a discovery

protocol to find all other nodes trying to join a vSAN cluster

• Multicast is a network communication technique utilized to

send information simultaneously (one-to-many or many-to-many)

to a group of destinations over an IP network

• Multicast needs to be enabled on the switch/routers of the

physical network

• Internet Group Management Protocol (IGMP) used within

an L2 domain for group membership (follow switch vendor

recommendations)

• Protocol Independent Multicast (PIM) used for routing

multicast traffic to a different L3 domain


Multicast added complexity to vSAN networking



bution

IGMP Considerations

• Consideration with multiple vSAN clusters

– Prevent individual clusters from receiving all multicast streams

– Option 1 – Separate VLANs for each vSAN cluster

– Option 2 - When multiple vSAN clusters reside on the same layer 2 network, VMware recommends changing the default multicast address

• See VMware KB 2075451




bution

https://kb.vmware.com/kb/2075451

Multicast Group Address on vSAN

• The vSAN Master Group Multicast Address created is 224.1.2.3 – CMMDS updates

• The vSAN Agent Group Multicast Address is 224.2.3.4 – heartbeats

• The vSAN traffic service will assign the default multicast address settings to each host node

16

# esxcli vsan network listInterface

VmkNic Name: vmk2

IP Protocol: IP

Interface UUID: 26ce8f58-7e8b-062e-ba57-a0369f56deac

Agent Group Multicast Address: 224.2.3.4

Agent Group IPv6 Multicast Address: ff19::2:3:4

Agent Group Multicast Port: 23451

Master Group Multicast Address: 224.1.2.3

Master Group IPv6 Multicast Address: ff19::1:2:3

Master Group Multicast Port: 12345

Host Unicast Channel Bound Port: 12321

Multicast TTL: 5




bution

vSAN 6.6 Introduces Unicast in Place of Multicast for vSAN Communication



bution

vSAN and Unicast

• vSAN 6.6 now communicates using unicast for CMMDS updates

• A unicast transmission/stream sends IP packets to a single recipient on a network

• vCenter becomes the new source of truth for vSANmembership

– List of nodes is pushed to the CMMDS layer

• The Networking Mode (unicast/multicast) is notconfigurable

18

vSAN 6.6 and above

Unicast




bution

vSAN and Unicast

The Cluster summary now shows if a vSAN cluster network mode is Unicast or Multicast:




bution

Member Coordination with Unicast on vSAN 6.6

• vCenter now becomes the source of truth for vSAN cluster membership with unicast

• The vSAN cluster continues to operate in multicast mode until all participating nodes are upgraded to vSAN 6.6

• All hosts maintain a configuration generation number in case vCenter has an outage.

– On recovery, vCenter checks the configuration generation number to see if the cluster configuration has changed in its absence.

20

vCenter




bution

New Unicast Considerations in vSAN 6.6



bution

Upgrade / Mixed Cluster Considerations with Unicast

22

vSAN Cluster

Software

Configuration

Disk Format

Version(s)CMMDS Mode Comments

6.6 Only Nodes* All Version 5 UnicastPermanently operates in unicast. Cannot switch to multicast.Adding pre-6.6 nodes will partition cluster.

6.6 Only Nodes*All Version 3 or

belowUnicast

6.6 nodes operate in unicast mode.Switches back to multicast if < vSAN 6.6 node added

Mixed 6.6 and vSAN

pre-6.6 Nodes

Mixed Version 5 with Version 3 or below

Unicast

6.6 nodes with v5 disks operate in unicast mode. Pre-6.6nodes with v3 disks will operate in multicast mode.

*** This will cause a cluster partition if mixed in a cluster! ***

Mixed 6.6 and vSAN

pre-6.6 Nodes

All Version 3 or Below

Multicast

Cluster operates in multicast mode. All vSAN nodes must beupgraded to 6.6 to switch to unicast mode.

*** Disk format upgrade to v5 makes unicast permanent ***




bution

Considerations with Unicast

• Considerations with vSAN 6.6 unicast and DHCP

– vCenter Server deployed on a vSAN 6.6 cluster

– vSAN 6.6 nodes obtained IP addresses via DHCP

– If IP addresses change, vCenter VM may become unavailable

• Can lead to cluster partition as vCenter cannot update membership

– This is not supported unless DHCP reservations are used

• Considerations with vSAN 6.6 unicast and IPv6

– IPv6 is supported with unicast communications in vSAN 6.6

– However IPv6 Link Local Addresses are not supported for unicast communications on vSAN 6.6

• vSAN doesn’t use link local addresses to track membership

23

vCenter




bution

Query Unicast with esxcli

• vSAN cluster node now displays the CMMDS networking mode - unicast or multicast

– esxcli vsan cluster get




bution

Query Unicast with esxcli

• One can also check which vSAN cluster nodes are operating in unicast mode

– esxcli vsan cluster unicastagent list:

• Unicast info is also displayed in vSAN network details

– esxcli vsan network list

25




bution

NIC Teaming and Load-Balancing Recommendations



bution

NIC Teaming – Single vmknic, Multiple vmnics (Uplinks)

• Route based on originating virtual port

– Pros

• Simplest teaming mode, with very minimum physical switch configuration.

– Cons

• A single VMkernel interface cannot use more than a single physical NIC's bandwidth.

• Route Based on Physical NIC Load

– Pros

• No physical switch configuration required.

– Cons

• Since only one VMkernel port, effectiveness of using this is limited

• Minor overhead when ESXi re-evaluates the load




bution

Load Balancing - Single vmknic, Multiple vmnics (Uplinks)

• vSAN does not use NIC teaming for load balancing

• vSAN has no load balancing mechanismto differentiate between multiple vmknics.

• As such, the vSAN IO path chosen is not deterministic across physical NICs

28

0

100000

200000

300000

400000

500000

600000

700000

800000

900000

1000000

Node 1 Node 2 Node 3 Node 4

KBps Utilization per vmnic -Multiple VMknics

vmnic0 vmnic1




bution

NIC Teaming – LACP & LAG (***Preferred***)

• Pros

– Improves performance and bandwidth

– If a NIC fails and the link-state goes down, theremaining NICs in the team continue to pass traffic.

– Many load balancing options

– Rebalancing of traffic after failures is automatic

– Based on 802.3ad standards.

• Cons

– Requires that physical switch ports be configured ina port-channel configuration.

– Complexity on configuration and maintenance




bution

http://www.ieee802.org/3/

Load Balancing – LACP & LAG (***Preferred***)

• More consistency compared to “Route based on physical NIC load”

• More individual Clients (VMs) will cause further increase probability of abalanced load

30

0

50000

100000

150000

200000

250000

300000

350000

400000

450000

500000

Node 1 Node 2 Node 3 Node 4

KBps Utilization per vmnic - LACP Setup

vmnic0 vmnic1




bution

vSAN Network on Different Subnets (air-gap)

vSAN networks on 2 different subnets?

• If subnets are routed, and one host’s NIC fails, host will communicate on other subnet

• If subnets are air-gapped, and one host’s NIC fails, it will not be able to communicate to the other hosts via other subnet

• That host with failing NIC will become isolated

• No software controlled failover mechanism

– TCP timeout 90sec on failure




bution

Supported Network Topologies



bution

Topologies

• Single site, multiple hosts

• Single site, multiple hosts with Fault Domains

• Multiple sites, multiple hosts with Fault Domains (campus cluster but not stretched cluster)

• Stretched Cluster

• ROBO/2-node

• Design considerations

– L2/L3

– Multicast/Unicast

– RTT (round-trip-time)




bution

Simplest Topology - Layer-2, Single Site, Single Rack

• Single site, multiple hosts, shared subnet/VLAN/L2 topology, multicast with IGMP

• No need to worry about routing the multicast traffic in pre-vSAN 6.6 deployments

• Layer-2 implementations are simplified even further with vSAN 6.6, and unicast. With such a deployment, IGMP snooping is not required




bution

Layer-2, Single Site, Multiple Racks – pre-vSAN 6.6 (multicast)

• pre-vSAN 6.6 where vSAN traffic is multicast

• Vendor specific multicast configuration required (IGMP/PIM)




bution

Layer-2, Single Site, Multiple Racks – 6.6 and Later (Unicast)

• vSAN 6.6 where vSAN traffic is unicast

• No need to configure IGMP/PIM on the switches




bution

Stretch Cluster Topologies



bution

Stretched Cluster – L2 for Data, L3 to Witness or L3 Everywhere

• vSAN 6.5 and earlier, traffic between data sites is multicast (meta) and unicast (IO).

• vSAN 6.6 and later, all traffic is unicast

• In all versions of vSAN, the witness traffic between a data site and the witness site has always been unicast




bution

Stretched Cluster - Why not L2 Everywhere? (Unsupported)

• Consider a situation where the link between S2 and S3 is broken

• Spanning Tree may discover a path between S2 and S3 exists via switch S1

• Possible performance decrease if data network traffic passes through a lower specification witness site




bution

2-Node (ROBO)

40



bution

2-Node vSAN for Remote Locations

• Both hosts in remote office store data

• Witness in central office or 3rd site stores witness data

• Unicast connectivity to witness appliance

– 500ms RTT Latency

– 1.5Mbps bandwidth from Data Site to Witness

41

Cluster

vSphere vSAN

vSphere vSAN

vSphere vSAN

Witness

vSphere vSAN

Witness

500ms RTT latency1.5Mbps bandwidth

500ms RTT latency1.5Mbps bandwidth




bution

2-node Direct Connect and Witness traffic separation


VSAN Datastore

witness

10GbE vSAN traffic via Direct Cable

management & witness traffic

• Separating the vSAN data traffic from witness traffic

• Ability to connect Data nodes directly using Ethernet cables

• Two cables between hosts for higher availability of network

• Witness traffic uses management network

Note: Witness Traffic Separation is NOT supported for stretch Cluster at this timeVMworld 2017 Content: Not fo


bution

vSAN andNetwork Performance



bution

General Concept on Network Performance

• Understanding vSAN concepts and features

– Standard vSAN setup vs. Stretch Cluster, FTT=1 or RAID5/6

• Understand Network Best Practice for optimum Performance – physical switch topology

– ISL trunks are not over subscripted

– MTU size factor

– No errors/drops/pause frames on the Network switches

44



bution

General Concept on Network Performance

• Understand Host communication

– No errors/drops/CRC/pause frames on the Network card

– Driver/Firmware as per our HCL

– Use SFP/Gbic certified by your Hardware Vendor

– Use of NIOC to optimize traffic on the protocol layer if links sharing traffic (Ex. VM/vMotion/..)

45



bution

http://www.vmware.com/go/hcl

DEMO: Adding 10ms Network Latency




bution

Summary: Graphical Interpretation IOPS vs. Latency

47

0

5000

10000

15000

20000

25000

30000

35000

40000

45000

50000

0 5 10 15 20 25

IOP

S

additional latency increase ms

latency ms Linear (latency ms)

+10ms latency = ~23100 IOPS

+5ms latency = ~33000 IOPS

Native = ~47000 IOPS




bution

DEMO: Network 2% and 10% Packet Loss




bution

Summary: Graphical Interpretation IOPS vs. Loss %

49

0

5000

10000

15000

20000

25000

30000

35000

40000

45000

50000

0 5 10 15 20 25

IOP

S

% loss

loss % Expon. (loss %)

1% loss = ~42300 IOPS

Native = ~47000 IOPS

2% loss = ~32000 IOPS

10% loss = ~3400 IOPS




bution



bution

Cormac Hogan@CormacJHogan

Andreas Scherr@vsantester



bution

Documents

STO1193BU A Closer Look at vSAN Networking Design and or … · 2017-10-12 · 3 NIC Teaming and Load Balancing 4 Network Topologies (incl. Stretched and 2-node) 5 Network Performance