Converged Networking for Windows Server 2012 R2 Hyper-V

Ben Armstrong DCIM319

• Virtualized Workloads• Hosting Infrastructure• Other Topics

Agenda

Historical Topology7 or more physical interfaces

Management

Storage times two…

Migration

Cluster

Virtual machine access times two

NIC

NIC

NIC

NIC

NIC

Management

Storage

Migration

Cluster

Team

NIC NIC

Virtual Switch

Converged Topology2 or more physical interfacesThey don’t have to be 10GB…

Network teaming

Virtual switch

Network QoS

Network isolation

vNIC

vNIC

vNIC

vNIC

vNIC

Management

Storage

Migration

Cluster

Team

NIC NIC

Virtual Switch

Converged Topology with RDMAFast, low latency, low overhead SMB

Leverages converged networking

Improved storage and migration perf

Can failover to converged

vNIC

vNIC

rNIC

rNIC

vNIC

Management

Storage

Migration

Cluster

Team

NIC NIC

Virtual Switch

Topology: converged plus RDMA, SR-IOVFast, low latency, low overhead guest networking

Leverages converged networking

Can target specific VMs

Can failover to converged

vNIC

vNIC

rNIC

rNIC

vNIC

Management

Storage

Migration

Cluster

Team

NIC NIC

Virtual Switch

Virtual Switch

NICSR-IOV

Enabled

virtualized workloadsdemands on networking

• Low demand• Lucky you…

• High demand• Throughput?• Latency?• Inbound, Outbound, Both?• North <-> South, East <-> West?• Availability?

Demands On Network

• What is it?• The grouping of 1 or more NICs to form a larger bandwidth virtual NIC that doesn’t fail

unless every member of the team fails• The way to expose VLANs in native hosts (NOT in Hyper-V!!)

• Why use it?• Better utilization of bandwidth• Loss of a NIC or NIC cable doesn’t cut off communication

• How does it get along with others• Doesn’t get along with SR-IOV or RDMA (the bypass technologies)• Works with all other Windows Networking Features

• Recommended mode: Switch independent teaming with Dynamic Load Distribution

NIC Teaming (in the host OS)

• NetLbfo Powershell Cmdlets• [Add|Get|Remove|Set]- NetLbfoTeamMember• [Add|Get|Remove|Set]- NetLbfoTeamNic• [Get|New|Remove|Rename|Set]- NetLbfoTeam

• *-NetAdapter cmdlets also work (where applicable)

• Invoke the UI: LbfoAdmin.exe• Or use the link on Server Manager

NIC Teaming

Host NIC Teaming in VMM$1 = New-SCLogicalNetwork -Name "Team"

$2 = New-SCLogicalNetworkDefinition -Name "Team_0" -LogicalNetwork $1 -VMHostGroup (Get-

SCVMHostGroup -Name "All Hosts")

-SubnetVLan (New-SCSubnetVLan -VLanID 0)

Host NIC Teaming in VMM$1 = New-SCLogicalNetwork -Name "Team"

$2 = New-SCLogicalNetworkDefinition -Name "Team_0" -LogicalNetwork $1 -VMHostGroup (Get-

SCVMHostGroup -Name "All Hosts")

-SubnetVLan (New-SCSubnetVLan -VLanID 0)

$3 = New-SCNativeUplinkPortProfile -Name "Team" -

LogicalNetworkDefinition $2 -

LBFOLoadBalancingAlgorithm "HostDefault" -LBFOTeamMode

"SwitchIndependent"

Host NIC Teaming in VMM$1 = New-SCLogicalNetwork -Name "Team"

$2 = New-SCLogicalNetworkDefinition -Name "Team_0" -LogicalNetwork $1 -VMHostGroup (Get-

SCVMHostGroup -Name "All Hosts")

-SubnetVLan (New-SCSubnetVLan -VLanID 0)

$3 = New-SCNativeUplinkPortProfile -Name "Team" -

LogicalNetworkDefinition $2 -

LBFOLoadBalancingAlgorithm "HostDefault" -LBFOTeamMode

"SwitchIndependent"

$4 = New-SCLogicalSwitch -Name "Team" -

SwitchUplinkMode "Team"

New-SCUplinkPortProfileSet -Name "Team" -LogicalSwitch $4 -NativeUplinkPortProfile $3

• What is it?• The grouping of 1 or 2 virtual NICs to form a larger bandwidth virtual NIC that doesn’t

fail unless every member of the team fails (i.e., every underlying switch loses connectivity)

• Why use it?• Better utilization of bandwidth• Loss of a NIC or NIC cable doesn’t cut off communication to the guest• Provides failure protection in a guest with SR-IOV VFs

• How does it get along with others• Works with all other Windows Networking Features in the guest

• Recommended mode: Switch independent teaming with Dynamic Load Distribution

NIC Teaming (in the guest OS)

Guest NIC Teaming in Hyper-VSet-VMNetworkAdapter

-AllowTeaming

Guest NIC Teaming in VMMNew/Set-SCVirtualNetworkAdapterNativePortProfile

-AllowTeaming<Boolean>

• What is it• A way to spread traffic processing across multiple processors• RSS for host NICs (vNICs) and SR-IOV VFs• VMQ for guest NICs (vmNICs)

• Why use it• Because multiple processors are better than one processor• vRSS provides near line rate to a VM on existing hardware

• How does it get along with others• RSS and VMQ each work with all other features except . . .• RSS and VMQ can’t be used at the same time on the same NIC

VMQ, RSS, and vRSS

Host VMq Information and ConfigurationPS C:\> Get-NetAdapterVmq

Name InterfaceDescription Enabled BaseVmqProcessor MaxProcessors NumberOfReceive Queues---- -------------------- ------- ---------------- ------------- ---------------Fabric Uplink Logical Switch Microsoft Network Adapter Mult... True 0:0 126SLOT 7 Port 2 Intel(R) Ethernet Server Adapt... True 0:0 16 63SLOT 7 Port 1 Intel(R) Ethernet Server Ada...#2 True 0:0 16 63

Host VMq Information and ConfigurationPS C:\> Get-NetAdapterVmq

Name InterfaceDescription Enabled BaseVmqProcessor MaxProcessors NumberOfReceive Queues---- -------------------- ------- ---------------- ------------- ---------------Fabric Uplink Logical Switch Microsoft Network Adapter Mult... True 0:0 126SLOT 7 Port 2 Intel(R) Ethernet Server Adapt... True 0:0 16 63SLOT 7 Port 1 Intel(R) Ethernet Server Ada...#2 True 0:0 16 63

63 + 63 = 126

Host VMq Information and ConfigurationPS C:\> Get-NetAdapterVmq

Name InterfaceDescription Enabled BaseVmqProcessor MaxProcessors NumberOfReceive Queues---- -------------------- ------- ---------------- ------------- ---------------Fabric Uplink Logical Switch Microsoft Network Adapter Mult... True 0:0 126SLOT 7 Port 2 Intel(R) Ethernet Server Adapt... True 0:0 16 63SLOT 7 Port 1 Intel(R) Ethernet Server Ada...#2 True 0:0 16 63

PS C:\> Disable-NetAdapterVmq

PS C:\> Enable-NetAdapterVmq

Guest VMq Configuration in Hyper-VSet-VMNetworkAdapter

-VmqWeight <0 -100>

*Weight of 0 == disabled*Weight of 1-100 == enabled

*Default is 100

Guest VMq Configuration in VMMNew/Set-SCVirtualNetworkAdapterNativePortProfile

-EnableVmq

*Default is enabled

Guest View of Virtual RSS Get-NetAdapterRssEnable-NetAdapterRssDisable-NetAdapterRss

• What is it• A way to get the NIC to segment a packet for you• The sender sends a large packet to the NIC, the NIC sends multiple

packets that fit within the MTU (packets are not marked as segments)

• Why use it• Reduces stack processing

• How does it get along with others• LSO gets along with all other Windows Networking Features

Large Send Offload (LSO)

Host Large Send Offload InformationPS C:\> Get-NetAdapterLso

Name Version V1IPv4Enabled IPv4Enabled IPv6Enabled---- ------- ------------- ----------- -----------vEthernet (Storage4) LSO Version 2 False True TruevEthernet (Storage3) LSO Version 2 False True TruevEthernet (Storage2) LSO Version 2 False True TruevEthernet (Storage1) LSO Version 2 False True TruevEthernet (Migration) LSO Version 2 False True TruevEthernet (Cluster) LSO Version 2 False True TruevEthernet (Fabric Managment... LSO Version 2 False True TrueFabric Managment Logical Sw... LSO Version 2 False True TrueFabric Uplink Logical Switch LSO Version 2 False True True10GBCopper-1 LSO Version 2 False True True10GBFibre-2 LSO Version 2 True True True10GBFibre-1 LSO Version 2 True True True1GBLOM-4 LSO Version 2 False True True1GBLOM-3 LSO Version 2 False True TrueRDMA-2 LSO Version 2 True True TrueRDMA-1 LSO Version 2 True True True1GBLOM-1 LSO Version 2 False True True1GBLOM-2 LSO Version 2 False True True10GBCopper-2 LSO Version 2 False True True

• What is it• A way to send large data units unbroken• Usually used for storage but can be used for anything

• Why use it• Reduces packet processing at every hop (savings estimated in the 10-

25% range)

• How does it get along with others• Has a slightly lower error detection probability• Requires knowledge of path-MTU• Otherwise gets along with all other Windows Networking Features

Jumbo Frames

Verifying Path MTU…C:\> ping -l 9014 <host>

Guest View of Jumbo FramesSet-NetAdapterAdvancedProperty

-RegistryKeyword

"*JumboPacket" -RegistryValue

9014

Topology with SR-IOV

vNIC

vNIC

vNIC

vNIC

vNIC

Management

Storage

Migration

Cluster

Team

NIC NIC

Virtual Switch

Virtual Switch

NIC

SR-IOV Enabled

• What is it• SR-IOV is a way to bypass the host OS and process the networking traffic directly in

the VM• RSS is the way the SR-IOV bypass traffic can get distributed to different processors in

the VM enabling higher throughput

• Why use it• Because with SR-IOV and RSS we’ve seen over 39 Gbps of traffic handled in a VM

(coming from a single 40 Gbps NIC Port)

• How does it get along with others• SR-IOV doesn’t play well with NIC Teaming (host) or other switch features• Plays well with Guest NIC Teaming

• ACLs, VM-QoS, etc., will prevent SR-IOV from being used• Should only be used in trusted VMs• Can’t have more VMs than NIC VFs/vPorts*

SR-IOV and RSS

Parent Partition

SR-IOVVirtual machine

Network stack

VM NIC

Virtual function (VF)

Hyper-V Extensible Switch

ExtensionsRouting Filtering

ACLs

SR-IOV network adapter VF

Virtual machine

Network stack

VM NIC

VM BUS

Parent Partition

SR-IOV + RSSVirtual machine

Network stack

VM NIC

Virtual function (VF)

Hyper-V Extensible Switch

ExtensionsRouting Filtering

ACLs

SR-IOV network adapter VF

Virtual machine

Network stack

VM NIC

VM BUS

SR-IOV Switch Configuration in Hyper-VNew-VMSwitch –EnableIov

Get-VMSwitch | filter-tableName, IovSupport,

IovSupportReasons

SR-IOV Virtual NIC Configuration in Hyper-VSet-VMNetworkAdapter

-IovWeight <0-100>

Set-VMNetworkAdapter-IovInterruptModeration

Default, Adaptive, Off, Low,

Medium, and High

-IovQueuePairsRequested <1 – Max Uint>

SR-IOV Switch Configuration in VMMNew-SCLogicalSwitch

-EnableSriov

SR-IOV Port Profile Configuration in VMMNew/Set-SCVirtualNetworkAdapterNativePortProfile

-EnableSriov

SR-IOV Guest Configuration in VMMGet-SCPortClassification |

where {$_.Name -eq “SR-IOV"}

Set-SCVirtualNetworkAdapter-PortClassification

• Low demand• Lucky you…

• High demand• Throughput? • Use NIC Teaming, SR-IOV, vRSS/Guest RSS, VMQ, LSO, Jumbo frames

• Latency?• Use SR-IOV, VMQ, vRSS/Guest RSS

• Inbound, Outbound, Both?• Inbound: SR-IOV, VMQ• Outbound: NIC Teaming, SR-IOV, LSO, Jumbo frames

• North <-> South, East <-> West?• North-South demands are generally lower than East-West. Optimize East-West.

• Availability?• NIC Teaming

Demands On Network - revisited

hosting infrastructuredemands on networking

• Services of the network• Bandwidth management (QoS)• Virtual machine mobility• Live migration• Storage migration

• SMB configuration

Demands On Network

Time Session Code Title

Wednesday

3:15 – 4:30 DCIM-B337 File Server Networking for a Private Cloud Storage Infrastructure in Windows Server 2012 R2

• What is it• Two forms: Software QoS (vmQoS) and Hardware QoS (DCB)• Software QoS provides per-VM (per vSwitch port) bandwidth

management for better sharing• Hardware QoS provides per-traffic class reservations• Software QoS and Hardware QoS cannot be used at the same time on

the same NIC

• Why use it• Software: To manage bandwidth allocations per VM or vNIC• Hardware: To ensure storage and data traffic play well together

• How does it get along with others• QoS can’t be used when SR-IOV is used

Quality of Service

• Watch out for . . . • Software QoS and Hardware QoS cannot be used at the same time on the same NIC• In Software QoS (vmQoS) • If weights are used, the weights should be such that the lowest weight is not

extremely different from the highest weight• If absolute bandwidth is used, the bandwidth should be such that the lowest

bandwidth is not orders of magnitude less than the highest bandwidth

• Once a switch is created in a mode (weight, absolute bandwidth) it can’t be changed.• Weights are better than absolute• Can’t migrate between systems where one is weights-based and the other is absolute• Migration will be blocked if absolute values can’t be honored

• DCB• For best results make sure PFC/ETS is on throughout your datacenter network

Quality of Service - Caution

Software QoS in Hyper-VNew-VMSwitch

-MinimumBandwidthMode<Absolute, Default,

None, Weight>

Set-VMSwitch-

DefaultFlowMinimumBandwidthAbsolute-

DefaultFlowMinimumBandwidthWeight

Set-VMNetworkAdapter-MaximumBandwidth-MinimumBandwidthAbsolute-MinimumBandwidthWeight

Software QoS in VMMNew/Set-SCVirtualNetworkAdapterNativePortProfile

-MaximumBandwidth-MinimumBandwidthAbsolute-MinimumBandwidthWeight

ALSO

Add/Set-SCVMHost-NetworkPercentageReserve

Enabling Hardware QoS in Hyper-VSet-VMNetworkAdatper

-IeeePriorityTag <On, Off>

Enabling Hardware QoS in VMMNew/Set-SCVirtualNetworkAdapterNativePortProfile

-AllowIeeePriorityTagging<Boolean>

• What is it• Enables migration of VM’s across Hyper-V servers• Three different transport options: TCP, Compression, SMB

• Why use it• TCP – reduced CPU utilization• Compression – faster migrations• SMB – multiple interfaces (SMB Mulitchannel), reduced CPU with SMB

Direct• Simultaneous migrations – reduces bandwidth demand

• How does it get along with others• Perfectly… except it can be a bandwidth hog

Live migration

Live Migration Settings in Hyper-V

Set-VMHost -MaximumVirtualMachineMigrations -

VirtualMachineMigrationPerformanceOptionTCPIP, Compression, SMB

Live Migration Settings in VMMAdd/Set-SCVMHost

-EnableLiveMigration-LiveMigrationMaximum-

LiveStorageMigrationMaximum-MigrationAuthProtocol-

MigrationPerformanceOption-MigrationSubnet

• What is it• Enables live migration of virtual machine storage• Traffic flows through the Hyper-V server• Maximum concurrent migrations configurable

• Why use it• Reduces bandwidth demands

• How does it get along with others• Perfectly… Just keep in mind it needs bandwidth

Storage migration

Storage Migration Settings in Hyper-VSet-VMHost

-MaximumStorageMigrations

Storage Migration Settings in VMMAdd/Set-SCVMHost

-LiveMigrationMaximum

• What is it• Quality of Service for SMB

• Why use it• Enables bandwidth management of SMB connections• Live migration• Provisioning• Virtual machines

• How does it get along with others• Works with SMB Multichannel• Works with SMB Direct• Works with RDMA

SMB Bandwidth Limits

SMB Bandwidth LimitAdd-WindowsFeature FS-SMBBW

Set-SmbBandwidthLimit -Category LiveMigration-BytesPerSecond 1000MB

Set-SmbBandwidthLimit -Category VirtualMachine

-BytesPerSecond 5GB

Key message: manage your bandwidth

hosting infrastructuredemands on networking

• Network Recommendations for a Hyper-V Cluster in Windows Server 2012• http://technet.microsoft.com/en-us/library/dn550728.aspx

• Windows Server 2012 R2 NIC Teaming (LBFO) Deployment and Management• http://www.microsoft.com/en-us/download/details.aspx?id=40319

• Windows Server 2012 NIC Teaming (LBFO) Deployment and Management• http://www.microsoft.com/en-us/download/details.aspx?id=30160

• What’s New in Windows Server 2012 R2 Networking• http://channel9.msdn.com/Events/TechEd/NorthAmerica/2013/MDC-B216

Additional Content and Documentation

Related contentBreakout Sessions (session codes & titles)

Hands-on Labs (session codes & titles)

Find Me Later At...

Required Slide*delete this box once you have listed content that is related to your session.

Speakers, please list the other Breakout Sessions and Virtualised Hands-on Labs that relate to your session.

Also indicate where and when they can find you, to continue the discussion. If you’re going to be at Hub Happy Hour (5.30-6.30pm Wed and Thu, let them know)

Resources

TechNet & MSDN FlashSubscribe to our fortnightly newsletter

http://aka.ms/technetnz http://aka.ms/msdnnz

TechNet Virtual LabsFree Virtual Hands-on Labs

http://aka.ms/ch9nz

Microsoft Virtual AcademyFree Online Learning

http://aka.ms/mva http://aka.ms/technetlabs

Sessions on Demand

Complete your session evaluation now and win!

© 2014 Microsoft Corporation. All rights reserved.Microsoft, Windows and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries.MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.