Storage Lunch

Cédric MégrozArchitect

22.11.2012

Why IP StorageIP storage NFS iSCSI SMB3

iSCSI conceptiSCSI architecture

iSCSI namehba or software iscsi

iSCSI dependant independant hardwareiSCSI Configuration

Binding or not binding that is the questionStacked or not stacked that is the question

Masking on the array with iqnMultipathing, Policy & Pluggable Storage Architecture

VMFS 5Miscellaneous

ALUAJumbo Frame

Delay AckEquallogic

Equallogic configurationsHost Software for VMware

Host Software for WindowsQuestions ?

Agenda

Why IP Storage

• Low cost infrastructure with 1 GB Ethernet• No switch and special connectivity to

implement it

• For small infrastructure with less 50 VMs

• BUT not so easy to implement (a lot of best practice to find perf and throughput)

IP Storage NFS iSCSI SMB3

• Two storage family

• Block (SCSI, iSCSI, FC, FCoE, AoE)

• NAS Network Attached storage ( NFS, SMB3 => Hyper-V)

Storage with VMware

Storagetechnology

Datastoretypes

FCoE iSCSIFibre

Channeldirect

attached

filesystem

NAS

NFSVMware vSphere® VMFS

VMware vSphere® ESXi™ hosts

iSCSI – What is it?

• iSCSI specifies a way to “encapsulate” SCSI commands in a TCP/IP network connection:

Explains how to extractSCSI commands and data

Provides information necessary toguarantee delivery

Contains “routing” informationso the message can find itsway through the network

SCSI commands and dataiSCSI Header

TCPHeader

IP Header

GigabitEthernet

Provides Link Addressing, large frame sizes, and flow control

iSCSI concept

• Internet Small Computer Systems Interface (iSCSI)

• Physical network isolation (dedicated sw) Logical network isolation (VLAN)

• Default Port Number: 3260• An optional CHAP Secret (e.g.,

"secretsarefun") do not use for masking• LUN (Logical unit number)

iSCSI architecture

ISCSI name

iSCSI target name:iqn.1992-08.com.mycompany:stor1-47cf3c25 oreui.fedcba9876543210iSCSI alias: stor1IP address: 192.168.36.101

iSCSI initiator name:iqn.1998-01.com.vmware:train1-64ad4c29 or eui.1234567890abcdefiSCSI alias: train1IP address: 192.168.36.88

LUN Masking

HBA or software iSCSI initiator

Network software iSCSI Configuration

• A VMkernel port must be created for ESXi to access software iSCSI.– The same port can be used to access NAS/NFS storage.

• To optimize your vSphere networking setup: Separate iSCSI networks from NAS/NFS networks.

• Physical separation is preferred.• If physical separation is not possible, use VLANs.

iSCSI Target-Discovery Methods

• Two discovery methods are supported:• Static• Dynamic (also known as SendTargets)

By default Dynamic

This functionality is used by Equallogic to redirect the host to the right target

192.168.36.101:3260

iSCSI Security: CHAP

• iSCSI initiators use CHAP for authentication purposes.• By default, CHAP is not

configured.• ESXi supports two types of

CHAP authentication:• Unidirectional • Bidirectional:

– Software iSCSI only

• CHAP is not masking !!!!

Software iSCSI properties > General tab

Target authenticates

host.

Host authenticates

target.

Multipathing with iSCSI

• Hardware iSCSI:

• Use two or more hardware iSCSI adapters.

• Software or dependent hardware iSCSI:

• Use multiple network interface cards (NICs).

• Connect each NIC to a separate VMkernel port.

• Associate VMkernel ports with iSCSI initiator.

Configuration iSCSI 1/4

• Create a dedicated vSwitch with 2 VMKernel and 2 VMNIC connected on the IP storage network

• If same subnet dedicate a VMNIC per VMKernel

• If not do nothing

Configuration iSCSI 2/4

• vmk_iscsi1 with VMNIC5

• vmk_iscsi2 with VMNIC4

• Idem for DVS

Configuration iSCSI 3/4

Configuration iSCSI 4/4

Binding or not binding VMKernel on iSCSI sw

Binding or not binding VMKernel on iSCSI sw

• Rescan times take longer than usual Incorrect • number of paths are seen per device • Unable to see any storage from the storage device• CISCO NEXUS with LACP impose this config. with AA array without ALUA

Stacked or not stacked that is the question

The architecture of the array can have two developement

Multi target: for each Ethernet port you have a iSCSI target => do not stack the switch like the FC word. (HDS or VNX)

Uni target: only one target is generated for all the Ethernet port => stack the switch to have a failover in case of lost connectivity (NetApp, Equallogic)

Dedicated network or not …

For small network yes it’s better

Think IOS update

With 10GB and Nexus… no choice

Generaly network infrastructure is stacked and it’s not a good thinks in case of lost of the master (old switch 15 sec without network)

Pluggable Storage Architecture

Multipathing, Policy

esxcli nmp satp setdefaultpsp -P VMW_SATP_ALUA -S VMW_PSP_RR for esxi 5 esxcli storage nmp satp set –P VMW_PSP_RR –s VMW_SATP_ALUA

What is Asymmetric Logical Unit (ALUA)?

ALUA Allows for paths to be profiled Active (optimized) Active (non-optimized)

Le VMFS 5

• VMFS-5:• Allows concurrent access to shared storage• Can be dynamically expanded• Uses a 1MB block size, good for storing large virtual disk files• Uses subblock addressing, good for storing small files:

– The subblock size is 8KB.• Provides on-disk, block-level locking

hosthost

VMFS datastore

VAAI

• vStorage APIs for Array Integration is a feature introduced ESX/ESXi 4.1 that provides hardware acceleration functionality.

• It enables your host to offload specific virtual machine and storage management operations to compliant storage hardware.

• Used for lock file, clone & copie, svMotion• NFS is supported with a ESXi agent• Need license Ent & Ent +

Datastore size

• Updating the metadata of VMFS requires a SCSI reserve and release

• 1. Starting and stopping a VM• 2. vMotioning a VM• 3. Using VMware snapshots• 4. Using vAPI Dataprotection (use VMware snapshots)

• . => 15 VMs / block datastore• VMware Best Practice 300 – 700 GB• But actually with VAAI => no limit• Use Large capacity datastores Avoid RDMs • NFS: 16TB • VMFS: vSphere 5 = 64TB • vSphere 4 = 2TB • Avoid extents

Miscellaneous

• Never route network IP storage !!! Warning with the loop !!!!

• Ask for compliant subnets with the Network team

• Use always the vCenter Plug-In to apply the best practice on the ESXi host

• Use always the vCenter Plug-In to create and mount the datastore

Jumbo Frame

• What is an Ethernet Jumbo Frame? Ethernet frames with more than 1500 bytes of payload (9000 is common – FCoE is 2240)

• Commonly ‘thought of’ as having better performance • Should I use Jumbo Frames?

• Adds complexity & performance gains (while existent) are relatively marginal with common block sizes

• Stick with the defaults when you can

Delayed Ack mandatory (EMC et HDS hyper-V)

TCP delayed acknowledgment is a technique used by some implementations of the Transmission Control Protocol in an effort to improve network performance. In essence, several ACK responses may be combined together into a single response, reducing protocol overhead. However, in some circumstances, the technique can reduce application performance.

http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1002598

http://support.microsoft.com/kb/981482

http://www.axiomdynamics.com/it-tweaks-and-tricks/iSCSI-best-practices/html/iSCSI-Best-Practices-VMware-EMC-Microsoft.html

Advanced network configurations

• Enable Rapid spanning-tree (portfast)• Enable flow control on all ports in the switch and array• Disable port storm control (broadcast- unicast)• Enable jumbo frames if possible

Alignment is Optimal I/O

Misalignment of filesystems results in additional work on storage controller to satisfy IO request

Affects every protocol, and every storage array VMFS & NFS Datastores

VMDKs & RDMs with NTFS, EXT3, LUN, etc

Filesystems exist in the datastore and VMDK

Create VMFS with GUI not vCLIWindows NT, 2000, 2003, & XP (use diskpart to set to 1MB) Linux (use fdisk expert mode and align on 2048 = 1MB)

Fixing Misalignment

Guest OS filesystem is misaligned

• Step 1: Take an array snapshot/backup • Step 2: Use offline tools to realign EMC

UBERAlign (open, works with all, scheduler, and in-guest reclaim) vSphere Migrator

Alternate: Use online tool to align NetApp Migrate & Optimize (VSC feature)

SIOC NIOC SDRS

• QoS on storage and network dVS

• NIOC offert QoS compatible 802.1p for Virtual machine traffic, Management traffic, iSCSI traffic, NFS traffic, Fault-tolerant traffic, VMware vMotion traffic, User-defined traffic, vSphere replication traffic

• SIOC use queue depth to manage contention. SIOC monitors the latency of I/Os to datastores at each ESX host sharing that device. When the average normalized datastore latency exceeds a set threshold (30ms by default), the datastore is considered to be congested, and SIOC kicks in to distribute the available storage resources to virtual machines in proportion to their shares.

• Need license Ent+ …

iSCSI Storage array

iSCSI Storage array

Switch stacked

Multi target Jumbo Frame 10GB ALUA

EMC VNX no yes yes yes yesNetApp depend Need different

subnet yes yes FC onlyHDS HUS no yes 1GB ? yes AAHDS AMS no yes 1GB no Only AMS

2500AA

Equallogic yes no yes yes -

Equallogic

• Can be combined with other EqualLogic PS Series arrays in the same SAN group, online. Up to 16 total members per group

• Automatic RAID 5, RAID 6, RAID 10 and RAID 50

• 1 mgmt target per SAN Group but each stack has these own tragets.

• The mgmt target redirect to the target with the LUN

• Need to use de MPIO of Dell Equallogic for Windows

Equallogic configurations

• There are three load balancers that operate within a pool:

• The NLB (Network Load Balancer) manages the assignment of individual iSCSI connections to Ethernet ports on the pool members

• The CLB (Capacity Load Balancer) manages the utilization of the disk capacity in the pool

• The APLB (Automatic Performance Load Balancer) manages the distribution of high I/O data within the pool.

Host Software for VMware

• Host Integration Tools for VMware® • Auto-Snapshot Manager/VMware Edition (ASM/VE): Integrates with

VMware VirtualCenter and PS Series snapshots to allow administrators to enable Smart Copy protection of VirtualCenter folders, datastores, and virtual machines.

• EqualLogic Datastore Manager: Allows administrators to create and manage datastores on EqualLogic PS Series arrays from within vCenter.

• Virtual Desktop Deployment Utility: Automates the deployment of virtual desktops in a VMware View™ environment.

• Storage Adapter for Site Recovery Manager (SRM): Allows SRM to understand and recognize PS Series replication for full SRM integration.

• Multipathing Extension Module for VMware® vSphere™: Provides connection awareness enhancements to the existing VMware multipathing functionality that understands PS Series network load balancing and facilitates host connections to PS Series volumes.

Host Software for Windows

• Host Integration Tools for Microsoft® • Remote Setup Wizard (RSW): Initializes new PS Series arrays, configures

host connections to PS Series SANs, and configures and manages multipathing.

• Multipath I/O Device Specific Module (MPIO DSM): Includes a connection awareness-module that understands PS Series network load balancing and facilitates host connections to PS Series volumes.

• VSS and VDS Provider Services: Allows 3rd party backup software vendors to perform off-host backups.

• Auto-Snapshot Manager/Microsoft Edition (ASM/ME): Provides point-in-time SAN protection of critical application data using PS Series snapshots, clones, and replicas of supported applications such as SQL Server®, Exchange Server®, Hyper-V™, and NTFS file shares.

• PowerShell Tools: Provides a comprehensive set of PowerShell cmdlets for managing one or many PS Series groups.

Questions ?

links

http://www.vmware.com/files/pdf/techpaper/vmware-multipathing-configuration-software-iSCSI-port-binding.pdf

http://blogs.vmware.com/vsphere/2011/08/vsphere-50-storage-features-part-12-iscsi-multipathing-enhancements.html

binding or nothttp://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2038869

PSAhttp://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1011375

Equallogichttp://docs.danielkassner.com/dell/ISCSI_optimization_EQL.pdf

iSCSI best practicehttp://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1002598

http://support.microsoft.com/kb/981482

http://www.axiomdynamics.com/it-tweaks-and-tricks/iSCSI-best-practices/html/iSCSI-Best-Practices-VMware-EMC-Microsoft.html

NetApp & EMC VMworld session Best Practicehttp://download3.vmware.com/vmworld/2012/top10/sto2980.pdf