Embed Size (px)
Citation preview
SMB 3 is the answer
Ned PyleSr. PM, Windows Server
MDC323B
AgendaWhy SMB 3?SMB 3 in WS 2012What’s new in SMB 3.02 for WS 2012 R2What SMB 3 brings to real life workloadsTroubleshooting SMB 3
Why SMB 3?
SMB lineage in MicrosoftSMB 1 – LANMan-XPSMB 2 – Vista/2008 SMB 2.1 – 7/2008 R2 SMB 3.0 – 8/2012SMB 3.02 8.1/2012 R2
BTW: You can remove SMB 1 in WS 2012 R2
Shared JBODStorage
Windows Server 2012 brought a new visionDramatically lowering the costs and effort of delivering Infrastructure as a Service storage services
Separate compute and storageIndependently manage and scale at each layer
Industry standard servers, networking
and storageInexpensive networksInexpensive shared JBOD storage
Scale-Out File Server ClustersStorage Spaces Virtualization and Resiliency
Hyper-V Clusters
SMB
This required a new storage stack
Cluster-Aware
Updating
SMB3
Virtual Fibre
Channel
Hyper-V Replica
8,000 VMs
per Cluster
VM Prioritizati
on
64-node clusters
Dedup
Scale-Out File Server
Storage Spaces
Offload Data
Transfer
VM Storage
Migration
iSCSI Target Server
ReFS VHDX
SMI-S Storage Service
NTFS Trim /
UnmapNFS 4.1SM API CSV online
CHKDSK
Windows Server 2012
SMB 3 mandatesEnable the IaaS compute and storage scenarioBecome the go-to remote file protocol for business Modernize for information workers
One ring to rule them all!
The industry is onboardEMC and NetApp have SMB 2 and subset SMB 3 implementations nowhttp://www.emc.com/collateral/white-papers/h11383-vnxe-introduction-wp.pdf http://www.netapp.com/us/media/tr-4172.pdf “NetApp is committed to providing support for SMB 3.0” – NetApp, 2012“[We have a] commitment to support Windows Server 2012 - including Server Message Block 3.0” – EMC, 2012
Even Apple prefers SMB now http://images.apple.com/osx/preview/docs/OSX_Mavericks_Core_Technology_Overview.pdf “SMB2 is superfast, increases security, and improves Windows compatibility.” – Apple, 2013
The workloads
Historically, FS cluster failover disruptsFailovers are not transparentWorks ok for traditional file serverMost user client software resumes gracefullyServer applications expect continuously available storage
Connection and file handles are lost on share failoverApplication disruptionAdministrator intervention required to recover
File Server Cluster
File Server Node A
File Server Node B
\\fs1\share
1
2\\fs1\share
3
SQL Server
Failover share and connections and handles lost
2
Normal operation1
Administrator intervention needed to recover
3
Hyper-VHyper-VHyper-VHyper-VHyper-VHyper-V
Hyper-V over SMBWhat is it?Store Hyper-V files in shares over the SMB 3.0 protocolStandalone and clusterFile storage used as cluster shared storage
File Server
File Server
SharedStorage
Hyper-V
SQLServer
IIS
VDIDesktop
Hyper-V
SQLServer
IIS
VDIDesktop
Hyper-V
SQLServer
IIS
VDIDesktop
Hyper-V Cluster
File Server Cluster
Hyper-VHyper-VHyper-VHyper-VHyper-VHyper-V
SQL over SMBWhat is it?Store SQL database files in shares over the SMB 3.0 protocolStandalone and clusterFile storage used as cluster shared storage
File Server
File Server
SharedStorage
SQL Server
SQLInstance
SQLInstance
SQL Server
SQLInstance
SQLInstance
SQL Server
SQLInstance
SQLInstance
SQLInstance
SQL Cluster
File Server Cluster
SQLInstance
SQLInstance
AdvantagesIncreases flexibilityEases provisioning, management and migrationLeverages converged networkReduces capital and operational expenses
How SMB gets you there
File Server Cluster
SMB Transparent FailoverFailover transparent to server applicationZero downtime – small IO delay during failover
Supports planned and unplanned failoversHardware/Software MaintenanceHardware/Software FailuresLoad Rebalancing
Resilient for both file and directory operations
RequiresFile Servers configured as Windows Failover ClusterWindows Server 2012 on both the servers running the application and file server cluster nodesShares enabled for “continuous availability” (default configuration for clustered file shares)Works for both classic file server clusters (cluster disks) and scale-out file server clusters (CSV)
Hyper-V
Failover share - connections and handles lost,temporary stall of IO
2
2
Normal operation1
Connections and handles auto-recoveredApplication IO continues with no errors
3
1 3
File Server Node A
File Server Node B
\\fs\share \\fs\share
SMB Scale-OutTargeted for server app storageExample: Hyper-V and SQL ServerIncrease available bandwidth by adding nodesLeverages Cluster Shared Volumes (CSV)
Key capabilitiesActive/Active file sharesFault tolerance with zero downtimeFast failure recoveryCHKDSK with zero downtimeSupport for app consistent snapshotsSupport for RDMA enabled networksOptimization for server appsSimple management
Single File System Namespace
Cluster Shared Volumes
Single Logical File Server (\\FS\Share)
Hyper-V Cluster(Up to 64 nodes)
File Server Cluster
(Up to 8 nodes)
Datacenter Network(Ethernet, InfiniBand or
combination)
SMB Client SMB Server
SMB ServerSMB Client
User
Kernel
Application
DiskR-NIC
Network w/RDMA support
NTFSSCSI
Network w/RDMA support
R-NIC
SMB Direct (SMB over RDMA)AdvantagesScalable, fast and efficient storage accessHigh throughput , low latency, minimal CPULoad balancing, automatic failover and bandwidth aggregation via SMB Multichannel
ScenariosHigh performance remote file access for application servers
Required hardwareRDMA-capable network interface (R-NIC)Three types: iWARP, RoCE and InfiniBandRDMA NICs use SMB Multichannel, not teaming
NIC Throughput
1Gb Ethernet ~0.1 GB/sec
10Gb Ethernet ~1.1 GB/sec
40Gb Ethernet ~4.5 GB/sec
32Gb InfiniBand (QDR)
~3.8 GB/sec
56Gb InfiniBand (FDR)
~6.5 GB/sec
HBA Throughput
3Gb SAS x4
~1.1 GB/sec
6Gb SAS x4
~2.2 GB/sec
4Gb FC ~0.4 GB/sec
8Gb FC ~0.8 GB/sec
16Gb FC ~1.5 GB/sec
Multiple RDMA NICsMultiple 1GbE NICsSingle 10GbE RSS-capable NIC
SMB Server
SMB Client
SMB MultichannelFull ThroughputBandwidth aggregation with multiple NICsMultiple CPUs cores engaged when NIC offers Receive Side Scaling (RSS)
Automatic FailoverSMB Multichannel implements end-to-end failure detectionLeverages NIC teaming (LBFO) if present, but does not require it
Automatic ConfigurationSMB detects and uses multiple paths
Requires least one of these configsMultiple network adapters One or more NICs that support RSS One or more NICs configured with teaming One or more NICs that support RDMA
SMB Server
SMB Client
SMB Server
SMB Client
Sample Configurations
Multiple 10GbE in LBFO team
SMB Server
SMB ClientLBFO
LBFO
Switch10GbE
NIC10GbE
NIC10GbE
Switch10GbE
NIC10GbE
NIC10GbE
NIC10GbE
NIC10GbE
Switch1GbE
NIC1GbE
NIC1GbE
Switch1GbE
NIC1GbE
NIC1GbE
Vertical lines are logical channels, not cables
Switch10GbE/IB
NIC10GbE/
IB
NIC10GbE/
IB
Switch10GbE/IB
NIC10GbE/
IB
NIC10GbE/
IB
Switch10GbE
SMB Encryption & SigningEnd-to-end encryption of SMB data in flightProtects dataAES-CCM
SMB Signing updatedPrevents packet tamperingAES-CMAC
Both leverage AES-NI CPUs (Nehalem+) for better performanceNote required, but highly recommendedNo need for IPSec or WAN accelerators
Configured per share or serverApplication workload over unsecured networksBranch Offices over WAN networks
Another good reason to remove SMB 1
ServerClient
SMB Encryption
Demo
What about SMB 3.02?
SMB
Scale-out File Server
Hyper-V host
Automatic Scale-Out Rebalancing SoFS clients redirected to the “best” node for access to a specific shareAvoids unnecessary redirection trafficDriven by ownership of Cluster Shared Volumes
SMB connections managed per share, not serverClients move as CSV volume ownership changes
Clustering now balances CSV automaticallyAutomatic behavior, no administrator action
File Server 2File Server 1
Storage Spaces
\\SOFS\Share2
Share1 Share2 Share1 Share2
\\SOFS\Share1
SMB Direct v2 Performance
Efficiency
Increased efficiency and density of hosting workloads with small I/O’s such as OLTP database in a VM
Optimizes 40Gbps Ethernet and 56Gbps InfiniBand
Performance
50% improvement for small IO workloads with SMB over RDMA
Increased 8KB IOPs from ~300K IOPS to ~450K IOPS per interface
Hyper-V Live Migration over SMB
SMB as a transport for Live VM Migration RDMA (SMB Direct)Streaming over multiple NICs (SMB Multichannel)
Provides highest bandwidth and lowest latency and CPU
0
10
20
30
40
50
60
70
Live Migration Times
Seco
nds
TCP/IP Compression
SMB w/RDMA(no compression)
Hyper-V host 2Hyper-V host 1
SMB Bandwidth ManagementFile Server for library Storage
Scale-out File Server
VHDX
Live MigrationLimit = 500
MB/s
StorageNo Limit
DefaultLimit = 100
MB/s
Control
Configurable SMB bandwidth limits per category
Default, VirtualMachine and LiveMigration
Common Infrastructur
e
SMB leveraged for VMs to access storage, distribution from VM library, and live migration
Desire to manage bandwidth of different types of SMB communication
Scale-Out File Server
File Server 2
Multiple SMB InstancesAdditional instance on each node in a SoFS for CSV trafficDefault instance handles incoming traffic from SMB clients accessing sharesOther instance handles only inter-node CSV traffic (metadata access or redirected traffic)Separate data structures (locks/queues) for regular client traffic and inter-node trafficImproves scalability and reliability of inter-node traffic between CSV nodes
File Server 1
Hyper-V Host 1
CSV1(Metadata
Owner)
Shared SAS Storage
SM
B S
erv
er
Defa
ult
In
stan
ce
SM
B S
erv
er
CS
V In
stan
ce
SM
B C
lien
t
SMB Client
SM
B S
erv
er
Defa
ult
In
stan
ce
SM
B S
erv
er
CS
V In
stan
ce
SM
B C
lien
t
Hyper-V Host 2
SMB Client
CSV2(Not
Metadata Owner)
CSV1(Not
Metadata Owner)
CSV2(Metadata
Owner)
Hyper-V over SMB (this is important)Hyper-V supports SMB version 3.0 onlyWS2012 and laterA few 3rd parties have a subset of SMB 3 nowHVBPA will confirm
Active Directory is requiredContinuously Available shares recommendedVirtual Machine Manager 2012 SP1 and 2012 R2 support Hyper-V over SMBFile Server and Hyper-V must be separate serversIf using Failover Clusters, File Server and Hyper-V must be on separate clusters
IW and SoFS (this is also important)Scale-out file server designed for App workloadsLong running data operationsFew metadata operationsWrite-through requirement
Traditional Information WorkerMany features incompatible with CSV, SoFSOpening files, closing files, creating new files, or renaming existing files are slower on SoFSUser client apps often very resilient to brief interruptionStill should cluster, just not as continuous availability!
File Server Configurations
Single-node File Server• Lowest cost for shared storage• Shares not continuously available
Dual-node File Server• Low cost for continuously available
shared storage• Limited scalability
(up to a few hundred disks)
Multi-node File Server• Highest scalability
(up to thousands of disks)• Higher cost, but still lower than
connecting all Hyper-V hosts with FC
Hyper-V Parent 1
Child 1Config
VHD Disk
Hyper-V Parent N
Child NConfig
VHD Disk
File Server
Share1 Share2
Disk Disk
Hyper-V Parent 1
Child 1Config
VHD Disk
Hyper-V Parent N
Child NConfig
VHD Disk
File Server 1
Share1 Share2
File Server 2
Share1 Share2
Shared SAS Storage
Disk DiskDisk Disk
Hyper-V Parent 1
Child 1Config
VHD Disk
Hyper-V Parent N
Child NConfig
VHD Disk
FS 1
Share1
Fibre Channel Storage Array
Disk Disk Disk DiskDisk Disk Disk
FS 2
Share2
FS 3
Share3
FS 4
Share4
A B C
Network ConfigurationsAll 1GbE Mixed 1GbE/High
SpeedAll High Speed
(10GbE/40GbE/56GbIB)
Hyper-V 1
File Server 1
Hyper-V 2
File Server 2
1GbE 1GbE
1GbE 1GbE
Hyper-V 1
File Server 1
Hyper-V 2
File Server 2
High Speed High Speed
1GbE 1GbE
Hyper-V 1
File Server 1
Hyper-V 2
File Server 2
High Speed High Speed
High Speed High Speed
Clients Clients Clients
File Server 1
File Server 2
High Speed High Speed
Clients
B CA D
Hyper-V 1 Hyper-V 2
Enterprise IW
IW ScenarioThe cloud is comingBut our new name is Cloud & EnterpriseThe on-premise information worker is still kingBusinesses still run on home folders and unstructured data shares, to the tune of 800PB and growingWindows 7 is an unstoppable juggernaut
Our WS2012 R2 IW File Server stackSMB 2.X/3.X
EncryptionSigningFile LeasingDirectory Leasing Energy efficiencySMB MultichannelDurable handles
Work Folders
AD RMS
DFSR
DFSN
iSCSINFSDynamic Access ControlStorage SpacesStorSimpleBranchCache & CSCDirectAccessFile Classification InfrastructureTraditional Cluster
SMB2.1 is no slouch for end usersHP StoreEasy 5000 results using WS2012+Windows 7
File Server Capacity Tool 1.2Validate and compare hardwareLocate existing bottlenecksPlanning for expansion in advance of exhaustion
Throughput capacity Maximum number of operations per secondMaximum number of users supported
http://www.microsoft.com/en-us/download/details.aspx?id=27284
Troubleshooting SMB3 in WS2012 R2
Improved SMB Diagnosibility in WS2012 R2Consistent design philosophyScenario-based events containing inter-machine info Useful events on by defaultInclude details on configuration and troubleshooting guidance.Less noisy event don’t wrap oftenCommon misconfigurations logged at boot up
Peel the onion (expect some tears)Is it enabled?SMB Windows PowerShellClient and Server
It’s often not SMBApplication?Storage?
When it is SMB, it’s often still not SMBNetwork?Security?
Use all the logs in the stackSMB, System, CSV, Spaces, ClusterMessage Analyzer
Vendor? Make them prove themselves
Message AnalyzerTime to learn a new tool But it’s a really great tool! Download from: http://connect.microsoft.com/site216 Designed with SMB3 troubleshooting in mind
Demo
Net cap or it didn’t happen!SMB2_GLOBAL_CAP_DFS - server supports the Distributed File System.
SMB2_GLOBAL_CAP_LEASING - server supports leasing.
SMB2_GLOBAL_CAP_LARGE_MTU - Connection.SupportsMultiCredit is TRUE.
SMB2_GLOBAL_CAP_MULTI_CHANNEL - Connection.Dialect is "3.000", IsMultiChannelCapable is TRUE, and SMB2_GLOBAL_CAP_MULTI_CHANNEL set
SMB2_GLOBAL_CAP_DIRECTORY_LEASING if Connection.Dialect is "3.000", server supports directory leasing, and SMB2_GLOBAL_CAP_DIRECTORY_LEASING set
SMB2_GLOBAL_CAP_PERSISTENT_HANDLES if Connection.Dialect is "3.000", SMB2_GLOBAL_CAP_PERSISTENT_HANDLES set, and server supports persistent handles.
SMB2_GLOBAL_CAP_ENCRYPTION if Connection.Dialect is "3.000", server supports encryption, and SMB2_GLOBAL_CAP_ENCRYPTION is set
Don’t disable SMB2/3I mean it, don’t disable SMB2/3ReallySeriously, don’t do it
Client IW issues in metadata caching, not all SMB2/3HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\Lanmanworkstation\Parameters Try setting FileNotFoundCacheLifetime to ZEROTry setting FileInfoCacheLifetime to ZEROTry setting DirectoryCacheLifetime to ZEROhttp://technet.microsoft.com/en-us/library/ff686200(v=WS.10).aspx All hurt perf, and directorycachelifetime really hurts perf – think SMB1
Ok, if you have to disable, use it as a test, not a solution
Related sessionsMDC331A - Storage Spaces: What's New in Windows Server 2012 R2MDC211 - Introduction to Windows Server 2012 R2MDC325C - Windows Server 2012 Storage Efficiencies Demonstrated
Related contentWindows File Server StackWindows File Server Team - http://blogs.technet.com/filecabJose Barreto - http://blogs.technet.com/josebdaWhat’s new in WS2012 R2 - http://technet.microsoft.com/en-us/library/dn250019.aspxWS2012 R2 - http://blogs.technet.com/b/josebda/archive/2013/06/26/updated-links-on-windows-server-2012-r2-file-server-and-smb-3-0.aspxWS2012 - http://blogs.technet.com/b/josebda/archive/2013/05/05/updated-links-on-windows-server-2012-file-server-and-smb-3-0.aspx
SOFS- http://technet.microsoft.com/en-us/library/hh831349.aspx
DFSR - http://technet.microsoft.com/en-us/library/dn281957.aspx
NFS - http://technet.microsoft.com/en-us/library/jj592688.aspx
iSCSI - http://technet.microsoft.com/en-us/library/dn305893.aspx
Related contentWindows File Server StackDAC - http://technet.microsoft.com/en-us/library/hh831717.aspx Work Folders - http://technet.microsoft.com/en-us/library/dn265974.aspx AD RMS - http://technet.microsoft.com/en-us/library/cc771234(v=ws.10).aspx FCI - http://technet.microsoft.com/en-us/library/dd758761(v=ws.10).aspx BranchCache - http://technet.microsoft.com/en-us/library/hh831696.aspx Storage Spaces - http://technet.microsoft.com/en-us/library/hh831739.aspxStorSimple - http://microsoft.com/storsimple Certified Windows JBODshttp://windowsservercatalog.com/results.aspx?&chtext=&cstext=&csttext=&chbtext=&bCatID=1573&cpID=0&avc=38&ava=0&avq=0&OR=1&PGS=25&ready=0
Developer Network
Resources for Developers
http://msdn.microsoft.com/en-au/
Learning
Virtual Academy
http://www.microsoftvirtualacademy.com/
TechNet
Resources
Sessions on Demand
http://channel9.msdn.com/Events/TechEd/Australia/2013
Resources for IT Professionals
http://technet.microsoft.com/en-au/
© 2013 Microsoft Corporation. All rights reserved. Microsoft, Windows and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries.The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.
