Tunneling SCSI over SMB: Shared VHDX files for Guest Clustering in Windows Server 2012 … · 2019-12-21 · For Hyper-V hosts and File Server Must be running Windows Server 2012

2013 Storage Developer Conference. © Microsoft Corporation. All Rights Reserved.

Tunneling SCSI over SMB:

Shared VHDX files

for Guest Clustering

in Windows Server 2012 R2

Jose Barreto, Program Manager, Microsoft

Matt Kurjanowicz, Software Development Engineer, Microsoft



The need for Shared VHDX

Virtualization is everywhere

High availability is a desired feature

There are two main methods to accomplish it

Host Clustering

Guest Clustering


Two types of Clustering

Host Clustering Guest Clustering

4

Hyper-V 2Hyper-V 1

VM

Hyper-V 2Hyper-V 1

VM2VM1

Shared Storage

App App

Shared Storage

FailoverFailover App

Requires iSCSI to the Guest or Virtual FC.

Exposes shared storage infrastructure to VM.

Could we simplify it?


File Storage

Block Storage

Introducing: Shared Virtual Disks

VHDX

VHDX

Guest Clustering

Simply share virtual disks to provide

shared storage for Guest Clustering

Hyper-V Guest Failover Clustering

with commodity storage

Maintains separation between

infrastructure and tenants

Virtual SAS

VM presented with virtual SAS disks

Shared SAS disks, ready to cluster

Used for data disk only

Cluster Shared Volumes (CSV)

for block storage

Scale-Out File Server

for file storage


Hyper-V Parent 2Hyper-V Parent 1

CSV

Shared VHDX Configurations

VM2

Config

VHDX2Boot

Disk

Shared SAS Storage

Disk2 Disk4Disk1 Disk3

Shared

Disk

VHDX3

P

Disk5

Storage Spaces

VHDX2

VM1

Config

VHDX1Boot

Disk

Shared

Disk

P

VHDX1

Space1

S

Regular VHDX

Shared VHDX

Hyper-V Parent 2

VM2

Config

VHDX

2

Boot

Disk

File Server 1

Share1

File Server 2

Share1

Shared SAS Storage

Disk2 Disk4Disk1 Disk3

Shared

Disk

VHDX3

P

Disk5

Storage Spaces

VHDX2

S

Hyper-V Parent 1

VM1

Config

VHDX

1

Boot

Disk

Shared

DiskP

VHDX1

Space1

CSV

SMB


Deploying Shared VHDX

Deployment options

Hyper-V Manager GUI

PowerShell

VMM 2012 R2 (Service Template)

Inside the VM

Disks behave just like regular disks

Two things to watch: anti-affinity and heartbeating


Shared Virtual Disks - GUI

In Hyper-V Manager, go to VM Settings

Add a new hard drive (make sure it’s a VHDX, not an old VHD)

Expand “Advanced Features” under “Hard Drive”

Check the box for “Enable virtual hard disk sharing”

Hit “Apply” only after you checked the box


Shared Virtual Disks - PowerShell

1 ShareVirtualDisk

2 ShareVirtualDisk

SupportPersistentReservations


Inside the VM – Just a regular disk


Experiencing Shared VHDX

Recorded demo from TechEd 2013 Conference

https://skydrive.live.com/redir?resid=DFBE7FC34A56

43C6!12108&authkey=!AABY-a4Zro0Kyus

Step-by-step for Shared VHDX

http://blogs.technet.com/b/josebda/archive/2013/07/3

1/windows-server-2012-r2-storage-step-by-step-with-

storage-spaces-smb-scale-out-and-shared-vhdx-

physical.aspx

https://skydrive.live.com/redir?resid=DFBE7FC34A5643C6!12108&authkey=!AABY-a4Zro0Kyus

http://blogs.technet.com/b/josebda/archive/2013/07/31/windows-server-2012-r2-storage-step-by-step-with-storage-spaces-smb-scale-out-and-shared-vhdx-physical.aspx


Supported OS versions

For Hyper-V hosts and File Server

Must be running Windows Server 2012 R2

For Guests (Virtual Machines)

Windows Server 2012 or 2012 R2 are supported

Install the Hyper-V Integration Components

Nothing architecturally preventing the use of older

versions, but we do not have test coverage


Supported virtual disk formats

Data disks, not OS disks

OS disks always require exclusive access

VHDX, not VHD

Metadata (persistent reservations) stored on the

VHDX file header

Fixed or Dynamic, not Differencing

Differing does not make sense for a data VHDX


Backup

As with all other guest cluster options (iSCSI to

the guest and Virtual FC), backup from the guest

Host-based backups and snapshots not

supported


Failover and Move

Supported Failover scenarios

Single guest (virtual machine) failover

Single Hyper-V host failover

Single File Server node failover

Also works when you

Move the CSV volume

Move the Virtual Machines

Storage Migration not supported


Comparing Guest Cluster Options

Capability Shared VHDX Virtual FC iSCSI to the

Guest

Supported storage back-ends SAS, FC, iSCSI,

SMB, Spaces

FC SAN iSCSI SAN

How storage is exposed in the guest as Virtual SAS Virtual FC LUN iSCSI LUN

Flows through the Hyper-V virtual switch No No Yes

VM storage configured at the Hyper-V host Yes Yes No

Provides solution for low latency/low CPU use Yes (RDMA, FC) Yes (FC) No

Requires specific hardware on Hyper-V host No Yes No

Request switch reconfiguration on migration No Yes No

Exposes hoster storage architecture to guests No Yes Yes


File Server

Hyper-V

File Server

Hyper-V

Comparing two stacks

“Classic” Hyper-V over SMB Shared Virtual Disk

17

VM

Disk

SMB Client (RDR)

SMB Server (SRV)

Storage VSP (StorVSP)

VHD Stack (VHDMP)

File System (NTFS)

VM

Disk

SMB Client (RDR)

SMB Server (SRV)

Storage VSP (StorVSP)

VHD Stack (VHDMP)

File System (NTFS)

Parser (PVHDPARSER)

Shared VHDX (SVHDXFLT)



Why RSVD?


General Protocol Characteristics

Built on top of SMB3

“Block Semantics” on top of “File Semantics”

Most SCSI commands simply encapsulated

and forwarded

Client driven

Failover through SMB Continuous Availability



Session establishment sequence

Client Server

SMB3 Connection Establishment

IRP_MJ_CREATE

Get Starting Information


Characteristics of a SVHDX Open


Open as Block Device ECP

GUID: {9ECFCB9C-C104-43E6-980E-158DA1F6EC83}

Contents:

Client version

Initiator ID

Initiator Hostname

Flags describing the open

Flags describing where the open came from

Request ID


Versioning

Client identifies version in ECP

Server identifies version in Get Starting

Information tunneled command

STATUS_SVHDX_VERSION_MISMATCH when

invalid tunneled operations are received



RSVD_SYNC_TUNNEL_REQUEST

0 1 2 3 4 5 6 7 8 910 1 2 3 4 5 6 7 8 9

20 1 2 3 4 5 6 7 8 9

30 1

OperationCode (ULONG)

StatusCode (NTSTATUS)

Request ID (ULONGLONG)

…

Payload (Variable Length)

64KB Max Tunnel Size


Get Starting Information (0x02000001)

When: Immediately after CreateFile

Why: Versioning, Layout of the underlying disk


Get Starting Information

0 1 2 3 4 5 6 7 8 910 1 2 3 4 5 6 7 8 9

20 1 2 3 4 5 6 7 8 9

30 1

Header

ServerVersion (ULONG)

SectorSize (ULONG)

PhysicalSectorSize (ULONG)

VirtualSize (ULONGLONG)

…


SCSI Command Request (0x02000002)

When: At client application request

Why: Issue a SCSI command to the virtual disk

Reads & Writes handled differently

Important SCSI commands for Clustering

PERSISTENT RESERVE IN

PERSISTENT RESERVE OUT

Not all SCSI commands have been tested


SCSI Command Request

0 1 2 3 4 5 6 7 8 9

1

0 1 2 3 4 5 6 7 8 9

2

0 1 2 3 4 5 6 7 8 9

3

0 1

Header

Length Reserved1

CdbLength SenseInfoExLength DataIn Reserved2

SrbFlags

DataTransferLength

CDBBuffer

….

…

…

Reserved3

DataBuffer (variable size)

…


SCSI Command Response

0 1 2 3 4 5 6 7 8 9

1

0 1 2 3 4 5 6 7 8 9

2

0 1 2 3 4 5 6 7 8 9

3

0 1

Header

Length SrbStatus ScsiStatus

CdbLength SenseInfoExLength DataIn Reserved

SrbFlags

DataTransferLength

SenseDataEx

….

…

…

…

DataBuffer (variable size)

…


Check Connection (0x02000003)

When: Periodically, at client application request

Why:

Ensure that the connection is still alive

Ensure that the disk is currently accessible


Check Connection

0 1 2 3 4 5 6 7 8 910 1 2 3 4 5 6 7 8 9

20 1 2 3 4 5 6 7 8 9

30 1

Header


Get Disk Information (0x02000005)


Why: To get extended SCSI status information

about the failure


Get Disk Information

0 1 2 3 4 5 6 7 8 9

1

0 1 2 3 4 5 6 7 8 9

2

0 1 2 3 4 5 6 7 8 9

3

0 1

Header

DiskType

DiskFormat

BlockSize

LinkageID

…

…

…

IsMounted Reserved

FileSize

…

VirtualDiskId

…

…

…


Validate Disk (0x02000006)


Why: To ensure that the disk is fully valid for IO

operations


Validate Disk

0 1 2 3 4 5 6 7 8 910 1 2 3 4 5 6 7 8 9

20 1 2 3 4 5 6 7 8 9

30 1

Header

IsValid Reserved



Moving large amounts of data

SMB optimizes File Read & File Write over

FSCTL

FSCTL 64KB Maximum Payload Size

40


Block addressing model

2 3

2 VirtualSectorSize3 VirtualSectorSize

1536B1024B


SCSI has a lot of error information


Extended Error Handling

Client Server

IRP_MJ_READ / WRITE

Key = STATUS_SVHDX_ERROR_STORED

IRP_MJ_FILE_SYSTEM_CONTROL(RSVD OpCode == RSVD_TUNNEL_SRB_STATUS_OPERATION,

ErrorKey = Key & 0xFF)

SRB Status, SCSI Status, Sense Data


Get SRB Status (0x02000004)

When: After a read or write has failed with

certain error codes

Why: To get extended SCSI status information

about the failure


Get SRB Status

0 1 2 3 4 5 6 7 8 9

1

0 1 2 3 4 5 6 7 8 9

2

0 1 2 3 4 5 6 7 8 9

3

0 1

Header

StatusKey

SrbStatus ScsiStatus SenseInfoExLength Reserved

SenseDataEx

…

…

…

…


STATUS_SVHDX_ERROR_STORED

00 FF

The last byte is passed to Get SRB Status



File Size

Valid Data Length

End of File



Sessions terminated through CloseFile



FSCTL_QUERY_SHARED_VIRTUAL_DISK_SUPPORT

0 1 2 3 4 5 6 7 8 910 1 2 3 4 5 6 7 8 9

20 1 2 3 4 5 6 7 8 9

30 1

SharedVirtualDiskSupport (0x01)

CurrentSharingStatus


STATUS_VHD_SHARED

•

•


Resources

[MS-RSVD]: Remote Shared Virtual Disk Protocol

Specification

TechEd

Application Availability Strategies for the Private Cloud

TechNet

Deploy a Guest Cluster Using a Shared Virtual Hard Disk

Virtual Hard Disk Sharing Overview

SCSI

SCSI Primary Commands – 3 (SPC-3)

SCSI Block Commands – 2 (SBC-2)

http://msdn.microsoft.com/en-us/library/dn393384.aspx

http://channel9.msdn.com/Events/TechEd/NorthAmerica/2013/MDC-B311

http://technet.microsoft.com/en-us/library/dn265980.aspx

http://technet.microsoft.com/en-us/library/dn281956.aspx


Q&A




Compatibility

•

•

•

•

•

•

•

•

•

•

•

•


Compatibility

•

•

•

•

•


Blocked things

Unaligned IO

Cached IO

Oplocks

IRP_MJ_ACQUIRE_FOR_SECTION_SYNCHR

ONIZATION

Direct IO

Copy Offload (for SVHDX files)



Troubleshooting

Performance counters

Events

Filter Manager (FLTMC.EXE)


Performance Monitor


Event Viewer

Get-WinEvent -LogName Microsoft-Windows-Hyper-V-Shared-VHDX/Operational

Get-WinEvent -LogName Microsoft-Windows-Hyper-V-Shared-VHDX/Reservation


Filter Manager (FLTMC.EXE)


Unsupported Bonus Scenario

Single host Shared Virtual Disk for test/dev/learning scenarios

Install Windows Server 2012 R2 on the host

Enable Failover Clustering feature

o Install-WindowsFeature Failover-Clustering

o No need to actually create a cluster for this scenario

Manually attach the Shared Virtual Disk filter

o FLTMC.EXE attach svhdxflt X:

o Use the step-by-step blog post to try it out… Not a supported scenario

Not really highly available

!

http://blogs.technet.com/b/josebda/archive/2013/07/31/windows-server-2012-r2-storage-step-by-step-with-storage-spaces-smb-scale-out-and-shared-vhdx-virtual.aspx



Hyper-V Cluster

Hyper-V

Host 1

Component view - CSV

Storage

VSC

Boot VM1

Storage

VSPVHD Stack

Physical

NIC

Shared

Data

Shared VHDX

Parser

Shared VDHX Filter

CSV

Hyper-V

Host 2Storage

VSC

Boot VM2

Storage

VSP

CSV2

VHD Stack

Physical

NIC

Shared

Data

Shared VHDX

Parser

Shared VDHX Filter

CSV

Guest Cluster

CSV1


Hyper-V ClusterHyper-V Cluster

Hyper-V

Host 1

Component view - SMB

Storage

VSC

Boot VM1

Storage

VSPVHD Stack

SMB Client

Shared

Data

Shared VHDX

Parser

Hyper-V

Host 2Storage

VSC

Boot VM2

Storage

VSPVHD Stack

SMB Client

Shared

Data

Shared VHDX

Parser

Guest Cluster

VHD Stack

SMB Server

Shared VDHX

Filter

CSV

SMB Server

FS2

VHD StackShared VDHX

Filter

CSV

FS1


Shared VHDx Open

•

•

•

•

•

Documents

Tunneling SCSI over SMB: Shared VHDX files for Guest Clustering in Windows Server 2012 … · 2019-12-21 · For Hyper-V hosts and File Server Must be running Windows Server 2012