Upload
bennial
View
28
Download
0
Embed Size (px)
Citation preview
1
Veritas Dynamic Multi-PathingOptimized, Centrally Managed Multi-Pathing
Ashish YajnikPrincipal Product Manager
Storage & Availability Mgmt GroupDynamic Multi-Pathing
SYMANTEC VISION 2011
Datacenter Multi-Pathing Trends & Challenges
Multi-Vendor storage & server environments increasingly a norm
Single tools across physical & virtual env critical for efficiency
Configurations are continually scaling up & scaling out
Bridging System & Storage Team Communication is a challenge
2Dynamic Multi-Pathing
SYMANTEC VISION 2011
DMP - Optimized Multi-Vendor Multi-Pathing Solution
• One interface for all OS, storage & architecture
– Enables multi-vendor environment & standardization
– Industry’s broadest hardware compatibility list
– ~1000 arrays supported
• Enterprise grade performance
– Best in class throughput
– Proactive path management
– Automated & reliable recovery
• Bridges storage & server boundaries
– Enables informed decision making
– Reduces risk of configuration errors
• Managed by Veritas Operations Manager
– Single-pane-of-glass datacenter wide path management
3Dynamic Multi-Pathing
SYMANTEC VISION 2011
Agenda
Dynamic Multi-Pathing 4
Advanced Usability & ManagementBridge the Storage & Server Information GapCentrally manage multi-pathing with Veritas Operations Manager (VOM)Identify bottleneck with I/O Stat tool
Maximize Availability & PerformanceProactive Path Failover & Reliable FailbackEffective Load Balancing
HeterogeneityStandardize across physical & virtual environmentsLeverage ANY StorageStandardize across any OS & any Storage Management Stack
SYMANTEC VISION 2011Dynamic Multi-Pathing 5
Advanced Usability & ManagementBridge the Storage & Server Information GapCentrally manage multi-pathing with Veritas Operations Manager (VOM)Identify bottleneck with I/O Stat tool
Maximize Availability & PerformanceProactive Path Failover & Reliable FailbackEffective Load Balancing
HeterogeneityStandardize across physical & virtual environmentsLeverage ANY StorageStandardize across ANY OS & Storage Management Stacks
SYMANTEC VISION 2011
Clustering
Native MP
Volume Manger
File System
Veritas Cluster Server
Volume Manger
File System
DMP
Veritas Cluster Server
DMP
Veritas Storage Foundation
Exploiting OS Native Stacks:Introducing DMP Standalone Product
Dynamic Multi-Pathing 6
Enable DMP EVERYWHERE
Simpler for You to Standardize on DMP
• Leverage DMP’s extensive array support
• Standardize on single multi-pathing across ALL storage management stacks
• DMP reporting and management through VOM
• Seamless migration from native multi-pathing to DMP
Any OS Stack
SYMANTEC VISION 2011
Exploiting OS Native Stacks:DMP Standalone SCL and HCL Overview
• Enabled by DMP Native Support feature
• Oracle ASM & RAC supported with DMP Standalone
• Leverages DMP’s existing HCL – 1000+ arrays supported
7
DMP Single node Native Stack
DMP
Native FS
Native LVM
Native Multi-Pathing
Native FS
Native LVM
DMP Native Stack with Clustering
DMP
VCS
Native FS
Native LVM
Native FS
Native LVM
VCS
Native Multi-Pathing
OS Version Architecture Clustering Storage Management Stack
AIX 5.3 & 6.1 PowerPC VCS AIX LVM, ASM
RHEL4/5, SLES10, SLES11 Intel/AMD X64 (no 32 bit) VCS Linux LVM, ASM
Solaris 9, 10 SUN Sparc & X86_64 VCS SUN ZFS, ASM
HP-UX 11.31 IA & PA VCS HP-UX LVM
Windows 2003, 2008 X86, X64, IA64 VCS, MSCS Windows LDM
Dynamic Multi-Pathing
Any OS Stack
SYMANTEC VISION 2011
Locked-In to Storage Vendors
Exploiting ANY Storage:Eliminate Vendor Lock-In With Dynamic Multi-Pathing
Multi-Vendor Benefits
• Enabling Multi-Vendor environments saves on storage cost
• Point tools are complex to manage or don’t support all environments
• Standardize on DMP across heterogeneous OS & Storage solutions
DMX Clariion USP-V
HDLMEMC PowerPath
Dynamic Multi-Pathing 8
Storage vendor lock-in
results in 30-100% price
premiums on storage
Free to Choose the Most Cost Effective Storage
DMX Clariion IBM XIVHDS – AMS
Dynamic Multipathing (DMP)
Any Storage
SYMANTEC VISION 2011
Exploiting ANY Storage: Modular Architecture to Optimize to Any Storage
9
• Challenge for Optimization
– Different Array modes are hard to support
• Array Support Library (ASL)
– HW specific plug in to DDL
– Automate DMP driver configuration
– Discover array specific attributes
• Array Policy Module (APM)
– HW specific plug in to DMP
– Dynamically loadable kernel module
– Allows array-specific integration
• Allows DMP to support 50+ array families & 1000+ array models
Dynamic Multi-Pathing
DDL vxdmp
ANY VM
ANY FS
SCSI Drivers
HBA Drivers
DISK
APM2
APM1
ASL1 ASL2
Any Storage
SYMANTEC VISION 2011
Exploiting Virtualization:Enhanced VIOS Support with DMP
Dynamic Multi-Pathing 10
LPAR LPAR
Virtualized AIX Server
VIO Server
DMP
VIO Server
DMP
DMP DMP
• DMP Enhances VIOS
– Enable standardization of DMP on AIX
– DMP resides inside the VIOS
– Ability to run DMP inside LPAR as well
• Leverage extensive DMP HCL
• Deep Visibility & Reporting
– DMP in LPAR gets deep storage visibility
– Enables consistency of device names
– Informed decision making
– Centrally managed by VOM
Deep Storage Visibility
Virtual Support
SYMANTEC VISION 2011
Summary of DMP’s Heterogeneous Support
Dynamic Multi-Pathing 11
• All Major Unix/Linux Support
• Windows LDM Support
• Oracle ASM support with DMP Standalone
OS & Native Stack
Support
• Adding value in VIOS environments
• Multi-Pathing in LDOMs, LPARs & KVM
• Utilize DMP across Physical & Virtual environments
Virtualization Support
• All Tier1, Tier2 & major Tier3 arrays supported
• 1000~ arrays part of extensive HCL
Extensive Storage Support
SYMANTEC VISION 2011Dynamic Multi-Pathing 12
Advanced Usability & ManagementBridge the Storage & Server Information GapCentrally manage multi-pathing with Veritas Operations Manager (VOM)Identify bottleneck with I/O Stat tool
Maximize Availability & PerformanceProactive Path Failover & Reliable FailbackEffective Load Balancing
HeterogeneityStandardize across physical & virtualLeverage ANY StorageStandardize across any OS
SYMANTEC VISION 2011
Bridging the Storage & System Information GapMapping OS Devices to SAN Devices
13
c0t5d65
c1t5d65
c4t5d65c3t5d65
c0t5d65 c3t5d65
Challenges
• Same LUN, different name across servers
• OS names do not map to storage terminology
c0t5d65
c1t5d65
c4t5d65c3t5d65
LSN: D600172E015B
EMC0_19 EMC0_19
LUN
c0t5d65
c1t5d65
c4t5d65c3t5d65
AVID: 0AF4
EMC0_0AF4 EMC0_0AF4
WITHOUT DMP
LUN LUN
LSN sorting algorithm
• Consistent names across servers
AVID naming
• Consistent names across servers
• Meaningful names for storage teams
Dynamic Multi-Pathing
Manageability
SYMANTEC VISION 2011
$>cat /etc/vx/device.custom
enclosure vendor=DGC product=DISK serial=CK200070400359 name=CLARiion
dmpnode serial=600601606D121B007C778BC48EDBDB11 name=MyApp-lun1
dmpnode serial=600601606D121B008FB6E0CA8EDBDB11 name=CLARiion-lun2
dmpnode serial=600601606D121B0091B6E0CA8EDBDB11 name=CLARiion-lun3
dmpnode serial=600601606D121B0093B6E0CA8EDBDB11 name=CLARiion-oralun1
dmpnode serial=600601606D121B00BFAB99C28FDBDB11 name=CLARiion-oralun2
Customized Device Naming
14
• Ability to change device names to user-specified names
– Admin can decide the names for the devices
– Framework for a file containing user-specified names to be uploaded to hosts
– Ability to clear the user-specified names with a CLI
– User-specified names will be made persistent
EMC Clariion
LUN
EMC Clariion
LUN
EMC Clariion
LUN
EMC Clariion
LUN
MyApp-lun1 CLARiion-lun2 CLARiion-oralun1 CLARiion-oralun2
#> vxgetdmpnames > /etc/vx/device.custom#> vxddladm assign names file=/etc/vx/device.custom
• Vxdisk list output
DEVICE TYPE DISK GROUP STATUS
MyApp-lun1 auto:cdsdisk - - online
CLARiion-lun2 auto:cdsdisk - - online
CLARiion-lun3 auto:cdsdisk - - online
CLARiion-oralun1 auto:cdsdisk - oradg online
CLARiion-oralun2 auto:cdsdisk - oradg online
SYMANTEC VISION 2011
Bridging Storage & Server Team Information Gap
Dynamic Multi-Pathing 15
Raid Levels – RAID5, RAID10Replication Attributes & SnapshotsThin Or Standard DevicesTier Information – SSD, SATA, FC, SAS
5
DMP Discovers Device Characteristics
My application is on tier 1 servers, what tier of storage is it using? SSD, SATA, SAS, FC?App
Team
Server Team
My mission critical file system is being protectedor not? RAID5, RAID10?
Is my critical data being replicated underneath or not?
#vxdisk –e list
DEVICE TYPE DISK GROUP STATUS OS_NATIVE_NAME ATTR
USP0_af14 auto USP0_53 DFDG2 online c0t50060482CAE053ECd25s2 PVOL,SSD
USP0_af15 auto USP0_54 DFDG2 online c0t50060482CAE053ECd26s2 SATA,RAID_1
USP0_af16 auto USP0_55 DFDG2 online c0t50060482CAE053ECd27s2 FC,RAID_10
Manageability
SYMANTEC VISION 2011
Proactively address risks to prevent downtime
• Reduce errors in your environment & increase HA
– Manage risks in your environment with proactive reporting & alerts
– Customized health check reports for different violations
16Dynamic Multi-Pathing
Manageability
SYMANTEC VISION 2011
Better environment control with DMP I/O Statistics
17
New options in vxdmpadm provides:
• Filter out ‘0’ entries
• vxdmpadm –z iostat show <selection-criteria>
• Displayed queued (-q) and errored (-e) I/Os
• vxdmpadm –q iostat show <selection-criteria>
• Per second IO statistics
• vxdmpadm –s iostat show <selection-criteria>
• Display output in units that make sense for normal admins
• vxdmpadm –u iostat show <selection-criteria>
Aggregate statistics for capacity management
• HBA port level statistics
• Groupby option to get enclosure, controller, and disk level statistics
• vxdmpadm iostat show groupby=ctlr|dmpnode|enclosure <selection criteria>
5.1 CLI Scalability & Accuracy Improvements
• Filtering & grouping of data in the kernel
• Reduces burden on CLI thus improving reporting accuracy
Dynamic Multi-Pathing
Manageability
SYMANTEC VISION 2011 18
$>vxdmpadm -u m –s iostat show groupby=ctlr interval=30
cpu usage = 878us per cpu memory = 32768b
OPERATIONS/SEC BLOCKS/SEC
CTLRNAME READS WRITES READS WRITES
c0 33200 11600 207.25m 248.13m
c1 33400 9040 233.24m 229.73m
c2 17600 5040 61.53m 64.62m
c3 18080 5680 98.45m 99.48m
$>vxdmpadm -u m -s iostat show groupby=enclosure ctlr=c1 interval=30
cpu usage = 1396us per cpu memory = 32768b
OPERATIONS/SEC BLOCKS/SEC
ENCLOSURENAME READS WRITES READS WRITES
emc_clariion0 17200 4000 118.97m 73.65m
ams_wms0 12000 4400 82.07m 80.01m
eva4k6k0 4000 4000 30.76m 73.65m
Better environment control with DMP I/O Statistics
Dynamic Multi-Pathing
Manageability
SYMANTEC VISION 2011Dynamic Multi-Pathing 19
Advanced Usability & ManagementBridge the Storage & Server Information GapCentrally manage multi-pathing with Veritas Operations Manager (VOM)Identify bottleneck with I/O Stat tool
Maximize Availability & PerformanceProactive Path Failover & Reliable FailbackEffective Load Balancing
HeterogeneityStandardize across physical & virtualLeverage ANY StorageStandardize across any OS
SYMANTEC VISION 2011
Optimize Application I/O Throughput
20
• MinimumQueue (MinQ) I/O Policy
– Smallest HBA queue depth path chosen
– Typical work load & SAN configurations
• Adaptive I/O Policy
– Paths with lowest response time are used
– End-to-End bottleneck identification
• Adaptive-MinQ
– Combines best of MinQ & Adaptive
• Balanced
– LUN segmented in equal size partitions,
– Each partition associated with a path
• Round-Robin, Single-Active, Priority
– Standard I/O policies for native multi-pathing
Dynamic Multi-Pathing
HBA - A HBA - B
SP-A SP-B
Maximize Performance
SYMANTEC VISION 2011
Proactive Path Management:Smarter & Faster I/O Error Analysis
21
• Device Discovery Layer (DDL)
– Multi-threaded for faster discovery
• DMP Device Driver (VxDMP)
– Multi-threaded core processes
• Normal I/Os go through SCSI layer
– Chance for SCSI to recover transient errors
• DMP Error Analysis bypasses SCSI layer
• “SCSI Bypass” Benefits:
– Faster error analysis
– Asynchronous interface
– More detailed information of I/O outcome
Dynamic Multi-Pathing
DMP
VxVM/ ANY VM/ASM
VxFS/ANY FS
HBA Driver
OS SCSI Drivers
ErrorAnalysis
RegularIO
Maximize Availability
SYMANTEC VISION 2011
Proactive Path Management
22
Throttling
• If I/O is backed up on the device for 10 seconds, mark the path “Suspect”
HBA API Notifications
• When Port logout notification happens, mark the associated paths “Suspect”
Sub-Path Failover Group Protocol
• Collectively taken an action by inquiring about few paths
• Good for scale out configurations and results in speedier analysis
Dynamic Multi-Pathing
Maximize Availability
DMP Proactive Path Management Capabilities
– Avoids build-up of I/O on failing paths by proactive methods
– Results in quicker I/O error analysis and speedier failover
SYMANTEC VISION 2011
Total Throughput Impact of a Fabric Failure
4096 LUN Paths Fail at Once, 2048 of them are Primary Paths with Live I/O
DMP Minimum Queue
0.00
1000.00
2000.00
3000.00
4000.00
5000.00
6000.00
0:00
:00
0:00
:20
0:00
:40
0:01
:00
0:01
:20
0:01
:40
0:02
:00
0:02
:20
0:02
:40
3:00
:00
0:03
:20
0:03
:40
0:04
:00
0:04
:21
0:04
:40
0:05
:00
0:05
:20
0:05
:40
0:06
:00
0:06
:20
0:06
:40
0:07
:00
0:07
:20
0:07
:40
0:08
:00
0:08
:20
0:08
:40
0:09
:00
0:09
:20
0:09
:40
0:10
:00
Run time (hh:mm:ss)
KB
/sec
Proactive Path Management:Maximize Uptime in Scaled Up Config
• 4 ports per controller, each acting as primary port for 128 LUNs. 1024 LUNs total (128 * 8).
• Workload: multiple vxbench, single striped volume, ensuring that all 4096 primary paths carry traffic.
• Each Fabric contains 2 switches. Fabric A failed by disabling all Inter Switch Links
Tier1 - ALUA
1024 LUNs, 8 Paths per LUN
A B
Failure of Fabric A
DMP Fails over of 4096
LUN-paths in 40 seconds
23
Maximize Availability
SYMANTEC VISION 2011
Real-Time & Reliable Path Recovery
24
• DMP Path Recovery is in Kernel for reliability
• DMP regularly checks the state of failed and idle paths
– Successful SCSI inquiry will cause DMP to bring the path back online
– SCSI inquiry failure will cause DMP to disable the path
• Path checking is triggered:
– By fabric events: a port login event triggers path recovery
– Utilizes Path Probing mechanism: Every 5 minutes (tunable)
Dynamic Multi-Pathing
Maximize Availability
SYMANTEC VISION 2011
Nimble Recovery of PathsLIPP - Low Impact Path Probing
25
• Ensure real time accurate path state awareness
• Optimized utilization of system resources
• Utilizes Sub-path Failover Groups (SFG)
• LIPP groups paths into logical entities
• Health of few paths determines the health of the rest in the group
• Reduces CPU overhead & avoids flooding of SAN with probes
• Tuning LIPP
• Tunable (dmp_low_impact_path_probe) to turn “on/off” LIPP
• Ability to change number of probes sent to “idle” or “disabled” paths
Dynamic Multi-Pathing
Maximize Availability
SYMANTEC VISION 2011
DMP - Optimized Multi-Vendor Multi-Pathing Solution
• One interface for all OS, storage & architecture
– Enables multi-vendor environment & standardization
– Industry’s broadest hardware compatibility list
– Utilize DMP to enable more arrays
• Enterprise grade performance
– Best in class throughput
– Proactive path management
– Utilize faster failover & effective load balancing
• Bridges storage & server boundaries
– Enables informed decision making
– Utilize VOM & DMP for deep visibility
• Managed by Veritas Operations Manager
– Download VOM & utilize it for storage/path management
26Dynamic Multi-Pathing
Thank you!
Copyright © 2011 Symantec Corporation. All rights reserved. Symantec and the Symantec Logo are trademarks or registered trademarks of Symantec Corporation or its affiliates in the U.S. and other countries. Other names may be trademarks of their respective owners.
This document is provided for informational purposes only and is not intended as advertising. All warranties relating to the information in this document, either express or implied, are disclaimed to the maximum extent allowed by law. The information in this document is subject to change without notice.
27
Ashish Yajnik - [email protected]
Chad Bersche – [email protected]
Dynamic Multi-Pathing