Upload
srinivas-gollanapalli
View
53
Download
6
Embed Size (px)
DESCRIPTION
dmx performance and tuning
Citation preview
1© Copyright 2008 Interscape Technologies Inc. All rights reserved.
Alok Jain
Performance Aspects of Open Systems on
EMC Platforms (Symmetrix)
09/12/08
2© Copyright 2008 Interscape Technologies Inc. All rights reserved.
Agenda
�Storage Area Networks from Performance Perspective
�Open Systems and Performance
�EMC Storage Platforms – Symmetrix
�Open Systems Tools
�Performance Metrics for Open Systems
�Performance Metrics for EMC Storage
�Performance Analysis using tools
�Storage Consolidation Planning & Considerations
�Use Case Study (time permitting)
3© Copyright 2008 Interscape Technologies Inc. All rights reserved.
Storage Performance (SNIA)
�Most data centers are dealing with ever increasing amounts of data storage. Although the need for storage seems insatiable, array performance typically plateaus or worse, degrades post installation. Storage performance tuning is a necessary and ongoing activity…..
4© Copyright 2008 Interscape Technologies Inc. All rights reserved.
Storage Area Networks and Performance
Storage IO performance issues can occur at any of these layers.
Data replication to remote site
5© Copyright 2008 Interscape Technologies Inc. All rights reserved.
Open Systems and Storage Performance
HARDWAREHARDWARE
DRIVER
LVMLVM
File System
SYMMETRIX
Path Manager
Host
Application
Symmetrix sees something very different than the initial size of the I/O!
Hardware delivers the I/O as requested
Driver may break up I/O (max_phys and other parameters)
Path manager may affect I/O channel and I/O size
File System changes I/O to conform to File System block size, fragment size, or memory page size
Database changes I/O subject to db_block_size, multiblock_read_count, and other parameters. Database
Application performs an I/O
SYMMETRIX
LVM breaks up according to stripe width & I/O parameter settings
6© Copyright 2008 Interscape Technologies Inc. All rights reserved.
Application and System Performance – Key Metrics
�Three Ways to Measure Application Performance
� Throughput (MB/s)
� Number of transactions per second
� Application run time
�Four Ways to Measure IO Sub-System Performance� IO/sec (IO operations per sec)
� MB/sec (Bytes transferred/sec)
� I/O Service Times (average time to do an I/O)
� I/O Queue Lengths (outstanding IO waiting to be sent)
7© Copyright 2008 Interscape Technologies Inc. All rights reserved.
Types of IO & Performance Impacting Points
�There is a limited number of basic I/O operations� Random read
� Sequential read
� Random write
� Sequential write
�Each type has “cache hit” and “cache miss” variants
�Block Size impacts the throughput (IO/sec) and Bandwidth (MB/s)
8© Copyright 2008 Interscape Technologies Inc. All rights reserved.
Application Work Load Characteristics
�All applications perform the following:� Read/Write data to/from disk systems (milliseconds)
� Process data in CPU (microseconds)
� Write data back to disk system (milliseconds)
Performing an IO is an order of magnitude time consuming compared to a CPU operation
�You need to know� Overall I/O rate
� % Read + read size
� % Write + write size
� % Random, % sequential
� % Cache hit (locality of reference)
9© Copyright 2008 Interscape Technologies Inc. All rights reserved.
Physical Drives – Key Consideration
� Number of disks and speed of disks can limit IO Subsystem performance
� The 7.2K RPM drives are good for sequential and large I/O sizes. A little slower at random access than the 10K RPM drives
� The 10K RPM drives give good, all-around performance
� 15K RPM Drives boost random I/O access rates� 30% more random-access, small transfers than the 10K RPM
drives
� They do not benefit much in large (64KB +) or sequential operations, as transfer rates are not as affected by rotational speed
� If I/O is hitting the cache, they’re all the same speed� For writes, disk type is unimportant as long as write cache is not
saturated
� Prefetch works at about the same rate for all of them
� What differs is how well they do random reads, and how fast theyflush random writes from the cache
10© Copyright 2008 Interscape Technologies Inc. All rights reserved.
Drives can be roughly divided into “families”
�The major distinction is the rotational speed� Faster rotation generally also means faster seek
Not Fast
146GB 15k
RPM
73GB 15k
RPM
300GB 10k
RPM
146GB 10k
RPM
73GB 10k
RPM
500GB 7200
RPM
Drive Capacity (Gbyte) 146 73 300 146 73 500
Internal data rate (MB/sec) 68 - 114 68 - 114 47 - 94 47 - 94 47 - 94 47 - 94
Disc Rotational Speed (r/min) 15,000 15,000 10,000 10,000 10,000 7,200
Avg Rotational Latency (ms) 2.00 2.00 3.00 3.00 3.00 4.17
Avg Read Seek (ms) 3.50 3.50 4.70 4.70 4.70 8.50
Avg Write Seek (ms) 4.00 4.00 5.40 5.40 5.40 9.50
Read Single Track Seek (ms) 0.27 0.27 0.20 0.20 0.20 0.80
Write Single Track Seek (ms) 0.45 0.45 0.50 0.50 0.50 1.00
Read Full Stroke Seek (ms) 7.40 7.40 9.70 9.70 9.70 16.00
Write Full Stroke Seek (ms) 7.90 7.90 10.40 10.40 10.40 16.55
Fastest Fast
Specification
11© Copyright 2008 Interscape Technologies Inc. All rights reserved.
RAID Level Considerations for IO Sub-Systems
High data availability. High cost. Best performance for random reads & writes
Striped + MirroredRAID 1+0
Highest data availability. Low cost. Slower than R5 performance
Striped set with dual distributed parity
RAID 6
High data availability. Lower cost. Cost effective performance
Striped set with distributed parity
RAID 5
Higher data availability. High cost. Best Sequential IO performance
Mirrored setRAID 1
Lowest data availability. Lowest cost. Highest performance. Not Supported in EMC Symmetrix
Striped set without parityRAID 0
CommentsRAID Group Configuration
RAID Level
12© Copyright 2008 Interscape Technologies Inc. All rights reserved.
RAID 1
� Writes go to two drives
� Read from 1 drive� DMSP policy
� High level of protection
� Most expensive $$$� 2x raw storage
Channel
ChannelDirector
Directory
�Global Memory
DiskDirector
Disk
Disk
M1
M2
EMC Symmetrix – RAID 1 (Sample Symm View)
13© Copyright 2008 Interscape Technologies Inc. All rights reserved.
Striping – At Host Level & Array level (Plaids for Best Performance)
�LUN = Logical Unit Number (drive letter/#)
�LUNs striped across multiple RAID groups (of same type)� Eliminate hot RAID groups, hot drives, hot back-end
links
� Variations include, RAID 0+1, 1+0, 5+0, 10, or 50
14© Copyright 2008 Interscape Technologies Inc. All rights reserved.
Open Systems Performance Measurement Tools
GUI based (with CLI options also)
PerfmonWindows
Iostat –x 5
Sar –d (many options)Iostat & sarAIX
Iostat –xt 5
Sar –d (many options)Iostat & sarLinux
Iostat –xtczn –T d 5
Sar –d (many options)Iostat & sarSolaris
Command ExampleToolOS Platform
15© Copyright 2008 Interscape Technologies Inc. All rights reserved.
Iostat – Unix Based Systems (Solaris)
ThroughputReads/sec & Writes
Cmds in Wait queue
Cmds activelyBeing executed
Wait tiimeIn queue
Actual ServiceTime
BandwidthRead & Write KB/sec
From Performance perspective the bottlenecks can be identified by looking at Wait Queues, Wait Service Time and
Active Service Time
16© Copyright 2008 Interscape Technologies Inc. All rights reserved.
Iostat – Unix Based Systems (Solaris)
�Performance Throughput Indicators� Reads/sec & Writes/sec
� Read Kbytes/sec & Write Kbytes/sec
�Performance Bottlenecks Indicators� Active Service Time – This can be application
dependent. It should be compared to a baseline to determine the changes when the problem occurs
� Wait Queues present – signifies IO sub-system is not able to keep up with demand at the LUN level
� Wait Service Time – Any wait service time will introduce IO latency
17© Copyright 2008 Interscape Technologies Inc. All rights reserved.
Sar – Unix based Systems (Solaris)
AvgQueues
Throughput Total IO/sec
Bandwidth total blocks/sec+
Avg. WaitTime (ms)
Avg. ServiceTime (ms)
SAR –d output. The metrics are different compared to IOSTAT output. Reads and writes data is shown together
%busy indicator is indicative only.
18© Copyright 2008 Interscape Technologies Inc. All rights reserved.
Perfmon – Windows Based Systems
Windows based systems can use Perfmon to do performance analysis
19© Copyright 2008 Interscape Technologies Inc. All rights reserved.
Windows Host – EMC Symmetrix Array
Windows Host Drives mapped to EMC Symmetrix physical disks. The color shows how utilized the physical disk spindles are in terms of IO performance.
This provides a comprehensive view of Host devices in the Storage Array
20© Copyright 2008 Interscape Technologies Inc. All rights reserved.
EMC Storage Platforms Performance Measurement Tools
GUI based
Navicli –h ip_address …..
Navisphere Analyzer
Navicli (CLI)Clariion
WLA is GUI based
Symstat –sid 1234 –c 5 –I 5
Work Load Analyzer (WLA)
Symstat (CLI)Symmetrix
ExampleToolEMC Platform
21© Copyright 2008 Interscape Technologies Inc. All rights reserved.
Symmetrix Components from Performance Perspective
� Symmetrix DMX Critical Component� %Cache Hits
� Fiber Adapter Channels
� Disk Directors
� Remote Fiber Director
� Device Utilizations
� Disk Utilizations (physical)
� Symmetrix Key Performance Metrics (at eac component level)� Read/write IO/sec
� Read/write Kbytes/sec
� %cache hits
� Block Sizes
� Write Pending Slot Counts
22© Copyright 2008 Interscape Technologies Inc. All rights reserved.
EMC Symmetrix Global Memory Management & Data Flow
23© Copyright 2008 Interscape Technologies Inc. All rights reserved.
EMC DMX-4 Symmetrix Array Architecture
Cache
Front-EndTo Hosts
Back-EndTo Disks
24© Copyright 2008 Interscape Technologies Inc. All rights reserved.
EMC Symmetrix – Tiering within arrays
25© Copyright 2008 Interscape Technologies Inc. All rights reserved.
EMC DMX-4 Symmetrix Disk Quadrant Layout
All drive types in all quadrants to leverage all Disk Adapters for Tiering
26© Copyright 2008 Interscape Technologies Inc. All rights reserved.
EMC Symmetrix – Performance Viewer (Work Load Analyzer)
27© Copyright 2008 Interscape Technologies Inc. All rights reserved.
EMC Symmetrix IO/sec & MB/sec Metrics
Monitor overall IO/sec and MB/sec for the whole array. Confirm the storage array is not over loaded
Reads MB/sec, Write MB/sec, Total MB/sec
Read IO/sec, Write IO/sec, Total IO/sec
28© Copyright 2008 Interscape Technologies Inc. All rights reserved.
EMC Symmetrix Write Pending Slots Vs. Max WP Slots
Monitor overall Write Pending slot counts for the whole array. Crossing Max thresholds will impact performance
29© Copyright 2008 Interscape Technologies Inc. All rights reserved.
EMC Symmetrix %Cache Hits
Monitor %Cache Hits. For Open Systems anything below 70% may impact performance negatively
30© Copyright 2008 Interscape Technologies Inc. All rights reserved.
EMC Symmetrix – Front-End Fiber Adapter CPU IOPS
Monitor Fibre Adapter CPU IO/sec on each individual FA CPUIdentify busiest FA CPUs
31© Copyright 2008 Interscape Technologies Inc. All rights reserved.
EMC Symmetrix – Front-End Fibre Adapter FA Port Utilization (based on IO/sec)
Identify FA Ports that are highly utilized. Anything over 90% may be of concern
32© Copyright 2008 Interscape Technologies Inc. All rights reserved.
EMC Symmetrix – Remote Adapters IOPS & MBPS (used for remote replication)
Identify RA Ports that are doing heavy IOs. Generally most RA’s will have similar IO profiles
33© Copyright 2008 Interscape Technologies Inc. All rights reserved.
EMC Symmetrix – Disk Adapters (DA)
Identify DA Ports that are doing heavy IOs. Generally most DA’s will have similar IO profiles if the hardware configuration is setup properly.
34© Copyright 2008 Interscape Technologies Inc. All rights reserved.
EMC Symmetrix – Devices – Sampled Average read and write times
Identify devices that are doing very high reads and writes. These devices then need to be mapped to the hosts that they are provisioned to
35© Copyright 2008 Interscape Technologies Inc. All rights reserved.
EMC Symmetrix – Disk SCSI Command Loads
Identify physical disks that are very busy. Any 10K RPM disk doing over 120-140 IOPS points to hot disks which will create bottlenecks
36© Copyright 2008 Interscape Technologies Inc. All rights reserved.
EMC Symmetrix – Disk to Device Mappings (Identify Bad Neighbors)
Identify busiest devices on the actual physical disks. This will point to hosts that may be starving other hosts for IO
37© Copyright 2008 Interscape Technologies Inc. All rights reserved.
EMC Symmetrix – Component %Utilization HEAT Chart
This provides a one page view into the Symmetrix array.The %utilization colors points to how hot each of the components are
38© Copyright 2008 Interscape Technologies Inc. All rights reserved.
Questions?
39© Copyright 2008 Interscape Technologies Inc. All rights reserved.
�Contact details
U.S. Office:200 Centennial Avenue,Suite 200, Piscataway, NJ 08854Office: 1.732.868.6320Fax: 1.973.302.8135
Alok JainCell: 848.248.0100Office: 732.868.6320Email id.: [email protected]
40© Copyright 2008 Interscape Technologies Inc. All rights reserved.
Storage Consolidation Analysis
Considerations & Use Case
41© Copyright 2008 Interscape Technologies Inc. All rights reserved.
IT Assets in General are Poorly Used
Typical Utilization Rates
Servers: 5-15%
PCs: 10-20%
Direct-attach storage: 20-40%
Network storage: 60-80%
A typical x86 serverconsumes between 30% and 40%of its maximum power when idle
Sources: VMware, Microsoft, EMC
42© Copyright 2008 Interscape Technologies Inc. All rights reserved.
Storage Consolidation Energy Benefits
�Storage consolidation saves energy
�Fewer and larger storage arrays� Consolidate and optimize
workloads
� Maximize storage utilization
�Requires understanding and planning� What are workloads?
� What are storage array capabilities?
� What are the performance characteristics?
� What are the data protection and business continuity requirements?
42
43© Copyright 2008 Interscape Technologies Inc. All rights reserved.
RAID 3+1 Annual Energy Cost Usable Terabyte vs
Drive Size
0.00
200,000.00
400,000.00
600,000.00
800,000.00
1,000,000.00
0 500 1000
Usable Terabytes (TB)
En
erg
y C
ost
@
$0.1
5/k
Wh
r
73 GB Yearly EnergyCost 3+1
146 GB YearlyEnergy Cost 3+1
300 GB YearlyEnergy Cost 3+1
500 GB YearlyEnergy Cost 3+1
Higher density disks brings the TCO down
44© Copyright 2008 Interscape Technologies Inc. All rights reserved.
Use Case: Storage Savings on Capital, Mgmt and Energy
Customer Example: Major Wireless Provider
1,296,480 kWh - Annual Energy Savings - $194,472
44 Mid-tier systems consolidated to 2 High-end systems
$222,066Annual
Cost @.15 kWh
$416,538
169 kWhPower and
Cooling317 kWh
High-end Array934 TB in 146 GB and 300 GB
drives
Mid-Tier885 TB in 146 GB drives
45© Copyright 2008 Interscape Technologies Inc. All rights reserved.
Use Case: Driving Factors Migration & Consolidation Planning
�Client is migrating Data Center form NY to NJ for various business reason
�Client would like to perform a detailed storage consolidation analysis as part of decision on migration strategy
�Client wants to look at their Tier-1 Storage environment which is mostly on EMC Symmetrix with SRDF Replication to another Data Center
�The consolidation analysis needs to model� Overall Storage Array Performance� Storage Capacity Consolidation Requirements� Technical Architecture at component levels
� LUN Limitations� Host Fan-Out ratios� Device masking thresholds
46© Copyright 2008 Interscape Technologies Inc. All rights reserved.
ALL EMC Symmetrix Performance – System IOPS & MBPS
S. No
Symmetrix
ID
Symmetrix
Pair Alias Location
Symm
Mode
Max
(IO/Sec)
95th
Percentile
(IO/Sec) Avg (IO/Sec)
Max
(MB/Sec)
95th
Percentile
(MB/Sec)
Avg
(MB/Sec) TimeStamp
1 190100031 190100032 NA DC 1 R1 35474 26244 16614 531 408 230 7/11/2008 21:452 190100011 190100012 NA DC 1 R1 32126 23554 12358 726 483 217 7/13/2008 12:45
3 190100015 190100016 NA DC 1 R1 30582 21398 12666 45 347 176 7/11/2008 22:154 187750019 187750020 NA DC 1 R1 24750 21812 12715 316 227 128 7/10/2008 22:155 187720023 187720024 NA DC 1 R1 16707 10049 5685 161 97 43 7/13/2008 0:45
6 187720025 187700026 NA DC 1 R1 14022 9341 4005 210 118 46 7/11/2008 22:007 187720027 187720028 NA DC 1 R1 13298 8976 4683 195 150 68 7/10/2008 21:15
8 187720013 187700014 NA DC 1 R1 12849 8696 4735 220 114 53 7/10/2008 21:309 287750029 287750030 NA DC 1 R1 12215 7796 4065 193 140 51 7/11/2008 22:30
10 187720021 187720022 NA DC 1 R1 11512 8739 4308 215 125 50 7/11/2008 21:30
11 187720017 187720018 NA DC 1 R1 9977 7741 4453 145 101 48 7/12/2008 2:3012 190100032 NA DC 2 R2 17003 11739 7147 249 168 78 7/11/2008 22:4513 190100016 NA DC 2 R2 14983 11123 6784 173 97 41 7/12/2008 19:30
14 190100012 NA DC 2 R2 11703 7677 3672 295 207 154 7/11/2008 23:1515 187750020 NA DC 2 R2 10755 6745 4174 90 70 28 7/12/2008 22:30
16 187720024 NA DC 2 R2 7720 3197 766 99 47 10 7/11/2008 20:1517 187720022 NA DC 2 R2 5764 2858 1381 27 18 8 7/12/2008 22:4518 287750030 NA DC 2 R2 5177 3356 1634 43 22 11 7/10/2008 22:30
19 187720018 NA DC 2 R2 5048 2180 971 24 18 6 7/10/2008 21:4520 187700014 NA DC 2 R2 2859 544 188 38 7 2 7/12/2008 19:4521 187720028 NA DC 2 R2 2387 1360 308 10 5 1 7/10/2008 20:15
22 187700026 NA DC 2 R2 212 67 46 3 0 0 7/11/2008 21:15
47© Copyright 2008 Interscape Technologies Inc. All rights reserved.
ALL EMC Symmetrix Performance – Max System IOPS & MBPS – R1+R2 View
Aggregate IOPS & MBPS for each interval across the pairs. This reflects a more realistic failed over view
14811510R1-R2 DC1 - DC2 2000S-M2 - 2000S-M2 0017-0018 11
21412927R1-R2 DC1 - DC2 2000S-M2 - 2000S-M2 0021-0022 10
15413213R1-R2 DC1 - DC2 2000S-M2 - 2000S-M2 0013-0014 9
21014066R1-R2 DC1 - DC2 2000S-M2 - 2000S-M2 0025-0026 8
16514388R1-R2 DC1 - DC2 2000S-M2 - 2000S-M2 0029-0030 7
9614621R1-R2 DC1 - DC2 2000S-M2 - 2000S-M2 0027-0028 6
15117205R1-R2 DC1 - DC2 2000S-M2 - 2000S-M2 0023-0024 5
32930579R1-R2 DC1 - DC2 2000P-M2 - 2000P-M2 0019-0020 4
88935214R1-R2 DC1 - DC2 DMX3-24 - DMX3-24 0011-0012 3
45939648R1-R2 DC1 - DC2 DMX3-24 - DMX3-24 0015-0016 2
72043278R1-R2 DC1 - DC2 DMX3-24 - DMX3-24 0031-0032 1
System Max Total MBPS (R1+R2)
System Max Total IOPS (R1+R2) Symm Mode Location Model Source Array No
48© Copyright 2008 Interscape Technologies Inc. All rights reserved.
Consolidation Analysis – Target Requirements
4+4RAID-1+010K400GBDMX-4T3
4+4RAID-1+010K300GBDMX-4T3
4+4RAID-1+015K300GBDMX-4T2
3+1RAID-515K300GBDMX-4T1
Target
RAID
Config
Target
RAID
Level
Target
Disk RPM
Target
Disk Size
Target
Model
Target
Config
ID
49© Copyright 2008 Interscape Technologies Inc. All rights reserved.
Symmetrix Performance IOPS & MBPS Profiles
50© Copyright 2008 Interscape Technologies Inc. All rights reserved.
Source Storage Arrays For Consolidation Targets
51© Copyright 2008 Interscape Technologies Inc. All rights reserved.
Target Storage Array Specifications
52© Copyright 2008 Interscape Technologies Inc. All rights reserved.
Source Storage #1 Performance Baseline
53© Copyright 2008 Interscape Technologies Inc. All rights reserved.
Source Storage #2 Performance Baseline
54© Copyright 2008 Interscape Technologies Inc. All rights reserved.
Symmetrix Sample – Aggregated System IOPS & MBPS
Sample shows two EMC Symmetrix IOPS and MBPS Consolidated View
55© Copyright 2008 Interscape Technologies Inc. All rights reserved.
Symmetrix Sample – Aggregated RA IOPS & MBPS
Sample shows two EMC Symmetrix Replication Channel Consolidated View
56© Copyright 2008 Interscape Technologies Inc. All rights reserved.
Symmetrix Sample – FA IOPS, MBPS, Masked Devices & FAN-OUT Analysis
Sample shown for two source array consolidation analysis view
57© Copyright 2008 Interscape Technologies Inc. All rights reserved.
Target Storage Array – Performance Heat Chart (Model)
58© Copyright 2008 Interscape Technologies Inc. All rights reserved.
Symmetrix Sample – Consolidated Target Storage Requirements
Sample shown for 2 source consolidation analysis view. Shows types of total device configs needed
59© Copyright 2008 Interscape Technologies Inc. All rights reserved.
ALL Symmetrix –2 to 1 Consolidation with RAID 1+0 – Overall Status
OK72%48030072043278Bi-DirDC1-DC2 DMX3-24 -DMX3-24 0031-0032 6
16514388Bi-DirDC1-DC22000S-M2 -2000S-M2 0029-0030 OK24%480300
9614621Bi-DirDC1-DC2 2000S-M2 -2000S-M2 0027-0028
5
21014066Bi-DirDC1-DC2 2000S-M2 -2000S-M2 0025-0026 OK27%480300
15117205Bi-DirDC1-DC22000S-M2 -2000S-M2 0023-0024
4
21412927Bi-DirDC1-DC2 2000S-M2 -2000S-M2 0021-0022 OK40%480300
32930579Bi-DirDC1-DC22000P-M2 -2000P-M2 0019-0020
3
14811510Bi-DirDC1-DC22000S-M2 -2000S-M2 0017-0018 OK60%480300
45939648Bi-DirDC1-DC2 DMX3-24 -DMX3-24 0015-0016
2
15413213Bi-DirDC1-DC22000S-M2 -2000S-M2 0013-0014 OK57%720300
88935214Bi-DirDC1-DC2 DMX3-24 -DMX3-24 0011-0012
1
Status
Target Storage
%Utilized
Recomm. Disk Count
Target Disk Size (GB)
System Max Total MBPS
(R1+R2)
System Max Total IOPS
(R1+R2) Symm Mode Location Model
Source Array Pair
Consol Group
60© Copyright 2008 Interscape Technologies Inc. All rights reserved.
Consolidation End Results
� 22 EMC DMX-2000 Symmetrix � 10 EMC DMX-4 Symmetrix
� 73GB 15K x 6336 Drives � 300GB 10K x 4800 Drives
� 90% Storage utilized � ~35% Storage Utilized with plenty of growth available
� ~500 Hosts � ~500 Hosts
� Data Migration using Symmetrix SRDF Array based utility
� Lowered Operational and Management Cost
� Much less spinning disks with plenty of growth left
� Performance modeling helped in minimizing performance impact
61© Copyright 2008 Interscape Technologies Inc. All rights reserved.
Questions?