Performance & Capacity Planning for SANs

04/11/23 1Performance And Capacity Planning Basics For Storage Networking

www.cmg.orgwww.inrange.com

Greg SchulzDirector Storage Networking Solutions

INRANGE Technologies

Performance And Capacity Planning Basics For Storage Networks

A Computer Measurement Group (CMG) Sessionwww.cmg.org



This Presentation Is Being Delivered On Behalf Of The Computer Measurement Group (CMG) For Educational Purposes

For more information about CMG, membership, conferences, and publications please see www.cmg.org

What Is CMG?



• Computer Measurement Group (CMG) is an International Organization with groups located throughout the United States and the World pursuing computer measurement, performance, and capacity planning activities. CMG looks at Capacity Planning and Performance Engineering and related tasks across all platforms (Mainframe, Open Systems, Servers, Workstations, Internet) and applications. CMG is a non denominational organization meaning it is not tied to any one vendor (hardware, software, network, etc) and is multi-disciplined providing a vehicle for learning about and sharing information on performance and capacity planning activities. CMG by its nature is also concerned with management software that lends itself to the overall computer measurement and performance engineering process.

• CMG activities include:– Annual international symposium (Dallas Texas, December 7-12, 2003)– Regional and international conferences, symposiums, and meetings– Online newsletter (MeasureIT), Journal, Conference Proceedings and papers

What Is CMG?



Customer Survey, 2001

Why The Concern For Capacity Planning and Performance



• Do you know how many Gbyte/Tbyte/Pbyte of storage you have?

• Do you know how many disk storage devices and/or sub-systems you have?

• Do you know where the storage is located?

• Do you know how much of it is free, allocated, actually in use?

• Do you know how many switch ports you have, how many are used, ISL, un-used?

• Do you know when you will need to add more storage, tape, cache, ports, HBAs, etc?

• Do you know how utilized your storage network is in terms of:– Bandwidth and throughput of servers, storage devices, switches

– I/Os, transactions, frames and packets being sent/received end-to-end and by component

– Channel delays and queuing for devices, adapters, and switches/gateways

– Response time and latency for the network and for storage

– Where the bottlenecks are, what is normal, what is abnormal?

– Workload profiles (reads, writes, I/O size, random, sequential, response time)

– How long tape mounts (real or virtual) take under different conciliations?

– much of it is disk, how much is tape, how many devices?

Can You Or Do You Need To Answer Any Of These?



• Performance and Capacity Planning activities have occurred in the enterprise environments for S/390 Mainframes and Open Systems platforms for many years.

• These activities have for the most part focused on large (expensive) components including processors, memory, storage and network interconnects, and storage sub-systems (disk and tape).

• Performance and Capacity Planning are independent yet related functions in that you can institute capacity planning in your environment without having to worry about performance engineering or vise versa not that it is a recommended practice.

• With the advent of lower cost processors (CPUs), memory, and storage, there is a tendency to go out and buy more. For some environments this may be applicable, however it brings with it an added management burden of more resources to see over.

• More recently Performance and Capacity Planning activities have focused on Internet related activities particularly response time and workload balancing.

• Capacity Planning, Performance, and Storage Resource Management (SRM) are related and depend upon each other.

• With storage and storage networking taking on a more prominent and lead role in many IT environments, more disciplined management activities are being employed to more effectively utilize IT resources particularly in tough business climates.

Getting Started - What Is It?



• Hardware is cheap, people are not, why tie someone up doing tuning?– While hardware is becoming less expensive, management (people and software is not)

• Your staff is already busy if not overworked, why give them more to do?– With planning, you can utilize your resources (people, hardware, software) better

• Why not buy more and have the vendor management it for you?– This may be an alternative if you can afford it from a dollar and business perspective

• Your environment is not growing so why be concerned with planning?– If your environment is stable, now is a good time to institute a plan for the future

• Your environment is dynamic so why do tuning and capacity planning?– During the dot-com bubble this was a common practice that lead to wild purchases and the

vendors absolutely loved it!

– Many sites over bought at what turned out to be higher prices than now available and have excess capacity that is consuming power, cooling, and management.

• You have your vendors take care of it all so why worry about this?– As part of the vendor community, can we talk….

Why Bother With Performance and Capacity Planning?



• Capacity Planning– Monitor how resources are being used including trends– Provides insight on how resources are being used including:

• Memory, CPU/Processors, I/O interfaces, Networks, Storage Devices• What the past looks like, what is normal, what is expected, what is planned• What is needed to support service level agreements (SLAs)

– Provides a view into the future as to when resources will be needed– Can help facilitate purchasing and upgrade decisions– Can help business decisions including do you have enough resources to support

growth, new functions, consolidation, and mergers/acquisitions.– Provides effective measurements to help manage resources better– Facilitate timely decision making to support dynamic and changing environments– Can be simple as tracking what you have, how it is used using a spreadsheet– Can be intermediate using a combination of tools and metrics– Can be advanced including modeling, sophisticated packages tightly integrated into

your environment.




• Performance Tuning and Engineering– Looks at how resources and application are performing

– Looks at applications and underlying infrastructure items including processors,

memory, file systems, operating systems, I/O interfaces, networks,and storage.

– Looks at how resources are being used including activity, bottlenecks, slowdowns,

and availability of resources that impact performance.

– Hardware, application, operating & file system, database, and network focused

– Can be simple or advanced depending on your needs and requirements

– Can support and drive Service Level Agreements (SLAs)

– Can range from simple maintenance like defragging the disk on your laptop to

advanced configuration and tuning of the entire I/O sub-system to application

optimization activities.




Planning for Growth• Capacity doubling every 18 months (Rich Media)• SANs getting more complex (Similar to early Lans)• Consolidation of servers, storage, SANs

Ultimately, how do you support growth, improve resiliency, while reducing costs?

TB/CapacityTB/Capacity

AvailabilityAvailability

$/Budgets$/Budgets

Data Availability Network Planning• More copies of data are needed (Regulatory)• Increased pressure for 100% uptime• DR networking costs expensive (Telco)

Shrinking Budgets and Cost Reductions• Lower management costs (Capital & Operating)• Optimize purchasing/Maximize Investment• Reduced Headcount/Improved efficient

Some Storage and Storage Networking Challenges…



Backup

Replication

Performance

Management

Capacity

Data Sharing

Security

Archive

Servers

S/390 and Open SystemsServers

Scalable

NFS ClientNAS & SAN

SAN/DAS

NAS SAN/DAS

SAN/DAS

NAS

Shared Storage & Tape

Servers

Consolidation

Block / File

A Unified Storage Networking Environment



53Copyright © 2001, Storage Networking Industry Association [draft SNIA TC Proposal]

Sto

rag

e d

om

ain

File/record layerFile/record layer

Block layerBlock layer

The SNIA storage modelwith Services subsystem

Application

Storage devices (disks, …)Storage devices (disks, …)

Block aggregationBlock aggregation

Database(dbms)

File system(fs)

Se

rvic

es

Se

rvic

es

Dis

cove

ry, m

onito

ring

Dis

cove

ry, m

onito

ring

Res

ourc

e m

gm

t, co

nfig

urat

ion

Res

ourc

e m

gm

t, co

nfig

urat

ion

Sec

urity

, bill

ing

Sec

urity

, bill

ing

Red

unda

ncy

mg

mt (

back

up, …

)R

edun

danc

y m

gm

t (ba

ckup

, …)

Hig

h av

aila

bilit

y (f

ail-o

ver,

…)

Hig

h av

aila

bilit

y (f

ail-o

ver,

…)

Cap

acity

pla

nnin

gC

apac

ity p

lann

ing

Capacity Planning – Part Of The SNIA Shared Storage Model Interconnect Usage Interconnect Capacity Storage Usage Storage Capacity



Have A Plan, Manage And Monitor Your IT Resource Capacity AndPerformance With That Plan

Managing Storage And Storage Networks

It Should Not Have To Be Complex!

Assess AndAnalyze

CollectData

MonitorResources

MonitorData Inputs

MakeAdjustments

TrackProgress



Capacity Planning And Performance Lifecycle Model

Collect, Assess, Report, Analyze, Model, Trend,

Track

Research, Adjust, Test, Implement

New DevelopmentBusiness Growth

Application ChangesConsolidation

Regulatory ComplianceAsset Management

Technology Options

How Are Resources Being Utilized and How Effective?

Change Control/Management

Configuration Management

DataCollection

DataReporting

What Changed? Positive Impact?

Test, Simulation, Modeling Engineer Changes

SRM



Assessment – What Do You Currently Have, How Is It Being Used• Do you have detailed configurations and topology information for your environment• Do you have an inventory or list of what equipment and software you have• Do you have profiles or information about your workloads and what is normal• Do you have any metrics or can you obtain them (sar, df, iostat, MXG, SAS, etc)

– Performance related includes bandwidth, I/O activity, MB/sec, I/Osec, response time– Capacity related includes disk and tape space consumption, ports and HBAs used

• Do you have any reporting and trending capabilities (paper, excel, web, SAS, etc)• Do you have any knowledge of business plans, growth, and activity to map to plan• Do you have any business tie back information to IT resources and consumption

– A business transaction requires “x” Mbyte and “y” I/Os at “z” response time– A device address can correspond to a branch office, bank teller machine, or terminal

• Do you have any Quality of Service (QoS), Rules of Thumb, or SLAs?– A port can support “x” Tbyte of storage at “y” Mbyte/sec– A server to storage port ratio is 6:1 assuming servers are doing 10-15MB/sec at 1Gb– You need to keep “x” ports free, “y” amount of storage free for growth on demand– You run servers at “x”% and interconnects at “y”% utilization, and storage “z” allocated

Capacity Planning Basics



Analyze, Model, Forecast – Interpret Data and Information• Identify trends and patterns based upon history, present activity, and projections• Understand business, applications, hardware and software activity• Create supporting reports and charts showing trends and activities• Track resource consumption (ports, memory, processor, storage)• Track resource activity (frames/packets/Ios, queuing, delay, bandwidth)• Model and forecast using tools or pencil, paper, and calculator• Identify options and various alternatives to support business needs

– Upgrades and expansion– Tuning and conservation– Combination of above

• Develop plans and strategies to support business needs and available resources• This is part science, part art, part intuitive (you need to understand your environment)• The better the data, the better the information, the better the plan!

– Your chances of success are as at least as good as your local weather forecaster .

Capacity Planning Basics



• Processor/Server (CPU)– CPU and Memory – Usage, paging, busy vs overhead vs. idle

– I/O bus and back plane – Is there enough capacity to support the adapters

– I/O Adapters and channels including HBAs and NICs – Are adapters busy or idle

• I/O interconnect/Channel– I/O path activity – How busy is the path, any congestion or queuing or channel

busy

– I/O interconnect (switches and directors) – How many ports, blocking/congestion

– Gateways, front end processors (FEP), bridges, and routers – Usage and profile

– Metropolitan and Wide Area Network interfaces – Bandwidth, Droop, Latency

• Storage Sub-System (Disk & Tape)– Cache capacity and usage – Cache hits, effectiveness and utilization

– Device capacity and allocation –How much allocated vs. used vs. available

– Number and type of devices – How many devices, what type, size, etc

– Control Unit usage and activity – How busy is controllers vs. devices

Storage Networking Focus Items



• CPU Processing Resource• Memory Capacity• I/O Channel (Adapters)

• Number Connection Ports• Non Blocking Bandwidth• I/Os and Frames/ Second

• Amount Of Capacity• Number Of Devices• Type Of Devices

CPU, Memory, I/O, And Storage• Activity (Idle, Overhead, Real Work)• Usage Patterns, Response Time• Allocation, Free, Used, Compression• Busy Channels, Blocking, Queuing• I/Os and Bandwidth, Reads, Writes• Tape and Volume Mounts and Delays

CPUHost

Processor

Switch Interconnect

Storage DeviceDisk & Tape

Capacity Planning And Performance Focal Points

LAN



• How busy are your ports (server, storage, ISLs) and effective are they?

• Do you need to upgrade your storage devices (e.g. cache) for faster ports?

• When will you need to add more ports?

• Will you need to upgrade or change your infrastructure to add ports?

• How utilized are your ISLs and Trunks?

• How busy and effective are your wide area interfaces?

• Are you incurring any blocking or congestion due to poor locality?

• Which of your systems benefit from 1Gb, 2Gb, 10Gb technology?

• Is the back plane on your servers over utilized or under utilized?

• Asset management (What do you have)– Servers, HBAs, cabling, switches/directors, gateways, metropolitan and WAN

interfaces, network links, disk, tape, software, and their configuration.

– What is owned, what is leased, who ones it, what is due for replacement

Storage Networking Focus Items



– Determine requirements and objectives• Why are you building a storage network and what do you want it to support• What are you success criteria, how will you know when it is done and works• Do you have a strategy to align with business, or technology• What are the growth plans, performance criteria, any SLAs or QoS?

– Assess what you have, what you need• Take inventory of what you have, where, and what shape it is in (Note: nothing is for Free)• Do you have components (software, hardware, networks, people) to factor in• What will you need to add, what will you need to remove

– Design based upon requirements and objectives• Develop a design that is scalable and stable to support growth and your objectives• Solicit multiple ideas from different vendors and have them explain pro’s and con’s• Determine what’s best for you, as opposed to what’s best for your vendor

– Implement and document the plan• Implement the plan yourself or have someone implement it for you• Training and documentation are important activities as well as software configuration

– Validate the plan, make adjustments and repeat steps above as needed…

Storage Networking Design Process



Block File Message

SCSICCW

S/390 Open Systems

FC-SB-2(FICON)

FC-SB(ESCON)

Fibre Channel1Gb/2Gb/[10Gb]

FCPpSCSI iSCSI SRP

Gb Ethernet1Gb/[10Gb]

InfiniBand1x/4x/[8x]

2.5Gb/10Gb/[30Gb]

SCSI(Copper)

IP

VIA RDMAMSDA

MSFT

IPDAFS NFS

OpticsxWDM/TDM & SONET/SDH & WANs

CIFS

Sockets

SDPSDP

LinuxOn zSeries

Local, MetroWide Area

Storage NetworkingxWDM SONET/SDH

Some Networks For Storage Networking

Fiber Optics At The Core



CIM

ISVn

IHVn

SANManagement

and SRM Tools SMIS (CIM/XML via HTTP)

CIM/SMIS Heterogeneous Simplified Storage Management

Backup andArchive

Applications

Switching and StorageVirtualization along with Local,Metro, and Global Connectivity

SMIS - CIM Clients

NAS ServerHead

CIM Server CIMOM

CIM Provider

CIM Server CIMOM

CIM Provider

CIM Server CIMOM

CIM Provider

SMIS - CIM Clients

S/390 FICON

AS400(p/Series) Unix NT

Storage DevicesIncluding Block and Object

Based Storage

Tape Devices andLibraries Including virtual

Tape

Switches & DirectorsDisk Storage Tape Storage

INRANGEFC/9000



Monitors SAN Performance (Trend/Real-time)

End-to-End (Delivering Service Levels)

Correlates Performance across

Devices, Hosts, and LUNs

Across Heterogeneous Storage Networks

Port and SessionStatisticsMB/sec

Frame/SecI/O Size

Read/WriteResponse Time

Collecting and Reporting On Storage Networking Activity



Utilization:• Frames (In/Out)• FC-2 MB/Sec (In, Out)• FC-4 MB/Sec (In, Out by ULP – SCSI, IP, VI, FICON,

and others)• Errors MB/Sec (In, Out)• SCSI IO/Sec (Read, Write, Other)• SCSI Read (avg, min, max, read percentage)• SCSI Write (avg, min, max, write percentage)• SCSI Other (other percentage)• SCSI Read/Write Payload Size Ranges (percentage)Throughput Errors:• Invalid CRC Errors• Primitive Sequence • Protocol Errors• Frames Too Long• Frames Truncated• Address ID Errors• Delimiter Errors• Disparity Errors• Discarded Frames• Invalid Transmission Words

Availability:• Link Resets (In/Out)• OLS (In/Out)• LOGIs (Port, Fabric)• %AvailableLink Integrity:• Sync Loss• Sig LossCapacity:• %capacity for all frames• FC-4 %capacity (SCSI,IP,VI,FICON, other)• % capacity link control• % capacity link servicesLatency:• SCSI Read/Write Duration (ms)

Storage Networking metrics (statistics) have been rudimentary and limited to what’s available from a given vendors switch, HBA, application, or storage device. The collection and flow of statistics has been either intrusive or proprietary formats. New capabilities now exist to collect in an non-intrusive manor complete statistics including detailed session level information not available from switches and directors across all platforms and storage devices. An example of this new capability is INRANGE PerformanceVSN Storage Networking probe and server with APIs.

Some Storage Networking Metrics



Summary of total Mbytes/secFor 24 selected across multiple

switches

Trend over past 8 hours of totalMbytes/se (In and Out) for a

selected port

Summary of top five LUNs(Storage Devices) being Read

From “Server_A”

Real-time error statisticssummary of a selected port

Performance Traffic Analysis

Diagnostics Capacity Planning

Performance, Workload Analysis, Capacity PlanningDashboard, Visualization Of Storage Networking Activity

Summary of Traffic by Protocol

Workload Analysis

Utilization Real-time Summary showing readexchanges by size from storage

Device “RAID_A” to all host.



• When Planning For Storage Networking Devices (Disk/Tape)

– Account for impact to cache with 1Gb, 2Gb, 10Gb

– Account for impact of mixed protocols (FCP, ESCON, FICON, Parallel SCSI)

– Account for device performance and configuration (e.g. RAID level etc.)

– Proper cache size for VTS devices depending on SLAs and requirements

– Single, Dual, and Quad pathing of storage interfaces for resiliency

Storage Networking Planning Items



• Server and Processors

– Account for any dual, quad HBA server port needs

– Account for adequate bandwidth on busses and back planes

– Software license of volume managers, path managers, fail over managers

– Some dual/quad HBAs share common bus and may not operate at full speed




• When Planning For Storage Networking Ports

– Account for any dual, quad HBA server port needs

– Account for storage ports (primary and secondary)

– Account for ISL and trunk ports for local and metro

– Account for connectivity ports to expand to other switches for growth

– Account for ports for diagnostics, test, and analyses

– Account for any spare and on-demand ports

– Account for ports for hardware mirroring

– Account for any SAN appliances including NAS heads and Virtualization devices




Droop• Phenomenon of performance degradation over distance• Performance drops before theoretical limit• Not to be confused with speed of light (5microsec/km latency)

• Function of Network Protocol vs. Physical medium• Synchronous access can be more impacted then Async.• Results from lack of Buffer Credits with flow control networks

Commonly found with Storage and Deterministic Interfaces

100

65

45

25

0 0 10 20 30 40 50 60 70 80 90

Data Droop Rate Distance in Km

Da

ta R

ate

M

B/s

ec

20

17

9

3.5

0 10 20 30 40 50 60

Data Droop Rate Distance in Km

Da

ta R

ate

M

B/s

ec

Lack of Buffer Creditsand Flow Control

Sufficient Buffer CreditsAnd Flow Control

Performance, Distance, and Latency Considerations



• Locality – The root of SAN performance bottlenecks– Locality describes where ports are located in a SAN with the higher locality, the

better performance will be with fewer or no hops to incur increased latency, simplified security and management. In the example below on the left, all ports are 100% local however this is determined by the number of available local ports. On a sixteen port switch, there would be 16 local ports, on a 64, 128, or 256 port director, there would 64, 128, or 256 local ports. Thus vertical scaling and locality go hand in hand. In the center example below, more ports are needed than what is available with vertical scaling so horizontal scaling is used to increase the number of ports however locality is reduced. On the right, is an example of vertical scaling to use a large port count device to improve locality to improve performance.

8

0% Locality

100%Locality

100%Locality100%

Locality

Vertical Scaling

Horizontal Scaling

Hops = Latency



Capacity Plan For Storage Networking Ports (All Fabrics)

Used/Allocated Ports Available Ports Trend

Q1 Q2 Q3 Q4 Q1 Q2 Q3 Q4

Time

Por

ts

16

32

48

64

96

128

256



Storage Usage Information

Device GB % Used Free I/Os MBsDev1 36 50 18 80 45Dev2 36 25 9 50 50Dev3 36 50 18 90 30Dev4 36 75 27 40 20Dev5 25 9 30 50 505 Devices 180 46% 81 290 195

Depending on device type, additional capacity may be needed for example to support RAID-5, mirroring RAID-1, striping RAID-0. These devices should appear in total capacity however may be hidden in terms of usable capacity and storage. Some devices may be used for capacity indicting a low I/O or

throughput (MB/sec) rate. Others may be high I/O with low utilization (possible candidate for SSD).



The best topology is the one that works for you as opposed to one you work for

Establish availability, performance, response time and other objectives

Create scripts that can check end-to-end response time and know what is normal

Capacity Planning concerns capacity, bandwidth, utilization, and activity

When migrating from 1Gb to 2Gb assess cache impact on your storage

ESCON to FICON migration can be done in phases using multi and quad paths

Poor metrics and information can lead to poor decisions and management

Understand what your storage network is doing and how it is being used

Understand what resources you have (servers, HBAs, disk/tape, ports, etc.)

Utilize capacity planning tools for reporting and monitoring

Simple homegrown tools can be used including sar, iostat,df, and excel

Be practical as it can be easy to get wrapped around the “axle”!

Some Storage Networking Performance and Capacity Planning Tips



“The Resilent Enterprise – Veritas Press

“Storage Networking Essentials” – Barker/Massiglia (Wiley)

“Blueprints for High Availability” – Marcus (Wiley)

“Leveraging MAN and WAN Connectivity” – SNWOnline – August 2002

InfoStor (www.infostor.com)• June 2001, Introduction To Storage Networking,• May 2002, Protocol options lead to “Storage Agnostic Networks”• August 2000, Storage Virtualization explained

Industry Associations (SNIA, FCIA, IETF, T11, SCTA) and their websites

SNIA Web Site and Education Committee Tutorials• www.snia.org• SNIA Dictionary and Tutorials (Metro Storage Networks, Infrastructures)

CMG Proceedings and handouts on CD’s (www.cmg.org)

“Storage Networking Primer and Update” & “Scaling Resilent Storage networks”

Some Additional Material and Reading



History – Enterprise Networking Solutions Since 1967

Thousands of World Wide Enterprise Customers• Telecommunications, Financial, Services, Transportation, Government, Manufacturing,

Retail, Insurance, Life Sciences and Medical• Direct touch exposure and experience with end users for consultive sales and solutions

Technology Leadership• Scalability and Reliability• Flexibility and Interoperability• Network Management Simplicity• Investment Protection

Balanced Business Model• 67% Product and 33% Services• Domestic and International presence• Core Storage (Switching)/Network Provisioning/Data Center Infrastructure

INRANGE Overview

Addressing Overall Storage and Storage Networking Functionality and Benefits

Multi-Protocol Switching(Fibre Channel and FICON)

Connection and Extension(FC and ESCON over xWDM, SONET/SDH, and IP)

Monitoring & Management Fiber and Cabling Infrastructure Professional Services



Questions?

[email protected]

www.cmg.org



Thank You

www.inrange.com

Documents

Performance & Capacity Planning for SANs