Capacity Planning - Managing the hardware resources for your servers

.

Capacity Planning

What is it?• A method to insure that a given set of hardware is adequate

for current and projected needs

Why is this important?• Determine current hardware utilization• Identify possible problems• Identify available capacity for new projects• Plan future hardware acquisitions• Forecast how multiple packages will behave if co-hosted

What information is needed?

Network Capacity• Relatively easily collected and analyzed

– sysmon or Monitor tables and netstat

Disk IO Capacity• Relatively easily collected and analyzed

– sysmon or Monitor tables and iostat or SAR

Memory Usage• Relatively easily collected and analyzed

– sysmon or Monitor tables and vmstat

CPU Utilization• Easily collected and Not easily analyzed

– sysmon or Monitor tables and SAR

Why CPU Utilization?

Host CPU utilization statistics do not reflect actual use• Includes time server spends spinning looking for work

Server Engine statistics do not reflect actual usage• System usage required to service server request not

included– Disk IO – Network IO

01020304050607080

Host

Server 1

Server 2

Servers

Host vs Server Statistics

Sybase Environment

Hardware• Large verity of host• Numerous host clusters• Multiple data centers

Servers• Varied usage, no two alike• An ASE may serve 1 application or more than 50

Replication• Used heavily for standard replication• Increased use warm standby

Hardware Environment

3 Data Centers 57 Solaris Hosts

• Varied hardware– Over 10 Different Sun Models (several are domains)– 4 Versions of OS

• 13 Veritas Clusters– 3 Clusters of 4 or More Hosts– 10 Clusters of 2 Hosts – SAN Disk Arrays

• 13 Standalone Hosts

2 HP Hosts

Sybase Environment

62 ASE Servers• 45 Production

– 34 Running on Clustered Hosts– 11 Running on Standalone Hosts– 4 Warm Standby – 8 are 3rd party venders

• 16 Test– 6 Running on Clustered Hosts– 10 Running on Standalone Hosts

• 1 Test/Integration

Replication Environment

3 Replication Servers• All on Clustered Hosts• Standard replication

– ASE to ASE– DB2 to ASE– ASE to DB2

• 4 Warm Standby

ASE Packages

ASE servers are installed in Veritas packages• A “package” or “service group” can contain:

– One or more file systems and mount points– Application processes– Sybase installation device – All disk devices– Dump devices for transaction logs

• A package has its own IP address (or virtual IP) and DNS entry and can be moved between the hosts of a cluster as needed

• Will automatically switch to another host upon any one of several types of system failures

Clustered Environment

SANA

SANB

Host BHost A

Host DHost C

Data Center

1

Data Center

2ASE

Goals of Project

Come up with a measurement that can be used to:• Evaluate the current host utilization• Identify problem server groupings• Compare servers running on different host in some

meaningful way• Predict the effects of moving servers around within a cluster

or to a new clusters • Gauge the impact of host and data center failures• Determine the effect of server failovers to warm standby

servers• Anticipate the hardware requirements of server expansion

and increased use of warm standby servers

What Data Needs to be Collected

Location of servers• Track movement of servers

Host SAR CPU statistics• Sample size and timing to match server engine

Server engine statistics• Sample size and timing to match host CPU

Tracking Server Movements

Servers can move from host to host for a number of reasons.• Semi-annual package switch test• Planed moves for hardware or OS maintenance• Hardware or software failures• To guarantee that the primary and secondary are not in the

same data center• Load balancing

Need to ensure that the calculations are using the correct server and host statistics

How is the Server Tracked?

Number of possible methods• Agent on each host that would report and update the

information• Check the mounted file systems on each host• Track the virtual IP to it’s current host

Record new host and the date and time the server moved from old host to new host

Side benefits• Determine at where each server quickly• Know when package was switched and from where

Host Map (Clustered)

Host Map (Clustered and Non-clustered)

What is a ‘Capacity Number’?

It is the statistical maximum of the total available host CPU capacity a server was using a specified hour on a specified day.

What does that mean?• A ‘capacity number’ is calculated using the data for the past

3 months for each hour of the day for each day of the week.– 24 capacity numbers per day– 168 capacity numbers for a week

• Maximum for that hour, for that day– limited by statistics for the 3 month sample

Usage 24 Hour period

0102030405060708090

100

0:10:00 4:40:00 9:10:00 13:40:00 18:10:00 22:40:00

0

2

4

6

8

10

12

1 11 21 31 41 51 61 71 81 91 101

111

121

131

141

151

161

171

181

191

Calculating Capacity Number

How is the ‘Capacity Number’ Calculated?

Each night the ‘Capacity Numbers’ for the previous day are calculated for all the servers• The allocation of host utilization is based on the number of

engines configured for each server on the host and how active the server is.– (cpu_busy * engines) / sum (cpu_busy * engines) for all servers

• Second step is to adjust the allocation based on the number of server engines vs total CPUs.– Host CPU / Server engines

or– Allocated CPU / Server engines

Calculating the ‘Capacity Number’

Host CPUs vs Allocated engines• Host CPUs > Allocated engines (servers represent all

activity) (pessimistic)70 * allocation

• Host CPUs > Allocated engines (servers do not represent all activity) (optimistic)

70 * allocation * (Allocated engines / Host CPUs)

• Allocated engines > Host CPUs70 * allocation

Calculating the ‘Capacity Number’ (Continue)

Calculate the range limits• Calculate the median for all data points within the selected

hour and .5 hours on both sides• Ignore all data points below the median within above range• Find the Standard Deviation of the remainder• Find the maximum for the previous day for the selected hour

– Maximum greater than mean + 2 * SD‘Capacity Number’ = mean + 2 * SD

– Maximum less than mean – 2 * SD‘Capacity Number’ = mean - 2 * SD

– Maximum within range‘Capacity Number’ = Maximum

Data Retained

• Server_id• Host_id – host the server was on during the period• Range_min – minimum based on allocated engines activity• Capacity_Number – based on allocated engines activity• Range_max – maximum based on allocated engines activity• Engines – configured for server• CPU – for the host• Allocated_engines – total allocated engines for all servers• Total_min – minimum based on all CPU activity• Total_Capacity_Number – based on all CPU acitivity• Total_max – maximum based on all CPU activity

Host/Servers combination that in the past has exceeded max capacity

Optimistic Capacity (Engines < CPUs)

Pessimistic Capacity (Engines < CPUs)

Optimistic Capacity (Engines >= CPUs)

Pessimistic Capacity (Engines >= CPUs)

Things that Effect the Server When Comparing Capacity Number on 2 Different Host

Memory – Server memory static straight comparison IO – Not considered to be significantly different Network – Not considered to be significantly different Number of CPU – Significant impact on performance

• Capacity number must be adjusted by a ratio based on the difference in number of CPUs

Processor speed – Significant impact on performance• Each host is assigned a comparative factor based on

processor speed• Capacity number must be adjusted by a ratio based on the

difference in comparative factors.

Calculating the ‘Relativity Index’ for a Server

Calculate the adjustment for CPU numbers.• Number CPU on source / Number CPU on destination

– Host a -> Host b 6 / 12 = .5– Host b -> Host a 12 / 6 = 2

Calculate the adjustment for Processor Speed• Comparison factor source / Comparison factor destination

– Host a -> Host b .45 / 1.2 = .375– Host b -> Host a 1.2 / .45 = 2.67

‘Relativity Index’ for the two host• CPU ratio * Speed Ratio

– Host a -> Host b .5 * .45 = .1875– Host b -> Host a 2 * 2.67 = 5.34

Predicting the Capacity when Rehosting Servers

Select a time period to be used a for comparison Select destination host Select servers that will be running on host Select ‘Capacity Number’ data for selected servers Calculate a ‘Relativity Index’ for each host pair (source,

destination) Adjust ‘Capacity Number’ by the ‘Relativity Index’

calculated for each server Plot results

Using ‘Capacity Number’ and ‘Relativity Index’

Predicated Host Utilization

Optimistic Prediction

Host Utilization with Optimistic Prediction

Pessimistic Prediction

Host Utilization with Pessimistic Prediction