A National Resource Working in the Public Interest © The MITRE Corporation. All rights reserved.Oct-15 MITRE Performance Testing: Load Testing With Usage

A National Resource Working in the Public Interest © The MITRE Corporation. All rights reserved.04/19/23

MITRE Performance Testing: Load Testing With Usage Analysis

The MITRE CI&T Performance Test Team

February 2009

A National Resource Working in the Public Interest © The MITRE Corporation. All rights reserved.

Page 2

04/19/23

Acknowledgements

This briefing was prepared by adopting material from the Fundamentals of LoadRunner 8.1 training manual

The final usage analysis method and model described in this briefing was pioneered by the MITRE CI&T Performance Test Team.

Initial methods and models were originated by Henry Amistadi and evolved with the input of Chris Alexander, Betty Kelley and Aimee Bechtle.


Page 3

04/19/23

Purpose & Goal

Provide background on our team and load testing at MITRE

Understand how the Project Lead can support load testing throughout the performance testing process

Introduce our Usage Analysis methods so the Project Lead

1. Understands how we target a realistic load in load testing

2. Has confidence that the right amount of testing is being performed on a project

Our goal is always to provide high quality, timely performance tests that meet the needs of the customer and the performance test team


Our Team

Comprised of 3 Performance Test Engineers and 1 Team Lead

Instantiated in 2001 as the Automated Test Team

Invested in Mercury (now HP):

– LoadRunner for Load Testing

– QuickTest Pro for Functional Testing

– Quality Center/TestDirector for requirements and test case management

In the beginning customers suspicious of the reality and accuracy of our testing

In 2005 stopped performing automated functional testing to focus on our core competency, performance testing

Have matured from single application and environment testing to multi-application and multi-environment testing

Preparing for a large-scale, enterprise load test for 2009-2010

Now, high level of confidence in our service

Page 4

04/19/23


Page 5

04/19/23

Background: What is Load?

What is Load? Traffic!!!!!

– Transactions distributed over time, expressed as rates

Transaction Per hour (TPH)

Transactions Per minute (TPM)

Transactions Per second (TPS)

Why Load Test? To assess how well your application is directing traffic

– Find the problems before your customers do

– Prevent poor response times, instability and unreliability (Road Rage!)

– The longer you wait to test the more costly the problem may become


Page 6

04/19/23

Background: When to Use Performance Testing When should I consider load testing? How can I use you throughout the project?

Contact performance test team during the project planning process and consider their incorporation into the plan

iSTEP Phase Examples/Activities

Requirements COTS Evaluation, Usage Analysis

Design Prototyping, architecture and server evaluation

Implementation Early assessment of application response times, Support capacity planning

Test Determining if the system is reliable enough for production

Post Rollout Isolating performance bottlenecks and degradation in performance after production. Regression testing.

Load testing can be used throughout the product’s life


Page 7

04/19/23

Background: The Process

Plan LoadTest

AnalyzeResults

ExecuteTests

Create Scenarios

Create Scripts

Tweak System or Test

Based on Analysis

§ Desc Goals & Objectives

§ Select Load Test Types

§ Analyze Application:

§ Document Plan

§ Define steps§ Create

Scripts

§ Create scenarios based on goals and targets

§ Execute scenarios:

PeakStressOther

1 2 3 4 5

TestTransaction

Rate

§ Analyze Data and pinpoint bottlenecks

§ Compare Test Transaction Rate to Target

Target Transaction RatesEnvironmentUsage

§ Supply Goals, Obj, Schedule

§ Initiate Data Collection

§ Select targets§ Review PlanP

roje

ct L

ead

/D

evel

op

er

Fu

nct

’l E

xper

t

§ Develop use cases as input into scripts

§ Provide Data

§ Supply transaction logs

Sys

an

d D

BA

dm

ins

Per

form

ance

Exp

ert

§ Provide Steps & Sample Data

§ Review scripts

§ Provide Steps & Sample Data

§ Review scripts

§ Prepare Test Environment

§ Monitor resources

§ Review results

§ Review results

§ Review Scenario design

§ Prioritize & schedule tests

§ Change settings (if applicable)

§ Change application


Page 8

04/19/23

Define Goals & Objectives

Begin working with performance test team upon completion of planning process

Start by discussing high-level goals & objectives.

And what are these objectives?

Objective For Example…

Measure Application Response Time

How long does it take to complete a business transaction?

Support Configuration Tuning Which configuration provides the best performance level?

Regression Does a new version of the software adversely affect response time?

Assess Reliability How stable is the system under a heavy work load?

Capacity Planning Can the application support a growth in usage?

Identify Bottlenecks What is the cause of degradation in performance?

Evaluate Products What is the best product (COTS, hardware, etc.)?

Plan LoadTest

1


Page 9

04/19/23

Select Load Test Types

What are the types of load tests?

Type Description

Typical or Peak Load Determine whether system handles real world loads

Stress Load Find the system’s breaking point. Test at 1.5, 2, 2.5, 3

times peak.

Exploratory or Goal Meet predetermined criteria or “What ifs?”

Architecture Verify architectural design, like failover, load

balancing

Extreme/Volume Load Check stability of system under extended periods of

high load

Plan LoadTest

1


Page 10

04/19/23

Analyze System - Environment

Need to understand the logical and physical test environment to make the proper tool, protocol selection and to know

our constraints

What is the historical CPU utilization on these machines?

What other apps are on these machines?

Plan LoadTest

1


Page 11

04/19/23

Analyze System – Usage Analysis

Usage analysis is appropriate when:

– Plan is to run a Typical, Peak or Stress test to meet the agreed upon objectives

– There’s an existing system to collect information from

Usage Analysis is the process of calculating load targets from log file data for the relevant URLS

– AKA Workload Analysis or Log File Analysis

– We employ a statistical evaluation of data collected

Results in a Transaction’s:

– Target Typical & Peak TPH

– Target Typical & Peak TPM

– Target Typical & Peak TPS

Plan LoadTest

1


Page 12

04/19/23

Analyze System – Usage Analysis Plan Load

Test

1

Identify Business

Trans.Convert TPS

Select Targets

Analyze Fine Grain

Analyze Coarse Grain

§ Select key business processes to be analyzed and tested

§ Select the days to be analyzed

§ Collect log files for these days

§ Parse and grep log file data

§ Aggregate data by hr, min and sec

§ Create cumulative freq. distrib.

§ Review cumulative freq. distrib. With PL

§ Select target typical and peak TPHs and TPMs

A B C D E

§ Convert TPS from TPM using Total Time Distrib. or

§ Convert TPS from TPM using Active Time Distrib.


Page 13

04/19/23

Usage Analysis – Business Transactions

“A business transaction is a set of user steps performed within an application to accomplish a business goal”

For example:

– Logon to Home Page

– Search

– Update profile

Focus on the business processes that are

– Most commonly used

– Business Critical

– Resource Intensive

We need the URLs that complete these business processes. These are the target URLs

Also, what are your expected target response times for these processes under typical load? Peak load?

Identify Business

Trans.

A


Page 14

04/19/23

Usage Analysis – Coarse Grain Purpose is to identify Typical Peak Usage

Patterns

Use WebTrends

We identify the Typical Peak by answering questions such as:

– Is this application used cyclical or quarterly?

– Are some months or days of the weeks higher use than others?

– Is this application heavily used on a daily basis?

– Is this application heavily used throughout the day or only at specific hours?

– What are the highest use days and hours?


B


Page 15

04/19/23

Usage Analysis – Coarse Grain Example

Corporate Portal:

– Looked at weekdays from Oct – Dec 2006

– Compared daily and hourly page views in WebTrends for portal/dt and amserver/UI/Login

– Ranked days in terms of daily totals

– Excluded two busiest days because of abnormal patterns

– Selected next 8 busiest days for further analysis

MyMII (Sun Portal) Daily Requests for portal/dtWEBTRENDS DATA

0

20000

40000

60000

80000

100000

120000

140000

160000

180000

200000

18-Sep-06 25-Sep-06 2-Oct-06 9-Oct-06 16-Oct-06 23-Oct-06 30-Oct-06 6-Nov-06 13-Nov-06 20-Nov-06 27-Nov-06 4-Dec-06 11-Dec-06 18-Dec-06

Requests Per Day


B


Page 16

04/19/23

Usage Analysis – Fine Grain

Once the Typical Peak timeframe is determined, log files from that period are collected for more detailed analysis.

Relevant fields are isolated for inclusion, such as:

– Requesting URL (as agreed upon with the project team), Response Code, Date and Time request completed, Referring URL, User information

Analyze Fine Grain

C


Page 17

04/19/23

The data is aggregated by time frame and frequency analysis is performed. For an example please see: Usage Frequency Analysis Example

The transaction rates are placed in a cumulative frequency distribution

Work Hours Cumulative FrequencyHTTP 200 (/portal/dt)

TPH TPM TPS

Mean (Ave) 4499 75 2

Median (50) 5103 81 2

60.0 5214 86 2

70.0 5381 91 2

80.0 5532 96 3

90.0 5778 104 3

95.0 5896 110 4

97.5 5988 118 5

99.0 6075 131 5

100.0 6075 309 26

Usage Analysis – Fine GrainAnalyze Fine Grain

C


Page 18

04/19/23

Usage Analysis – Select Targets

Select your Typical and Peak targets

We recommend using the TPH and TPM rates at the Median, 50th Percentile, as the Typical Transaction Rates

We recommend using the TPH and TPM rates at the 99th percentile as Peak Transaction Rates

The TPS in this analysis is a reference point.

This index is a satisfaction index:

– What percentage of the time are you willing to have users be satisfied/unsatisfied?

– If you design a system to handle 131 TPMs your users would be satisfied with response time performance 99% of the time

Work Hours Cumulative FrequencyHTTP 200 (/portal/dt)

TPH TPM TPS

Mean (Ave) 4499 75 2

Median (50) 5103 81 2

60.0 5214 86 2

70.0 5381 91 2

80.0 5532 96 3

90.0 5778 104 3

95.0 5896 110 4

97.5 5988 118 5

99.0 6075 131 5

100.0 6075 309 26

Select Targets

D


Page 19

04/19/23

Usage Analysis – Convert TPS

We can choose from two methods

– Method 1: Total Time Distribution (How?)

Assumes transactions are evenly distributed and every minute or second is active

Represents the lowest transaction rate in terms of load distribution

– Method 2: Active Time Distribution (How?)

Accounts for the wave patterns of transaction activity

Assumes that not every minute and second is active

The active and inactive times are calculated

Method 1 Total Time Distribution

0

0.5

1

1.5

2

2.5

9:00 9:01 9:02 9:03 9:04 9:05

Time Sample

TP

S actual trans

avg trans

Method 2 Active Time Distribution

0

0.5

1

1.5

2

2.5

9:00 9:01 9:02 9:03 9:04 9:05

Time Sample

TP

S actual trans

avg trans

Purpose is to know the target TPS, and at what time intervals, the transactions are going to occur in the test

Convert TPS

E


Page 20

04/19/23

Pulling It All Together

At the conclusion of the usage analysis we meet with the Project Lead and review the plan.

Ideal if conducted during development, prior to integration testing

Review Business Processes and Target Transactions Rates

Target Transactions Rates Example: “The system shall sustain a load home page transaction time of 3 seconds or less during typical hours at a rate of 2.02 transactions per second, and 3-5 seconds maximum during peak hours at a rate of 3.3 transactions per second.”

Transaction Typical Targets Peak Targets

TPH TPM TPS PRT* TPH TPM TPS PRT*

Portal Home Page 5103 81 2.02 <3s 6075 131 3.3 3-5s

* PRT = Preferred Response Time

Plan LoadTest

1


Page 21

04/19/23

Create Scripts

Scripts contain the recorded steps, test data, and think times that define the transactions

Functional test cases, use cases, design documents and training manuals are a good source for this information

A sample of the detail we need:

Step Data

Click link for “My Insurance Plan Summary”

Logon Username

Password

Click on the “Life” link under Type of Benefit

Click on the Edit button under Covered Beneficiaries Section

Enter New Primary allocation box for first beneficiary 100

Click “Save”

Click “Ok”

A real example: Detailed Steps

Create Scripts

2


Page 22

04/19/23

Create Scenarios

Scenarios define the overall load test execution. The instructions.

Elements of a scenario are:

• Scripts

• Vusers: Emulate the steps of real users. Steps are recorded into a script.

• Ramp Up Rate: The rate at which vusers are suddenly or gradually introduced into the test

• Think Times: Emulates how long the user pauses when:

• Executing a transaction or business process. I.e. pauses between clicks or keystrokes

• Between transactions (iterations), between starting the script over

• Test Duration: The length the test will run. 1 hour, 4 hours, 8 hours. Varies by test type.

Create Scenarios

3


Page 23

04/19/23

Favorites

Home Page(Default Config)

Project Services

RSS

HomePage (Configured)

Center Portlet

Typical Day 1x(Home Default Config)

ExploratoryEdits

Typical Day 3x(Home Config)

Typical Day 1x95% Home (Default)

+ 5% Edits

Typical Day 1x(Home Config)

ExploratoryProject Services

Script

Test Run2007-03-07-D

Scenarios Test Runs

Test Run2007-03-04-A

Test Run2007-03-09-F

Test Run2007-03-09-H

Test Run2007-03-09-I

Peak Second Bursts +1x(Home Default Config)

Test Run2007-03-12-N

Test Run2007-03-12-L

Home Page(Default Config)

Bursts

Pulling it All TogetherExecuteTests

4


Page 24

04/19/23

Prepare Test Environment

– Account Setup

– Data Setup

Plan for Multiple test runs

– Debug

– Baseline

– Full execution

Tests will be run in priority order as we agreed upon

Execute tests as a team!!!

– Systems and Database administrators should monitor the performance of each tier of the application

– Ideally a network administrator could monitor the network

Execute TestsExecuteTests

4


Page 25

04/19/23

Analyze Results

At the end of the test run(s) an analysis file is produced that reports:

• Running Vusers

• Duration

- Response Times By Transaction:

o Average & Maximum

o Percentiles

• Test Transaction Rates

- Test Total Transactions (Passed/Failed)

- Test Transactions Per Second

Example analysis file: Sample LoadRunner Analysis File

The Target Transaction Rates from the Usage Analysis are compared to the Test Transaction Rates. If they don’t reconcile then…

AnalyzeResults

5


Page 26

04/19/23

Tweak The Load Test or The System

The test will need to be updated and/or rerun if:

1. The test goals have not been met then change the script or scenario

– Test Transaction Rates are not similar to Target Transaction Rates

For an hourly test the Total Transactions should be close to the Target TPH

The TPM and TPS rates should meet or exceed the TPM or TPS from the usage analysis

Increase vusers and decrease think time to increase load

2. Performance has not met expectations (i.e. poor response time or CPU utilization)

– Change Code– Change System Configuration– Change Architecture

This is an iterative process which takes time. This needs to be planned for.

Tweak System or Test

Based on Analysis


Page 27

04/19/23

Finally…

Roll out and repeat!!– When the load test and performance goals are met you are good

to go

– Contact performance test team when application changes are made that affect performance:

Significant GUI or Code changes

Changes in the architecture or environment

Performance degradation in production

Adding more users


Page 28

04/19/23

In Conclusion…

Partner with the peformance test team:

– Engage team from the beginning – during planning

– Have goals and objectives about what you want your performance test to accomplish

– Provide:

Architecture diagrams

Transactions and steps to be scripted

Test data

Webtrends and log file data

Server permissions and DBA/Sysadmin support

– Select Targets

– Review the test plan, prioritize and schedule the tests

– Plan enough time for an iterative testing process

– Keep team informed of schedule or resource changes


Page 29

04/19/23

Background

Detailed Calculation Examples


Page 30

04/19/23

Method 1: Even Distribution - Example

Evenly divide the TPH and TPM transaction rate by 60

TPM = TPH/60

TPS = TPM/3600

For example:

TPM = 6075/60

–TPS = 131/60

101.3

2.2 (We would use this one)

% TPH TPM TPS


Page 31

04/19/23

Method 2: Active Time Distribution

Calculate Active Time from log as the percentage of time requests are written to the log

We calculate the Active Time per hour of day in terms of:

– Active Minutes per Hour

– Active Seconds per Minute

For example:

– At 9 o’clock every minute in the hour had at least 1 transaction, but every second did not.

– Only 46.7 seconds of the minute were active

Active Time

hour mins / hour secs / min

6 60.0 19.0

7 60.0 34.1

8 60.0 45.4

9 60.0 46.7

10 60.0 46.0

11 60.0 44.3

12 60.0 42.5

13 60.0 44.5

14 60.0 45.3

15 60.0 44.7

16 59.3 40.2

17 60.0 28.8

Average 59.9 40.1


Page 32

04/19/23

Method 2: Active Time Distribution - Example

Evenly divide the TPH transaction rate by Average Active Time

– TPM = TPH/Active Minutes in a Hour

– TPS = TPM/Active Seconds in a Minute For example:

TPM = 6075/59.9

TPS = 131/40.1

101.4

3.3

4b

% TPH TPM TPS


Page 33

04/19/23

Add Inactive Time--beginning point for determining think times or pacing

– Percentage of time when no requests are completed or written to the logs or the time between completed

requests

– It includes the processing time of the second request

We calculate the Inactive Time per hour of the day in terms of:

– Inactive Minutes per Hour

– Inactive Seconds per Minute

Inactive Time needs to be sprinkled evenly between the Active Times to avoid creating a

compressed hour

10-second Snapshot

120-second Snapshot

Method 2: Active Time Distribution – Example (con’t)

Documents

A National Resource Working in the Public Interest © The MITRE Corporation. All rights reserved.Oct-15 MITRE Performance Testing: Load Testing With Usage