Performance Testing Highlights

Sharad Jain

Date – 07th Feb 2011

What is Performance Testing

• Performance testing – refers to test activities on checking system performance

The major objectives of performance testing:

• To confirm and validate the specified system

performance requirements.

• To check the current product capacity to answer the questions from customers and marketing people.

• To identify performance issues and performance degradation in a given system

Differences b/w Performance, Load and Stress Test

• Performance Test determine the run time “behavior” of the application and its supporting infrastructure, under certain conditions.

• Performance testing is used to measure several system characteristics, such as processing speed, response time, resource consumption, throughput and efficiency.

• Load Test determine the applications “behavior under load”, up to and including its limits (not just as its limits).

• Load tests specifically refers to the load size (number of concurrent users) and related values.

• Stress Test determines the application ability to handle large amount of data

• Stress testing can be much more successful with the full load applied to the server.

Why Performance Testing?

• Scalability – Will the application handle the expected load and beyond?

• Stability – Is the application stable under expected and unexpected user loads?

• Availability – Is the application available to the end user without any interruption?

• Serviceability – Can the system quickly recover from a Failure?

• Speed - Does the application respond quickly.

• Confidence – Are you sure that clients will have a positive experience on go-live day?

Performance Requirements

• Following are some representative performance requirements which system would be expected to meet

• The system should support peak load of x active users and y transactions/sec. (for web based application it could be requests/sec)

• At peak load, system response to the X% of users should be Y Sec.

• System should be able to support 24 X 7 operation (reliability)

• System should be scalable to meet growth in demand.

• The response time requirement could vary based on

• Geographical location of users

• Connection bandwidth (especially true for Internet users)

• Transaction complexity.

Performance parameters to consider

• Throughput: The number of requests processed per unit time (per second)

• Latency: The time taken in establishing the network handshake.

• Efficiency: Throughput - Latency

• Degradation: The Throughput of the application when the concurrent requests are gradually increased – Test to see if the performance of the application degrades when the number of concurrent requests is increased.

• Longevity : Execute tests for a pre-determined time interval with a pre-determined load. Monitor the health of the application throughout the test execution and check for memory leaks and analyze the GC dump

What Performance Problem may arise ?

• Memory-related problems:

• Application uses more memory than it should.

• Memory leaks.

• Excessive garbage allocation, i.e. application creates a lot of temporary objects.

• Code-related problems:

• Application algorithms are not optimal, and there are performance bottlenecks.

• Hardware/Software related Problem:

• Hardware you select for your database is not in vendor Hardware Compatibility List

Scalability

• This section talks about the impact of scaling up the Hardware resources from the suggested minimum Hardware requirements

• Processor impact

• Database impact

• I/O impact

• Network bandwidth Impact

Processor Impact

• Single Processor v/s Multi Processor The response time for searching or archiving a document decreased by 50 - 80% on adding an additional processor in a multi-threaded scenario (info based on assumptions)

Data base Impact

• Single Processor v/s Multi Processor

• As the number of concurrent users using application services are increased, it is observed that Oracle performs better than MS SQL Server with a 5 - 8% improvement in the performance (info based on assumptions)

I/O Impact

• I/O has considerable impact on the performance of the Application with 15% to 18% reduction in the response time especially when there are huge reports to be generated on the system.

Network bandwidth Impact

• A good Network connectivity between the Client, Application Server and database improves the performance considerably. There was 40% (approx) improvement in the throughput of the application when the Network Connectivity between Database system and Application Server system was increased from 100 mbps to 1Gbps (info based on assumptions).

Identify the Performance issues

Application

Database

Application CodeApplication Design.JVM settingsApplication server configuration and usage

Web Server Configuration and usage

System OSSystem OS

Client Side

Database Schema, design, configurations, resource usage, SQLs, Indexes etc

OS Resources – CPU, Network, Disk, Memory,

OS kernel, Storage etc

Browser settings, Client system, Page size,

amount of data displayed

Performance Bottleneck Sources

Network8% Web Server

12%

App Server35%

DB Server45%

Database Server Causes

Other5%

Slow specific query response22%

Slow overall query response16%

Slow overall DB response13%

Overuse of row-at-a-time processing

11%

Missing indexes8%

Poor multi-user response7%

Query plan cache too small7%

Data cache too small5%

Deadlocks3%

Old optimizer statistics3%

App Server Causes

Memory leak15%

Inefficent garbage collection12%

Expired sessions still active12%

Poorly configured App Server12%Insufficient hardware resources

10%

Poorly configured DB connection pool

9%

Inefficiently coded transaction11%

Inefficient DB access architecture

4%

Inefficient object access method

5%Other10%

Network Causes

Bad network architecture20%

Other20%

Poorly configured/insufficient NICs10%

Security too tight15%

Insufficent overall bandwidth13%

Load balancing ineffective22%

Software and Hardware Requirements

• Hardware Requirement

• Details of system to be tested

• Workstation required for load simulation

• Workstation required for running monitoring utilities

• Network Equipment (e.g. Router, Firewall, Modems etc.)

• Software Requirement

• System Software

• Load Testing Tools

• Performance Monitoring utilities

• Miscellaneous utilities like graph generation etc.

Performance Testing Process

• Understand system and identify performance requirements

• Identify performance test objectives.

• Define performance test strategy:

• Identify the needs of performance test tools and define performance test environment

• Write performance test plan

• Set up the target system and performance test beds

• Design performance test cases and test suite

• Performance test execution and data collection

• Performance analysis and reporting

Performance Testing Approach

• Performance testing: (during production)

• Measure and analyze the system performance based on performance test data and results

• Performance simulation: (pre-production)

• Study and estimate system performance using a simulation approach

• Performance measurement at the customer site: (post-production)

• Measure and evaluate system performance during system operations

Performance Modeling Project Overview

In order to analyze performance behavior, improve product performance, and serve as the foundation for capacity planning

• The overall goal was to design and develop a model that would be:

• realistic and general enough to be widely applicable across a variety of customer configuration and application scenarios in Altair target market.

• simple enough to be easily usable by the field sales and support organizations to assist customers make system configuration decisions

• possible to validate so that it could be applied with confidence.

What is the resultant deliverable

The resultant of this performance testing exercise is a “Performance and Scalability document” with results of the tests performed and hardware recommendations. The document would not be a sizing guide per se but will have:

• Throughput and scalability numbers for the end users to consume

• Test metrics - numbers and graphs related to the tests executed

• And an internally consumable document containing performance recommendations for the developers to work on.

Also, note that the resultant document is not a Sizing guide. But this document will definitely be a good input to the performance team to come up with a sizing guide for the future releases.

Thank You