SAP best practices for performance testing

1 ©2010 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice

SAP Best practices for performance testing

Andreas Gloege, QRM Solutions, SAP Americas

“Quality & Testing” - Biggest Pain Point of Upgrades

83%

17%

% Members

In the

Process of

upgrading

Not

upgrading

Major Cost and Time Factors during Upgrade or Implementation,

% Customers

QA & Testing

Change Mgmt.

Planning

Training & Doc.

Resource Mgmt.

Complexity

Project schedule

n=52

0%

5%

7%

12%

16%

30%

75%

Testing and coverage is the #1

stated pain point for all

customers

Over 86% of customers are

concerned about risks due

to lack of comprehensive

testing

Complexity and project

schedule are rated much

lower compared to the rest

Source: ASUG Test influence council member survey

© SAP 2007 / Page 3

Why Performance Test?

To ensure implementation complies to non-functional requirements at both

individual business processes and a system wide level

Identify and remove bottlenecks in individual

transactions

Improve codeChange hardware configurations

Modify the business process

Determine system scalability under load

Measure KPIs at reasonably expected peak volumes (100% load scenario)

at increased volumes (100% + n% load scenario)

Understand the impact of peak processing times by simulating the exact processes in a non-production

system before go-live

© SAP 2007 / Page 4

Performance Testing Objectives

Note: Proper testing methodology is required to identify *real* bottlenecks. Improper testing will lead to bottlenecks that will never occur in real world conditions

What are the Risks ?

The primary objectives of System Performance Tests are to reduce risks associated with performance issues before go-live

Data processing/transformation not able to complete in required time

User productivity is decreased due to poor system response time

User perception of the system is poor due to transactional response time

How To Reach The Objectives ?

Testing against Service Level Agreements (SLA) SLAs may be a function of a business requirement Define a strategic approach to testing based on Best

Practice and follow the plan Eliminate Bottlenecks occur in infrastructure

(hardware) and/or the application layers test approach must provide coverage in both areas

© SAP 2008 / Page 5

Project Phase Overview

Preparation In the first phase we will work in conjunction with the project managers, process teams and technical teams to perform such preparation tasks as planning, designing, and configuring the environment.

Development In the second phase test scripts and scenarios will be developed using LoadRunner following the test plans prepared in phase one.

Execution/Analysis The third phase includes running the scenarios, gathering data, analyzing system performance, identifying bottlenecks and tuning. This phase is cyclical requires multiple iterations to meet load test goals.

Reporting In the final phase reports are developed and reviewed with project stakeholders.

© SAP 2008 / Page 6

What to Test - Conceptual

What uses system resources?

Online Users

Data processing online/background

Goal:

Identify peak usage times

Simulate peak load (users and data processing)

SAP System

Users executing online

transactions

Incoming Interfaces

Outgoing Interfaces

Background Processes

(batch)

© SAP 2008 / Page 7

What to Test - Modeling

The first step in setting up the test scenario is choosing the business processes to be investigated.

Work out a 24-hour time-grid showing the planned execution time-windows and associated processing volumes for each business step involved in the test. Identify all load scenarios that may differ from this model. The test scenarios that are non-critical or already included in another scenario can be neglected.

Example:

Similar modeling required for business activities happening• Weekly, Monthly• Quarterly, Yearly

© SAP 2008 / Page 8

What To Test: Online Transactions

Core business processes as well as other processes known to be time-critical or business critical.

The processes chosen for testing online transactions are selected based on the following criteria:

Transaction CriticalCore transactions which must be run in order to establish conditions for running other processes (Example: Login, Search Invoices, Create Billing Docs).

Heavy ThroughputHigh volume with respect to concurrent users and high transactional throughput (Example: Period End Reporting).

Dynamic ContentPrograms that require high level of interaction with the database (Example: Create orders w/ multiple schedule lines, ).

Customized ComponentsRICEFW Development Objects / Z programs / User exits have higher risk of failure than standard (“out of the box”) functionality.

© SAP 2008 / Page 9

What To Test: Batch/Interface

Custom interfaces and batch jobs from each process being implemented are selected for testing based on the following criteria:

Heavy Throughput during business hours with expected heavy user load

Complex logic on SAP or middleware tier

Time sensitive batch processes. (Example: Starts at 3AM and must be complete by 6AM)

Business Critical

Is *NOT* A One Time Data Load

Interfaces can be considered as critical if they have one or more of the following requirements:

High throughput requirements (Rules of thumb)

– Maximum data volume per hour > 1,000 or

– Average data volume per day > 10,000 or

– Maximum number of documents per hour X Response time [sec] > 3600

– High data volume >10.000 data records or documents x line items per day

Requirements for good response time are (rule of thumb):

– < 2 seconds for synchronous interfaces (sender waits for response of receiver)

– < 1 Minute for asynchronous interfaces (sender does not wait)

When Is Performance Testing Required?

Test performance of your projects:

New SAP installations/ deployments

Upgrades/Updates

Application of Support Packages and Legal Change Packages

In-house developments

Consolidations

Capacity planning and sizing

SAP Application Lifecycle

Capacity

planning for

hardware

Building

performance

scripts

Load

testing Stress testing

Diagnostics

for quick

problem

resolution

Project PrepContinuous

ImprovementRealizationBlueprint

Final

Preparation

Go Live

and Support

Manual or Automated Performance Testing

What Is Automated Load Testing?

Replaces real users with thousands of “virtual” users

Generates accurate, measurable and repeatable load on the

system from a single point of control

Pinpoints bottlenecks in the system

Web Server App. Server DatabaseInternet/

WANSimulated

UsersController

Emulates production workloads on an IT system

What Tools Are Recommended by SAP ?

“The only load testing vendor to increase market share every year since 2000 …..”

Source: Yankee Group 2005 Source: Gartner Research (March 2005)

HP77%

Other 1%

Empirix

2%

Spirent

2%

Rational

9%

Other

6.7%

Compuware

3% Segue

4%

SAP LoadRunner by HP and SAP Performance Center by HPThe industry standard

Manual

Automation

Standardization

Centralization

Performance Authority

Level 0

Project based manual testing

Project-by-project

automation

Standard common testing

infrastructure standardization

Center of Excellence,

best practice processes

shared across projects and

teams

Centralized testing

services, standardized performance

lifecycle for all projects

Level 1

Level 2

Level 3

Level 4

SAP Performance Center by HP

SAP LoadRunner by HP

Performance Testing - Maturity Model

Why SAP LoadRunner by HP ?

Ease of use reduces the learning curve and script

development time

Breadth of environment support helps assure that all

enterprise applications are covered

Depth of analytical data allows fast triage of

problems and the information needed to fix them

quickly

SAP Internal Usage

Industry Standard

Market-leading capabilities mitigate

performance risk and accelerate application delivery

LoadRunner Components

© SAP 2007 / Page 15

LoadRunner ControllerRun the Load and Performance Testing Scenarios.

LoadRunner Virtual User GeneratorCreate Virtual User scripts by using recording on a specified single or multi-protocol.

LoadRunner Virtual UsersOne Virtual User script can be executed several hundreds or thousands of times in parallel to simulate hundreds or thousands of real users. While this is happening the response times for specified end-user steps will be measured for every single user.

LoadRunner Monitors and DiagnosticsWhile Load Test is running the LoadRunner Controller can use Monitors and Diagnostics capabilities to monitor the different components of the infrastructure under load. Using the Diagnostics capabilities LoadRunner provides the insight in different applications and infrastructure to identify directly where the response times was lost

LoadRunner AnalysisThe LoadRunner Analysis module is now providing the offline analysis where the response times from the virtual users, plus the monitoring information and the diagnostics details come together to enable the performance team for full offline results and route cause analysis.

How Does LoadRunner Work?

Record an application business process into an automated scripte.g. “Create New Order” in an ecommerce web application

Add end-user transactions, parameterize input data, add verification checkpoints

Apply user behaviors such as network connectivity, pacing etc…

Application

Scripts

“CREATE NEW ORDER”

VUGen Recorder

LoadRunner VuGen

SAPGUI or

Browser


Design and execute Load Scenarios that mimic production conditions

Emulate hundreds or thousands of concurrent users from one or more locations

Gather end-to-end transaction response times

e.g. “Logon”, “Select Product”, “Process Credit Card”, and total “Create Order” process

Collect application, network and system data across all architectural tiers and components

DIAGNOSTICS

MONITORS

LOADBALANCER

WEBSERVER

APPLICATION SERVERS

DATABASENETWORKCONTROLLER LOADGENERATORS

LoadRunner Controller

VUsers

US

Europe


© SAP 2007 / Page 18

SAP Load Runner by HP

Analyze Online &

Offline Results

Test SAP R/3

Diagnostics

Isolate/pinpoint bottlenecks in SAP R/3

Comprehensive alerts for proactive analysis

Delegate problem resolution to the right team

Performance Testing

Load test SAP GUI, SAP R/3 transactions and Enterprise Portal based Applications

Capture the results

Analyze results, fix issues & rerun tests

LoadRunner SAP Diagnostics

SAP LoadRunner by HP Summary

Industry Standard solution for application performance testing

Single platform that enables a consistent, repeatable process for:

• Performance testing critical business processes

• Testing across a wide breadth of applications and protocols

• Monitoring system and application data during tests

• Result analysis and problem isolation

Support a broad set of technologies including performance testing for web 2.0 applications

Isolate performance bottlenecks using analysis and diagnostics

Shorten performance testing cycle time and reduce total cost of ownership

As Customers Mature…

…the Next Set of Problems

Multiple projects• Scheduling

• Auditing

• Reporting

• Security

Multiple people • Global access

• 24x7 access

• Privilege management

• Asset sharing

Growing Testing Needs

Apps

Expertise

Infra

……

Localized testing teams with individual tools have to

buy more for new projects

Apps

Expertise

Infra

Apps

Expertise

Infra

Testing needs

increase for major

initiatives:

• ERP Upgrades

• Web 2.0

• SOA

• Virtualization

Need for Centralization

Team B

Team DTeam C

Applications

Expertise

Infrastructure

Team A

Improve efficiency by moving to Center of

Excellence

Central testing team with

resources to support multiple

projects

Reasons for Center of Excellence

Eliminate inefficienciesCan’t utilize or reach available resources to maximize application testing

Drive best practicesLack processes required to reach performance goals

Test more applications more completelyApplication complexity and number quickly overtake available resources

Minimize IT riskPerformance varies greatly from project to project and from LOB to LOB

Forces driving centralization


Summary

SAP Performance Center by HP is the de-facto standard for enterprise level performance validation

24x7, global access

Pooled licenses and vusers

Shared and reusable testing assets

Central and standard testing

Schedule / Auto Start tests

IncreasedProductivity

• Standardized best practices (less duplication)

• Increased expertise due to knowledge sharing

• 24x7 concurrent usage model (follow the sun)

Why Moving from SAP LoadRunner by HP to


Increased Efficiency, Better Application Quality

IncreasedUtilization

• Load testing available to all projects

• Visibility promotes ubiquitous load testing

• More flexible access to assets promotes use

ReducedCosts

• Lower acquisition costs (pooled model)

• Lower maintenance costs (central infrastructure)

• Lower staff costs (training)

© SAP 2010 / Page 26

ASAP Methodology v7.0

Defines How to Use SAP LoadRunner by HP and SAP Performance Center by HP

Contact Information

Andreas Gloege

QRM Solutions

SAP America

[email protected]

+1-650-690-2167

Thank

you!

mailto:[email protected]

Q&A

29 ©2010 Hewlett-Packard Development Company, L.P.

To learn more on this topic, and to connect with your peers after

the conference, visit the HP Software Solutions Community:

www.hp.com/go/swcommunity

30

Documents

SAP best practices for performance testing