Ramesh Padmanabha's slides on Mercury Interactive LoadRunner

April 16, 2010April 16, 2010 Entegration SoftwareEntegration Software 11

Performance Load TestingPerformance Load TestingCase Study – Agilent Case Study – Agilent

TechnologiesTechnologies


AgendaAgenda

IntroductionsIntroductions BackgroundBackground Testing ObjectivesTesting Objectives Preparation PhasePreparation Phase Execution PhaseExecution Phase Analysis Analysis Lessons LearntLessons Learnt Contact InformationContact Information


IntroductionIntroduction

Ramesh PadmanabhanRamesh Padmanabhan Entegration SoftwareEntegration Software Consulting & product company based Consulting & product company based

in San Josein San Jose Proud to be service partners ofProud to be service partners of

Oracle CorporationOracle Corporation

Mercury InteractiveMercury InteractiveYash TechnologiesYash Technologies


IntroductionIntroduction

Agilent TechnologiesAgilent Technologies $6 Billion Global Mfg Company$6 Billion Global Mfg Company Over 30,000 employees in more than Over 30,000 employees in more than

50 countries50 countries One of the largest global single One of the largest global single

instance installs of Oracle E-business instance installs of Oracle E-business suitesuite

Consolidated over 150 legacy systemsConsolidated over 150 legacy systems Expect a maximum 5,000 concurrent Expect a maximum 5,000 concurrent

usersusers


BackgroundBackground

Largest single instance installLargest single instance install 3 HP Superdomes –Production, 3 HP Superdomes –Production,

Reporting, PlanningReporting, Planning Single US based data centerSingle US based data center Over 50 operating unitsOver 50 operating units Significant business volume in Significant business volume in

Asia & EuropeAsia & Europe Consolidating over 125 Consolidating over 125

different legacy systemsdifferent legacy systems Implemented all Financial & Implemented all Financial &

MFG ModulesMFG Modules


Testing ObjectivesTesting Objectives


Testing ObjectivesTesting Objectives

Validate single instance strategyValidate single instance strategy Validate network and hardware Validate network and hardware

infrastructureinfrastructure Scalability to 5000 concurrent usersScalability to 5000 concurrent users Stress test for “high water mark”Stress test for “high water mark” Set user response time expectationsSet user response time expectations Identify and fix significant performance Identify and fix significant performance

tuning issues within Oracle Applicationstuning issues within Oracle Applications Identify and drive solutions for hardware Identify and drive solutions for hardware

issuesissues


Preparation PhasePreparation Phase


Data GatheringData Gathering

Identified major transactions within each Identified major transactions within each application moduleapplication module

Questionnaires sent for legacy data Questionnaires sent for legacy data volumes by geography (US, Asia, Europe)volumes by geography (US, Asia, Europe)

Short listed transactions with high volume Short listed transactions with high volume or data intensive processingor data intensive processing

Identified user distribution by region and Identified user distribution by region and by application areas by application areas

Determined estimation methodology for Determined estimation methodology for inquiry transactions inquiry transactions


Hardware PreparationHardware Preparation

Ensure that the production configuration of Ensure that the production configuration of back-end server and middle tier machines back-end server and middle tier machines were set-up and configuredwere set-up and configured

Procure the Load generation agent boxes Procure the Load generation agent boxes and have them installed and setup at the and have them installed and setup at the right locations right locations

Ensure that the Cisco load balancing router Ensure that the Cisco load balancing router was correctly set upwas correctly set up

Set up network ‘sniffing’ devices to get Set up network ‘sniffing’ devices to get detailed metrics of network traffic detailed metrics of network traffic


Software PreparationSoftware Preparation

Procure and install LoadRunner on the Procure and install LoadRunner on the agent and controller boxesagent and controller boxes

Install LoadRunner and the Oracle Install LoadRunner and the Oracle Applications client on the machines of Applications client on the machines of the scriptersthe scripters

Install/Setup other database Install/Setup other database monitoring softwaremonitoring software

Prepare scripts for detailed Prepare scripts for detailed transaction analysistransaction analysis


Data PreparationData Preparation

Validated various application Validated various application setups setups

Initial cycles required all key Initial cycles required all key master data to be fabricatedmaster data to be fabricated

Developed numerous scripts to Developed numerous scripts to extract key data elements like extract key data elements like items, customers, vendors etc. to items, customers, vendors etc. to be used in transactionsbe used in transactions

Ensured adequate breadth of data.Ensured adequate breadth of data. Identified key data and parameters Identified key data and parameters

for background loadfor background load


Develop LoadRunner ScriptsDevelop LoadRunner Scripts

Recorded scripts for all the critical and Recorded scripts for all the critical and high volume transactionshigh volume transactions

Adequate mix of inquiry and update Adequate mix of inquiry and update txns.txns.

Parameterized all the critical pieces of Parameterized all the critical pieces of data like item, customer, orders etc.data like item, customer, orders etc.

Identified activities for which server Identified activities for which server response times were key and set up response times were key and set up transaction timers around them e.g. transaction timers around them e.g. commits, quick-picks etc.commits, quick-picks etc.


Execution PhaseExecution Phase


Build Test ScenariosBuild Test Scenarios

Develop matrix for users by Develop matrix for users by geography by transactiongeography by transaction

Manual scenariosManual scenarios Goal oriented scenariosGoal oriented scenarios Transactions split into three Transactions split into three

groups based on data groups based on data dependency conditionsdependency conditions


Run Tests…Run Tests…

5 cycles of testing5 cycles of testing 1- validation cycle1- validation cycle 2 – complete cycle with converted data2 – complete cycle with converted data 3- Stress test cycle3- Stress test cycle 4- Complete integrated test with key 4- Complete integrated test with key

interfaces and customizationsinterfaces and customizations 5- Production simulation run5- Production simulation run

Each cycle comprised of two major Each cycle comprised of two major runs/day for two weeks. Each test runs/day for two weeks. Each test run was about 4-7hrs longrun was about 4-7hrs long


Run Tests…Run Tests…

5000 concurrent user load 5000 concurrent user load generated from 8 LoadRunner generated from 8 LoadRunner agents – 4 in US, 2 each in agents – 4 in US, 2 each in Europe & AsiaEurope & Asia

LoadRunner monitors set up for LoadRunner monitors set up for network, backend server & network, backend server & middle-tier boxesmiddle-tier boxes

Dedicated DBA and performance Dedicated DBA and performance tuning experts monitored the HP tuning experts monitored the HP Superdome serverSuperdome server


Analysis Analysis

Used LoadRunner Analysis toolUsed LoadRunner Analysis tool Real time graphical interface to Real time graphical interface to

monitor the test progressmonitor the test progress Post run analysis includes numerous Post run analysis includes numerous

graphs and transaction timersgraphs and transaction timers More detailed analysis was done More detailed analysis was done

from the result data stored by from the result data stored by LoadRunner in an Access databaseLoadRunner in an Access database


AnalysisAnalysis

Data from the analysis used toData from the analysis used to Set up realistic response time Set up realistic response time

expectations from the end usersexpectations from the end users Modify various database parameters Modify various database parameters

in the init.ora to better performancein the init.ora to better performance Tweak settings of the Cisco load Tweak settings of the Cisco load

balancer for middle tier machinesbalancer for middle tier machines Identify and tune some of the Identify and tune some of the

application code that had bad application code that had bad performanceperformance


LimitationsLimitations

Some performance intensive processes Some performance intensive processes could not be tested due to data could not be tested due to data dependency issues e.g. lock-box receiptsdependency issues e.g. lock-box receipts

Some dynamic and interactive processes Some dynamic and interactive processes could not be tested very well e.g. could not be tested very well e.g. configured ordersconfigured orders

Some custom code not stable till the last Some custom code not stable till the last cyclecycle

Some of the newer application modules Some of the newer application modules not stable for a reasonable testnot stable for a reasonable test

Application version and patch set lagsApplication version and patch set lags


Lessons LearntLessons Learnt

Performance test will only be as Performance test will only be as good as the data collected in the good as the data collected in the analysis phaseanalysis phase

While performance test can While performance test can significantly reduce risk of poor significantly reduce risk of poor performance, it is not a guarantyperformance, it is not a guaranty

Initial performance testing cycles Initial performance testing cycles should focus more on non-code should focus more on non-code related performance variables related performance variables


Lessons LearntLessons Learnt

Intensive code related performance testing Intensive code related performance testing & tuning should take place after custom & tuning should take place after custom solutions have been put into testing and solutions have been put into testing and application patch sets are frozenapplication patch sets are frozen

Performance testing should be in the Performance testing should be in the critical path of project plan and critical path of project plan and performance testing instances should be performance testing instances should be patched just like the BST instancespatched just like the BST instances

Should plan on at least one marathon Should plan on at least one marathon testing run that extends for 3 or 4 daystesting run that extends for 3 or 4 days


Contact Information

Ramesh PadmanabhanEntegration Softwarerpadmanabhan@entegration.com408-674-3701www.entegration.com

mailto:[email protected]

Documents

Ramesh Padmanabha's slides on Mercury Interactive LoadRunner