IBM Performance Automation Using RPT, Build Forge and Open Source- Amitabh

5/16/2018 IBM Performance Automation Using RPT, Build Forge and Open Source- Amitabh - slidepdf.com

http://slidepdf.com/reader/full/ibm-performance-automation-using-rpt-build-forge-and-open-source-amitabh 1/14

IBM Testing Services

© 2008 IBM Corporation

TTQ

@ 2009 IBM Corporation

Amitabh Kaushal – Performance Test Manager

Vivek Sharma – Performance Test Lead

20 November 2010

Performance test automation using IBM Rational BuildForgeand IBM Rational Performance Tester




© 2009 IBM Corporation2

Agenda

Case Study

Introduction & Background

Performance Test Life Cycle

Urgence Performance Framework

Automation of Performance Test

Test Execution Sequence

Comparison with Urgence Performance Test Cycle

Major Benefits of using Urgence Framework

Summary





Largest European Bank in UK : Case StudyClient Business

Areas Served

Engagement Details

LargestEuropean

Bank

Trades & SettlementLifecycle - Trade

Capture & vesting,

Messaging likeSWIFT, CREST

Positions & Balance,Settlements,

Statutory reportingand Reconciliations

Global Markets

Derivatives

Type of Service : Managed Test Services

IBM is providing managed testing services, to assistone of the largest European bank in performingoffshore Managed Testing Services for trading andsettlement application In financial market service.

IBM responsibility:

Provide expertise in testing transition in strategic level

Responsible for transition planning and execution

Help obtain knowledge transfer and documentation

from Customer during transition period Designs test process and framework

Formulate test strategy, test plan and requirementtraceability

Develop test scenarios, functional test cases, testscripts, execute testing, record and compile results

Performance Testing, Test Automation andRegression improvement

Day-to-Day management of testing services

IBM has successfully completed testing of severalreleases of developed applications

Benefits realised

1. 50% Reduction in Performance Regression

Test Execution Timelines

• Indexing/cataloging of Regression Test

Pack

• Complete Automated Solution

• Effective risk-based approach

1. SLA-Based Performance KPIs

• Test Quality

• Schedule Adherence

• Test Effectiveness

1. Significant Knowledge Management

Improvements

• Reduction in reliance on Customer’s

SMEs

• Greater flexibility in test execution

1. On-track for 16% reduction in fixed-costs of

Service

2. Service Improvement initiatives around

• Performance Testing effectiveness and

methods

• Further Test Automation in Other

projects

• Driving Quality Methods further up the

SDLC





Introduction and Background

Customer’s application is a ‘Trades Settlement Platform’ which processes ~2.5M trades/day, supports 33Back Offices in all major financial Geographies, Europe’s biggest Oracle Database

IBM currently delivers a range of Testing Services for this Platform. Customer challenged IBM to provide“… specialised, on-demand, repeatable and cost effective (offshore) performance test service for thisapplication. This will be a different delivery model to SIT, Regression testing etc.”

IBM did deliver a repeatable, on-demand and cost effective Performance Regression Testing solution withthe following attributes:

– Cover end to end Performance Test Lifecycle (Test Preparation – Shakedown – Execution – InterimReporting - Final Reporting), using IBM Rational tooling and IBM Research innovation.

– Test Phase duration got significantly reduced (target was 50%)

– An automated “chassis” with rules-based Environment deployment, performance tests automationcapabilities, threshold driven automated report generation.

– Based around the capabilities of IBM Build Forge product which forms the foundation of the Service

– Uses IBM's Rational Performance Test (RPT) tooling for test execution of the online testingfunctionality

– Combines existing monitoring augmented by Unix level Monitoring tools / integration with Optier-Corefirst / online transaction timing metrics.

– Combined uses of Open Source tool with others to monitoring server side metrics for analysis andscalability - PMRE, SQL Developer, Oracle OEM, NaGIOS etc.





Performance Test Life Cycle

P e r f o r m a n c e R is k

W o r k s h o p

W o rk s h o p

E x i t R e p o r t

T e s t

P r e p a r a t i o nC1 I m p l e m e n t

P E R F O R M A N C E

L CD e t a ile d P e r f o r m a n c e

T e s t P l a nC1C1 R e p o r t

The performance test

lifecycle primarily

consists of:

1. Performance Risk

Workshop to decide

the scope of the

performance testingand the test mitigation

for the risks the project

will introduce.

2. Workshop Exit

Report

3. Detailed

Performance test Plan

which consists of the

scope, approach, risks,

dependencies and

estimated schedule for

the performance test.





Deploy Test Data

Install Test Tools

Validate Test

Environment

Readiness

Generating Test

Reports

GUI Application

Define and use

Build Forge to

set up Test

environment

Storage of RPT

scripts and data Preparation,

Execution and

monitoring of

Performance Test

BF AgentBF Agent

RationalBuild Forge

Test Data

Test preparation, Execution and Monitoring are automated with Rational Build Forge andRational Performance Tester

RationalPerformance Tester

RationalPerformance Tester

Performance Test Environment

Solaris, Win, Linux boxes with

Database Servers

Performance Test Environment

Solaris, Win, Linux boxes with

Database Servers

Urgence Performance Framework

Open Source Tool for

monitoring

Open Source Tool for

monitoring





Automation of Performance Test – Preparation, Execution, Monitoring

Test Preparation

• Prepare test input files using IBM Build Forge

• Execute Performance Health Check using IBM RPT, BuildForge, OpenSource tool for monitoring

• Environment validation – Automatically raise JIRA/GIMS (incidentmanagement system) if required processes are not running in the test system

Test Execution

• Schedule multiple Test runs for various regions – APAC, EUROPE, USA• 12-20 hrs online tests running in-parallel with IBM Rational Performance

Tester. This whole end-to-end execution is fully automated now.

Test Monitoring & Reporting

• Continuous monitoring of different databases to assess the trade flow usingOEM, SQL Developer, build forge

• Server resource utilization monitoring using PMRE and native Solariscommands - Prstat/Top, IOstat, netstat. Use in-house shell scripts to capturechannel throughput.

• Raise JIRA automatically and send notification mail to stakeholders.





T: 1 T: 1 T: 1 T: 1

eSpear Required Process Monitoring ( minute? Loop)11

Server Monitoring ( minute ? Loop)1

T: 1 T: 1 T: 1

Online RPT T esting

TK Batch

T: 1 T: 1 T: 1 T: 11 T: 11 T: 11 T: 11 T: 11 T: 11T: 11 T: 11 T: 11 T: 11 T: 11 T: 11 T: 11T: 11 T: 11

HK Batch

UK Intraday

UK Settlement

UK Monitoring

UK Batch

US Intraday

US Monitoring

US Batch

Batch Results

TK Intraday

TK Monitoring

HK Intraday

HK Monitoring

TK Settlement

Intraday Results

Online Te st Results

Urgence Test Execution SequenceFollowing scheduled tasks are driven by Buildforge – comprises of the entire performance test steps

Test ExecutionScheduling

Test execution consists of

running online tests for 12

hrs in parallel to the

intraday tests for various

regions at scheduled

time.

The servers are

monitored the entire

duration of the tests to

view CPU utilisation.

These tasks are

sequenced as shown and

executed at the specifiedtime in Build Forge.





S t e p

P l a n n

S t a r t T i

( B S T )

P l a n n

D u r a t i

( H H : M

A c t u a

S t a r t T i

( B S T )

A c t u a

E n d T i

( B S T )

P r e r e q u

S t e p sO w n e r J I R A E x e c u t io n - O n l in e , I n t r a d a y ( T K / H K / U K / U S ) , B a t c h , S e t tl e m e n t sS t a tu s A s s u m p t i o n s /C o m m e

11 :1 11 :0 00 :0 00 :1 11 N A I B M U r g e n c eP O P S -0 0 0 0 0R a i s e J I R A t oB o u n c e w e b A p p l i c a ti o n s & C l u s t e r sh o s te d o n e S P E A R A p pP e r fo r m a n c e T e s t

C o m p le t e d

11 :1 11 :1 11 :1 11 :1 11 11 D S O P S D S O P S w i l l b o u n c e th e s e r v e r a n d n o t i f y u s . T h e y w i l l t a k e h e lp fr o m T A t e a m i f n e e d e d .C o m p le t e d

11 :1 11 :1 11 :1 11 N A I B M U rg e n c e C h e c k B u s in e s s d a t e s fo r a l l r e g io n s C o m p le t e d

11 :1 11 :1 11 :1 11 11 I B M U r g e n c e C h e c k P u b l i c H o l id a y f o r c u r r e n t w e e k C o m p le t e d

00 :1 11 :1 11 11 I B M U r g e n c eI f to d a y ' s d a t e ( F u l l- ru n ) i s a P u b l ic H o l id a y f o r a n y r e g i o n t h

d a y t o n o r m a l b u s in e s s d a y .N A

11 :1 11 :1 11 11 D S O P S D S O P S w i l l c h a n g e t h e d a y a s b u s i n e s s d a y f r o m h o l id a y a n d n o t i f y u s .N A

00 :1 11 :1 11 :1 11 :1 11 11 I B M U r g e n c eO E T A -1 1 1 1C h e c k s t a tu s o f C h a n n e l S p a c e J I R Ar a is e d a g a in s t T A o n T + d a y , u p d1c h a n n e l s t a t u s

C o m p le t e d

11 :1 11 :1 11 :1 11 11 T A O E T A -1 1 1 1T A w i ll c h e c k t h e c h a n n e l s p a c e a n d p u r g e t h e d a t a , i f n e e d e d .C o m p le t e d

11 :1 11 :1 11 :1 11 :1 11 N A I B M U rg e n c e C h e c k B F A g e n t s S ta t u s o n a l l b o x e s in U A T C o m p le t e d

11 :1 11 :1 11 11 T A C h e c k B F A g e n t s S t a t u s - I f d o w n , e s c a l a t e i t t o T A N A

11 :1 11 :1 11 :1 11 :1 11 11 I B M U r g e n c e C h e c k th e s ta t u s o f A p a c h e S e r v e rs , I f s e r v e r s t a tu s is d o w n , r a is e O E T A J IR A a g a i n s t T AC o m p le t e d

11 :1 11 :1 11 11 T A T A w i l l b r in g u p t h e A p a c h e s e r v e r a n d n o t i f y u s N A

11 :1 11 :1 11 :1 11 :1 11 N A D B A s O E D B A -1 1 1 1 1R a i s eO E D B A J I R At o R u nD B i n v a l id R e p or t o n U A T e n v a n d m a k e s u r e D

g o o d h e a l t h b y c o n f i r m in g w i t h C o m p o n e n t T e a m

C o m p le t e d

11 :1 11 :1 11 :1 11 11 C o m p o n e n ts I f D B I n v a l i d c a n c a u s e p ro b le m , c o m p o n e n t w i l l d e p lo y re c o m p i le b a s e l i n e .C o m p le t e d

11 :1 11 :1 11 :1 11 :1 11 N A I B M U r g e n c e C h e c k f o r J A V A P o l l e r p r o c e s s s t a t u s , r a is e J I R A i f n e e d e d C o m p le t e d

11 :1 11 :1 11 :1 11 :1 11 N A I B M U r g e n c eC h e c kM M t a s kc o u n t i n t a b l ee x c e p t i o n _ l o g , m m _ o p e n t r a d e , m m _

m m _ e n g i n e _ r e s o u r c ea n d t r u n c a t e th e m i f n e e d e d .C o m p le t e d

11 :1 11 :1 11 :1 11 11 D B A s O E D B A -1 1 1 1 1R a i s e r e q u e s t t o D B A t e a m t o t r u n c a t e t h o s e t a b le i f d o n ’ t h a v e a c c e s s t o d o s o .C o m p le t e d

111 :1 11 :1 11 11 I B M U r g e n c eC h e c kT a s k s u b s c r ib ea n d t a k e a c t io n t o c l e a r it , i f n e e d e d t a k e s u

t e a mC o m p le t e d

111 :1 11 :1 11 111U r g e n c e +

e t t l e m e nM a k e s u r e t h o s e T a s k C h a n n e l a r e c le a r e d . N A

U R G E N C E P e r fo r m a n c e T e s t E x e c u t i o n P l a n & A c t i v i t y C h e c k l i s t





Process Monitoring Reporting Environment Status





Standard Performance Test Cycle

~2.2 weeks cycle time reduced to 5 days

6 days effort saved on plan per Cycle (typically critical-path)

60 days saved on plan per year (based on 4 releases per year) and being utilised for more run now

Within Test Execution Task there can be many ‘runs’, and therefore more robust performance metrics

ID Task Nam e Start F in ish Duratio n

Dec 11 1111 Ja n1 1111 Jan11 1111 Ja n11 1111

1 1 1 1 1 1 1 1 1 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11

1d1/ /11111111/ /11111111Preparation

1 d1/ /11111111/ /00000000Reports

1 d1/ /11111111/ /11111111Test Execution

1 d1/ /11111111/ /11111111Deploy/Refresh

Ja n11 1111 Fe b1 1111

11 11 11 11 1 1 1 1 1 1

1 d1/ /11111111/ /11111111 An a lysis

ID Task Nam e Sta rt F in ish Du ra tion

D ec 11 1111 Ja n1 1111 Ja n11 1111 Jan11 1111

1 1 1 1 1 1 1 1 1 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11

1 d1/ /11111111/ /00000000Preparat ion

1 d1/ /11111111/ /00000000Reports

1 d1/ /11111111/ /11111111Test Execut ion &

An a lys is

1 d1/ /11111111/ /11111111Deploy/Refresh

Ja n11 1111 Fe b1 1111

11 11 11 11 1 1 1 1 1 1

Urgence Performance





Major Benefits of using Urgence Performance Framework

Re-usability and Modularity

- Used Build Forge feature to create common projects across environments wherever

it is applicable

- Parallel execution possible at a time in 2 or more environments

Reduced / Eliminated dependency

- Reduce Support Team and SME involvement for providing result.

- Usage of licensed software like TeamQuest, Loadrunner etc.

- Dependency on Technical Architect and DBA group to provide data and analyse result- Performance tester involvement in checking process down and raising JIRAaccordingly, reduce tester effort

- Resource Utilization (Server Stats) using external tool like – TeamQuest, Analyzer

Automation – Reduction in manual efforts

- Creation of test data (bulk trade files and online data)

- Trade Monitoring (to collect vesting results)

- Use Multi-threading concept to speed up the execution process

- Can run unattended performance test (using scheduling feature in BF)

Central Repository for all test results





With Urgence Framework, Customer has achieved significant…

Reduction in test timelines Shorter test execution runs (5

days for 3 run)

Fewer attempts to get to first

good run

Get the analysis done and find

the bottleneck by comparingdifferent run results

Increases time dedicated toanalysis

Automated testing will provide

answers for “how does release Y

equate to release X”

Opens the door to aggressive

tuning activities “tweak a

variable, run the test, tweakagain, run the test”.

Raise the consistency and

confidence in Performance

testing.

Reduction in human resourcecosts

Resources running the tests will

have lower rates – completely

offshore

SMEs will spend less time on

Performance Testing

Reduced dependency on Support

Team(TA, DBA) to fetch baseline

stats

Eliminating manual, error prone

tasks

Repeatable and consistent

processes for improved efficiency





Summary

Using Build Forge, RPT and Open source tool we have automated the end to end

performance testing for customer’s application.

We have demonstrated reduction in overall test cycle time, ability to execute the tests 'On

Demand' in a cost effective way.

We have reduced manual efforts by introducing automation (i.e. creation of test data,

Trade Monitoring, JIRA, running unattended performance test with the scheduling feature

in Build Forge).

Performance test scope is now decided early in the release lifecycle by conducting a ‘risk

workshop’ to discuss the non-functional requirements.

Specific deliverables like risk workshop exit report, test strategy, test plan are delivered in

different phases of the performance test life cycle and obtained a sign off before the

execution of tests.

In Summary, we were able to realise the customer's vision for an 'On

Demand' performance test capability.

Documents

IBM Performance Automation Using RPT, Build Forge and Open Source- Amitabh