©2010 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice

Session ID: BTOT-TU-1800/1Twitter hashtag #HPSWU

©2010 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice

Speaker Name: Test Manager Maja Usbeck Cao and Test Specialist Claus Haslund AndersenDate: 30. november 2010Session ID: BTOT-TU-1800/1

How to use QTP and Quality Center for daily automated regression test Type: BTO Applications

Agenda ATP – the company and the test organisation

System landscape and regression test

Requirements for our testing tools and automated test

setup

The setup for automated testing

Communication with relevant stakeholders

Daily mail with graphs

Daily follow-up on the nightly runs

Progress on developing automated tests

How did we get there? Prerequisites for our autotest

setup

Brief presentation of ATP: Company tasks

Bankruptcy(pay compensation)

Maternity(salaries refunding)

Occupational disease(Workers’ compensation insurance)

HolidayHoliday pay)

Pension(Pension services)

Vocational training (internship grants)

Pension DanmarkJØP (Unit Link)

DA-Barsel

AERAESFerieKontoLGBarsel.dk

ATPSPSUPPLD

Pension og SecuritySecurityPension

Regulated tasks

Customers contact with the ATP Group

4.4 million Danes come in contact with the ATP Group's products. Total productsensures our customers a basic economic security during and after their working

lives.

Test OrganizationThe test resources

• 3 Head of Test

• 6 Test Managers (RTM, TI and TM)

• 9 Test coordinators

• 3 Test specialists (automated test and test data)

• Up to 4 IT-Testers and students workers (ITU and DTU)

• 23 Test consultants

Responsibility area

• Test in projects and Sustained Engineering

• Test methods, TDM and test tools (QC, QTP and Loadrunner)

Test Organization - automatisationCentralized automation team

• Building principles and guidelines• Coordination across the organization• Upgrade & support of the tool• Building regression test • Maintenance of automated test• Responsibility for the daily runs

Sometimes QTP developers are lent to projects for a period of time

ATP Business Platform - Logical overview

Execution of:• GUI transactions• Service requests • Batch runs

WebsphereWebsphere

Mainframe DB2

Portal

Kerne/.NET/DB2

Broker / Websphere MQ

SAP Doc./Papyrus/WAS Masterdata/WAS WFM/WAS

Client

Client

HTTP

Client

Portal 360o

SAP clientGuitalis

Regression test purposesRegression testing aims to test whether any errors occurs in already implemented functionality, when implementing new or changed functionality.

The extent of regression test and risk of missing regression test increases in proportion to the implementation of new functionality (see illustration).

Missing/incomplete regression test represent a quality risk

Release 1 2 3 54 6 7

Test of new functionality

Regression test of previously delivered functionality

Requirements for our Testing Tools• We want to use our current tools Quality Center and QTP

• We have to be able to run our test set on a scheduled basis: Daily, weekly and

individual dates

• Automatic re-running tests that fails the first time (based on percentage of failed

tests in a test set)

• Distribution of loads on x number of machines (so we continuously can expand our

setup)

• Grouping of servers, so test on different releases is possible (Sustained Engineering

and release)

• Troubleshooting - current screenshots of the machines and embedded error

detection in scripts

• We need to know the expected execution time for a test set (Crucial when time is

short eg. emergencies)

Don’t think: What you can get – Think : What you want!

Automation focus areas

Testdata Creation of data Deleting data

Environments Component Integration Pilot

ServicesAliveFunctional

Batch runs (via gui)Start of batch runCheck result

System testingDeep functional

Test of Service requests - solutionThere are 6 main systems providing services.

Application for communication (Broker).

The portal using these services.

WebsphereWebsphere

Portal

Kerne/.NET/DB2 SAP Doc./Papyrus/WAS Masterdata/WAS WFM/WAS

Client

HTTP

Portal 360o

Broker / Websphere MQ

One automated BPT component in Quality Center.

QTP

QCATSA

Automation Test environmentAutomated Test Scheduler Application (ATSA)

What is the ATSA and how does it work?

Daily status on the test executionDaily mail with information on the execution of automated tests

Daily monitoring of test executionThe main reason that there are 26 failed tests, time out and changed data.

Errors are described in details below (note that you can not compare the numbers directly, as part of the failed setup test cases are not counted astest cases)

Portal:PD: Passed.DFO: Errors due to environmental instability. There is created POB-action on that.SUPP: Passed

Service requests:Alive test:ATP: 2 Broker Time out (Defect 16159). PD: 5 Broker Time out (Defect 16159). Functional:AER: Passed.ATP: 3 Broker Time out (Defect 16159). PD: 2 fails because of environment setup (service call is send to wrong environment).

1 Broker Time out (Defect 16159). SUPP: 1 Broker Time out (Defect 16159).

Kernel:ATP: 2 errors in death batch job because of time out of the stakeholder replication

3 data / environment errors on K060.PD: 3 kernel time outs

5 Data / rounding errors on the letters and H1503 fails due to customer agreement does not work (H150). Defect has been created

SUPP: 9 Data / rounding error on the letters and F024 fails because of data / setup

Number of automated tests over time

8 10 11 13 16 18 22 23 25 26 27 28 29 30 32 33 34 35 36 38 39 40 41725

775

825

875

925

975

1025

1075

1125

Automated regression test progress over time

Week

Tes

tcas

es

Approach on automated test Standard for BPT (content, size and structure)

It is mandatory to use BPT in all regression tests

Education in BPT components (workshops on our

customers applications)

Review of the BPT components (both manual and

automated)

Code guideline

Code review

Automation team located in the same room

(communication)

Data discipline in QC is crucial (data for our graphs)

Test data and change of test environments

Test data used for the automated tests must be

reserved in all test environments

Solution established for deleting and reload of data in

our component test environments

The automated test generates data and cleans up data

How did we get this far Management focus on automation - it has taken a long time to

reach this far (maturity)

Visibility on automated testing - we get that by sending out a daily

mail with graphs and mark all defects found

We can test a large part of our regression tests automatically in

one night and it can be used for bug fixes in production

Over time - lower costs on our regression test

Centralized automation team is crucial for our success

Important to focus on maturity of the company (we are now TMMi

level 2 – moving towards level 3)

Continue the conversation with your peers at the HP Software Community hp.com/go/swcommunity