Customer Testing & Quality in Outsourced Development

- A Story from an Insurance Company

Version : 1.0

Created by: Gitte Oberbossel

Agenda

• RSA Scandinavia

• Background for the new test model

• RSA test model

• Implementation

• Today

2

OK, now you do exactly as I tell you!

RSA Scandinavia

Short facts

Part of RSA Insurance PLC UK

Scandinavia's third largest non-life insurance company

42 offices in Sweden, Denmark, Norway and Finland

Offers non-life insurance in the Personal and Commercial segments

2.5 million Personal customers and 200,000 Commercial customers

4,000 employees

3

• Locations

• RSA Scandinavia is located within Denmark, Sweden and Norway

• Brands we are known for:

RSA Scandinavia – Seen from a testing perspective

• Our area of expertise is insurance

− Most employees involved in testing have little or no knowledge of testing and testing models when they are first assigned to a project.

− Test managers can be in-house insurance employees or they can be consultants.

− We have a small test centre that works as process owner, and also supports and attends in quality evaluation.

• We are not an IT house

− The majority of the company does not think in or use IT terminology and processes – we use business terminology, and talk about business processes and market advances.

− The solutions we implement need to work all the time, or we lose a lot of money.

• Major parts of development and maintenance are outsourced

− We have decided to outsource most of our IT development and technical infrastructure to various suppliers, both within development and maintenance.

− In some cases one supplier develops and another is responsible for maintenance.

− Sometimes one project has multiple suppliers.

4

We need a way to run test handling that fits our world

• Outsourcing development and maintenance of solutions demanded new requirements of testing and quality ensuring within the company. It also raised a lot of questions:

− How do we ensure that suppliers perform a test which provides a solution that is not filled with defects and has high quality?

− What are the responsibilities with regard to test activities between supplier and customer?

− How do we ensure effective testing without delays due to misunderstandings between supplier and tester?

− What are the test criteria to the supplier, and how should they report these?

− How do we ensure that test material used by one supplier for development can be re-used by another supplier for maintenance testing in the future?

− How is defect handling, test reporting etc. best done between supplier and customer?

5

We need a way to run test handling that fits our world

• We need a test model that:

− Is fit to handle suppliers:

• Setting test and quality requirements for the supplier.

− Resembles Lego:

• Easy to use and quickly understood – everybody knows it already in some form and can put it together to match different types of solutions to be tested.

− Has build-in quality ensuring and easy reporting.

− Is fit to handle many different:

• Types of projects / maintenance:

• New developments / changes of existing system.

• Standard solutions.

• Technical – Infrastructure.

• Sizes of tests.

6

7

Agenda

• RSA Scandinavia

• Background for the new test model

• RSA test model

− Overall

− Test policy

− Overall test strategy

• Implementation

• Today

Making a product that matches the people using it

Results

Budget

Quality

Time

Risk

RSA test model

8

RSA test model – mandatory models within the model

− Risk-based testing:

• Defining test scope

• Risk handling

− Test estimations

− The use of different test types

− Test design

− Test review

− Re-use of test material

− Defect handling

− Test reporting

9

The main RSA test model is made up of detailed models within:

It’s like riding a bicycle – Once you learn how, it works the same way each time.

Only the type of bicycle and the equipment changes!

Points to be followed by RSA and supplier

• Test contract appendix:

− Responsibility.

− Test scope requirements:

• Amount of testing.

• Test type to be performed.

− Requirements for test approach:

• Example: Defined test design approach is according to RSA test model, to ensure reuse possibility between Supplier and RSA-Suppliers.

− Test meetings.

10

Points to be followed by RSA and supplier

• Test deliveries according to project model gates:

− Test to be performed.

− Documents.

• Test exit criteria.

− Example of one exit criteria: 100% of severity 1 defects are solved (fixed and tested ok)

• Suspension criteria and resumption requirements.

− Example of one suspension criteria: When the agreed deliverables (e.g. from a supplier) from previous test phase turns to be not kept during the next test phase.

• Test data and environments.

11 Based on the god Old V-Model

Setup between RSA and supplier and responsibility

Test activity, and who to perform each activity, is defined in the model for both RSA and supplier!

12 Well defined roles and responsibility

RSA test policy

• The RSA Scandinavia test policy is the “law book” of testing within RSA Scandinavia and applies to both RSA and supplier test activity.

• RSA have, within the test policy, defined the purpose of testing which:

− Reduces risk for RSA and the customer.

− Ensures solutions are:

• Delivered with agreed quality.

• Deployed with known quality.

• Deployed without creating problems in existing production:

• Regression test and new solutions can work with existing technical solutions.

− Ensures that things work according to defined requirements.

13

Test policy – testing principles

• Test before deployment to production.

• Test according to RSA Scandinavia testing standard:

− All tests are reviewed, measured and signed off according to standard.

− All reporting of test results is done according to test control gates.

• Tests are performed by educated tester and test manager according to RSA Scandinavia test education standard.

• Test scope of a solution is defined in the overall test planning phase – and is a priority!

• Focus on re-using test material.

• All tests must use test tools when mapping requirements to test cases, test design, test execution, test defect handling and test measurement.

14

Test model – focus on standard

15

• The standard is overall defined by the policy and in detail in the test models and overall test strategy, thereby enabling RSA and suppliers to have:

− Same overall test process.

− Same deliveries within each gate.

− Same requirements overall for test scopes.

− Same milestones.

− Same exit criteria.

− Same templates.

− Same method of documenting test cases.

• Benefits:

− Streamlines the monitoring, quality handling and quality reporting.

− Can handle many suppliers at the same time.

− Enables re-use and sharing of test material across RSA and suppliers.

− You do not need to known each contract, just the test model.

RSA Scandinavia test quality model

16

Test policy Overall test strategy

Templates

Defined quality requirement

Who should do it and the

way to ensure it is kept

Test guidelines

Measurement of quality and way to

follow up

Standard test reports

RSA Scandinavia Definitions of quality

Concept

Education

RSA Scandinavia test quality

RSA Scandinavia V-model

17

Test phase

Test execution Test close downTest phase planning Test design

Test setup

Test execution Test close downTest phase planning Test design

Test setup

Test execution Test close downTest phase planning Test design Test setup

Test execution Test close downTest phase planning Test design Test setup

Test execution Test close downTest phase planning Test design Test setup

Test execution Test close downTest phase planning Test design

Test setup

Overall Test planning

Acceptance test

System integrations test

System test

Component integrations test

Unit test

Deployment test

Overall Test planning

V-model Main test activity for each test phase

Startactivity

End

Test performed by RSA / supplier

18 RSA IT – Only test execution if internal development

Control and quality evaluation within each main test activity

For both RSA and supplier, within the overall test planning and each test phase, there are standard exit criteria for:

− What to deliver.

AND

− What and who to approve – (Within the project and by RSA Test Centre).

Shown main test activity within each test phase and the overall test planning

19

Test policy – entry and exit criteria

20

• The exit criteria and acceptance criteria are set to ensure a satisfactory result; ensuring it is possible to proceed with the next test phase, and in the end go to production with the required quality:

− Defined overall in the RSA test policy.

− Details documented in the overall test strategies.

Makes it possible to control that we are on track and have the correct quality!

Overview of entry and exit criteria

Test phaseTest exit criteria

Test entry criteria

Next test phase/

production

Test policy – entry and exit criteria

• Examples of exit criteria for projects:

• Test case/test checklist has been executed with the following results:- 100% of priority 1 test cases/test checklists have been successfully executed- 99%-80% of priority 2 test cases/test checklists have been successfully executed- 79%-60% of priority 3 test cases/test checklists have been successfully executed- 59%-00% of priority 4 test cases/test checklists have been successfully executed

• Defects corrected with the following results:- 100% of severity 1 defects are solved (fixed and tested ok)- 100% of severity 2 defects are solved (fixed and tested ok)- Maximum outstanding of severity 3 defects are 0-10 errors - Maximum outstanding of severity 4 defects are 0-20 errors - Maximum outstanding of severity 5 defects are 0-30 errors

21

Exit criteria are used to define and ensure that quality is okay before continuing to the next phase – because bad quality in one phase has an

impact on the next phase!

Test policy – entry and exit criteria

22

• Amount of testing for each defined, prioritized test scope:

• For each defined and prioritized test scope, the mandatory amount of testing within each priority is as defined in the table:

 

Test policy – entry and exit criteria

23

Test execution phase Scope to be tested according to defined priority

Priority 1 Priority 2 Priority 3 Priority 4

Unit test 100%-94% 94%-80% 79%-50% 49%-20%

Component Integrations test 100%-94% 94%-80% 79%-50% 49%-20%

System test 100%-94% 94%-80% 79%-50% 49%-20%

System Integrations test 100%-94% 94%-80% 79%-50% 49%-20%

Acceptance test 100%-94% 94%-80% 79%-50% 49%-20%

Deployment test 100%-94% 94%-80% 79%-50% 49%-20%

• Amount of testing for each defined, prioritized test scope.

• Defined set of rules for what we mean by each priority:

• In practice -> Use the risk-based test model to define priority.

• All test cases within a scope have the same priority, thereby making it possible to monitor and use it as an exit criteria.

Test policy – RSA test reporting policy

• It is policy to keep a close focus on test quality at:

− Overall level,

− Release level,

− Within development of each solution.

To ensure that it is possible to be pro-active as soon as it is detected

“That there is a risk that the defined quality and/or supplier contract requirements will not be met!”

• It is therefore policy, and a mandatory requirement for tests of all solutions, to keep focus on test quality, and to report on risk and quality:

− During all test phases, both for RSA and supplier test phases.

− A final test report documenting the quality is produced at the exit of each test phase.

24

25

Agenda

• RSA Scandinavia

• Background for the new test model

• RSA test model

− Overall

− Test policy

− Overall test strategy

• Implementation

• Today

Making a product that matches the person using it

Results Risk

Budget

Quality

Time

Overall test strategy – introduction

• All testing within projects is run according to the RSA Scandinavia overall test strategy for projects – both for RSA and supplier:

− This document contains the overall test strategy used by all projects, and it defines the things that are special to the project, e.g. test meetings and test scope.

26

RSA Scandinavia test policy

RSA Scandinavia project test strategy

RSA Scandinavia overall test strategy for project

RSA Scandinavia test planSystem test

RSA Scandinavia test planSystem Integrations test

RSA Scandinavia test planAcceptance test

RSA Scandinavia test planDeployment test

Overall test strategy – project types

• For projects, the detailed test process is split because there are different test focuses and amounts of test. Therefore, to ensure effective testing, the test template for planning, designing and reporting is tailored to match specific needs.

• Projects are split into 3 types:

27

Overall test strategy – test steering

• Steering of all test phases and ensuring test quality is upheld by using RSA Scandinavia Project Excel Sheet for RSA or supplier:

− For each test phase there is one sheet documenting:

• What test activity to perform.

• Test entry and exit criteria.

• What reviews to perform and who should approve them.

− The excel sheet is updated with a status throughout test activity.

− There is one excel sheet to be filled in by RSA and one to be filled in by the supplier.

28

29

Agenda

• RSA Scandinavia

• Background for the new test model

• RSA test model

• Implementation

• Today

Implementation

• Used for all projects initiated since 1 January 2011, and currently being determined for Maintenance and Incidents.

• What we did when we started:

− The test model was fully implemented into the RSA project process and the contract negotiations process.

− Communication – a lot of it – at all levels of the organisation.

− Education:

• Internal courses in RSA made up of small modules.

− Support:

• Each project, when starting up, is appointed someone from the test centre assigned to give support.

− High focus on approval of each test phase:

• Working with release management to ensure that the known non-approved tests from projects are not being approved for production.

30

Today – status on key areas after implementation

• Contract with supplier:

− Standard test appendix is an automated part of the contract for suppliers.

• Support:

− Projects have learned to contact RSA Test Centre for support.

• Review and approval:

− Test review and approval are becoming a natural part of projects.

• Education:

− We have already educated many and are still running courses internally.

− New test manager consultants within testing receive a 1-2 hour introductory session.

• Overall reporting:

− Now running for releases and is on the way for IT management.

31

TIPS

• Tailor the communication to the people you are talking to:

− Have standard presentations that match different audiences, thereby ensuring all in the test centre are able to tell the same story.

• Write down the model – without making it too IT nerdy

• Reviews should be used for everything:

− When you are within the IT world and a test expert, you easily forget that the users of the model are not. Therefore, a review is the best way to ensure that everything is understood.

• Engage in team work with all process owners that interact with the test model.

32

TIPS

• Align and build the test model into the already existing models in the company, and do it in cooperation with existing model owners.

• Education is important – It should be ongoing throughout the year.

• Support and help is necessary.

• High focus on the supplier and whether they are upholding the contract.

• We are a performance company, therefore, define SMART goals within testing.

33

THANK YOUFrom Gitte Oberbossel

Firm email gio@codan.dk

Privat email goberbossel@gmail.com