Sogeti Webinar Effective Test Process Improvement 220709

© Sogeti Group 2009 Page 1

Sogeti Ireland

Effective Test Process Improvement

Fran O’Hara

[email protected]

22nd July 2009


Contents

• Introduction

• TPI

• TMMi

• Process improvement challenges and sample solutions

• Summary

• Q&A


3

SOGETI Overview: A Worldwide Leader in IT Services

• Sogeti Group� 20,000 professionals

� 200 locations in 14 countries

• Providing worldwide standards� Copyright TMap® and TPI® brands

� Dedicated R&D on new developments

� Test thought leadership

Part of• Leader in consulting, IT &

outsourcing

• 88,000 staff

• Revenue €8.7 billion

• Sogeti:

• £1.6 billion revenues

• Testing revenues

£150M (2007), £190M

(2008)USA

France

Spain

Sweden

BeluxGermany

Switzerland

NL

DenmarkUKIreland

India

Test Centre of Excellence

provides for • Reduced cost (>25% over 2 yrs)

• Improved quality

• Improved flexibility

• Reduced time-to-test/market

• Reduced risk

• Increased capacity


People Process Technology

Capability

Products/services

World ClassProcess focus improves

predictability of

performance (reduces

variation), as well as

performance itself

(difference between

desired and actual)

Test Process Improvement - learning to work smarter,

to test software better, cheaper, faster


Triggers

• Efficiency/cost?

• Effectiveness?

• Growing Complexity?

• Different levels of Test Maturity?

• Time Pressure?

• Are we doing the right thing(s)?

Development

Agile

XMLGUI’s

Integration

Automation

Webserv

ices

CBDCOTS

Development

Agile

XMLGUI’s

Integration

Automation

Webserv

ices

CBDCOTS

Business

Time-to-market

Error free

Quality

Cheap

Reuse

User-friendlinessMaintainability

Standard.

GrowingComplexity

Business

Time-to-market

Error free

Quality

Cheap

Reuse

User-friendlinessMaintainability

Standard.

GrowingComplexity

Tester


Key areas

Levels Test

Maturity

Matrix

Improvement suggestionsCheckpoints

The TPI® Model


• Test strategy

• Life-cycle model

• Moment of involvement

• Estimating and planning

• Test specification techniques

• Static test techniques

• Metrics

• Test automation

• Test environment

• Office environment

• Commitment and motivation

• Testing functions and training

• Scope of methodology

• Communication

• Reporting

• Defect management

• Testware management

• Test process management

• Evaluation

• Low-level testing

20 Key Areas


Key area / Scale 0 1 2 3 4 5 6 7 8 9 10 11 12 13

1 Test strategy A B C D

2 Life -cycle model A B

3 Moment of involvement A B C D

4 Estimating and planning A B

5 Test specification techniques

A B

6 Static test techniques A B

7 Metrics A B C D

8 Test automation A B C

9 Test environment A B C

10 Office environment A

11 Commitment and motivation

A B C

12 Test functions and training A B C

13 Scope of methodology A B C

14 Communication A B C

15 Reporting A B C D

16 Defect management A B C

17 Testware management A B C D

18 Test process management A B C

19 Evaluation A B

20 Low -level testing A B C

Current Situation (example)

Dependencies!


Process Improvement ….

Key area / Scale 0 1 2 3 4 5 6 7 8 9 10 11 12 13

1 Test strategy A B C D

2 Life - cycle model A B

3 Moment of involvement A B C D

4 Estimating and planning A B

5 Test specification techniques

A B

6 Static test techniques A B

7 Metrics A B C D

8 Test automation A B C

9 Test environment A B C

10 Office environment A

11 Commitment and motivation

A B C

12 Test functions and training A B C

13 Scope of methodology A B C

14 Communication A B C

15 Reporting A B C D

16 Defect management A B C

17 Testware management A B C D

18 Test process management A B C

19 Evaluation A B

20 Low - level testing A B C

Controlled EfficientOpti

mizing


Testing Maturity Model Integration (TMMi®)

• Currently under development by the TMMi Foundation� See www.tmmifoundation.org� Levels 2 & 3 published

• The TMMi® Foundation will provide:� A standard TMMi® Model that can be used in isolation or

in support of other process improvement models and is either staged or continuous.

� An independently managed data repository to support TMMi® assessment method accreditation, assessor and assessment certification/validation and validated assessment data and certificates.

� Assessment Method Accreditation/Audit Framework for TMMi® in accordance with ISO15504 and the process to certify commercial assessment methods against the standard model.

� Certification and training/examination process, procedures and standards for formal, public accreditation of Assessors and Lead Assessors and the on-going management.


TMMi® Model

Test Organisation

Test Training Program

Test Life Cycle and Integration

Non-Functional Testing

Peer Reviews

3 Integration

1 Initial

Test Policy and Strategy

Test Planning

Test Monitoring and Control

Test Design and Execution

Test Environment

2 Managed

Test Measurement

Software Quality Evaluation

Advanced Peer Reviews

4 Management & Measurement

Defect Prevention

Test Process Optimisation

Quality Control

5 Optimisation

Process AreasLevel


Problems with Process Improvement

Typical Issues with Process Improvement:

� Lack of management commitment

� Lack of prioritised goals to focus/prioritise

� Excessive emphasis on documenting and assessing process

� Difficulty in defining and deploying the process

� Inadequate buy-in from staff

� Lack of process architecture

Results from Sogeti ‘Worldwide survey on TPI’ :

� Lack of management commitment **

� Unrealistic expectations *� Unclear scope and target (i.e. goals) *� Unclear communication� Only 17% had measurable results

� Results unknown for too long a time

� Lack of change process� Change team without influence

� Involvement of people within the test process*� Top down approach (‘Thou shalt..’)

� 44% combined with other improvement projects


Improvement Strategy - 1

Achieve sign-off on documented requirements and constraints and associated improvement (project) plan

Align and prioritise improvement efforts with business goals through a goal-problem tree

Focus on early and measurable results

Lack of management commitment

- Lack of management commitment**

Lack of prioritised goals to focus/prioritise

- Unrealistic expectations*

- Unclear scope and target (i.e. goals)*- Unclear communication- Only 17% had measurable results - NB

Solutions

Problems


Problem Scenario - 1

0

10

20

30

40

50

60

70

80

1 2 3 4 5 6

Months

Reported Production Defects

# Severity 1 & 2 defects

Customer

satisfaction

rating dropped

20% in June

survey

compared to 6

months

previously.

Cost reduction

of 10% is also

a department

target


Problem Scenario -2

Origin of Production Defects

User requirements

System requirements

Architecture

Detailed design

Code & Build

Integration

Data also

shows that

rework of

production

defects is

costing 40% of

development

budget


• Perform formal peer reviews of System requirements� Involve system testers

• Use a risk-based approach to system test involving business users in risk workshops

• Use a test driven development approach to unit test (unit test cases developed and automated before code written)

• Design system tests as soon as system requirements become available

• Resource system test with some dedicated test professionals

• Perform formal planning of system test

• Measure DDP for peer reviews, unit test and system test

• Some ways to identify improvement actions:� Brainstorm process problems and vote

� Brainstorm improvement ideas/actions

� Model gap analysis – assessment results

Examples of test-related improvement actions


Goal Tree & Cost/Benefit

Satisfied customers

0As aboveReduce number of production defects per month by 50% (based on equivalent appn. throughput)

Improve Customer satisfaction ratings by 20%

1m

(10% of IT budget)

200kImprove effectiveness of Peer reviews, Unit test, and System Test by 75% (Currently running at 50% DDP)

Reduce rework costs of production defects by 50%

Reduce IT Appn. Development costs by 10%

Est. Benefit

Est. Cost of required actions

Test Process Goals

Dev/Test Department Goal

Business Goal


Goal Metric Scenario – Sample Results

Defect Detection Percentage per Release

0

20

40

60

80

100

1 2 3 4 5 6

Release number (or subsequent projects)

DDP



Use a more agile approach to PI - Iterate in short improvement cycles (2-3 months) – how do you eat an elephant?

Focus on lightweight, rapid process modelling for early deployment (80/20 rule for any gap analysis)

Perform formal assessments to confirm process maturity results

Excessive emphasis on documenting and assessing process

- Results unknown for too long a time

Solutions

Problems



Use a facilitated workshop approach to process modelling/improvement

Process owners involved in workshops, ‘change team’become facilitators

Goal–problem driven focus

Use a continuously improving well-defined change process

Difficulty in defining and deploying the process

�Lack of change process

�Change team without influence

Inadequate buy-in from staff

�Involvement of people within the test process*

�Top down approach (‘Thou shalt..’)

Solutions

Problems


Example Modeling Session

•Picture of process-level modeling (Image2)

•session inserted here.


Example “Mapped” Element


When a standard framework is required –

sample process level template

Inputs Entry Exit Outputs

Activity Name Activity Purpose Activity Controls

Measurements

Costs, Efforts,

Time, Quality

RolesIssues

Planning,

Tailoring, …

Tasks

Action

Action

Action

Action

Artifact

Artifact

Artifact

Action

Condition

Condition

Condition

Condition

NameName

ArtifactArtifact

Artifact

Artifact

Level of Abstraction of tasks is

key to a common frameworkActivity Controls or ‘Work Aids’ provide options

for supporting templates, tools, procedures, etc.

Tailoring can also be used to highlight

key differences of approach required


Sample Test Process Element

Define the Test Strategy

Metrics1. Planned versus actual effort

Roles1. Test Manager (T.1-T.7)

2. Test Designer (T.5)

3. Business Analyst (T.3, T.7)

4. System Expert (T.3-T.7)

5. End User (T.3, T.7)

Tailorings

Outputs1. (Risk Based) Test Strategy.

2. Prioritised Risk List.

Tasks1. Determine level of Requirements definition for the

project

2. Divide the System under Test into its components, if relevant. Prioritise component parts

3. Identify and analyse Product Risks, taking Project Requirements, Statement of Work, Prioritized Quality Characteristics and System Components into consideration

4. Identify and Define Test Levels.

5. Identify and Define Test Level Attributes - Methods and Test Techniques, exit criteria, coverage, automation approach, etc.

6. Document the Testing Strategy.

7. Review and sign-off the Testing Strategy.

Inputs1. Project Requirements

Document(s) [User, System, Functional/Technical]

2. Statement of Work

3. Test Project Scope

Exit Criteria1. Test-ware is placed under

Configuration Management control.

Controls / Work Aids1. Project Lifecycles with defined Test Stages

2. Test Methodology (identifying Test Methods & Test Techniques.)

3. Review Meeting Minutes Template

4. Risk List Template

5. Test Strategy Template

Entry Criteria1. Inputs are under

Configuration Management control


Performevaluation

Formulateplan

Implementimprovement

actions

Obtainawareness

Executeassessment

Defineimprovementactions

Determine target, area of consideration

and approach

Process of change


Suggestion: Use a performance-based ‘agile’

process improvement approach

Phase 1: Selected

goals and problems

for process

improvement

Phase 2:

Documented

baselines of

current practice

and performance

Phase 3: Process

documents to be

deployed

Phase 4: Revised

process and

performance

baselines


• Clear scope and target 16%

• Clear communication 9%

• Results periodically visible 8%

• Champions in change team 8%

• Use of a change process 7%

• Other ... 5%

Biggest DO’s: • Management commitment 24%• Involvement of “own” people 23%

Do’s – from TPI® survey


Summary – key suggestions

• Use an agile approach to test process improvement – incremental with 80/20 prioritisation� Driven by and linked to measurable business goals

� Addressing key staff process problems

• Involvement and ownership by test staff

• Lightweight, usable process definition (80/20 again!)


Sogeti Support

• Capability building

� test process and how to improve it

� seminars, workshops and training

• Process assessments

� Model based (e.g. TPI®) and/or benchmarking

• Consulting/mentoring during process improvement

• [email protected]

• www.sogeti.ie