Improving the Test Process T-Systems

Test Process Improvementusing the TPI®-Next Model:

Graham Bath

Graham Bath ANZTB Test 2010V.1.0 Page 2 SOFTEC 2010 Graham Bath© T-Systems GEI, PDC Test, 2010

Trademarks and ReferencesTrademarks

The following registered trademarks and service marks are used in this tutorial:

CMM ®, CMMI ®, TMMi ®, IDEALSM, ISTQB ®, TPI ®

CMM and CMMI are registered in the U.S. Patent and Trademark Office by Carnegie Mellon University.

IDEAL is a service mark of Software Engineering Institute (SEI), Carnegie Mellon University

ISTQB is a registered trademark of the International Software Testing Qualifications Board

TMMi is a registered trademark of TMMi Foundation

TPI is registered trademark of Sogeti Nederland B.V.

References

[TPI Next] TPI Next: Business driven test process improvement, de Vries et al, UTN Publishers, ISBN 9789072194978

[Expert ITP] ISTQB Expert level syllabus, “Improving the test process”, Graham Bath, Erik van Veenendaal,Isabel Evans (available from www.istqb.org)

[Craig02] “Systematic Software Testing”, Craig, Rick David; Jaskiel, Stefan P., Artech House, 2002, ISBN: 1-580-53508-9

[Black03] “Critical Testing Processes”, Rex Black, Addison-Wesley, 2003, ISBN: 0-201-74868-1


Test Process Improvement with TPI-Next

Contents.

IntroductionSetting the scene for test process improvementUsing models for improvementA closer look at the TPI®-Next ModelThe improvement cycleUsing TPI-NextCritical Success FactorsThe ISTQB® Expert Level syllabus “ITP”Key messages


Setting the Scene.How well do we perform test processes?

Common complaints:

We find the faults too late

Testing is too expensive

Testing isn’t systematic enough

Something’s wrong with our testing

What the testing organization does is a mystery to me


Setting the Scene.What is “improvement”?

Many of these common complaints are symptoms.

The causes can be varied, but more often than not, the test process is in need of improvement.

What is “improvement”?

It depends on who you ask.CustomersManagersUsersOperations staffTesters

We need to consider all of these people (stakeholders)



Contents.

IntroductionSetting the scene for test process improvementUsing models for improvementA closer look at the TPI®-Next-ModelThe improvement cycleUsing TPI-NextCritical Success FactorsThe ISTQB Expert Level syllabus “ITP”Key messages


Using models for test process improvement

“Best practices” in software testing described as a model

The assumption is: “Follow the model and you will improve”

Different levels of test process maturity are defined

Very bad„initial“

Very good„optimal“

Use the model to find out

Where the test process is now

Where should it be improved

What steps to take


Using models for test process improvementTwo principal types of model can be used

1. Process modelsStep-by-step approachSteps to be taken are defined by the model

Examples:Test Maturity Model Integrated (TMMi)Test Process Improvement Model (TPI-Next)

2. Content modelsDefine specific activities which can benefit a test process if applied well Do not require that improvements occur in a specific order

Examples:The Systematic Test and Evaluation Process (STEP), [Craig02] Critical Testing Process (CTP), [Black03]



Contents.IntroductionSetting the scene for test process improvementUsing models for improvementA closer look at the TPI®-Next-ModelThe improvement cycleUsing TPI-NextCritical Success FactorsThe ISTQB Expert Level syllabus “ITP”Key messages


Test Maturity Matrix

Structure of the TPI-Next Model.

Key AreasC

lusters

Improvementsuggestions Enablers

Maturity Levels

Checkpoints

Go to contents


Key Areas represent different aspects of the test Process

StakeholderRelations

TestManagement

TestProfession

Reporting

Communication

Test organization

Test strategy

Degree ofinvolvement

Stakeholdercommittment

Test processmanagement

Estimating andplanning

Metrics

Defectmanagement

Testwaremanagement

Methodologypractice

Testerprofessionalism

Test case design

Test tools

Test environment

DetailsAppendix D


Maturity levels

Controlled

Initial

Efficient

Optimizing

Stages of increasing test process maturity

Key areas can be assessed at any maturity level

Each maturity level of a given key area:is achieved by fulfilling all specific checkpointshas associated improvement suggestionshas associated enablers

„Doing the right things“

„Ad-hoc activities“

„Doing things the right way “

„Continuously adapting toever-changing circumstances“


Checkpoints

Stakeholdercommittment 01.c.1 The principal stakeholder is defined

(not necessarily documented) and known to the testers.

01.c.2 Budget for test resources is granted by and negotiable with the principal stakeholder.

01.c.3 Stakeholders actually deliver the committed resources.

01.c.4 The principal stakeholder is responsible for a documented product risk analysis (the input for the test strategy).

Checkpoints for Controlled

objective statementscan be confirmed by a simple ‘yes’ or ‘no’.

TPI NEXT contains:157 Checkpoints across16 Key Areas and 3 Maturity Levels.

Key Area 1:

Source : [TPI Next]


Clusters

It makes no sense to improve the test process by proceeding sequentially according to key area (i.e. improving key area 1, then key area 2 etc.)

Clusters are defined to help us to improve the test process in logical steps

Clusters define an improvement path for test process improvementcan also be defined by the user to place emphasis on specific business goals (Business-driven approach)


Clusters

A cluster is a group of related checkpoints across different key areasTPI-Next defines 13 “base” clusters which relate directly to the test processThe clusters are shown on a Test Maturity matrix (here, just a section is shown. More on the Test Maturity Matrix follows later) Cluster A, for example, contains 12 checkpoints from 8 different key areas

Source : [TPI Next]Source : [TPI Next]


ClustersExample for base cluster A

KA Checkpoint1 01.c.1 The principal stakeholder is defined (not necessarily documented) and known to the testers.2 02.c.1 The test assignment, scope and approach are negotiated early with the principal stakeholder

as one of the first test activities.3 03.c.1 The principal stakeholder agrees with the documented test strategy.3 03.c.2 The test strategy is based on a product risk analysis.4 04.c.1 People involved know where to find the persons (or department) responsible for test services.6 06.c.1 The reporting contains aspects of time and/or costs, results and risks.7 07.c.1 At the start of the test project a test plan is created. The test plan includes at least the test assignment,

the test scope, the test planning, the roles and responsibilities.7 07.c.2 The test plan is agreed with the principal stakeholder.10 10.c.1 The defect lifecycle is defined (including a retest) and applied10 10.c.2 The following items are recorded for each defect: unique ID, related test case ID (if applicable),

person reporting the defect, date, severity category, description (the actions to reproduce the defect, expected and observed result) and defect status.

14 14.c.1 The test cases are recorded on a logical level.14 14 c.2 The test cases consist of a description of: a) initial situation, b) change process = test actions to be performed,

c) predicted result.

Base Cluster A contains 12 checkpoints from 8 different key areas (KA)Achieving these checkpoints is a first step away from initial “ad-hoc” testing

Source : [TPI Next]


Test Maturity Matrix

BenefitsVisual representation of the overall test processEasy to show what the current maturity is across all key areasPlanning and showing improvement paths Provides a instrument for discussion with stakeholders

Two different viewsCheckpoint viewCluster view


Test Maturity Matrix: Checkpoint view

Checkpoint view to show achieved maturity (all checkpoints shown)

The matrix shown above is blankCurrent maturity is shown by coloring in the checkpoints achieved

Source : [TPI Next]


Test Maturity Matrix: Cluster view

Clusters define an improvement path for test process improvement

The matrix shown above is blankSource : [TPI Next]


Enablers

Clarify how other SDLC processes can support improvement of the test process Creates a link to other software process improvement (SPI) models such as CMMI and ISO 15504

Example

Test strategy, Controlled : Use risk management for product risks


Improvement suggestions

Tips on how to achieve a particular maturity level for a given key areaExample (summarized)

Locate the person who orders test activities or is heavily dependant on test resultsResearch the effect of poor testing on production and make this visible to stakeholders. Show which defects could have been found earlier by testing. Indicate what the stakeholders could have done to avoid the problems.Keep to simple, practical examples.Focus on “quick-wins”

Stakeholder commitment, Controlled



Contents.

IntroductionSetting the scene for test process improvementUsing models for improvementA closer look at the TPI-Next®-ModelThe improvement cycleUsing TPI-NextCritical Success FactorsThe ISTQB Expert Level syllabus “ITP”Key messages


The improvement cycle

Activities.

Identify the need and objectivesPlan and prepare the assessmentPerform the assessment (interviews)Analyse results and make recommendationsCreate and present assessment reportEstablish an improvement planMonitor progress, learn and set new objectives


The need for improvement often results froma negative event (e.g. production failure)a negative realization (e.g. “our testing is not effective”)a planned activities (e.g. improvement is part of company policy)

The need for improvement results in the definition of objectives, e.g.More efficiency in all aspects of the test process.Generally lower costs (but not at the expense of quality!)Better transparency of achieved test coverage.More effective use of available testing staff. It may be sufficient to just say „improve on our current situation“?

The TPI-Next model allows key areas to be priorized to account for specific business objectives

Identify the need and objectives


Capture objectives in the TPI-Maturity Matrix

Objectives are shown here in blue

For example: Establish testing at the “controlled” maturity level




Activities.


Plan the assessment

Week 1 Week 2 Week 3 Week 4

Establish objectivesand set scope

Plan & Prepare forassessment

Perform Assessment

Analyse results andmake

recommendations

Create and present

assessmentreport

1 day

2 days

Total effort:10 days

2 days

1 day

4 days

Typical example


Separate interviews for stakeholders:

EssentialTest manager 2 hours (minimum)Tester 1 1,5 hours (minimum)Tester 2 1,5 hours

Optional (1 hour each, as required)Test environment managerProject leaderRelease managerAutomation specialists

Plan the interviews

Where possible:In this sequence.On the same day


Prepare for the assessment interviewsMaybe you can review some test documents beforehand

Test plansTest specifications

Get some opening questions preparedPrevent “black-outs”

Create summaries of the checkpointsMindmapsKeywordsPath diagrams (sequence of key areas)




Activities.


Perform the assessment (interviews)Cover all checkpoints for all key areas at the Controlled maturity level

Ask the right questions

Ask the questions right Interviewing skills are essential!Open questions are better than closed questionsConduct a conversion, not an inquisition

React to events and the answers you getControl the interview but be flexible

Don’t be afraid to deviate from the modelIf it doesn’t make sense to ask a question, don’t ask itIf it makes sense to ask an additional question, ask it




Activities.


Analyze results

The TPI-Next Model can helpScoring toolBuilt-in improvement suggestions

This does not replace the skills of the test process improverAbility to take project-specific aspects into account Can adapt to different development life-cycles (sequential, agile)


Use the TPI-Matrix (Checkpoint view) to show the assessment results

No key areas achieve controlled maturity at presentOnly the stakeholder commitment is close to the controlled levelFour key areas are still at the ad-hoc “initial” level




Activities.


Create an assessment report

Minimum content

Management summaryTPI-Matrix showing objectives and achieved maturity levelResults for each key area

What was goodWhat could be improved

Proposed improvement stepsSpecific recommendations

PriorityCost of implementationImpact on specific objectives

A test improvement plan is usually agreed using the recommendations


Assessment report: Results for each key area

Positive aspectsA test manager role is defined for this projectA number of testers are availableA project hand book defines the roles and responsibilitiesAll testing staff have in-depth knowledge of the application

Problem areasNo specific test trainingTesting skills are not evaluatedTest managers are relatively inexperienced and have no formal training

Recommendations

For example: Key Area: Tester professionalism, Level: controlled

Will be completed later


Propose improvement steps Use the TPI-Matrix (Cluster view)

Step1: Achieve remaining checkpoints for cluster B


Step2: Achieve remaining checkpoints for cluster C



Step3: Achieve checkpoints for cluster D



Step4: Achieve Controlled maturity level by completing checkpointsfor cluster E



Add recommendations to the assessment report

Positive aspectsA test manager role is defined for this projectA number of testers are availableA project hand book defines the roles and responsibilitiesAll testing staff have in-depth knowledge of the application

Problem areasNo specific test trainingTesting skills are not evaluatedTest managers are relatively inexperienced and have no formal training

Testers should complete the ISTQB Foundation Level training Test managers should complete the ISTQB Advanced Level training within the next 18 monthsLine management should employ a consultant to evaluate testing skills each yearThe training will cost approximately 20,000 USD

Recommendations

For example: Key Area: Tester professionalism, Level: controlled


Tips for proposing recommendationsGather all recommendations into a single list

Prioritize Which recommendations do you consider urgent?The customer sets the final priority

Costs, benefits Will it have any noticeable effect?Why would I want to implement

Estimate implementation risks

Estimate chances of success

Indicate when payback can be expected (short term / long term)




Activities.


Establish an improvement plan

Sets priorities

Obtain resources IndividualsImprovement team

Plan the improvement activitiesMilestonesDeliverablesMeasures for success




Activities.


Monitor progress, learn and set new objectivesThe situation at a later date could be:

Controlled maturity has now been achieved

Our overall objective now is to achieve the Efficient maturity level

Repeat the cycle



Contents.IntroductionSetting the scene for test process improvementUsing models for improvementA closer look at the TPI-Next®-ModelThe improvement cycle Using TPI-NextCritical Success FactorsThe ISTQB Expert Level syllabus “ITP”Key messages


Practical session: Contents.

Background and expectationsAssessment planInterviewsResults and analysisReport

Background and expectationsT-Bank is a new on-line banking system which was started six months ago. and the first delivery of software has already been tested. It is planned that the system will be operational within two years. Company standards require that a first test process assessment using the TPI-Next model should now be conducted. It is expected that the test process has achieved cluster A in the TPI-Next model.It is considered desirable (“nice to have”) at this stage to have at least partially completed subsequent clusters B and C. At this early stage, clusters beyond C are not considered to be in scope, but a perspective should be given on how to improve before the next assessment takes place in six months.The company standards require that a controlled test process is established by the time the next assessment takes place.The base practices of the test process are in focus for this assessment. The key areas are all set to “neutral” priority.

Objectives shown on the TPI-Next Maturity MatrixInitial

1 Stakeholder commitment2 Degree of involvement3 Test strategy4 Test organization5 Communication6 Reporting7 Test process management8 Estimating and planning9 Metrics

10 Defect management11 Testware management12 Methodological practice13 Tester professionalism14 Test case design15 Test tools16 Test environment

Cluster A "must achieve"Cluster B "nice to have"Cluster C "even nicer to have"

Key Area Controlled

We need to achieve „controlled“ before the next assessment: this sets our overall scope for the current assessment





Assessment planFeb.25 March 1 March 8 March 15

Establish objectivesand scope

Prepare forAssessment

Createassessment

report

1 day

3 days

Total effort:11,5 daysTalk to the customer(Mr. Primbo) and withinternal quality management

Perform Assessments:•Test manager•Tester

Make arrangementsGather templates for assessment Perform required tailoring

1/2 day

3 days

Preliminary analysisand recommendations

3 days Detailed analysisand recommendations

Presentation(1 day)

March 19





Information to participantsI will read through an interview which I had with Mr.Testwell, the Test Manager.

Please listen carefully to the interview.

Make notes as you go along if you want to.

At the end of the interview, we will judge how good the test process is, based on what we learned from Mr. Testwell.

We will need the red and the green cards for that.


Interview with Mr.Testwell (Test Manager)

Script





What did we learn from Mr.Testwell?(Test Manager)

Lets judge the test process now.

We will do this by using the TPI-Next scoring tool and our cards.

This is just a quick check on what we think right now

In real life we would have more time!!


This is how our TPI-Matrix might appear after the test manager interviewCluster view generated by TPI-Next scoring tool

This is how our TPI-Matrix might appear after the test manager interview

Other maturitylevels not in scopefor this assessment

Initial

1 Stakeholder commitment2 Degree of involvement3 Test strategy4 Test organization5 Communication6 Reporting7 Test process management8 Estimating and planning9 Metrics

10 Defect management11 Testware management12 Methodological practice13 Tester professionalism14 Test case design15 Test tools16 Test environment

Key not achievedachievednot evaluated

111

Key Area

1 2 3

Controlled

41 2 3 4

1 2 3 41 2 3 4

1 2 31 2 3 4

1 2 3 4

1 2 31 2 3 4

1 2 3 4

1 2 31 2 3 4

1 2 3 4

1 2 31 2 3

1 2 3 4

Other interviews should cover these gaps


What did we learn about using the TPI-Next model?



Contents.



Critical success factors

Many factors can influence the success of a test process improvement programIgnore any one of them and our improvement efforts may fail.

1: Choose the right approach

2: Consider improvement as a process

3: Obtain sponsorship

4: Understand people and get “buy-in”

5: Manage expectations

6: Manage change as a process

7: Acquire the right skills and knowledge



Contents.IntroductionSetting the scene for test process improvementUsing models for improvementA closer look at the TPI-Next®-ModelThe improvement cycleUsing TPI-NextCritical Success FactorsThe ISTQB Expert Level syllabus “ITP”Key messages


ISTQB® Certified Training: structure

TestAnalyst

TestManager

Technical Test

Analyst

Certified Tester Foundation Level

Certified Tester Advanced Level

Test Manager

Test Analyst

TechnicalTest

Analyst

Certified Tester Expert Level

Improving the Test Process

TestManagement

TestAutomation

TTCN-3…


Analytical-based Improvement

Selecting the approach for test process Improvement

The improvement process

Organization, roles and skills

Managing change

Critical success factors

Model-based Improvement

Fundamental approaches and frameworks

The ISTQB® Expert Level Syllabus.Improving the Test Process: Principal Contents



Contents.



Using a model to improve the testing process is a tried and trusted approach.

The TPI-Next model has been shown to provide benefit to those using it.

Use it as a tool to guide you through test process improvements.

Talking to and understanding different stakeholders is essential.

Key messages.

Thanks for participating!

Graham Bath


Appendix:

Checkpoints to be assessed in the interview with Mr. Testwell


Key Area 1: Stakeholder commitmentMaturity level:

01.c.1 The principal stakeholder is defined (not necessarily documented) and known to the testers.

01.c.2 Budget for test resources is granted by and negotiable with the principal stakeholder.

01.c.3 Stakeholders actually deliver the committed resources.01.c.4 The principal stakeholder is responsible for a documented product risk

analysis (the input for the test strategy).

Checkpoints

Controlled

Source: [TPI Next]


Key Area 2: Degree of involvementMaturity level:

Checkpoints

Controlled

02.c.1 The test assignment, scope and approach are negotiated early with the principal stakeholder as one of the first test activities.

02.c.2 Test activities are started early, timely before test execution, with the goal of keeping the test activities of the project's critical path.

02.c.3 A tester is involved in project planning: dependencies between the test process and other processes are taken into account.

02.c.4 A tester is involved in the analysis and mitigation of overall project risks.

Source: [TPI Next]


Key Area 3: Test strategyMaturity level:

Checkpoints

Controlled

03.c.1 The principal stakeholder agrees with the documented test strategy.

03.c.2 The test strategy is based on a product risk analysis.03.c.3 There is a differentiation in test levels, test types, test coverage and test

depth, depending on the analyzed risks.

03.c.4 For retests and regression testing a simple strategy determination takes place.

Source: [TPI Next]


Key Area 4: Test organizationMaturity level:

Checkpoints

Controlled

04.c.1 People involved know where to find the persons (or department) responsible for test services.

04.c.2 There is a structure of control and accountability within the test organization.

04.c.3 Test tasks and responsibilities are defined (and documented) and are assigned to a person or organizational unit.

04.c.4 The products and services of the test organization are clear to its clients.

Source: [TPI Next]


Key Area 5: CommunicationMaturity level:

Checkpoints

Controlled

05.c.1 Every team member is aware of decisions being made and of internal progress.

05.c.2 The test team actively gathers relevant information from stakeholders.

05.c.3 It is possible to trace back points of action, agreements and decisions of the test team.

05.c.4 The test team deliberates with stakeholders about progress, product quality and risks and signals proactively potential delays as well.

Source: [TPI Next]


Key Area 6: ReportingMaturity level:

Checkpoints

Controlled

06.c.1 The reporting contains aspects of time and/or costs, results and risks.

06.c.2 The reporting frequency and content matches the basic demands ofstakeholders for the decision making process.

06.c.3 Reporting is done in writing.

Source: [TPI Next]


Key Area 7: Test process managementMaturity level:

Checkpoints

Controlled

07.c.1 At the start of the test project a test plan is created. The test plan includes at least the test assignment, the test scope, the test planning, the roles and responsibilities.

07.c.2 The test plan is agreed with the principal stakeholder.

07.c.3 Each test activity is monitored and when necessary adjustments are initiated.

07.c.4 The test plan is agreed with the relevant stakeholders.

Source: [TPI Next]


Key Area 8: Estimating and planningMaturity level:

Checkpoints

Controlled

08.c.1 For test effort estimation, simple techniques are used such as ratios.

08.c.2 For each test activity there is an indication of the period in which it runs, the resources required and the products to be delivered. Activities to be identified are: test planning and management, defining test cases and executing test cases.

08.c.3 The dependencies between test phases or test activities are plotted in a test planning. A certain overlap of test phases and test activities is allowed.

08.c.4 Test estimations and test planning are discussed with the principal stakeholder.

Source: [TPI Next]


Key Area 9: MetricsMaturity level:

Checkpoints

Controlled

09.c.1 In the test process metrics are defined and used to estimate and control the test project.

09.c.2 The necessary input for the metrics is recorded uniformly and the defined metrics are systematically stored.

09.c.3 The input (data) for metrics is probably accurate.

Source: [TPI Next]


Key Area 13: Tester professionalismMaturity level:

Checkpoints

Controlled

13.c.1 Testers have received specific test training and/or have sufficient experience in the field of structured testing.

13.c.2 Testers are familiar with the adopted test method and they apply it.

13.c.3 All required expertise - whether industry, business or technical - is available to the test team.

13.c.4 Testers are regularly evaluated on specific testing skills as well as general IT abilities in an employee performance appraisal.

Source: [TPI Next]

Documents

Improving the Test Process T-Systems