ISTQB Expert level Improving the testing process - ie-net.be · PDF fileISTQB Expert level Improving the testing process Version 1.1 Isabelle Robrechts Training content Agenda Practical

© 2014 Improve Quality Services BV 01‐01 Course Introduction ‐ 1

First practical experience

ISTQB Expert levelImproving the testing process

www.ip-consult.be

Version 1.1

Isabelle Robrechts

●Training content

Agenda

●Practical experience

●Conclusion

2


ISTQB Structure

IP-Consult bvba 3

Bron : www.istqb.org

Goal ISTQB Expert ITP

• Coaching/Leading test process improvement tracks in own organisationtracks in own organisation

• Goal part 1 : Assessing the test processes• Improvement models• Linking and situate

4

• Goal part 2 : Improving the test processes• Change Management• Organisation and human factors


IDEAL

Part 1: Assessing

© 2014 Improve Quality Services B.V. 5

Part 2: Implementing

Approaches

Overview Improvement Approaches

Model based

IT/SW Testing

Analytical

GQMMeasuresMetrics

Indicators

Hybrid / Mixed

Analytical & Models

Analytical & Content

Other

People Skills

Standards & Regulations

Infrastructure & Tools

© 2014 Improve Quality Services B.V. 6

CMMIIEEE15504

Processmodels

TMMiTPI Next

Content models

CTPSTEP

(TMap)

CausalAnalysis

Infrastructure & Tools

Evaluations


●Training content

Agenda

●Training content


●Conclusion

7

●OverviewEnvironment

Practical experience:

− Environment− Business goals

●Retrospective− Starting point− Transfer to improvement: Causal analysis ?

exercise

● Improvement proposition− Short term− Long term

8


Environment

•Financial institution•SDLC : Combi V-model / ‘Agile’ (inhouse invention)•SDLC : Combi V-model / Agile (inhouse invention)•IS : tester is not a profession but a role (<-> Bsn)

9

●To go quickly to the market with new products− Risk-based testing !

Business goals

Risk-based testing !− Regression testing !

●Good quality of the requested products− Fit for purpose− Passive testing !g− Less nuisance for the operations department

10


Retrospective

• Assignment 2009 : IS•Set-up :Set-up :

• test organization• test methodology• test culture• testing

•Representing the department in an organizational

11

g gtesting competence center•Representing the IS testing at business meetings•Report to the department management

• just CTAL

●What is present ?− Test work products ?

Retrospective : where do I start ?

Test work products ?Test PolicyTest Strategy (Master)Test PlanTest Cases

− Test culture− Test process ?− Test Tooling

Test ManagementDefect Logging

12


●Knowledge through measuring is power...

Retrospective : where do I start ?

●But what are you going to measure ?− Inspired by the circumstances ?− Inspired by others (lecture, management,...)− Inspired by own interestsInspired by own interests

13

● Defects : − # incidents in production

Retrospective : Data Collect

p− # scope changes per phase− # defects per phase (incl.priority)

● Test cases : − # written test cases per phase (incl.priority)− # executed, (in-)correct test cases

● Budget : − Planned & Actual test budget− Planned & Actual total budget− Planned & Actual Realization (Analysis & Development)− Planned & Actual repair budget

14


●Can we obtain information from this data ?Test Effectiveness

Retrospective : first metrics...

− Test EffectivenessHow many defects are found per test case ?How many defects are found per test phase ?

HOW EFFECTIVE ARE OUR TESTS?

− Test EfficiencyTest EfficiencyWhat’s the price of a defect found during the ST/SIT phase vs a defect found during the UAT phase ?

15

●How effective are our tests ??

Retrospective : Test Effectiveness

16

Beware of the numbers !


Retrospective : Test Efficiency

17

●Pre-condition : an effective defect and/or incident management process

Transfer to improvement : Causal Analysis ?

management process

●4 steps : 1. Select the items for causal analysis2. Collect and organize the information3. Identify any root causes3. Identify any root causes4. Draw conclusions

18


●Defects/incidents● Irregularities reported by stakeholders

1. Select the items for Causal Analysis

● Irregularities reported by stakeholders●Problems found during the analysis●Aids:

− The pareto principle : 80/20− Categorization of defects− Analysis of statisticsAnalysis of statistics− Results of retrospectives

19

Analysis of statistics

Reality:• incidents doubled the last two years

• normal requests (no projects) decreased

20

Peaks are indicating the close-out of a project.


Analysis of statistics

21

●Let someone take the lead●Aid while structuring the data :

2. Collect and organize the information

●Aid while structuring the data :− Ishikawa

22


2. Collect and organize the informationManagementInfrastructurePlanning

Minimal budgets for resources

Tests are delayed by strict authorizations

Unavailability of the test environment is 1 on 5

UnrealisticProject planning

# Incidents doubles in the last

2 years

No money for trainings

authorizations

Monthly refresh of test dataTesting is a role, no profession

No budget for resources

Lack of van PM capacities

No commitment to any test workMissing milestines is easily Accepted.

Bsn is often seen as the enemy by IS

23

Communication Culture Human

No professional IS testers

Motivation/work spirit often very low

IS : why loose time testing now if it is thoroughly tested later ?

Defects are found much too late

No transparancy in the IS tests

● Identify the ‘most-likely’ causes●Work top down through the causes

3. Identificy the root causes

●Work top-down through the causes●Add (if necessary) extra information to the analysis● Identify any Tipping Points or Broken windows●Maintain brainstorm rules

24


●Are there any conclusions ?●Be Aware for single events ?

4. Draw conclusions

●Be Aware for single events ?●Do we find any problems in the test process which

can be avoided in the future ?●Are there any other processes involved ?●Mutual causes ?

25

− Short term : some quick-wins are defined: More transparancy between IS and Bsn

Improvement proposal

p y● Visualisation of the businessplan with all milestones● Weekly reporting on planning at the Division Management● Definition of quality gates and its controles● Better communication (1N)● Usage of 1 Test Management Tool (creates responsibility)

26


− LongTerm : elaborate an improvement track based on TPI Next, with possiblity to expand to CMMi

Improvement Proposal

, p y pWhy TPI Next ? And not TMMi ?● Test process exists (TMAP)● Test process has already a certain maturity● Emotional sensitive to improvement● Clear visibility of the impact on the business goals● Link with CMMi exists

Fit in the model ... ● How does it look dd 2014Q1 ?● TPI model in oktober● Where do we want to go ?

27

Improvement Proposal : Long Term

TPI Next : Test Maturity Matrix (dd 2014Q1)

TPI Next : Test Maturity Matrix (dd 2014Q4)

28


●TPI Next : Test Maturity Matrix (2016)

Improvement Proposal : Long Term

Maturity Level : - Controlled on all key areas- Evolution from Efficient on the business critical key areas

29

Don’t forget the soft skills !

IP-Consult bvba 30


●Training content

Agenda


●Conclusion

31

●Models and causale analyse are complement

Conclusion

− Models for Long Term visions− Causale analyse for selection and short term activities

● In the training links between techniques are shown

●Assignments connect the theory with own practical experiences

●Don’t forget the soft skills32


33

Documents

ISTQB Expert level Improving the testing process - ie-net.be · PDF fileISTQB Expert level Improving the testing process Version 1.1 Isabelle Robrechts Training content Agenda Practical