Test Process Improvement - Paul Gerrard

8/4/2019 Test Process Improvement - Paul Gerrard

1/27

2000 Systeme Evolutif Ltd Slide 1

Test Process Improvement

Paul Gerrard

Systeme Evolutif Limitedemail: [email protected]://www.evolutif.co.uk


Agenda

u Introduction

uWhat Can Be Improved?

uTest Methodology

uTest Organisation Maturity (TOM)

and Testing Assessment

u Implementation.


2/27


Introduction


Is testing a strategic issue?

uTesting, in its broadest sense, comprises

more than 50% of your development cost

and has influence over all these risks

uSoftware development productivity isincreasing, testing productivity decreasing

uTester psychology at odds with developer

and most management mentality

uDemand for, appreciation of testing and

testers is increasing.


3/27


Testing perceptions

uTesting is innate - anyone can do it

everyone tests, but it takes a good tester to find

bugs in good code

anyone can find bugs in poor quality code

uTester psychology is counter-intuitive,

destructive, all-wrong

uGood testers are awkward employees

uSome code doesnt need testing

uTesting doesnt add value.


Testing perceptions (2)

uSoftware is getting easier to test

uSoftware quality is improving

uTesting has little value - destructive,

pointless

uTesting is random and cannot be

systematised

test techniques are prone to mathematical

treatment

development is still a craft, in comparison.


4/27


Some of the problems to be

addressed

uToo many errors being found in production

or being reported by customers

uTesting is taking too long, and delaying

delivery

uTesting is too expensive

uDifficulty in finding volunteers to test

uTesters dont see a career path in your

company.


What can be Improved?


5/27


Testing know-how

uTraining is the quick-win stream of work

uFocus on principles, process and techniques

uTraining and qualification schemes are

emerging (ISEB scheme in the UK):

grades of certification

industry-relevant syllabus

accredited training organisations, courses and

instructors

examination schemes.


Standards and procedures

uServe four main purposes:

a consistent baseline for the quality of testing

across the organisation

alignment with your development anddeployment processes

appropriate use of industry best practices

provide a shortcut to using the best, most

appropriate techniques.


6/27


Organisation

uStreamlining testing usually implies

fostering closer liaisons between

projects and software users

developers, maintainers and support groups

uA common improvement is to create an

independent test group

set up to offer testing services to other

departments, projects.


Environment

uExisting test environments are often

inadequate and need improvement

scale - too small to run some tests

control - chaos is a barrier to systematic testing isolation - some tests too risky to implement

dedication - shared environments problematic

uTools can make testers more productive

the only realistic way to regression test.


7/27


Improvement investment

uAll improvements require investment

uPay back by ensuring resources are focused

uBenefits vary with type of improvement and

by installation

e.g. one site found that ...[There was] a seven-

fold pay-back for the cost of the training over a

three month period.


Test Methodology


8/27


Typical test strategy

Integration

System

Acceptance

Unit

Ad hoc


V model: waterfall and locks

Requirements

Logical

Design

Physical

Design

Code and test

Integration

System test

Acceptance

test


9/27


System

Integration

Typical test practice

Acceptance

Unit

Ad hoc x


Time/Cost

Development LiveRunning

Economics of errors


10/27


Front-loading

uThe principle is to start testing early

uReviews, walkthroughs and inspections of

documents during the definition stages are

examples of early tests

uStart preparing test cases early. Test case

preparation tests the document on which

the cases are based

uPreparing the user manual tests the

requirements and design.


Front-loading advantages

uRequirements, specification and design

errors are detected earlier and are therefore

less costly

uRequirements more accurately captured

uTest cases are a useful input to designers and

programmers

uSpreads the workload of test preparation

over the whole project.


11/27


Early test case preparation

Requirements

Logical

Design

Physical

Design

Code and test

Integration

System test

Acceptance

test

Prepare

tests

Preparetests

Prepare

tests


Testing throughout the lifecycle: the W model

AcceptanceTest

Code

BuildSoftware

Install

Unit Test

Write

LogicalDesign

PhysicalDesign

Test the

Test theDesign

Test theSpecs

IntegrationTest

System

Test

BuildSystem

RequirementsRequirements


12/27


Acceptance

Test

Code

BuildSoftware

Install

Unit Test

Wri te

LogicalDesign

Physical

Design

Test the

Test the

Design

Test theSpecs

IntegrationTest

SystemTest

BuildSystem


Static test techniques

Inspections StaticAnalysis

Requirements

Animation

BehaviourAnalysis

ScenarioWalkthroughs

Reviews

Inspections

EarlyTest CasePreparation


AcceptanceTest

Code

BuildSoftware

Install

Unit Test

Wri te

LogicalDesign

Physical

Design

Test the

Test the

Design

Test theSpecs

IntegrationTest

SystemTest

BuildSystem


Dynamic test techniques

FeaturesTesting

Performance,

TestingVolume, Stress

Restart & Recovery

TestingInstallation

PathTesting

LoopTesting

Input ValidationTesting

Multi-userTesting

SecurityTesting

Equivalence PartitioningBoundary Value Analysis

Cause-effect Graphing

Transaction Flows


13/27


Test Process Assessment


Information gathering

uNeed to determine:

where you are today

where you want to go tomorrow

uData gathering using interviews

managers, developers, testers, users

checklist type and open-ended questions

uOther research

examination of project and test plans, test

records, fault reports.


14/27


The challenges of testing

improvement

uMajor barriers are organisational and

personal, not technical

changing management perceptions so they

support testing and improvement

overcoming management and practitioner

resistance to change

design and implementation of workable

processes and management controls

uPeople, not tools, implement test strategies.


Problems or symptoms?

uNeed to separate problems from symptoms:

management doesn't understand the objectives of

testing

the cost of testing is high but difficult to pindown

developers, testers, users may never have been

trained

the quality of the product delivered into testing

is poor, so takes longer to system test.


15/27


Improvement mix

uA mix of improvements is most likely to be

required:

management awareness

tester training

improved definition of the test stages and their

objectives

measurement of the quality of the product at

each stage etc. etc.


Staged improvements

uNot all improvements are a good idea

straight away

some improvements are expensive

some save time, but bring dramatic change some improve the quality of the testing, but take

longer to implement

uVery few improvements save time, improve

quality, cause minimal change and pay back

after two weeks.


16/27


TOM

uRecommended improvements must take

account of broader organisational objectives,

constraints and priorities

uTest Organisation Maturity model.


Test Organisation Maturity

(TOM)

and Testing Assessment


17/27


Process maturity

uProcess maturity is a product-independent

measure of an organisations capability

uFramework for process improvements

uWhat are the most sensible things to improve

next?

uAssessment: where are we now?

uMonitoring: are we there yet?


Capability Maturity Model (CMM)

uCMM for software widely adopted

uUsed to assess capability and identify

improvements

uMaturity 'levels' define a graduated scale and

provide a roadmap of improvements

uCMM represents a high-level, process-

oriented description of the capability of a

development organisation.


18/27


CMM (2)

uHeritage of the CMM is large, long term,

defence projects in the US

uSEI agenda remains high-integrity, but has

little to say about testing

uRelevance to commercial IT organisations is

often tenuous

uMost organisations at levels 1 (0, -1?)

uWhose process is it anyway?


Several testing maturity modelsexist

uTest Process Improvement (TPI) Model,

Koomen, Pol (Holland)

20 key areas scored, improvement suggestions

uTestability Maturity Model, Gelperin, USA 20 key areas, score between 1-2

uTesting Maturity Model, Burnstein,

Suwannasat and Carlson, USA

aligned with the CMM


19/27


Problems with existing models

uRemedy oriented, not problem-oriented

here are the pills, which are you taking?

heres the solution, does it solve the problem?

uLittle guidance on priorities or constraints

hackers and high integrity get same treatment

assumes a click and go mentality and culture

no comparison of costs v benefits

uFor process hypochondriacs - you didnt

know you had a problem until now.


TOM

uTest Organisation Maturity model - TOM

uAssessment

self assessment or consultant-assessment

questions based on symptoms of poor testing

assessment score from 20-100

uBuilt-in improvements model

improvements selected, based on assessment

improvements prioritised, based on assessment.


20/27


TOM assessment to action plan

Symptoms(current maturity)

Priorities(target maturity)

Action Plan

u Document objectives and

constraints and prioritise

u Identify the testing-related

problems

u Which problems cause

most concern?

u Select improvementswhich best meet

objectives.

Objectives/

Constraints


Assessment questions

uQuestions focus on organisational rather than

technical issues

can be answered by management or practitioners

(try both and compare!)uQuestions relate directly to the symptoms

how bad is the problem? (the score)

how much does it hurt? (the priority).


21/27


Improvement objectives and

constraints

uDecrease time required to test

uDecrease cost of testing

u Increase quality of testing (and systems)

uMinimal change to current practice

uQuick payback

uGive each a priority of 1-5 (low-high).


Example assessment symptom

Score 1 Tests are not based on requirements or design documents,

there are no test inventories or means of measuring coverage

against requirements or specifications

Score 3 Test inventories are used to define the scope of system and

acceptance tests and cross-reference requirements, formal

test techniques are sometimes used to design black-box test

cases.

Score 5 Test inventories are used for all testing and are reviewed

against requirements and specifications, formal test

techniques are used for test case design, tools are used to

measure code coverage

There are gaps in the testing - features of thesystem may be released untested


22/27


Score 1 Errors are found in acceptance tests which should have

been found in sub-system and system tests

Score 3 Errors are found in system tests which should have been

found in sub-system tests

Score 5 Errors found would not be expected to have been

detected earlier

Of the errors that are found, there is a

perception (based on evidence) that manyshould have been found in earlier test stage(s).

Example assessment symptom (2)


Potential improvements

uTrain developers, system testers in testing

u Improve test design by adopting techniques

u Involve users in definition of system tests

uMotivate developers to do better testing

uSeparate code and test activities in plans.


23/27


Objective/Constraint Score Description

Decrease time required to test 0 Increase dev.test

Decrease sys.test

Decrease cost of testing 0 No change

Increase quality of testing(and systems)

+1 Yes

Minimal change to currentpractice

-1 Change likely

Quick payback +1 Yes

Improvement: train developers,

system testers in testing


Implementation


24/27


Pilot projects

uTest the improvements and gain experience

uPilot projects should be:

low risk, in a familiar area, low criticality

in their earliest stages - budget yet to be set

project managed by someone willing and

capable of managing additional responsibilities

staffed by people willing to learn and apply new

techniques, and capable of providing objectivefeedback.


Pilot project objectives

uGet consistent and comprehensive incident

and defect information for a project

where are the bugs found?

where should the bugs have been found?

u Identify where costs are incurred in testing

uMeasure the amount of testing done

uReduce or eliminate duplicated tests

uReduce the frustrations of testing.


25/27


Pilot projects - preparation

uProject manager needs to be briefed and

measurable objectives for the pilot agreed

uProject staff

need to be aware of the purpose of doing things

differently

training in the use of unfamiliar process,

techniques and tools

uNeed to provide support to the projectmanager and staff involved.


Reviewing the results of the pilot

u If the pilot objectives were met and the

changes accepted, then the changes are ready

for roll-out to other projects

uOtherwise, the conduct of the pilot should be

investigated:

were the objectives too ambitious?

could the changes ever have delivered the

benefits anticipated?


26/27


Roll-out - refinement and full

implementation

uRefinement and re-issue of materials based

on pilot experiences

u Internal publicity of the results of the pilot

and the imminent roll-out of the changes

uSchedule and conduct training

uNominate members of the pilot project and

implementation team as consultants to other

projects.


Post-implementation review

uWere benefits achieved? If not why not?

u If you exceeded expectations, celebrate!

uAre there new opportunities for additional

improvements?

uAre additional refinements required?

uWhat would you do different? The same?

u Is testing now providing the information to

provide an informed decision to release?


27/27


TOM assessment forms can be

downloaded and completed at

www.evolutif.co.uk

Documents

Test Process Improvement - Paul Gerrard