PSP Quality Strategy [SE-280 Dr. Mark L. Hornick 1

PSP Quality Strategy

[ SE-280Dr. Mark L. Hornick

1


2

What is software quality?

Basic definition: Meeting users' needs

What about “wants”? True needs may be hard to

establish

Hierarchy of needs Do required tasks Be

Usable & convenient Install quickly & easily Run consistently

Economical Timely Dependable

No destructive or unexpected behavior


3

How do you measure quality?


4

In the PSP, defects are the basic quality measure; is this reasonable?

On the one hand, defects are only important to the customer or user if they cause visible adverse effects.

• Affect operations• Cause inconvenience• Cost time or money• Result in a loss of confidence

However, defects have a big impact on the software development process, costing money and taking time that could be better spent in adding value for those customers and users.


5

Why focus on defects?

Defects have a big impact on the software development process:

Defects must be managed, costing money and taking time that could be better spent in adding value for those customers and users.

With poor quality, there is no time to address other quality issues such as:

• Ease of installation• Safety• Recovery & robustness• Usability & convenience

Generally, defects are more expensive to remove if they are not found until later in the software development process.

Average defect fix time (minutes) [Xerox TSP team]

5

22

2

25

32

1405

1 10 100 1000 10000

Design review

Design insp.

Code review

Code insp.

Unit test

System test


7

CMM – Capability Maturity Model

CMM is an assessment of an organization’s software capabilities; an SEI-developed metric with 5 levels:

Level 1 = “no defined process” Level 5 = “process exhibiting continuous

improvement”


8

A mature organization at CMM level 5:Creates and reviews requirements

Does design inspections

(design is reviewed by a group)

Does code inspections

(code is reviewed by a group)

Does a lot of testing And still leaves about 1 defect/KLOC


9

With the growth of software systems, we are facing a significant quality challenge

What if we want 1-10 defects per MLOC? Level 5 CMMI is about 1000 defects/MLOC!

You probably can't "test in" this quality level

And, probably can't "inspect it in" either: Assume 1000 LOC 30 pages of listings Then, 1 MLOC 30,000 pages Can we inspect 30K pages well enough to leave

only 1-10 defects??


10

Quality management (defect removal and prevention) is a critical part of the PSP.

Q. Who injects software defects?A. Individual software engineers!

Fundamental Axiom: The personal process of the individual software engineer is the place to remove defects, determine their causes, and learn to prevent them.


11

Remove

Quality Metrics: Yield is a measure of how well we find and remove defects.

Developmentphase

Yield is the % of existing defects we remove in a phase.

Generally, defects are injected in development phases like Design and Coding, but it is possible for

defects to be injected or removed in any phase..

Inject

Developmentphase

RemoveInject

Developmentphase

RemoveInject

Escapes are defects that pass from one phase to the next.

PhaseInjectedInhasesFromPriorP

haseRemovedInPPhase DefectsEscapes

DefectsYield

100

Process Yield is the composite yield for all phases before Test.

CodeReview

Planning

CodeReview

Planning100injected

removed

Defects

DefectsYield

The process yield (or just "yield") target is about 70%; that is, no more than 30% of defects should be left to be found during testing, since testing is expensive!


13

There are different kinds of costs associated with quality.

Appraisal costs are the costs of inspecting for defects, (in the PSP, Design and Code review times).

Failure costs result from rework, repair, and scrap • in the PSP, this is time you spend in Test

Both of these metrics are costs of quality; that is,the “price” you pay to ensure the quality of your code.

The relationship between these two kinds of quality costs are described by a ratio:

Failure

Appraisal

COQ

COQFRA /

In the PSP, the A/F ratio should be about 2.0.

A/FR tracks ratio of Appraisal cost (review time) Failure cost (test time)

SE-280Dr. Mark L. Hornick

15

If measured in many organizations, A/FR would be near zero!

but (mature) PSP processes typically exceed 2.0

High A/FR is associated with low test defect density and high product quality

The five PQI components are calculated in the following manner.

OCfectsPerKLUnitTestDe

10PQI

CectsPerKLOCompileDef

20PQI

CodeTime

TimeCodeReviewPQI

DesignTime

ewTimeDesignReviPQI

CodeTime

DesignTimePQI

fectsUnitTestDe

ectsCompileDef

TimeCodeReview

ewTimeDesignRevi

TimeDesignCode

5

10

2

2

Note that all PQI values are limited to a range of 0.0 … 1.0

iOverall PQIPQI

Target value: < 10 defects/KLOC A+M

Target value: < 5 defects/KLOC A+M

Overall PQI is computed by taking the product of the components.

Target: > 0.4

Here is an example of a PQI calculation.

PQIRaw PQI

AdjPQI

DesignCodeTime

DesignReviewTime

CodeReviewTime

CompileDefects

UnitTestDefects

Total PQI

Design Time 125

Design Review Time 38

Code Time 67

Code Review Time 32

Compile Defects/KLOC (A+M) 0

UT Defects/KLOC (A+M) 6.1

2

2

10

5

DesignCodeTime

DesignReviewTime

CodeReviewTime

CompileDefects

UnitTestDefects

DesignTimePQI

CodeTimeDesignReviewTime

PQIDesignTime

CodeReviewTimePQI

CodeTime20

PQICompileDefectsPerKLOC

10PQI

UnitTes

tDefectsPerKLOC

PQIDesignCodeTime

PQICode Review Time

PQICompile DefectsPQIUnit Test Defects

PQIDesign Review Time

Note that Adjusted PQI values are limited to a range of 0.0 … 1.0

Results of the PQI calculations

PQIRaw PQI

AdjPQI

DesignCodeTime 1.87 1.00

DesignReviewTime 0.61 0.61

CodeReviewTime 0.96 0.96

CompileDefects 2.00 1.00

UnitTestDefects 0.90 0.90

Total PQI 0.52

Design Time 125

Design Review Time 38

Code Time 67

Code Review Time 32

Compile Defects/KLOC (A+M) 0

UT Defects/KLOC (A+M) 6.1 OCfectsPerKLUnitTestDe

10PQI

CectsPerKLOCompileDef

20PQI

CodeTime

TimeCodeReviewPQI

DesignTime

ewTimeDesignReviPQI

CodeTime

DesignTimePQI

fectsUnitTestDe

ectsCompileDef

TimeCodeReview

ewTimeDesignRevi

TimeDesignCode

5

10

2

2

PQIDesignCodeTime

PQICode Review Time

PQICompile DefectsPQIUnit Test Defects

PQIDesign Review Time

The process quality index (PQI) is a compact representation of five quality measures.

Text, page 150

The value at each vertex is 1.0 (good), while the center is zero (bad).

TimeDesignCodePQI

ewTimeDesignReviPQI

TimeCodeReviewPQI

ectsCompileDefPQIfectsUnitTestDePQI

The Process Dashboard tool computes a number of quality metrics.

Start looking at your own quality metrics!

What about the use of automated defect-finding tools?

If you have a tool that can remove 100% of a certain type of defect, use it as early as possible to get rid of them!

If you have a tool that can remove about 10% of a certain type of defect, it is not very effective, but also relatively harmless since you won’t rely on it.

If you have a tool that can remove 90% of a certain type of defect, it can be dangerous because you will tend to believe that it found them all, and not aggressively seek out the remaining ones.


22

Conclusion

Inspections and Testing will not catch enough defects

It is more much more expensive to inspect and test than to review

Personal Design and Code reviews are key to high quality software

Documents

PSP Quality Strategy [SE-280 Dr. Mark L. Hornick 1