... there is no particular reason... there is no particular reasonwhy your friend and colleague why your friend and colleague cannot also be your sternest critic.cannot also be your sternest critic.

--Jerry Weinberg

Reviews and Inspections

Underlying principles◦ We should not review our own work◦ Group synergy

A walkthrough team consists of from four to six members

It includes representatives of◦ The team responsible for the current workflow◦ The team responsible for the next workflow◦ The SQA group

reviewreviewleaderleader

producerproducer

recorderrecorder reviewerreviewer

standards bearer (SQA)standards bearer (SQA)

maintenance maintenance oracleoracle

user repuser rep

Select Participants Schedule meeting Develop agenda

The walkthrough is preceded by preparation◦ Evaluate product before walkthrough◦ Lists of items

Items not understood Items that appear to be incorrect

be prepared—evaluate be prepared—evaluate product before the reviewproduct before the review

review the product, not review the product, not the producerthe producer

keep your tone mild, ask keep your tone mild, ask questions instead of questions instead of making accusationsmaking accusations

stick to the review agendastick to the review agenda

raise issues, don't resolve themraise issues, don't resolve them

avoid discussions of style—stick to technical correctnessavoid discussions of style—stick to technical correctness schedule reviews as project tasksschedule reviews as project tasks

record and report all review resultsrecord and report all review results

1.1.

2.2.

3.3.

4.4.

5.5.

6.6.

7.7.

8.8.

The walkthrough team is chaired by the SQA representative

In a walkthrough we detect faults, not correct them◦ A correction produced by a committee is likely to

be of low quality◦ The cost of a committee correction is too high◦ Not all items flagged are actually incorrect◦ A walkthrough should not last longer than 2 hours◦ There is no time to correct faults as well

A walkthrough must be document-driven, rather than participant-driven

Verbalization leads to fault finding

A walkthrough should never be used for performance appraisal

An inspection has five formal steps◦ Overview◦ Preparation, aided by statistics of fault types◦ Inspection◦ Rework◦ Follow-up

An inspection team has four members◦ Moderator◦ A member of the team performing the current

workflow◦ A member of the team performing the next

workflow◦ A member of the SQA group

Special roles are played by the◦ Moderator◦ Reader◦ Recorder

Faults are recorded by severity ◦ Example:

Major or minor

Faults are recorded by fault type◦ Examples of design faults:

Not all specification items have been addressed Actual and formal arguments do not correspond

For a given workflow, we compare current fault rates with those of previous products

We take action if there are a disproportionate number of faults in an artifact◦ Redesigning from scratch is a good alternative

We carry forward fault statistics to the next workflow◦ We may not detect all faults of a particular type in

the current inspection

IBM inspections showed up◦ 82% of all detected faults (1976)◦ 70% of all detected faults (1978)◦ 93% of all detected faults (1986)

Switching system◦ 90% decrease in the cost of detecting faults (1986)

JPL◦ Four major faults, 14 minor faults per 2 hours (1990) ◦ Savings of $25,000 per inspection◦ The number of faults decreased exponentially by

phase (1992)

Warning

Fault statistics should never be used for performance appraisal◦ “Killing the goose that lays the golden eggs”

Walkthrough◦ Two-step, informal process

Preparation Analysis

Inspection◦ Five-step, formal process

Overview Preparation Inspection Rework Follow-up

Reviews can be effective◦ Faults are detected early in the process

Reviews are less effective if the process is inadequate◦ Large-scale software should consist of smaller,

largely independent pieces◦ The documentation of the previous workflows has

to be complete and available online

Review Options Matrix

trained leadertrained leaderagenda establishedagenda establishedreviewers prepare in advancereviewers prepare in advanceproducer presents productproducer presents product““reader” presents productreader” presents productrecorder takes notesrecorder takes noteschecklists used to find errorschecklists used to find errorserrors categorized as founderrors categorized as foundissues list createdissues list createdteam must sign-off on resultteam must sign-off on result

IPR—informal peer review WT—WalkthroughIPR—informal peer review WT—WalkthroughIN—Inspection RRR—round robin review (no face to face meeting)IN—Inspection RRR—round robin review (no face to face meeting)

IPRIPR WTWT ININ RRRRRRnonomaybemaybemaybemaybemaybemaybenonomaybemaybenononononononono

yesyesyesyesyesyesyesyesnonoyesyesnonononoyesyesyesyes

yesyesyesyesyesyesnonoyesyesyesyesyesyesyesyesyesyesyesyes

yesyesyesyesyesyesnonononoyesyesnonononoyesyesmaybemaybe

**

*

Inspection rate (e.g., design pages inspected per hour)

Fault density (e.g., faults per KLOC inspected)

Fault detection rate (e.g., faults detected per hour)

Fault detection efficiency (e.g., number of major, minor faults detected per hour)

Does a 50% increase in the fault detection rate mean that◦ Quality has decreased? Or◦ The inspection process is more efficient?