Formal Technical Reviews

Matt Graham18 November 2004

EECS 814University of Kansas

Outline

Introduction Generic Inspection and Variations Inspection throughout Development Cycle

FTR Metrics Tools Conclusion

FTR: What it is...

A method involving a structured encounter in which a group of technical personnel analyzes an artifact according to a well-defined process.

::Output::

A structured artifact that assesses or improves the quality of the artifact as well as the quality of the method.

::Input::

FTR: What it is...

A structured FTR is a forum for:

Finding defect information for the Author.

Educating peers about product.

Providing fault likelihood data for testers.

Providing detailed status report for managers.

Allowing process improvement group a test to measure.

FTR: What it isn’t...

Walk-through No Measurements Often informal, impromptu Main purpose is developer training

Why Review?

Improves overall product quality Early detection of defects Reduces rework and its associated costs Educate the participants and provide training Setting standards of excellence / maintain process

improvement momentum Improve schedule performance IBM reported 83% and AT&T 92% defect detection

through inspections

Inspection Process Overview

Planning

Overview

Defect Detection

Defect Collection

Defect Correction

Follow-Up

6 Steps Team

2 - 6 members Experienced and involved in product development Distinct and important roles

Inspection Process Overview

Planning

Overview

Defect Detection

Defect Collection

Defect Correction

Follow-Up

Roles: Organizer – plans the inspection activities Moderator – ensures that procedures are followed and moderates the meetings. Inspectors – responsible for detecting defects in the product Presenter – presents the product in logical fashion paraphrased at a suitable rate Author – author of the developed product. Recorder – records the defects during the meeting Collector – collects the defects if there is no meeting.

Planning

Planning

Overview

Defect Detection

Defect Collection

Defect Correction

Follow-Up

Objective – organize inspection Check to see if work products pass the entry criteria Select inspection participants and assign roles Schedule inspection meeting Distribute inspection material Prepare inspection documentation Ensure product inspection readiness

Objective

Planning

Overview

Defect Detection

Defect Collection

Defect Correction

Follow-Up

Objective – educate participants on the product being inspected

Author explains the inspection materials Can be beneficial when:

The inspection artifact is complex If the artifact is part of large system If new members participate in inspection

Could be rolled into previous or next phase if product is simple enough Communicate inspection goals!

Defect Detection

Planning

Overview

Defect Detection

Defect Collection

Defect Correction

Follow-Up

Objective – to find defects Can be divided into 2 phases:

Preparation (individual) phase. Meeting (group) phases.

Individual preparation with the goal of detecting defects can help inspectors to be more prepared.

Defect Collection

Planning

Overview

Defect Detection

Defect Collection

Defect Correction

Follow-Up

Objective – collectively agree and document the defects (triage)

Decide on further inspection needed or not Decisions made subjectively as a group.

Defect Collection

Planning

Overview

Defect Detection

Defect Collection

Defect Correction

Follow-Up

Objective – correct the defects collected in the previous phases

The author makes the detected changes and sends it for follow-up

Follow Up

Planning

Overview

Defect Detection

Defect Collection

Defect Correction

Follow-Up

Objective – to make sure all defects are resolved

Only one person involved and he/she verifies the defect resolution Verifier should “return and report” to inspection team

Formal Inspection Methods

Fagan Inspection Asynchronous Inspection Phased Inspections N-Fold Inspections

Fagan Inspection

Overview

Preparation

Inspection

Rework

Follow-Up

4 Roles Author Reader Moderator Scribe

Meeting centric – cost of scheduling and time Meetings add little to defect detection

Asynchronous Software Inspection

Initialization

Inspection

Review

Inspection

Compile Final Defect List

Team Author not involved Moderator and Inspectors

Re-Work Re-Work

Asynchronous Software Inspection

Initialization

Inspection

Review

Inspection

Compile Final Defect List

No meetings Easy to assess participation Process improvement from documenting all correspondences Allows parallel communication Can be distributed in space and time Eliminates group approval

Re-Work Re-Work

Asynchronous Software Inspection

Initialization

Inspection

Review

Inspection

Compile Final Defect List

Moderator sends out material Initial Individual Review – create list of defects Circulate the copy of defect list to all inspectors and discuss via email Individual Review – update defect list and send to Moderator Moderator compiles final defect list, send it to author and follow up – eliminates group approval

Re-Work Re-Work

Phased Inspections

Consists of several coordinated partial inspections called phases Each phase inspects for a specific property or small set of related properties Each phase responsible for thorough checking of the properties Some phases have single inspectors and others have multiple inspectors Reference documents are provided to inspectors at the beginning of each phase and each inspects using checklists

Domain specific checklists Application specific checklists

N-fold Inspection

N independent teams inspect the same product using traditional inspection method Collective effort of multiple teams more faults than a single team Moderator collects faults from independent teams and composes the final defect list

Requirements Inspection

Reading Techniques are techniques to analyze work product during defect detection Ad-hoc Checklist based reading Scenario based readings

Perspective based reading Defect based reading

Perspective based Reading

Inspectors stand in for specific stakeholders in the document to verify the quality of requirements.

Designer – detail for creating system components Tester – detail for constructing test plans User – correct functionality

For each perspective, reviewer creates high level work products from the requirements document

Code Inspections

Cognitive models for program comprehension Bottom-up Top-Down Integrated Systematic and as-needed comprehension

FTR Metrics

Helps you measure the effectiveness of the inspection Aids in continuous process improvement Provides feedback to management A unit of work product varies often:

Usually 1000 lines of source code (not including comments) Page counts and line counts for text

Some Generic Metrics

Average preparation effort and preparation rate per unit of material Average examination effort per unit of material Average explanation rate per unit of material Average number of defects and major defects found per unit of material Average hours per defect and per major defect

Nine Key Metrics by AT&T for Code Inspections

Total Non Comment lines of source code inspected, in KLOC

Average lines of code inspected Average preparation rate Average inspection rate Average effort per KLOC Average effort per fault detected. Average faults detected per KLOC Percentage of re-inspections Defect-removal efficiency

Tools

Various tools for different inspection methods. ICICLE – for inspection of C & C++ programs Scrutiny & InspeQ for specific inspection processes ASSIST –supports generic inspection process

Choosing a tool...

Threads of discussion Sharing of Information Train of thought Visual Cues Reaching a consensus Coordination

The Future of FTR

FTR adoption is slow: Managers see added cost of inspections instead of the benefits of greatly reduced defect leakage Practitioners already have tight schedules and reluctant to take on additional responsibilities Companies postpone adoption since peer reviews are part of CMM level 3.

FTR Barriers

Insufficient preparation Moderator domination Incorrect review rate Ego-involvement and personality conflict Issue resolution and meeting digression Recording difficulties and clerical overhead

Conclusion

Organization’s size, culture and industry should be considered in deciding on the FTR method to use. Review does not replace testing but can make it easier Success of inspection lies in:

Process Adherence Systematic Reading technique Optimization

FTR: References

● Philip Johnson: http://www2.ics.hawaii.edu/~johnson/FTR/● Tom Gilb: http://www.result-planning.com/● IEEE STD 1024-1197