The Nuts, Bolts and Duct The Nuts, Bolts and Duct Tape of Establishing a Tape of Establishing a System Engineering System Engineering

Measurement ProgramMeasurement ProgramPaul J. FrenzPaul J. Frenz

General Dynamics Advanced General Dynamics Advanced Information SystemsInformation Systems

Team AssignmentTeam Assignment

• Establish a Systems Engineering Measurement Program

• Support CMMI Level 5

• Support Organizational Initiatives

• Provide Quantifiable evidence of improvements achieved

Measures ResearchMeasures Research

Investigation of industry practices

• Software Productivity Consortium (SPC)

• Practical Software and Systems Measurement (PSM)

• INCOSE

• Generic web and literature searches

• Summer Intern did industry survey

Research FindingsResearch Findings

Measures such as:• requirements per hour • defects per requirement

can be useful to an organization:• current capabilities and setting goals • don’t provide insight into the daily

management of a program

Measures GuidelinesMeasures Guidelines

Measurements selected must be tied to business goals!

This lead to measurements that supported the goals of faultless execution, customer satisfaction, and quality

Measurement ChallengeMeasurement Challenge

The objective was to give the program leads the measures they needed to support these goals

The measures needed to guide the leads with knowing what to pay attention to and when to pay attention!

Initial Measures EstablishedInitial Measures Established

Building off of our lessons learned within Software Engineering, we established the initial measures using these principles:

• Measure collection should be uncomplicated - preferably automated

• Measures are justified

Initial Measures EstablishedInitial Measures Established

• We use a standard time period - calendar month

• Although we permit tailoring it is discouraged

• Avoid frequent replans and focus on using measures to return to the planned baseline

• Consistent criteria and an approval when a measurement baseline is replanned

Initial Measures SelectedInitial Measures Selected

• Productivity Measure – Requirements per Hour• Inch-stone Schedule Performance• Requirements Volatility• Requirement Volatility Impact • Problem Report Status Aging• Effort Expended to Planned Effort• Peer Review Scheduled to Plan• Peer Review Execution and Results• Requirements Defect Density

Productivity Measure – Productivity Measure – Requirements Per HourRequirements Per Hour

SE Productivity Predictability

0

2

4

6

8

10

12

Jan-0

4

Feb-0

4

Mar-

04

Apr-

04

May-0

4

Jun-0

4

Jul-04

Aug-0

4

Sep-0

4

Oct-

04

Nov-0

4

Dec-0

4

Jan-0

5

Feb-0

5

Re

qu

ire

me

nts

Pe

r 1

00

Ho

urs

EAC SE Requirements*100/EAC HoursOriginal EAC SE Requirements*100/EAC Hours

Inch-stone Schedule Inch-stone Schedule Performance to PlanPerformance to Plan

All Monthly Schedule Performance

0

20

40

60

80

100

120

140

Jan-

04

Feb

-04

Mar

-04

Apr

-04

May

-04

Jun-

04

Jul-0

4

Aug

-04

Sep

-04

Oct

-04

Nov

-04

Dec

-04

Jan-

05

Feb

-05

Mo

nth

ly T

asks

to

Pla

n P

erce

nta

ge

Monthly Tasks Percentage to Plan

Inch-stone Schedule Inch-stone Schedule Performance to PlanPerformance to Plan

All Schedule Performance

0

100

200

300

400

500

600

Jan

-04

Fe

b-0

4

Ma

r-0

4

Ap

r-0

4

Ma

y-0

4

Jun

-04

Jul-

04

Au

g-0

4

Se

p-0

4

Oct

-04

Nov-

04

Dec-

04

Jan

-05

Fe

b-0

5

Ta

sk

s

Planned Tasks Complete Actual Tasks Complete

Requirements VolatilityRequirements Volatility

Requirement Requirement Volatility Impact AnalysisVolatility Impact Analysis

Problem Report (PR) Status Problem Report (PR) Status AgingAging

All Problem Report Status Aging

0

10

20

30

40

50

60

1 2 3 4 5 6+

Months

Pro

ble

m R

ep

ort

s

Open Analysis CCB ReviewAssigned Verification Closed

Effort Expended to Effort Effort Expended to Effort PlannedPlanned

Peer Review Schedule to PlanPeer Review Schedule to Plan

Peer Review Execution and Peer Review Execution and ResultsResults

SE Peer Review Saves Per Page

0

0.5

1

1.5

2

2.5

3

3.5

4R

eq

SR

R

DD

PD

R

CD

R

CIR

L

Saves

Escapes Per PageMajor Saves Per PageMinor Saves Per PageNew Reviews

Peer Review Execution and Peer Review Execution and ResultsResults

SE Peer Review Hours Per Page

0

0.5

1

1.5

2

2.5

3

3.5

4

Req

SR

R

DD

PD

R

CD

R

CIR

L

Ho

urs

Hours Per PageNew Reviews

Requirements Defect DensityRequirements Defect DensitySE Requirement Defect Density

0

0.2

0.4

0.6

0.8

1

1.2

Jan-0

4

Feb-0

4

Mar-

04

Apr-

04

May-0

4

Jun-0

4

Jul-04

Aug-0

4

Sep-0

4

Oct-

04

Nov-0

4

Dec-0

4

Jan-0

5

Feb-0

5

De

fec

t P

erc

en

tag

e

Req Defects PercentageGoal Requirement Defect Density Percentage

Guidance for Implementing Your Guidance for Implementing Your Own Measurement ProgramOwn Measurement Program

Critical to Success:

• Strong measurement champion

• Strong management support

• Start small – limit your base measures to less than 7

Guidance for Implementing Your Guidance for Implementing Your Own Measurement ProgramOwn Measurement Program

Critical to Success:• Provide clear definitions for base measures

• Measure at a high enough level to identify issues without burdening

• Minimize collection effort through the use of tools and automation

Guidance for Implementing Your Guidance for Implementing Your Own Measurement ProgramOwn Measurement Program

Pitfalls:• Measures that have little value to the program• Not being able to explain how to use the data• Trying to collect too many base measures• Poorly defined, ambiguous base measures• Not creating a change in culture where

managing with measures is the norm

SummarySummary

• The development and sustainment of a measurement program requires top down support

• There must be a champion to continually sell the benefits of a measurement program

SummarySummary

• Champion needs depth and breadth of understanding of measures and the ability to enthusiastically communicate this understanding

• The measures need to be overwhelmingly geared to assisting the program in its day-to-day execution

SummarySummary

• Our measurement program has assisted our organization in dramatically improving the execution of our development programs by quantitatively managing key processes and focusing attention on critical tasks to guide our leads through the development process.