March 2002

COSYSMO: COnstructive SYStems Engineering Cost MOdel

Ricardo Valerdi

USC Annual Research Review

March 11, 2002

March 2002 2

Outline• Background on COSYSMO

• EIA632

• Approach

• Delphi Survey

• Delphi Round 1 Results

• Analysis/Conclusions

• Lessons Learned/Improvements

• The Next Step

• Q & A

March 2002 3

“All models are wrong, but some of them

are useful”

- W. E. Deming

Source: www.deming.org

March 2002 4

What is it?

The purpose of the COSYSMO project

is to enhance the current capability of

the COCOMO II model by accounting

for costs that are outside the realm of

software engineering by introducing

system engineering drivers.

March 2002 5

The Challenge

To develop a preliminary model for estimating the cost impact of front-end System Engineering tasks in the design of software intensive systems.

These include system definition, integration, and test activities as defined in standard EIA632.

                                       

       

March 2002 6

Approach

• Begin with front-end costs of information systems engineering

• Follow 7-step modeling methodology– Steps 5,6, & 7 (gather data and refine loop)

• Use model parameters compiled by TRW, SAIC, Raytheon, and USC/CSE– System size (requirements, TPMs, I/F)– Effort drivers (maturity, cohesion, stability)

March 2002 7

COCOMO Suite

COCOMOII

COQUALMOCOPSEMO

CORADMO

COSYSMO

COPROMO

COCOTS

March 2002 8

COSYSMO Operational Concept

COSYSMO

SizeDrivers

CostDrivers

Effort

Duration

Calibration

# Requirements# Interfaces# TPM’s# Scenarios# Modes# Platforms# Algorithms

- 7 Application factors- 8 Team factors

WBS guidedBy EIA 632

COCOMO II-based model

March 2002 9

EIA63233 activities organized into 5 groups:

1. Acquisition and supply2. Technical management

• Planning process• Assessment process• Control process

3. System design• Requirements definition process• Solution definition process

4. Product realization5. Technical evaluation

March 2002 10

Delphi Survey

• 3 Sections:– Scope, Size, Cost

• Used to determine the range for size driver and effort multiplier ratings

• Identify the cost drivers to which effort is most sensitive to

• Reach consensus from systems engineering experts

March 2002 11

7 Size Drivers

1. Number of System Requirements 2. Number of Major Interfaces3. Number of Technical Performance

Measures4. Number of Operational Scenarios5. Number of Modes of Operation6. Number of Different Platforms7. Number of Unique Algorithms

March 2002 12

15 Cost Drivers

1. Requirements understanding 2. Architecture understanding 3. Level of service requirements,

criticality, difficulty 4. Legacy transition complexity 5. COTS assessment complexity 6. Platform difficulty 7. Required business process

reengineering

Application Factors (7)

March 2002 13

15 Cost Drivers (cont…)

1. Number and diversity of stakeholder communities

2. Stakeholder team cohesion

3. Personnel capability

4. Personal experience/continuity

5. Process maturity

6. Multisite coordination

7. Formality of deliverables

8. Tool support

Team Factors (8)

March 2002 14

Delphi Round 123 Surveys returned

Aerospace 2

Galorath 1

Lockheed Martin 8

Raytheon 4

SAIC 6

TRW 1

USC 1

March 2002 15

System Engineering Effort Per EIA Stage

StageSupplier Performance Technical ManagementRequirements DefinitionSolution DefinitionSystems AnalysisRequirements ValidationDesign Solution VerificationEnd Products Validation

Delphi5.2%

13.1%16.6%18.1%19.2%11.3%10.5%

6.6%

Suggested5%

15%15% 20%20%15%

5%5%

3.054.254.544.285.974.586.073.58

Std. Dev.

March 2002 16

Delphi Round 1 Highlights (cont.)

Range of sensitivity for Size Drivers

# A

lgo

rith

ms

# R

eq

uir

emen

ts

# In

terf

aces

# T

PM

’s

# S

cen

ario

s

# M

od

es

# P

latf

orm

s

5.57

Relative

Effort

1

2.232.54

2.212.10

6.48

6

4

2

March 2002 17

Two Most Sensitive Size Drivers

Suggested Rel. Effort

Delphi Respondents EMR

Rel. Effort

Standard Deviation

# Interfaces 4 5.57 1.80

# Algorithms 6 6.48 2.09

March 2002 18

Delphi Round 1 Highlights (cont.)

Range of sensitivity for Cost Drivers (Application Factors)

EMR

1.93

2.812.13

2.432.24

4

2

Req

uir

em

ents

un

d.

Arc

hit

ectu

re u

nd

.

Lev

el o

f se

rvic

e re

qs.

Leg

acy

tra

nsi

tio

n

CO

TS

Pla

tfo

rm d

iffi

cult

y

Bu

s. p

roce

ss r

een

g.

1.741.13

March 2002 19

Delphi Round 1 Highlights (cont.)

Range of sensitivity for Cost Drivers (Team Factors)

1.28

2.461.91 2.161.94

1.25

To

ol

sup

po

rt

Sta

keh

old

er c

om

m.

Sta

keh

old

er c

oh

esio

n

Per

son

nel

cap

ab

ilit

y

Per

son

al

exp

erie

nce

Pro

cess

mat

uri

ty

Mu

ltis

ite

coo

rd.

Fo

rmal

ity

of

del

iv.

1.841.78

EMR4

2

March 2002 20

Suggested EMR

Delphi Respondents EMR

MeanStandard Deviation

Arch. Under. 1.66 2.24 0.83

Reqs. Under. 1.73 2.43 0.70

Pers. Cap. 2.15 2.46 0.66Serv. Req. 2.5 2.81 0.67

Four Most Sensitive Cost Drivers

March 2002 21

Conclusions

Not only do we need to better manage requirements, we also need to manage:

1) # of Interfaces2) # of Algorithms3) Personnel Capability4) Level of service requirements, criticality,

difficulty 5) Level of understanding

“Control the controllables”

March 2002 22

Lessons Learned/Improvements

                                            

  

Lesson 1 – There are lots of people and groupsInterested in more precisely estimating system costs- And they are willing to help do it free.Lesson 2 – Currently, system engineering effort isestimated using activity-based costing heuristicsLesson 3 – When mounting a Delphi, clearly identify what you are trying to do.- Else, system engineers will attack you with a shotgunLesson 4 – We could use help in developing a betterdesigned template for Delphi instruments

March 2002 23

The Next Steps

• Incorporate suggestions from Delphi 1• Write report• Get Masters degree

• Data from completed systems will then be used to statistically confirm or deny initial ratings• Round 2 of Delphi

…in the future

March 2002 24

Special Thanks to:

• Advisors

Dr. Boehm, Dr. Axelband, Don Reifer

• Affiliates

Gary Hafen, Tony Jordano, Chris Miller, Karen Owens, Don Reifer, Garry Roedler, Evin Stump, Gary Thomas, Marilee Wheaton

March 2002 25

Where can I get more info?

• COSYSMO Working Group meeting

Thursday March 14th 12:00 - 5:00

• valerdi.com/cosysmo

March 2002 26

March 2002 27

Delphi Round 1 Participants1 Don Reifer USC2 Gary Thomas Raytheon3 John Rieff Raytheon4 Deke Dunlap Raytheon5 Michael McBride SAIC6 Tony Jordano SAIC7 James Evans LMCO8 Albert Cox TRW9 Abe Santiago Aerospace

10 Craig Hayenga LMCO11 Robert Kaufman SAIC12 Greg Kaszuba LMCO13 Phill Rowell SAIC14 Keith Young LMCO15 Charles Zumba SAIC16 Susan Ruth Aerospace17 Denton Tarbet Galorath18 Don Greenlee SAIC/INCOSE19 Jeffrey Shupp LMCO/INCOSE/Vertex Inc.20 David Lindstrom LMCO21 Garry Roedler LMCO22 Rocky Hudson LMCO23 John McDonald Raytheon