COSYSMO Extension as a Proxy Systems Cost Estimation

1

Reggie ColeLockheed Martin Senior [email protected]

Garry RoedlerLockheed Martin [email protected]

April 30, 2014

COSYSMO Extension as a Proxy Systems Cost Estimation

mailto:[email protected]

mailto:[email protected]

2

AgendaCOSYSMO as a Proxy for System Cost Estimation

Deep Dive on the Proxy/Bias Function

Key Use Cases

Workshop Results and Recommendations

3

COSYSMO as a Proxy for System Cost Estimation

4

COSYSMO as a Proxy Estimator for System Cost

• Need for a Top-Down System Cost Model– Better Buying Power 2.0 Mandate

• Perform ongoing should-cost analysis– Better Buying Power 2.0 Implication

• Perform design-to-cost analysis and design for affordability– There is Currently a Gap in Tools

• Primarily in early-stage analysis – when directions can still be changed without significant repercussions

• Extending COSYSMO for System Costing– Evaluate the Viability of Extending COSYSMO

• COSYSMO seems to have the right parameters

5

Cost Modeling Needs Change Over Time in Terms of Speed and Accuracy

Problem-Space Description

Cost Estimate ± 25%

High-Level Solution Description

Cost Estimate ± 10%

Detailed Solution Description

Cost Estimate ± 5%

High-Level Solution Assumptions Cost Estimate ±

20%

Increasing Effort and Cost-Modeling Expertise

Increasingly Refined

Information About the

Solution

Increasingly Refined Cost Estimate

Incre

asin

gly Re

fined

Solu

tion

We Have a Good Selection of Tools for Late-Stage Cost Modeling

We Have Gaps in Early-Stage Cost Modeling

6

SE Effort as a Proxy Measure of Overall System Size and Complexity

• Proxy Measures– Proxy measures are used when you cannot directly measure what you want to

measure – and when an indirect measure provides sufficient insight– Proxy measures are often used in clinical studies since direct measurement is often

infeasible or can even alter the outcome– It is not always possible to directly measure what you want to measure – or directly

estimate what you want to estimate

• System Engineering Effort is a Proxy Measure for System Cost– There is strong evidence for the link between systems engineering effort and program

cost – dating back to a NASA study in the 1980s– The optimal relationship between systems engineering effort and overall program cost

is 10% - 15%– Industry has long used a parametric relationship between software cost and systems

engineering cost for software-intensive systems– Systems engineering effort can be an effective proxy measure for overall system cost

7

COSYSMO 2.0 Model Parameters Provide a Rich Assessment of System Size and Complexity

Number of System RequirementsNumber of Major System Interfaces

Number of Critical AlgorithmsNumber of Operational Scenarios

Size Drivers

Requirements Understanding Architecture Understanding

Level of Service Requirements

Migration Complexity

Technology Risk

Level of Documentation Required

Diversity of Installed Platforms

Level of Design RecursionStakeholder Team Cohesion

Personnel / Team Capability

Personnel Experience / Continuity

Process Capability

Multisite Coordination

Level of Tool Support

Cost Drivers

Managed ElementsAdopted ElementsDeleted ElementsModified Elements

New Elements

Reuse FactorsInitial Estimate of System Size

Scaled Estimate of System Size

Consolidated Cost Driver Factor

Estimate of Systems Engineering Effort…Also a Biased Proxy Estimator for System Scope…And System Cost

8

Relationship Between SE Effort and Total EffortTotal Program Overrun

32 NASA Programs

R2 = 0.5206

0

20

40

60

80

100

120

140

160

180

200

0 5 10 15 20

Definition Percent of Total Estimate

Prog

ram

Ove

rrun

Definition $Definition Percent = ---------------------------------- Target + Definition$

Actual + Definition$Program Overrun = ---------------------------------- Target + Definition$

0.6

1.0

1.4

1.8

2.2

2.6

3.0

0% 4% 8% 12% 16% 20% 24% 28%

SE Effort = SE Quality * SE Cost/Actual Cost

Act

ual/P

lann

ed C

ost

NASA data supports a 10%-15% optimal allocation of systems engineering effort as a portion of overall program effort

W. Gruhl, Lessons Learned, Cost/Schedule Assessment Guide,” Internal Presentation, NASA Comptroller’s Office, 1992

E. Honour, “Understanding the Value of Systems Engineering,” INCOSE, 2004

INCOSE study on the value of systems engineering also supports a 10%-15% optimal allocation of systems engineering as a portion of overall program effort

9

Putting It All Together

Size Drivers (Problem Space) Customer Requirements System Interfaces Major Algorithms Operational Scenarios

Complexity Drivers (Problem/Solution) Requirements Understanding Architecture Understanding Level of Service Requirements Migration Complexity Technology Risk Documentation Needs Installations/Platform Diversity Levels of Recursion in the Design Stakeholder Team Cohesion Personnel/Team Capability Personnel Experience/Continuity Process Capability Multisite Coordination Tool Support

Reuse Factors (Solution Space) New Modified Deleted Adopted Managed

0.6

1.0

1.4

1.8

2.2

2.6

3.0

0% 4% 8% 12% 16% 20% 24% 28%


Act

ual/P

lann

ed C

ost

SE Effort is an estimator for total system cost…but it is a biased estimator

Estimator Bias Function is Based on the Well-Established Relationship Between SE Effort and Overall Program Effort

Estimation of Total System Cost

10

Adjusting Our View of COSYSMO ParametersOur view of the size drivers can be preserved – their context doesn’t change under COSYSMO extension

Our view of the cost drivers needs to be broadened to include all aspects of the system, not just systems engineering

Our view of reuse requires the most extreme adjustment – we are not just talking about systems engineering artifacts

11

Expanding Our View of Cost Drivers




Technology Risk






Process Capability



Cost Drivers

Precedented systems and unprecedented systems are fundamentally different

Need to consider the entire team – including subcontractors

Need to consider the all processes and tools

System modification and reuse have a significant effect on some cost drivers

Expand to a view of all aspects of the system for the Cost Drivers

12

Expanding Our View of Reuse Factors


New Elements

Reuse Factors

New Elements – These are elements that need to be engineered and developed. Just reusing systems engineering artifacts is not sufficient.

Modified Elements – These are elements that offer some form of reuse. Enhanced COTS or reusable components that need modification fall into this category.

Adopted Elements – These are elements that offer significant reuse with minimal modification and do not require full retesting. COTS typically falls into this category.

Managed Elements – These are elements that are already in the system and require minimal regression testing. A previously deployed element falls into this category.

For Requirements… Think Functional Components

For Interfaces… Think Connection Points

For Algorithms… Think Functional Components

For Scenarios… Think Implications to Both

Need to include the system elements, as well as the SE artifacts

13

Example

Consider the case of large C2 system. Initially developed 20 years ago, the system was unprecedented. Twenty years later, a replacement system is needed. While the initial development was unprecedented, the replacement system is not, which drives down the size drivers (through reuse) and cost drivers. Case 1 – Unprecedented System Case 2 – Developed Replacement System Case 3 – COST/GOTS-Based Replacement System

0.00

2.00

4.00

6.00

8.00

10.00

0

500

1000

1500

2000

Pessimistic Expected Optimistic

Cost

Driv

er F

acto

r

Size

(Effe

ctive

Req

uire

men

ts)

Case 1 - Large Unprecedented System

Requirements System I/F

Algorithms Scenarios

Cost Driver Factor

0.00

0.20

0.40

0.60

0.80

0

200

400

600

800

1000

1200


Cost

Driv

er F

acto

r

Size

(Effe

ctive

Req

uire

men

ts)

Case 2 - Replacement System (Developed)



Cost Driver Factor

0

0.1

0.2

0.3

0.4

0.5

0.6

0

100

200

300

400

500

600


Cost

Driv

er F

acto

r

Size

(Effe

ctive

Req

uire

men

ts)

Case 3 - Replacement System (COTS/GOTS)



Cost Driver Factor

Similar Size Drivers – But Significantly Different Cost Drivers

Similar Cost Drivers – But Significantly Different Size Drivers

14

Part 1 Wrap-Up

• The Approach Based on Well-Established Approaches– COSYSMO provides the basis for estimation of systems engineering

effort – and a biased proxy estimator for overall system cost– There is a well-established relationship between systems

engineering effort and overall effort used to de-bias the COSYSMO-modeled effort

• The Approach Can Improve System Cost Modeling– It occupies an important niche – fully parametric system cost

modeling in the early stages of system definition– It can serve as a powerful affordability analysis tool – supporting

rapid-turnaround analysis of alternatives– But…it is not a replacement for existing models

15

Deep Dive on the Proxy/Bias Function

16

Context of the Bias Function

Deeper Discussion of the Proxy/Bias Function is Necessary – As Well as a Technique for Generating Cumulative Distribution of System Costs

17

The Bias Function

stic)(Determini etc.) Fee, MR, ODC, A,&(G Factors Additional : minc)(Stochasti etc.) material, travel,(e.g., Factors Additional :

c)(Stochasti RateLabor : c)(Stochasti COSYSMO UsingComputedEffort SE :

c)(StochastiEffort Program Total Effort to SE Convertingfor Factor : stic)(DeterminiFactor n Calibratio COSYSMO :

:Where

min

Cost

Cost

Labor

SE

Conv

Cal

CostCostConv

LaborCalSESystem

sisticAdderDeterAddersStochastic

RateEffortFF

sisticAdderDeterAddersStochasticFRateFEffortCost

Variable Type DescriptionCOSYSMO Calibration Factor Deterministic Scalar Value Organization-specific calibration factor

Effort Conversion Factor Triangular Distributed Random Variable Three-point estimate of factor to convert SE effort to total program effort (nominally 0.08, 0.12 and 0.16)

SE Effort Triangular Distributed Random Variable Three-point estimate for SE effort, generated using COSYSMO

Labor Rate Triangular Distributed Random Variable Three-point estimate for composite labor rate

Material Costs Triangular Distributed Random Variable Three-point estimate for material costs

Travel Costs Triangular Distributed Random Variable Three-point estimate for travel costs

We are going to discuss each of these factors!

18

SE Effort (EffortSE)

Size Drivers (Problem Space) Customer Requirements System Interfaces Major Algorithms Operational Scenarios

Complexity Drivers (Problem/Solution) Requirements Understanding Architecture Understanding Level of Service Requirements Migration Complexity Technology Risk Documentation Needs Installations/Platform Diversity Levels of Recursion in the Design Stakeholder Team Cohesion Personnel/Team Capability Personnel Experience/Continuity Process Capability Multisite Coordination Tool Support

Reuse Factors (Solution Space) New Modified Deleted Adopted Managed

Three different COSYMO scenarios – optimistic, expected & pessimistic – provide the basis for a sampling distribution.

COSYSMO provides a Proxy Estimate of the system cost. We will not try to de-bias it right now….that is the next step.

Since we want to perform Monte Carlo simulation of costs, we would like to generate a distribution of the proxy costs.

Optimistic Expected Pessimistic

Triangular Distribution Beta Pert Distribution

COSYSMO Estimate of Hours Becomes the Parameters for Either Triangular or Beta Pert Distribution

19

Effort Conversion Factor (FConv)

Total Program Overrun32 NASA Programs

R2 = 0.5206

0

20

40

60

80

100

120

140

160

180

200

0 5 10 15 20

Definition Percent of Total Estimate

Prog

ram

Ove

rrun

Definition $Definition Percent = ---------------------------------- Target + Definition$

Actual + Definition$Program Overrun = ---------------------------------- Target + Definition$

0.6

1.0

1.4

1.8

2.2

2.6

3.0

0% 4% 8% 12% 16% 20% 24% 28%


Act

ual/P

lann

ed C

ost

Studies provide some insight into what the value should be

This is probably the most important factor in the bias function!

Heuristic approaches for determining SE costs for software intensive systems are also consistent with these studies

All indications point to a range of 0.08 to 0.16 for FConv

And this range is consistent with all the data we’ve collected to date…for relatively healthy programs

And it provides the basis for our random variable parameters

Triangular Distribution Beta Pert Distribution

0.08 0.12 0.16OptimisticExpectedPessimistic

20

Other Stochastic “Adder” Factors

• Labor Costs– Labor costs vary – especially in early stages – and needs to be treated as a random variable– Any number of distributions would probably be OK – Beta Pert would be a good default– If hours are the item of interest rather than cost, this factor can be omitted

• Material Costs– Material costs vary – especially in early stages – and needs to be treated as a random

variable– Any number of distributions would probably be OK – Beta Pert would be a good default but

there is at least one study that looks at using a normal distribution– If hours are the item of interest rather than cost, this factor can be omitted

• Travel Costs– Travel costs vary – especially in early stages – and needs to be treated as a random variable– Any number of distributions would probably be OK – Beta Pert would be a good default– If hours are the item of interest rather than cost, this factor can be omitted

• Other– Any number of other “stochastic adders” can be treated similarly

21

Other “Deterministic Adders”

• Some Factors Are Well Known– To the Point They Can Be Considered Deterministic– They are often set and apply across programs– Examples include:

• G&A Costs• Other Direct costs• Management Reserve• Fee

22

COSYSMO Calibration Factor

• Local Calibration is Important– Keeps us from over-tuning the Effort Conversion

Factor

• This Also Serves as a Type of Organizational Efficiency Factor– Can vary across organizations within an enterprise

• It is a Simple Scalar Factor– Optimally, it should be 1.0

23

Calibration for Proxy Estimation

• It is Not Necessary to Revalidate or Recalibrate COSYSMO– The strength of this approach is that it rests on the

COSYSMO foundation

• It is Necessary to Validate and Calibrate the Bias Function– Important to validate the relationship between

system costs and systems engineering costs– Important to calibrate the COSYSMO Calibration

Function

24

Our Current Approach for Validation

• Use a Few Long-Running Programs– They tend to have good data collection and good

process discipline – so the data is reliable

• Treat Each Major Release as a Separate Entity– That really allows us to dig into reuse between

releases– It is necessary to collect data on reuse – we found

that to be a little challenging– Use some releases for validation and others for

calibration

25

Run Monte Carlo Simulations and Generate Cumulative Distribution of Costs

Define Parameters for Remaining Factors in Bias Function

Construct the COSYSMO Scenarios

Revisiting the Overall Approach

1

2

3

260.6

1.0

1.4

1.8

2.2

2.6

3.0

0% 4% 8% 12% 16% 20% 24% 28%


Act

ual/P

lann

ed C

ost

Revisiting the Overall Approach

Requirements Baseline

Architecture Baseline

Optimistic Expected PessimisticRequirements

Interfaces

Algorithms

Scenarios

Bias Function

Risky Range Target Cost Target

Reserve

2. Define Parameters for Remaining Factors in Bias Function - Effort Conversion Factor- Stochastic Adder Factors - Deterministic Adder Factors

3. Run Monte Carlo Simulations and Generate Cumulative Distribution of Costs

stic)(Determini etc.) Fee, MR, ODC, A,&(G Factors Additional : minc)(Stochasti etc.) material, travel,(e.g., Factors Additional :

c)(Stochasti RateLabor : c)(Stochasti COSYSMO UsingComputedEffort SE :

c)(StochastiEffort Program Total Effort to SE Convertingfor Factor : stic)(DeterminiFactor n Calibratio COSYSMO :

:Where

min

Cost

Cost

Labor

SE

Conv

Cal

CostCostConv

LaborCalSESystem

sisticAdderDeterAddersStochastic

RateEffortFF

sisticAdderDeterAddersStochasticFRateFEffortCost

1. Construct the COSYSMO Scenarios

1

2 3

27

Part 2 Wrap-Up

• The Basic Bias Function– Concerns with the basic bias function– Suggested improvements on the basic bias

function

• The Approach for Validation and Calibration– Concerns with the basic approach– Suggested improvements for validation and

calibration

28

The Key Use Cases

29

The Key Use Cases

• Should-Cost Analysis– Establishing a should-cost for which cost estimates from

bidders can be evaluated– Establishing a DTC target for performing DTC analysis

• Design-to-Cost Analysis– Performing cost vs. capability trades as way to provide more

affordable solutions

• Analysis of Alternatives– Evaluating alternative solution strategies– It’s not just about the problem space – the solution space can

be evaluated too

30

Should-Cost Analysis

• Goal is to Establish a Target Cost– Can also be a cost range– Usually performed very early in the lifecycle

• Approach– Given a requirements baseline, use the extended

COSYSMO approach to estimate the cost– Use “plug” numbers for adders

• e.g., labor, material, fee, etc.

31

Part 3 – Wrap-Up

• There Are Three Very Important Use Cases– Should-Cost Analysis– DTC Analysis– Analysis of Alternatives

• There Are At Least a Couple More– Change Evaluation for Operational Systems– Scope Creep Monitoring– Probably Others We Haven’t Thought Of

• Discussion on Use Cases

32

Workshop Results and Recommendations

33

Conclusions and Recommendations (1 of 3)

• Validity of overall approach– Unanimous support – approach is valid and should

continue to be developed and refined for wider application

– Feedback was all focused on ensuring all factors had been considered and areas for refinement – no discussion resulted in conclusion that the approach had major issues

– Appropriately uses the concepts of COSYSMO and tailors the perspective for systems

34


• Improvement of Approach– Best distribution to use? Cumulative, frequency, or other? – Preference is to not change the Scale Factor

• Want to retain as close to COSYSMO as possible – ready to leverage COSYSMO 3

– Review and refine the bias function and calibration of the bias function

• May consider adding in other COCOMO factors, as applicable• May need to establish rules for when to leverage a HW model using the

same approach to use as input for HW, when highly HW intensive• May need to consider calibration for different types/classes of systems• Bottom line – User needs to consider what tailoring/adaptation of the

bias function is needed for the system application– Determine under what conditions it OK to eliminate a cost factor– Additional use cases?

• Should include Impact Analysis / Change Evaluation

35


• Thoughts on moving forward– Form small, diverse working group– Periodic meetings (USC ARR, COCOMO Forum,

INCOSE IW, …)– Validation through pilots

• Use on completed programs – Access the more robust data points from COSYSMO data – Comparison of estimates

• Use on non-LM programs – Incorporate feedback from validation and refine– Can this support the SERC project for COSATMO?

36

BACK-UP CHARTS

37

COSYSMO SetupOptimistic

Expected

Pessimistic

Assume a mature supplier, experienced in the domain, minimal scope creep, and cooperative stakeholders

Assume an average supplier, with some experience in the domain, average scope creep, and generally cooperative stakeholders

Assume a new supplier who will have some challenges, a fair amount of baseline volatility, and stakeholders who need to be corralled

38

Cost Analysis Approach & Results



Optimistic Expected PessimisticRequirements

Interfaces

Algorithms

Scenarios

Use a “Plug” Number for Adders Anticipated Distribution of Labor Rates Anticipated Distribution of Material Costs Anticipated Travel Costs Anticipated Supplier Fees

Bias Function

Risky Range Target Cost Target

Reserve

39

Design-to-Cost Analysis

• Goal– The goal is, given a target cost, design a solution to

meet the target cost

• Approach– Given a requirements baseline, identify and prioritize

capabilities– Use the extended COSYSMO approach to estimate

the cost for each capability– Evaluate the “cost for capability” against the

capability priority

40

Capability Decomposition



Problem: TELECOM Operations Support System Major Upgrade, Budget Limited to $40M

1 – Service Provisioning

2 – Service Order Orchestration

3 – Pre-Provisioning Support

4 – Service Order Validation

5 – Service Activation

6 – Network Management

7 – Dashboards for Awareness & Reporting

8 – Service Catalog Management

9 – Service-to-Subscriber Mapping

10 – Auto-Discovery

11 – Service Reconciliation

12 – Billing - Service Order Creation

13 – Billing - Trouble Ticketing

14– Service Coverage Mapping

15– Resource Management

Major Capabilities

Evaluate Each Capability with Respect to Cost and Enterprise Utility to Determine Best-Value

Baseline

41

Mission Utility Analysis of Capabilities

Operational Burden

Very High

High

Mod High

Med

Mod Low

Low

Low Mod Low Med Mod High High Very High

Operational Tempo

1 – Service Provisioning

2 – Service Order Orchestration

3 – Pre-Provisioning Support

4 – Service Order Validation

5 – Service Activation

6 – Network Management

7 – Dashboards for Awareness & Reporting

8 – Service Catalog Management

9 – Service-to-Subscriber Mapping

10 – Auto-Discovery

11 – Service Reconciliation

12 – Billing - Service Order Creation

13 – Billing - Trouble Ticketing

14– Service Coverage Mapping

15– Resource Management

15

14

13

12

11

10

1

2

34 5

67

8

9

Operational Tempo is a combination of the frequency with which the capability is used operationally and its criticality to the overall mission.

Operational Burden is a combination of the skill level required for the capability and the time it takes to perform.

42

Cost Analysis Approach



Requirements

Interfaces

Algorithms

Scenarios


Three COSYSMO Scenarios for Each Capability

Use a Common Number for Adders Anticipated Distribution of Labor Rates Anticipated Distribution of Material Costs Anticipated Travel Costs Anticipated Supplier Fees

Bias Function

Take the 80% Confidence Cost as the Capability Cost

A Cost Curve is Produced for Each Capability

43

Simplified Cost Analysis Approach

Expected



Requirements

Interfaces

Algorithms

Scenarios

A Single COSYSMO Scenario for Each Capability


Bias Function



A Simpler Approach is Slightly Less Robust…Bust Still Just as Valid

44

Analysis Results

Capabilities Sorted By Cost

Total Cost = $81.3M

Colors Map Mission Utility Priorities RED = High YELLOW = Medium Green = Low

Capabilities Sorted By Mission Utility

$34.4M$39.9M

$64.1M

$81.3M

(Cost for Priority 1 Capabilities Only)(Capabilities at DTC Target)

(Cost for Priority 1&2 Capabilities)

(Cost for All Capabilities)

45

Analysis of Alternatives

• Goal– The goal is, given a requirements baseline, determine

the most affordable solution that meets requirements

• Approach– Given a requirements baseline, identify possible

solution alternatives– Use the extended COSYSMO approach to estimate the

cost for each capability– Evaluate the cost to find the most affordable

alternative that meets all requirements

46

Case Study Overview

• 20-Year Old C2 System– The system was unprecedented at the time– Twenty years later, a replacement system is needed due

to obsolescence and needed changes• Alternative Solutions

– Full Replacement System• Develop and deploy a new replacement system

– COTS/GOTS/NDI-Based Replacement System• Use a combination existing non-obsolete components and

COTS components• Modify components as necessary to meet requirements

47

The Two Key Alternatives

Newly Developed System

Refurbished System Using Modified NDI

Ground-Up Development of System – Requirements Refinement, Architecture, Detailed Design – Soup-to-Nuts

But…It is No Longer an Unprecedented System – So Requirements/Architecture Understanding, etc. Are High

Use of NDI is Maximized – Additional Components Developed as Necessary to Meet Requirements

Reuse Considerations Drive This Alternative

48

COSYSMO Size and Cost Drivers




Technology Risk






Process Capability



Cost Drivers

System modification and reuse have an effect on some cost drivers

Size Drivers for the Two Alternatives Are Largely the Same Cost Drivers for the Two Alternatives Are Very Similar – With

a Few Notable Exceptions

49

COSYSMO Reuse Factors


New Elements

Reuse Factors

New Elements – These are elements that need to be engineered and developed. Just reusing systems engineering artifacts is not sufficient.

Modified Elements – These are elements that offer some form of reuse. Enhanced COTS or reusable components that need modification fall into this category.

Adopted Elements – These are elements that offer significant reuse with minimal modification and do not require full retesting. COTS typically falls into this category.

Managed Elements – These are elements that are already in the system and require minimal regression testing. A previously deployed element falls into this category.

Most Elements Are New for the Development Alternative

Elements That Are “Wrapped” Can Be Treated as Adopted for the NDI Alternative

Most Elements Are Modified for the NDI Alternative

Elements That Stay in Place Without Modification (or Wrapping) Can Be Treated as Managed

Deleted Elements – For the NDI Alternative, Some Elements May Need to Be Deleted/Retired

Elements That Need to Be Retired Should Be Treated as Deleted Elements

50

Cost Analysis Approach



Requirements

Interfaces

Algorithms

Scenarios


Three COSYSMO Scenarios for Each Alternative


Bias Function



51

End of Workshop Wrap-Up

• Validity of the Overall Approach

• Improvement of the Approach

• Thoughts on Moving Forward

52

Documents

COSYSMO Extension as a Proxy Systems Cost Estimation