2013 COCOMO Forum Stoddard, 24 October 2013 © 2013 Carnegie Mellon University Harvesting Reference Points for Cost Estimation: A Step in the SEI’s Cost

2013 COCOMO ForumStoddard, 24 October 2013© 2013 Carnegie Mellon University

Harvesting Reference Points for Cost Estimation: A Step in the SEI’s Cost Estimation Method

Software Engineering InstituteCarnegie Mellon UniversityPittsburgh, PA 15213

Robert W. StoddardDennis GoldensonRhonda Brown

24 October 2013


Introduction

QUELCE (Quantifying Uncertainty for Early Lifecycle Cost Estimation) is a multi-year research project led by the Software Engineering Measurement and Analysis (SEMA) team within the SEI Software Solutions Division.

Research team membership comprises SEI technical staff with cost estimation background in collaboration with several external faculty (Dr. Ricardo Valerdi, Univ of Arizona, & Dr. Eduardo Miranda, CMU).

This research is motivated by (1) the WSARA Act requiring cost estimates pre-Milestone A and (2) DoD’s need for more accurate cost estimation methods that provide continuous monitoring of changing assumptions and constraints.


• Mission / CONOPS• Capability-based analysis

...

• KPP selection• Systems design• Sustainment issues

...

• Production quantity• Acquisition mgt• Scope definition/responsibility• Contract award

Technology Development Strategy

Operational CapabilityTrade-offs

System CharacteristicsTrade-offs

Proposed Material Solution & Analysis of Alternatives

Information from Analogous Programs/Systems

Program Execution Change Drivers

Probabilistic Modeling (BBN) &

Monte Carlo Simulation

Exp

ert

Jud

gm

ents

Information Flow for Early Lifecycle Estimation

Plans, Specifications, Assessments

• Analogy• Parametri

c Models

Cost EstimatesProgram Execution

Scenarios with Conditional Probabilities of Drivers/States

Driver States & Probabilities

• Engineering

• CERs


Repository: Analyze Existing Data to Model Program Execution Uncertainties – 1

Materiel Solution Analysis Phase – Pre-Milestone Estimate A

Program Change Repository

Prog State Driver

DDG51 cond 1 CONOPS

cond 2 System

cond 3 CapDef

JTRS cond 1 InterOp

cond 2 Produc

F22 cond 1 Contract

cond 2 Function

cond 3 CONOPS

For C2 systems, how often does Strategic Vision

change?

Records show that Strategic Vision changed in 45% of the

MDAPS

Change Driver Nominal State Alternative States

Scope Definition

Stable Users added Additional (foreign) customer

Additional deliverable (e.g. training & manuals)

Production downsized

Scope Reduction (funding reduction)

Mission / CONOPS

defined New condition New mission New echelon Program becomes Joint

Capability Definition

Stable Addition Subtraction Variance Trade-offs [performance vs affordaility, etc.]

Funding Schedule

Established Funding delays tie up resources {e.g. operational test}

FFRDC ceiling issue

Funding change for end of year

Funding spread out

Obligated vs. allocated funds shifted

Advocacy Change

Stable Joint service program loses particpant

Senator did not get re-elected

Change in senior pentagon staff

Advocate requires change in mission scope

Service owner different than CONOPS users

Closing Technical Gaps (CBA)

Selected Trade studies are sufficient

Technology does not achieve satisfactory performance

Technology is too expensive

Selected solution cannot achieve desired outcome

Technology not performing as expected

New technology not testing well

● ~~~~ ~~~~ ~~~~

● ~~~~ ~~~~ ~~~~ ~~~~

● ~~~~ ~~~~ ~~~~ ~~~~ ~~~~ ~~~~

Driver State Matrix

The Materiel Solution of a global network command and control system anticipates a possible change in Strategic Vision that will include allied participation.

Sharing information with allies creates new encryption requirements (a change in Mission/CONOPs).

These changes lead to changes in Capability Definition.

Repository identifies probability of change in MDAP cost drivers.

1. Identify Change Drivers &

States

3. Assign Conditional

Probabilities to BBN Model

4. Calculate Cost Factor

Distributions for Program Execution

Scenarios

5. Monte Carlo Simulation to Compute Cost

Distribution

2. Reduce Cause and Effect Relationships

via Dependency Structure Matrix

Techniques




If Strategic Vision changes, what else changes?

70% of the time the Mission/CONOPS changes


Scope Definition





Mission / CONOPS




Funding Schedule


FFRDC ceiling issue


Funding spread out


Advocacy Change













● ~~~~ ~~~~ ~~~~

● ~~~~ ~~~~ ~~~~ ~~~~

● ~~~~ ~~~~ ~~~~ ~~~~ ~~~~ ~~~~

Driver State MatrixChange Drivers - Cause & Effects Matrix

Mis

sio

n /

CO

NO

PS

Ch

an

ge

in

Str

ate

gic

Vis

ion

Ca

pa

bili

ty D

efin

itio

n

Ad

vo

ca

cy C

ha

ng

e

Clo

sin

g T

ech

nic

al G

ap

s (

CB

A)

Bu

ildin

g T

ech

nic

al C

ap

ab

ility

& C

ap

acity (

CB

A)

Inte

rop

era

bili

ty

Mission / CONOPS 3 3 0 6 0Change in Strategic Vision 3 3 3 2 2 2 2 2 3 2 3 2 29 0Capability Definition 3 0 2 1 1 0 0 2 2 2 0 1 0 2 0 0 16 0Advocacy Change 2 1 1 1 1 6 0Closing Technical Gaps (CBA) 2 1 3 1 2 2 1 2 2 1 1 2 1 0 2 2 1 1 2 2 1 2 34 0Building Technical Capability & Capacity (CBA) 1 1 2 1 2 2 1 2 3 2 2 1 2 2 1 1 1 27 0Interoperability 1 2 1 1 1 1 1 1 1 1 2 1 1 2 1 1 3 1 2 2 2 29 1Systems Design 1 2 2 2 2 1 1 1 1 1 2 2 3 21 3Interdependency 1 2 2 1 1 1 1 1 1 1 1 1 2 1 2 2 1 1 1 1 1 2 2 3 33 5Functional Measures 2 2 2 1 1 1 1 1 1 1 2 1 16 0Scope Definition 1 1 3 5 0Functional Solution Criteria (measure) 1 2 2 1 1 2 1 10 1Funding Schedule 1 1 2 1 5 0Acquisition Management 1 1 2 3 1 1 2 2 1 1 1 2 1 19 2Program Mgt - Contractor Relations 1 1 2 1 1 1 1 2 2 12 2Project Social / Dev Env 1 1 1 2 2 1 1 2 1 1 1 14 2Prog Mgt Structure 1 2 1 2 6 1Manning at program office 2 1 2 5 2Scope Responsibility 1 1 1 1 1 1 6 5Standards/Certifications 1 1 1 1 1 1 3 1 10 2Supply Chain Vulnerabilities 1 1 1 1 2 1 7 4Information sharing 1 1 1 1 1 1 1 7 3PO Process Performance 2 2 4 0Sustainment Issues 0 0Contract Award 0 0Production Quantity 2 2 0Data Ownership 2 2 0Industry Company Assessment 0 0Cost Estimate 0 0Test & Evaluation 0 0Contractor Performance 2 2 0Size 0 0Project Challenge 0 0Product Challenge 0 0Totals 0 0 6 4 1 9 5 12 8 7 7 13 4 10 15 18 7 7 8 8 14 17 17 15 12 9 10 13 11 20 19 5 5 17 0Below diagonal 0 0 0 1 1 4 4 4 1 2 0 3 1 3 2 2 3 1 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0

Effects

Causes

DSM Cause-Effect Matrix


Prog State Driver

DDG51 cond 1 CONOPS

cond 2 System De

cond 3 CapDef

JTRS cond 1 InterOpera

cond 2 Production

F22 cond 1 Contract

cond 2 Functional

cond 3 CONOPS





States





Scenarios


Distribution

2. Reduce Cause and Effect Relationships via Dependency Structure Matrix Techniques





Scope Definition





Mission / CONOPS




Funding Schedule


FFRDC ceiling issue


Funding spread out


Advocacy Change













● ~~~~ ~~~~ ~~~~

● ~~~~ ~~~~ ~~~~ ~~~~

● ~~~~ ~~~~ ~~~~ ~~~~ ~~~~ ~~~~

Driver State Matrix Change Drivers - Cause & Effects Matrix

Mis

sio

n /

CO

NO

PS

Ch

an

ge

in

Str

ate

gic

Vis

ion

Ca

pa

bili

ty D

efin

itio

n

Ad

vo

ca

cy C

ha

ng

e

Clo

sin

g T

ech

nic

al G

ap

s (

CB

A)

Bu

ildin

g T

ech

nic

al C

ap

ab

ility

& C

ap

acity (

CB

A)

Inte

rop

era

bili

ty

Mission / CONOPS 3 3 0 6 0Change in Strategic Vision 3 3 3 2 2 2 2 2 3 2 3 2 29 0Capability Definition 3 0 2 1 1 0 0 2 2 2 0 1 0 2 0 0 16 0Advocacy Change 2 1 1 1 1 6 0Closing Technical Gaps (CBA) 2 1 3 1 2 2 1 2 2 1 1 2 1 0 2 2 1 1 2 2 1 2 34 0Building Technical Capability & Capacity (CBA) 1 1 2 1 2 2 1 2 3 2 2 1 2 2 1 1 1 27 0Interoperability 1 2 1 1 1 1 1 1 1 1 2 1 1 2 1 1 3 1 2 2 2 29 1Systems Design 1 2 2 2 2 1 1 1 1 1 2 2 3 21 3Interdependency 1 2 2 1 1 1 1 1 1 1 1 1 2 1 2 2 1 1 1 1 1 2 2 3 33 5Functional Measures 2 2 2 1 1 1 1 1 1 1 2 1 16 0Scope Definition 1 1 3 5 0Functional Solution Criteria (measure) 1 2 2 1 1 2 1 10 1Funding Schedule 1 1 2 1 5 0Acquisition Management 1 1 2 3 1 1 2 2 1 1 1 2 1 19 2Program Mgt - Contractor Relations 1 1 2 1 1 1 1 2 2 12 2Project Social / Dev Env 1 1 1 2 2 1 1 2 1 1 1 14 2Prog Mgt Structure 1 2 1 2 6 1Manning at program office 2 1 2 5 2Scope Responsibility 1 1 1 1 1 1 6 5Standards/Certifications 1 1 1 1 1 1 3 1 10 2Supply Chain Vulnerabilities 1 1 1 1 2 1 7 4Information sharing 1 1 1 1 1 1 1 7 3PO Process Performance 2 2 4 0Sustainment Issues 0 0Contract Award 0 0Production Quantity 2 2 0Data Ownership 2 2 0Industry Company Assessment 0 0Cost Estimate 0 0Test & Evaluation 0 0Contractor Performance 2 2 0Size 0 0Project Challenge 0 0Product Challenge 0 0Totals 0 0 6 4 1 9 5 12 8 7 7 13 4 10 15 18 7 7 8 8 14 17 17 15 12 9 10 13 11 20 19 5 5 17 0Below diagonal 0 0 0 1 1 4 4 4 1 2 0 3 1 3 2 2 3 1 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0

Effects

Causes

DSM Cause-Effect MatrixBBN Model

When

both Strategic Vision & Mission/CONOPs

experience change, the BBN calculates that

Capability Definition will also change

95% of the time.





States





Scenarios


Distribution

2. Reduce Cause and Effect Relationships via Dependency Structure Matrix Techniques


Develop Efficient Techniques to Calibrate Expert Judgment of Program UncertaintiesSolution

Calibrated

Un-Calibrated

Estimate of SW Size


1) Size of ground combat vehicle targeting feature xyz in 2002 consisted of 25 KSLOC Ada

2) Size of Army artillery firing capability feature abc in 2007 consisted of 18 KSLOC C++

3) …

Step 1: Virtual training using reference points

Step 2: Iterate through a series of domain-specific tests

Step 3: Feedback on test performance

Outcome: Expert renders calibrated estimate of size

Calibrated = more realistic size and

wider range to reflect true expert

uncertainty

Used with permission from Douglas Hubbard Copyright HDR 2008 [email protected]


Research Challenge

QUELCE team found the “mining” of domain reference points challenging in terms of requisite knowledge, effort, and schedule.

The use of an NVivo-like platform was hypothesized to address the intermediate step of identifying potential change driver experiences documented in historical MDAP artifacts•without losing the context of a given artifact, •enabling queries by attributes across MDAPs, and•achieving review by others prior to committing to a refined domain reference

point statement


How CAQDAS May Help

Computer-Assisted/Aided Qualitative Data AnalysiS (CAQDAS) is the “use of computer software to aid qualitative research such as transcription analysis, coding and text interpretation, recursive abstraction, content analysis, discourse analysis, grounded theory methodology, etc.”1

CAQDAS software initially developed for social science researchers needed a way to assemble, annotate, and make sense out of a myriad of textual artifacts.

The QUELCE team conceived of the use of CAQDAS in the sense that “research” has always been inherent in the use of analogy for cost estimation. CAQDAS enables a more formal treatment of that activity.

1 Wikipedia, http://en.wikipedia.org/wiki/CAQDAS

http://en.wikipedia.org/wiki/CAQDAS

http://en.wikipedia.org/wiki/CAQDAS


NVivo Experiment

Phase 1: Domain experts coded an assigned set of DoD program artifacts likely to exhibit potential change drivers

• Checkpoint synchronization in the middle of this phase

Phase 2: Domain experts and the research team conducted queries against the artifacts using NVivo text analytic and search capabilities

Phase 3: Feedback gathered from all regarding NVivo usage• On the NVivo experience• On high-value documents• On the co-occurrence and/or cascading of change drivers


Experimental Procedure

Approximately 200 artifacts of an inventory of 1,500 artifacts were identified for coding.

We operated in standalone mode, with each domain expert assigned a unique NVivo project on the server.

Each NVivo project possessed a unique subset of the 200 artifacts (we balanced the workload among 7 experts).

Experts were asked to open assigned artifacts, one at a time, within NVivo and identify the apparent change drivers, if any.

We asked that experts to track their time in hours during the coding so that we could gauge the productivity rate in forecasting future coding projects.


Detailed Experimental Work Flow

STEP 1: Open NVivo project

STEP 2: Open a file to code

STEP 4: Sequentially identify change events

STEP 5: Within a change event, identify one or more

change instances

STEP 6: For each change instance, highlight the text

and tag with a Change Driver code

STEP 7: For each change instance, optionally highlight and tag

separate elements of Stimulus, Response, and Outcome

STEP 3: Peruse the entire artifact for

context


An NVivo Demonstration Platform


NVivo Coding Artifacts with Change Drivers


NVivo Summary of Change Driver Coding


NVivo Simple Query on a Change Driver


NVivo Search on “Aerospace” and Schedule Change Driver


NVivo Word Frequency


NVivo Word Search Cluster Analysis


NVivo Analysis of Reproducibility


NVivo Search on Co-occurrence of Change Drivers Within Artifacts


Early Experimental Feedback

Although individual NVivo projects may be employed and then merged together, efficiencies may be possible via the team server setup.

The NVivo performance across sensitive networks can be troublesome.

Installing NVivo with today’s security infrastructure remains quite challenging.

Experts found the coding within NVivo to be painless, e.g., as easy as highlight and drag or highlight and click on a pull-down list of change drivers.

Queries provide a means to scan a large portfolio of artifacts for key words, change driver codes, co-occurrence of change drivers, etc…


Future Work

Need to enlist additional experts to code the same artifact set and evaluate reproducibility.

Need to design and implement expert judgment experiments for improvement in judgment “calibration” using a coded NVivo repository as a reference.

Need to refine the query results from an extensive NVivo code repository with the aim of producing refined domain reference points.

Need to consider the at-scale implications and logistics using a tool such as NVivo, e.g., as a living DoD asset.


Contact Information

Robert W. StoddardPrincipal ResearcherSoftware Solutions Division, SEAPTelephone: +1 412-268-1121Email: [email protected]

U.S. MailSoftware Engineering InstituteCustomer Relations4500 Fifth AvenuePittsburgh, PA 15213-2612USA

Webwww.sei.cmu.eduwww.sei.cmu.edu/contact.cfm

Customer RelationsEmail: [email protected]: +1 412-268-5800SEI Phone: +1 412-268-5800SEI Fax: +1 412-268-6257

Documents

2013 COCOMO Forum Stoddard, 24 October 2013 © 2013 Carnegie Mellon University Harvesting Reference Points for Cost Estimation: A Step in the SEI’s Cost