ISA 201Intermediate

Information Systems Acquisition

Lesson 10Software Cost Estimating

&Budgeting

Learning Objectives

3Cost Estimating

OVERALL: Given a DoD IT/SW system scenario, develop the software program office lifecycle cost and schedule effort projections for budget purposes.• Describe the DoD processes for generating program

office cost and schedule estimates.• Describe the basic cost and schedule estimating

methodologies used to budget for a software-reliant system.

• Given a DoD IT/SW scenario, develop cost and schedule budget estimates.

• Determine the impact of COTS and other cost drivers on achieving an accurate estimation of Program costs.

Today we will learn to:

Lesson Overview

4Cost Estimating

• Program Office Budgeting• Software Cost Estimating Intro• Costing Exercise/Demo• COTS/ERP Cost Estimation• JTAMS CE Exercise

Lesson Plan

Lesson Overview

5Cost Estimating

• Program Office Budgeting- Life cycle costs- Building the program budget- Required cost estimates

• Software Cost Estimating Intro• Costing Exercise/Demo• COTS/ERP Cost Estimation• JTAMS CE Exercise

Lesson Plan Status

6Cost Estimating

• Life cycle cost (LCC) “The total cost to the government of a program over its full life, including costs for research and development, testing, production, facilities, operations, maintenance, personnel, environmental compliance, and disposal.”

• Costs are viewed in differing ways depending on the stakeholder involved (Program Office, Congress, Contractor):

Life cycle cost categoriesFunding appropriation Work breakdown structure (WBS)

The Life Cycle Cost

7Cost Estimating

Ref: DoD 5000.04-M-1, Cost Analysis Guidance and Procedures

Congressional appropriation categories

RDT&E—Research, Development, Test and

EvaluationPROC—Procurement

O&M—Operations and Maintenance

MILCON—Military ConstructionMILPERS—Military Personnel

DoD life-cycle cost categories

R&D—Research & DevelopmentInvestment

O&S Operating and SupportDisposal

Life-cycle cost vs. Funding Appropriation

8Cost EstimatingRef: MIL-HDBK-881D, 9 April, 2018

• The WBS serves as a basis for:- Program and technical planning, cost estimating, resource allocations,

performance measurements, and status reporting. • It displays the total system as a product-oriented hierarchy composed

of hardware, software, services, data, and facilities; and relate the elements of work to one another, as well as to the end product.

• Where used: Cost breakouts by WBS elements are useful to the program office and contractors in managing the program.

By Work Breakdown Structure (WBS)

9Cost Estimating

All ACAT I programs require an ICE

• ICE (Independent Cost Estimate)- Done by: Director, Cost Assessment and Program

Evaluation (CAPE)• CCE (Component Cost Estimate)

- Component Cost Agency• PLCCE (Program Life Cycle Cost Estimate)

- Program Office• EA (Economic Analysis)

- Requirement for MAIS programs

Required Cost Estimates

10Cost EstimatingNote: FY 2014 constant dollars per DoDI 5000.02

ACAT Reason Decision AuthorityACAT 1 (MDAP)

>$480M RDT&E, >$2.79B PROCMDA designation as special interest

ACAT ID: DAE or asDelegated

ACAT IC: Head of the DoDComponent or, if delegated, the CAE

ACAT IA (MAIS)

>$40M Single year cost>$165M from Material Solution Analysis through full deployment>$520M from Material Solution Analysis through full useful lifeMDA designation as special interest

ACAT IAM: DAE or asDelegated

ACAT IAC: Head of the DoD Component or, if delegated, the CAE (not further delegable)

ACAT II >$185M RDT&E or >$835M PROC CAE or as designee

ACAT III Does not meet criteria for ACAT II or above AIS that is not a MAIS

Designated by CAE

ACAT Level Drives Cost Estimate Requirements

11Cost Estimating

Ref: Defense Acquisition Guidebook Chapter 3.6 - Economic Analysis (MAIS Only)

Economic Analysis is IT system focused

• Independent Cost Estimate- An independent estimate of program life-cycle costs

• Economic Analysis- Determining the best acquisition alternative by assessing

the net costs and benefits of the proposed program relative to the status quo.

Cost Estimate vs. Economic Analysis

12Cost Estimating

Reference: DoDI 7041.3, Economic Analysis for Decision Making

Alternatives1. Status quo or

current functional baseline

2. New acquisition3. Leasing 4. Modification of

existing assets

ECONOMIC ANALYSIS• Objective• Assumptions• Alternatives• Costs and Benefits of each

alternative• Comparison of the costs

and benefits of each alternative

• Sensitivity and Uncertainty Analyses

• Results and Recommendations

Elements of an Economic Analysis

13Cost Estimating

SummaryCost Analysis Organizations

Lesson Overview

14Cost Estimating

• Program Office Budgeting

• Software Cost Estimating Intro- Overview- Estimating methods- Parametric Modeling Tool

• Costing Exercise/Demo• COTS/ERP Cost Estimation• JTAMS CE Exercise

Lesson Plan Status

Software is Here to Stay!

0.00

5.00

10.00

15.00

20.00

25.00

30.00

F-22 Raptor(2005)

Boeing 787(2009)

F-35 Fighter(2015?)

Line

s of

Cod

e in

Milli

ons

Aircraft

0.00

10.00

20.00

30.00

40.00

50.00

60.00

Windows 2000(2000)

Windows XP(2001)

Windows Vista(2006)

Line

s of

Cod

e in

Milli

ons

Operating Systems

0.00

100.00

200.00

300.00

400.00

500.00

600.00

Facebook (withoutbackend code)

Healthcare.gov (2013)Line

s of

Cod

e in

Milli

ons

Web Pages

0.00

20.00

40.00

60.00

80.00

100.00

120.00

Chevy Volt (2010) Software in TypicalNew Car (2013)

Line

s of

Cod

e in

Milli

ons

Automobiles

15Cost Estimating

16Cost Estimating

Software Challenges and DoD Acquisition Results

Nunn-McCurdy

17Cost EstimatingSource: Standish Group Chaos Report 2015

History of Software Project Success

18Cost Estimating

Poor at Estimating SW Costs

• Cost overruns decreased from 54% in 2008 to 46% in 2010.

• Time overruns also have gone down, from 79% in 2008 to 71% in 2010. The high point in time overruns was 2004 (84%).

Source: Standish Group Chaos Database (2004)

So, what happened after 2004? ……

19Cost Estimating

• Intangible nature• Complex• Ill defined requirements• Code Growth during development• Optimism of Reuse• Optimism for COTS solutions• Software Tool unavailability/inexperience• Poor interface design

Causes of Poor Cost Estimating

20Cost Estimating

Estimating Approaches for Software

21Cost Estimating

• Compares a new or proposed system with one other system in which the form, fit, and function are alike

• Done early in the life cycle when costing data points are lacking

• Used as a “sanity check” for other type estimates• Requires identifying “adjustment factors” and “cost

to cost relationships”• Pros: Fast and Inexpensive, easy to change• Cons: High degree of cost risk

Analogy

22Cost Estimating

Adjustment Factors:Inflation: $250,000 x 1.2184 = $304,601Complexity: $305,000 x 1.05 = $319,831

Estimating Cost of a 2017 Ferrari 612 GTO Skaglietti

Analogy Example

Available Data:• $250,000 in 2009• Inflation 2.5% over 8 yrs (1+.025)8 =

1.2184• Complexity/Innovation 5% (expert opinion)

23Cost Estimating

Usually done using software-based tools

1. Model derived from historical data of many developmental efforts.

2. Determine historical CERs (Cost estimating relationships) and SERs (Schedule estimating relationships).

3. Apply the CERs and SERs to a planned project.

Parametric Model

24Cost Estimating

The parametric model gives us a great understanding of the drivers of software cost.

Let’s dive into this model more deeply…

• Pros- Provides excellent visibility of software cost drivers - Computer-based tools allow for excellent conducting

“what-if” analyses• Cons

- Constrained by the amount and quality of the data- Sensitivity of the tool requires an expert assessment of

expected cost drivers- Must estimate project size prior to using the model (expert

judgment?)

Parametric Pros and Cons

25Cost Estimating

Parametric Model Cost Drivers

26Cost Estimating

ESLOC = New Code + Modified Code + Reused Code

Cost Factors in Software Development

27Cost Estimating

Each software estimating model calculates ESLOC a little differently

Measures the size of a SW program by counting the number of lines of code that are designed, documented, and tested, but … which ones count?• Executable (Yes, but don’t count End or debug Statements)• Data Declarations (yes)• Compiler Directives (yes)• Format Statements (yes)• Comments (nope)

Equivalent or Effective Lines of Code = Resource estimates based on physical LOC for modified code and reusable code cannot account for the additional resource demands to reverse-engineer, test and integrate, so …we have to account for• New code• Modified code• Reused code and • Off-the-shelf components

Estimating Size

28Cost Estimating

• Required Software Reliability

• Data Base Size • Product Complexity• Developed for Reuse • Documentation Match to

Life-Cycle Needs

Cost Factors in Software Development

29Cost Estimating

• Execution Time Constraint • Main Storage Constraint • Platform Volatility

Cost Factors in Software Development

30Cost Estimating

• Analyst Capability • Programmer Capability • Personnel Continuity • Applications Experience • Platform Experience • Language and Tool

Experience

Cost Factors in Software Development

31Cost Estimating

• Use of Software Tools • Multisite Development• Required Development

Schedule

Cost Factors in Software Development

32Cost Estimating

• Precedentedness• Development

Flexibility• Architecture / Risk

Resolution• Team Cohesion • Process Maturity

Cost Factors in Software Development

33Cost Estimating

How do these effect the cost of software development?

What management decisions will reduce costs?

Cost Factors in Software Development

34Cost Estimating

The result of analyzing 100’s of software development efforts

(In Person Months)

Effort MultipliersNominal Values = 1Scale Factors

COCOMO IIA = 2.94B = 1.1 (Nominal)C = 3.67D = 0.28

COCOMO II—How it Works

Lesson Overview

35Cost Estimating

• Program Office Budgeting• Software Cost Estimating Intro

• Costing Exercise/Demo• COTS/ERP Cost Estimation• JTAMS CE Exercise

Lesson Plan Status

36Cost Estimating

• What if we doubled the size of the code?

• Example: How many person-months will my project take if the software size is 100 KSLOC and all cost drivers (scale factors and effort multipliers are nominal. Using COCOMO historical parameters we know that A =2.94 and B=1.1 and nominal values for EAF are all normalized to 1.

How does size effect the effort?

37Cost Estimating

• Scenario- As a result of uncertainty with funding from the

government for software development projects, your contractor is reporting personnel turnover moved from 12% per year to 48% per year. You originally estimated turnover to be at 12%. Given that all other effort multipliers (parameters) and scale drivers are nominal how may this effect the cost of your SW project?

Parametric Model Exercise 1 - Effort Multipliers(Ratio Technique)

38Cost Estimating

Refer to COCOMO Cost Estimation

Workbook, p34–35, 38

29% more

• What Effort Multiplier parameter does this apply to? • What value applies for 12%? 48%? • How much more will it cost?

Parametric Model Exercise 1 - Effort Multipliers Answer (Ratio Technique)

39Cost Estimating

The COCOMO II Cost Estimating Tool – so we don’t worry about the math!!

1. Open file “DAUCostApplication”2. Select “New” for Project Type3. Highlight Project 1 folder4. Click on new software item5. Highlight Software Item 1

COCOMO II Software Tool

40Cost Estimating

Cost Estimating Tool

41Cost Estimating

Scale Factors cause EXPONENTIAL increases to development effort

From a common sense viewpoint, what can we do to manage these?

• Precedentedness• Development Flexibility• Architecture / Risk Resolution• Team Cohesion • Process Maturity

Parametric Model-Scale Factors

42Cost Estimating

Your program office has been planning a new C4I capability for the Navy. The current estimate for the SW is 200KSLOC and 26 months at a cost close to $8M. This assumes nominal values for all cost drivers and scale factors and average SW Labor Rates of $8K per person-month. You, however, have a plan to significantly improve team cohesion from moderately cooperative to highly cooperative. Using the COCOMO Cost Estimation Workbook answer this question:

- What does it mean to move the team from nominally to highly cooperative?

• Using the online COCOMO modeling tool, answer these questions:- How will this change the cost?- Which scale drivers are manageable by the PMO to lower costs?

Run a sensitivity analysis (Instructor assignment)- What scale driver has the biggest effect on the cost of software?

Parametric Model Demo 2—Scale Factor

43Cost Estimating

1. Which three multiplicative cost drivers account for the largest variation in costs? Hint: Use Cost Factor Cheat Sheet.

2. Using a 350 KSLOC SW with no reuse or modification of code and $8K/person-month cost, what are the costs and schedule with factors rated (Hint: Use CE Tool to calculate):- all nominal - all 3 identified factors optimal (less effort/lower cost) - all 3 identified factors pessimistic (greater effort/higher

cost)

Costing Exercise 2: Biggest Cost Drivers

44Cost Estimating

Source: “Modern Tools to Support DoD Software Intensive System of Systems Cost Estimation”, By Jo Ann Lane and Barry Boehm, University of Southern California, 2007

• Software Engineering- COCOMO II — Boehm eta, 2005- CostXper—CostXpert Group, 2003- Price-S (PRICE, 2006)- SEER-SEM (Galorath, 2001)- SLIM (QSM, 2006)

• COTS Integration- COCOTS (Abts, 2004)- SEER-SEM (Galorath, 2001)

Other Cost Estimation Apps

Lesson Overview

45Cost Estimating

• Program Office Budgeting• Software Cost Estimating Intro• Costing Exercise/Demo

• COTS/ERP Cost Estimation-Commercial off the Shelf (COTS)

Software-Enterprise Resource Planning (ERP)

Software• JTAMS CE Exercise

Lesson Plan Status

46Cost Estimating

• Can provide “out-of-the-box functionality but …

• Consider- May be difficult to customize and integrate (glue code)- Might need additional testing- Upgrades might require re-integration (rewrite glue code)- Must still perform requirements definition, design and test- May be incompatible with other software packages- Use proprietary languages (hard to support in-house)- Need training and access to documentation- New releases (may cause unpredictable problems—time/$

to find and fix)

COTS

47Cost Estimating

Source: Pat Phelan, Estimating the Time and Cost of ERP Implementation Projects Is a 10-Step Process (Stamford, Conn: Gartner Inc., Apr. 27, 2006, p. 3.

• Special kind of COTS—business support system• Consider

- Contains thousands of packages—built from database tables—that need to be configured to match end business processes = requires a deep understanding of existing business operating processes

- Integrating and testing software links- Interfacing with legacy systems- Customizing—can be costly and should be carefully managed - Converting and analyzing data—did you account for converting data

into new format? Data warehouses—renaming? Programming customized?

- Following up installation—is your team changing?- Training

• Through 2006, actual time spent on 40% of ERP implementations will exceed original estimates by at least 50% (0.7 probability)*

ERP

48Cost Estimating

• Estimates are necessary to - Establish systems engineering and software development

budgets - Facilitate trade-offs/negotiations

• Estimation is not easy • Different approaches have different inherent problems • Best way to estimate is to use a combination of

techniques • Local calibration of parametric models provide best

accuracy

Cost Estimating Recap

Lesson Overview

49Cost Estimating

• Software Cost Estimating Intro• Costing Exercise• Program Office Budgeting

• JTAMS CE Exercise

Lesson Plan Status

50Cost Estimating

1. Task 1. (20 min) Using the chart (Individual Estimate of Cost Factors), individually analyze the scenario and estimate the value of each factor (Very Low-Extremely High). (Use the COCOMO II Cost Estimating Workbook/Factor cheat sheet used in the lesson.)

2. Task 2. (10 min) Compare your results with that of your team. Discuss and decide on an agreed set of parameters. Record your team results and justification. (whiteboard)

3. Task 3. (10 min) Using the COCOMO II modeling tool calculate effort, schedule and cost for your team estimate – place these values on your Whiteboard.

4. Task 4. (15 min) Compare your team results with the other teams. Where there are differences of opinion, share your insights as you discuss/defend your results. The object is to have a clearer understanding of the ratings so we all eventually agree. (Instructor facilitated discussion)

If time permits:1. Answer the program manager’s challenge. Record results in worksheet.2. Identify total cost savings using CE Tool if all suggested improvements were

implemented? Record results in worksheet.

JTAMS CE Exercise

Summary

51Cost Estimating

Today we learned to:

OVERALL: Given a DoD IT/SW system scenario, develop the software program office lifecycle cost and schedule effort projections for budget purposes.• Describe the DoD processes for generating program

office cost and schedule estimates.• Describe the basic cost and schedule estimating

methodologies used to budget for a software-reliant system.

• Given a DoD IT/SW scenario, develop cost and schedule budget estimates.

• Determine the impact of COTS and other cost drivers on achieving an accurate estimation of Program costs.

BACK-UP

53Cost Estimating

Ref: DoDD 5000.04-M-1, Cost Analysis Guidance and Procedures

• Program Acquisition Cost- Development Costs

- RDT&E- Procurement Costs

- Weapons System Costs- Flyaway Cost

- Procurement (PME, SE/PM, STE)- Procurement (Support Items)

- Procurement (Initial Spares & Repair Parts)- MILCON (Facilities)

• O&S (MILPERS, O&M, Disposal)

More Costing Terminology

54Cost Estimating

Appropriation Categories

55Cost Estimating

Ref: DoD 5000.4 M, "DoD Cost Analysis Guidance and Procedures," Sec1

More than just documenting the life cycle costs

• Cost Analysis Requirements Description (CARD) Requirement:- Initial submission required at MS A and MS B- Updates required at Development RFP, MS C and Full Rate- Production/Fielding Decision- Due any time an Independent Cost Estimate or Economic

Analysis is required• Serves as the basis of all cost estimates• Prepared by the PMO and approved by the PEO

Building a program’s budget

56Cost Estimating

Ref: DoDD 5000.4 M, "DoD Cost Analysis Guidance and Procedures," Sec1

CARD• Program description• Program operational concept• Program data management

requirements• Program quantity requirements• Program manpower

requirements• Program fielding strategy• Program milestone schedule• Program acquisition plan or

strategy

SOURCE DOCUMENTSTechnical Readiness Assessment

(TRA)Capability Docs (ICD, CDD, CPD)

Acquisition Strategy (AS)Life Cycle Sustainment Plan

(LCSP)Technical Evaluation Strategy/

Test and Evaluation Master Plan (TES/TEMP)

Manpower EstimateSystems Engineering Plan (SEP)

CARD Content

57Cost Estimating

80% of F-22 functionality and other modern day A/C depend on SW

• The role of software as the most critical part of weapons systems is growing as is the % of DoD Budget

• $28B annually to support operations, maintenance and modernization of various DoD IT systems: - $9B for C3I systems - $19B for business systems..

Why Care About SW?

58Cost Estimating

Ref: Teaching Note “Budget Exhibits”

Used to report appropriations for FYDP by program element

“Code Word”“P-Forms” or “P-Docs”“R-Forms” or “R-Docs”“O-Forms”“M-Forms”“C-Forms”

Appropriation CategoryProcurement

RDT&EO&M

MILPERS MILCON

Documenting a Program Budget

59Cost Estimating

Source: Standish Group Chaos Database (2004)

Poor at Estimating SW Costs

60Cost EstimatingFigure Source: Boehm, Barry. Software Estimation with COCOMO II. Prentice Hall: Upper Saddle River, NJ (2000)

Some studies have shown that historically cost

estimates are best accurate to within 25% of actuals

only about 70% of the time!

We need to improve our abilityto predict costs