Dan galorath

Estimation & Planning Processes Decide Project SuccessNASA Project Management Challenge 2012

Dan Galorath, Founder & CEO

Copyright Galorath Incorporated 2011

Introduction

• Estimating is critical for all kinds of systems

• Yet many treat is as a second rate process

• Having a repeatable estimation process is critical to both estimating AND to successful projects

• Estimation and measurement go hand in hand

© 2009 Copyright Galorath Incorporated 2

Delusions of Success: How Optimism Undermines Executives' Decisions (Source: Richard Hartley, HBR)

• Problem:

Humans seem hardwired to be optimists

• Optimism from cognitive biases & organizational pressures• Exaggerate talents & degree of control

• Attribute negative consequences to external factors

• Anchoring (relying too heavily on one piece of information) magnifies optimism• Most pronounced for new initiativesSolution: Temper with “outside view”

Supplement traditional forecasting w/ statistical Parametrics

Don’t remove optimism, but balance optimism & realism

Example of Tempering: (Source How To Measure Anything)

• German Mark V Tanks

• Allies estimated production by analyzing serial numbers

• Used the captured tanks as a random sample and predicted probability of various production levels

An Estimate Defined

• An estimate is the most knowledgeable statement you can make at a particular point in time regarding:• Effort / Cost

• Schedule

• Staffing

• Risk

• Reliability

• Estimates more precise with progress

• A WELL FORMED ESTIMATE IS A DISTRIBUTION

5

Estimation Methods 1 of 2

Model Category

Description Advantages Limitations

Guessing (Common)

Off the cuff estimatesQuick Can obtain any answer desired

No Basis or substantiationNo ProcessUsually WrongGenerally optimistic

Analogy (Common)

Compare project with past similar projects.

Estimates are based on actual experience.

Truly similar projects must exist.Less optimistic

Expert Judgment (Common)

Consult with one or more experts.

Little or no historical data is needed; good for new or unique projects.

Experts tend to be biased; knowledge level is sometimes questionable; may not be consistent.Generally optimistic

Top Down Estimation (Common)

A hierarchical decomposition of the system into progressively smaller components is used to estimate the size of a software component.

Provides an estimate linked to requirements and allows common libraries to size lower level components.

Need valid requirements. Difficult to track architecture; engineering bias may lead to underestimation.Generally optimistic (miss non-WBS items)

6

Estimation Methods 2 of 2

Model Category Description Advantages Limitations

Design To Cost (sometimes)

Uses expert judgment to determine how much functionality can be provided for given budget.

Easy to get under stakeholder number

Little or no engineering basis.Optimistic

Simple CER’s (common)

Equation with one or more unknowns that provides cost / schedule estimate

Some basis in data

Simple relationships may not tell the whole storyHistorical data may not tell the whole storyLess optimistic

Comprehensive Parametric Models (common)

Perform overall estimate using design parameters and mathematical algorithms.

Models are usually fast and easy to use, and useful early in a program; they are also objective and repeatable.Easy tradeoffs can provide better plans

Models can be inaccurate if not properly calibrated and validated; historical data may not be relevant to new programs; optimism in parameters may lead to underestimation. More or less optimistic depending on parameters

7

Why should we care: Each method has challenges and we should be familiar with each

Common Challenges In Estimating

• High effort and expenses in producing estimates Proposals - Acquisition Strategy inputs

• Informal request - POM budgets

• Significant investment in time to generate numbers

• Weak accuracy in numbers leading to cost overruns

• Customer frequent request for revisions cause even most expenses

• Numbers susceptible to many forces without robust cost generation systems

• Lack of understanding of cost risk

• Difficulty in calibrating “Engineering Judgment


Poor Estimates Effect on Projects• Inaccurate estimates can reduce project success:

• Poor implementations

• Critical processes don’t scale

• Emergency staffing

• Cost overruns caused by underestimating project needs

• Scope creep• Forever changing project goals

• Frustration

• Customer dissatisfaction

• Cost overruns and missed schedules

• Project Failures

• Important project business decisions made early with minimum knowledge & maximum uncertainty

9

Why should we care: Poor estimates and plans are a root cause of program failure

Initial Cost Estimation Problems (Software Program Managers Network)

• Many programs that have been evaluated tend to initially estimate using a very optimistic method.

• Low bid to win contract

• Naïve level of effort estimation

• Human optimism (HBR & Rich Hartley USAF Undersec Acquisition)

• Often wrong since they are not based on a thorough analysis of requirements. The formula

Cost = Size x Complexity/Productivity • Has three unknowns upfront: size, complexity, and productivity.

• Methods & estimating tools determine size given system requirements

• Organizational databases of productivity on comparable size and complexity projects can be used

• Or other bottom-up estimation techniques

“In any event, all initial cost estimates should be considered as potential high risk and should be reviewed at each program review” SPMN

Software Is A Critical Component of Military & Aerospace Systems Failures Ariane 5

• Ariane 5 video

• Note: $500 Million payload

• Failure due to reused software from (Ariane 4)

• with incompatible assumptions for exception condition that was not required

ariane_5_explosion_video.mp4

“the culture within the Ariane program…” only addressed “random hardware failures… which can quite rationally be handled by a backup system”“the view had been taken that software should be considered correct until it is shown to be at fault”!

Why should we care: Software cost & schedule should be sufficient for successful missions

Software Is A Key Risk Item In Weapons Systems

• Navy Mobile User Objective Satellite Communication System delays to the Joint Tactical Radio System, a set of software-defined radios causes advanced MUOS capabilities to drastically underused… GAO

• GAO identified 42 programs at risk for cost & schedule 1. military requirements changes

2. software development challenges

3. workforce issues

• National Institute of Standards and Technology (NIST)

• Software defects cost nearly $60 Billion Annually

• 80% of development costs involve identifying and correcting defects

Software, not Hardware or technology readiness levels were called out

Death March Projects Are Likely To Fail

Why should we care: If you have a project on a death march failure is highly probably

Balancing Resources & Schedule Is A Science

For a given Size, Complexity and Technology

Impo

ssib

le

Minimum TimeTo Complete

(Effort Increases

to Reduce Schedule)

Work ExpandsTo Fill Time

(Effort Increases

due to lack of pressure)

Inefficient

Minimum Time

Optimal Effort(Lower Effort for Longer Schedule)

Calendar Time

Effo

rt M

onth

s

Effort Increasedue to Longer Schedule

Why Total Cost Grew for Two Space ProgramsDavid Graham, NASA

5 “The Success Triangle of Cost, Schedule, and Performance: A Blueprint for Development of Large-Scale Systems in an Increasingly Complex Environment” - (Booz|Allen|Hamilton, 2003)

Development Growth Causes

25%

11%

25%

9%

11%

11%

8%

RequirementsGeneration & Translation

Budget/Funding

Cost Estimation

Underestimation of Risk

Schedule Slips (Govt &Contractor)

Price Increases

Other

Quantitative Framework

Use Earned Value To Quantify Progress Versus Effort (Oversimplified)

• Main EVM concern is what has been accomplished in a given time and budget, versus what was planned for the same time and budget• Project generally deemed healthy if what has been

accomplished is what was planned, or more

• Project deemed unhealthy if accomplishment lags expectations

• Definition: Earned value = budgeted value for the work accomplished (what you got for what it cost you)

16

Time = Now

Budget$

EV

HealthyBudget$

Time = Now

EV

Unhealthy

Deploying Before Complete Leads To Program Disasters: You Better Understand Schedule

17

0 4 8 12 16 20

Schedule ProbabilityExample Application 1

Probability

Time (calendar months)

1%10%20%30%40%50%60%70%80%90%99%

Understand Project Risks Include Them In Planning Decisions (Example SEER-SEM Outputs)

0 1800 3600 5400 7200 9000

Effort (person-hours)

1%10%20%30%40%50%60%70%80%90%99%

Effort ProbabilityExample Application 1

Probability

0 12 24 36 48 60

Defects ProbabilityExample Application 1Probability

Defects (count)

1%10%20%30%40%50%60%70%80%90%99%

18

Considering Maintenance During Planning Can Yield More Successful Long Term Results

Staff Vs Maintenance Rigor

0500

100015002000250030003500

1 7 13 19 25 31 37 43 49 55 61 67 73 79 85

Time

staf

f hou

rs p

er m

onth

develop

Rigor vhi+

Rigor nom

Rigor vlo

19

Data Doesn’t Have To Be Perfect To Be Useful: But Is Has To Be Viable

• 80 Calories per serving

• 2.5 Servings per can

• 4 Ounces, Condensed, 8 Ounces With Water

You have an estimate …Now what?

21

• Correlation does not imply causation• Just because two data points may sit side by side

doesn’t mean they are the same or will have the same outcome

• Casual analysis is a recognized error in medicine

Perhaps ???

The Error of Causal AnalysisCreating a False Association

Tumor Can Cause Headache

Headache doesn’t mean a tumor

22

Use Historical Measurement to evaluate your estimate!

It’s easy to dig deeper and deeper to justify an estimate!23

24

Data Beware Apples and Oranges:

Phase : All activities may not be included.

System Concept missing

Integration missing

Phases

System Concepts System Req & Design

System Req Analysis Preliminary Design

Detailed Design Code / Unit Testing

Software Test System Integration / OT&E

Estimation Process

25

26

Basic Cost Estimating Process (Source CEBOK)

• Work Breakdown Structure (WBS) Development

• Program/System Baseline Development

WBS

Baseline

Data Collectio

nData

Analysis

Methodology

Validation

Reports

27

US GAO process for Credible Estimates

28

10 Step System Estimation Process 2011

1. Establish Estimate Scope

2. Establish Technical Baseline, Ground

Rules, Assumptions

4. Refine Technical Baseline Into

Estimable Components

4. Collect data / estimation inputs

5. Estimate Baseline Cost, Schedule, Affordability Value

6. Validate Business Case Costs &

Benefits (go / no go)

6. Quantify Risks and Risk Analysis

8. Generate a Project Plan

9. Document Estimates and Lessons

Learned

10. Track Project Throughout

Development

Estimation Organizational Maturity V1.7Level 0 Informal or no estimating Manual effort estimating without a process

Level 1 Direct Task Estimation Spreadsheets Ad Hoc Process

Level 2Formal Sizing (e.g. function

points)

Direct Task Estimation

Simple model (Size * Productivity) or

informal SEER Use

Some measurement &

analysisInformal Process

Level 3 Formal Sizing

Robust Parametric estimation

(SEER)

Estimate vs. actual capture

Formalized Multiple Estimate Process

Rigorous measurement & analysis

Parametric planning & Control

Risk Managem

ent

Repeatable

process

Level 4 Formal sizing

Repeatable process

Robust parametric estimating

(SEER)


Parametric estimation with

tracking & control

Risk Managemen

t

Process improvement via lessons

learned

Level 5 Formal sizing

Repeatable process

Robust parametric estimating

(SEER)


Parametric estimation with

tracking & control

Risk Managemen

t

Continuous process

improvement

Why should we care? Maturity is related to estimate viability… With better estimation process, projects more likely to be successful in execution

Estimation Should Be Key Part of Process (Source Q Redman, APMP Just Say No)

30

Scop

e &

Accu

racy

Us e

1 2 3 4Phase 0-1

ROM

ROM

ROM

ROM

AcquisitionPlanning/ POMand Plus Ups

Market Assessment/

“What If’s”

OpportunityCreation/ Customer

Decision PlansProcurement

Initiation Draft RFP RFP

DDE

Modified Budgetary Estimate

Draft RFP/Gate 36-8 people, 3

weeks(Bid Stds + History

)

Formal BidGate 4

15-20 people4 weeks

(Bid Stds+ History)

EARLY ESTIMATING3-5 people, 3 - 5 days

Top Down, parametric model based price estimating

Vs.Current state: 90 people, 6wks

31

• This chapter discusses a 1972 GAO report on cost estimating• We reported that cost estimates were understated and causing unexpected cost growth

• Many of the factors causing this problem are still relevant today

• We also discuss a 12 step process for producing high quality cost estimates

GAO Publication: Characteristics of credible cost estimates and a reliable process for creating them

32

GAO Publication: Why cost estimates are required for government programs and challenges developing credible results

• Introduces why cost estimates are required for government programs

• Developing annual budgets, supporting management decisions about which program to fund, and evaluating resource requirements at key decision points

• Discusses various challenges associated with developing credible results

Ask These Questions When Identifying Estimate Scope• Challenged projects

• Would you still go forward if you knew• Schedule would be significantly longer?

• Cost would be dramatically higher?

• Probably: but perhaps more insight could identify mitigation

• Plan functionality differently

• Certainly you could notify stakeholders of real costs

• Ensure staffing is appropriate for the constraints

• Failed Projects

• Would you start a project you knew was unaffordable? Or if schedule was completely unrealistic?

• If knowing up-front could you do something about it?

• Often better to kill project before it begins than waste resources & let the organization down

33

Lesson Learned: Estimate Must Quantify Risk & Uncertainty

3434

What is likely to happen

Feel lucky?

Firm Fixed Price?

Understand the risk before you commit!

Estimation Best Practices

© 2011 Copyright Galorath Incorporated

35

36

Cost Estimate Qualities (Adapted from CEBOK)

• The characteristics of high quality cost estimates are:• Accurate (Viable Within Range)

• Comprehensive

• Replicable and Auditable

• Traceable

• Credible

• Timely

37

System Description (Parametrics Can Estimate More, Earlier) Adapted from CEBOK

“If you can’t tell me what it is, I can’t tell you what it costs.”

-Mike Jeffers

“If you can tell me the range of what it might be I can tell you the

range of cost, schedule & probability”

-Dan Galorath

38

Types of Cost Estimates (Adapted From CEBOK)

• Life Cycle Cost Estimate (LCCE)

• Independent Cost Estimate (ICE)

• Budget Estimate

• Rough Order of Magnitude (ROM)

• Estimate At Completion (EAC)

• Independent Cost Assessment (ICA)

• Analysis of Alternatives (AoA)

• Economic Analysis (EA)

1

39

Aircraft Example: Estimating Techniques Adapted from CEBOK)

• Where applicable, use primary methods

• Analogy• Model X100 series jet engines have only been used on one other

plane, but weight is 100% higher on this model; estimate 2x other model

• Simple Parametric• As shown, need to estimate 2 lb brake rotors, should be roughly $4M from regression curve

• Commercial Parametric• Key characteristics range are

• 30-50k lines of code and 600-800

• kilos engine & 7 PC boards

• Engineering Build-Up• Know Air Conditioning (AC) system costs on plane because received

quotes from HVAC vendor for all duct work and AC blower off the shelf

Parametric Estimate

0

2

4

6

8

10

12

0 2 4 6 8 10

Parameter (ie. w eight)

Co

st

Manual Estimates: Human Reasons For Error (Metrics Can Help)

• Manual Task estimates yield SIGNIFICANT error

• Desire for “credibility” motivates overestimate behavior (80% probability?)• So must spend all the time to be “reliable”

• Better approach: force 50% probability & have “buffer” for overruns

• Technical pride sometimes causes underestimates

40

41

Lesson Learned: The Risk In Risk Analysis

"It's tough to make predictions, especially about the future." -- Yogi Berra.

42

Cost Readiness Levels at Low TRLs and/or Less Than Firm Requirements

• If the project has critical items at less than TRL 4…• Like asking Edison in 1876 “How

much longer for the light bulb”

• “Hard to say”, he would have said as he had you thrown out

• Note that this is not the same as asking, in 1879, once he had found a workable carbon filament, “How much will a production version of the light bulb cost to develop and produce Tom?”

• This would have been a TRL 4 question

• Tom’s cost estimators could have begun to model this

• So if 1<TRL<3 CRL < 4

• Likewise, if requirements are not firm, CRL < 4

TRL 9

TRL 8

TRL 7

TTRRLL 66

TTRRLL 55

TRL 4

TRL 3

TRL 2

TRL 1

TRL9: Actual system “flight proven” thorough successful mission operations TRL8: Actual system completed and “flight qualified” through test and demonstration TRL7: System prototype demonstration in a space environment TRL6: System/subsystem model or prototype demonstration in a relevant environment TRL5: Component and/or breadboard validation in relevant environment TRL4: Component and/or breadboard validation in laboratory environment TRL3: Analytical and experimental critical function and/or characteristic proof-of-concept TRL2: Technology concept and/or application formulated TRL1: Basic principles observed and reported

43

Table 1: CRL Rating Prior to Availability of Probabilistic Risk Analysis

CRL9 5 5.5 6 6.5 7 7.5 8 8.5 9

CRL8 4.5 5 5.5 6 6.5 7 7.5 8 8.5

CRL7 4 4.5 5 5.5 6 6.5 7 7.5 8

CRL6 3.5 4 4.5 5 5.5 6 6.5 7 7.5

CRL5 3 3.5 4 4.5 5 5.5 6 6.5 7

CRL4 2.5 3 3.5 4 4.5 5 5.5 6 6.5

CRL3 2 2.5 3 3.5 4 4.5 5 5.5 6

CRL2 1.5 2 2.5 3 3.5 4 4.5 5 5.5

CRL1 1 1.5 2 2.5 3 3.5 4 4.5 5

Extremel Minimal

Very Very Very Very Very Very Very Very

Basic “Complexity*”

of the to go work at the time

of the estimate

Adequacy of “Estimating Methods”, experience of the estimators, quality of CARD, availability of analogous data and

cost tools, time allowed for estimate, independence of estimate, number of

cross checks performed

*Complexity considerations include human rating, launch system requirements, planetary destination, operational vs experimental requirements, materials complexity, use of deployables, parts count, challenging thermal requirements, high data rates, electronic parts class, stabilization requirements, power generation type, propellant choice, propulsion requirements and many other factors. Programmatic complexity includes team size, team experience, schedule and many other factors.

CRL Description

9End of project actual cost

8Cost fit for very firm

engineering decisions and very firm budget

commitments (+/-5%)

7Cost fit for firm

engineering decisions and firm budget commitments

(+/-15%)

6Cost fit for PDR

engineering decisions and PDR budget use

(+/-25%)

5Cost fit for preliminary

engineering decisions and preliminary budget use (+/-

35%)

4Cost fit for very

preliminary engineering decisions and very

preliminary budget use (+/-45%)

44

Table 2: CRL Rating After Availability of Probabilistic Risk Analysis

25th Percentile Cost

Median Cost 75th Percentile CostLookup

Ratio of 75th Percentile Cost to 25th

Percentile Cost

ReadCRL

Description

100 100 100 1.00 9End of project actual cost

95 100 105 1.11 8Cost fit for very firm

engineering decisions and very firm budget commitments

(+/-5%)

85 100 115 1.35 7Cost fit for firm engineering decisions and firm budget

commitments (+/-15%)

75 100 125 1.67 6Cost fit for PDR engineering

decisions and PDR budget use (+/-25%)

65 100 135 2.08 5Cost fit for preliminary

engineering decisions and preliminary budget use (+/-35%)

55 100 145 2.64 4Cost fit for very preliminary

engineering decisions and very preliminary budget use (+/-45%)

Use S Curve inter-quartile cost range to translate to a CRL rating– Calculate ratio of 75th percentile cost to 25th percentile cost; then lookup ratio on chart to read CRL

Estimation Best Practices

• Decide Why You Want An Estimate

• Map Estimation Goals To Estimate Process Maturity & Develop Plan To Achieve The Maturity

• Have A Documented, Repeatable Estimation Process

• Make The Estimating Process As Simple As Possible; But No Simpler

• Be Proactive: The Process Is Important, The Tools Go Along With The Process

• Get Buy-in From Program Managers

• Hold People Accountable: Center Of Excellence Can Prepare Estimate But Program Managers Must Own Them

• Tie The Estimate To The Plan

45

Estimation Best Practices 2

• Evaluate Total Ownership Cost; Not Just Development

• Estimate A Range And Pick A Point For The Plan

• Re-estimate The Program When It Changes

• Avoid Death Marches: Programs With Unachievable Schedules Are Likely To Fail And Drain Morale

• Keep A History: Start An Enterprise Database NOW…

• Business Case: Evaluate ROI In Addition To Costs

• Convert Expert Spreadsheets Into A Common Language

46


• Track Progress Vs. Estimate Throughout The Life Cycle

• Estimate Schedule As Well As Effort (Cost) For Complete Picture

• Tie The Business Case Into The Estimating Process

• Attack Non-productive Rework As Part Of The Process

47


• Have clear definitions:

• What does “complete” mean

• What activities are included and excluded (E.g. development only or total ownership; help desk included or excluded, etc.)

• Which labor categories are included and excluded in the estimate (e.g. are managers included? Help desk? Etc.)

• Don’t ignore IT infrastructure and IT services costs

• Tracking defect sources can go along with the process

48

A1-49

Project Management Challenges Relate To Estimation Planning• “No good deed will go unpunished.” Unreasonable

expectations on the next project are supported by heroic behavior on the previous project

• The most important business decisions about a software project are made at the time of minimum knowledge and maximum uncertainty.

• Adding and/or changing means more time and/or more effort

• When a project is in trouble ask for more time rather than more people. Deferring functionality common approach to asking for more time

• Increasing concurrency is usually not a solution (e.g. designing several releases concurrently)

50

7 Characteristics of Dysfunctional Software Projects (Source: Mike Evans, et al.)

• Based on 350 projects:

• Failure to Apply Essential Project Management Practices

• Unwarranted Optimism and Unrealistic Management Expectations

• Failure to Implement Effective Software Processes

• Premature Victory Declarations

• Lack of Program Management Leadership

• Untimely Decision-Making

• Lack of Proactive Risk Management

Additional Information

• www.galorath.com

• Dan on estimating BLOG: www.galorath.com/wp

• Email: [email protected]


http://www.galorath.com/

http://www.galorath.com/wp

mailto:[email protected]