Upload
others
View
0
Download
0
Embed Size (px)
Citation preview
UK NDA Application of
Reference Class Forecasting
in Optimism Bias Adjustment Karl Sanderson
Head of Cross-Industry Learning
3 October 2019
NDA Historical Context
• 2012 report by the
National Audit Office
detailed historical
problems with cost &
schedule estimates at
the NDA - it notes that no
OBA was applied to past
estimates and that
contingency was
inadequate
• 2018 update report noted
that overall lifetime cost
estimating had improved
but large uncertainties
still remained
3
Reference: Amyas Morse., The Nuclear Decommissioning Authority:
Progress with Reducing Risk at Sellafield, London, National Audit Office,
2018
Optimism Bias Adjustment
• Optimism Bias (OB) defined in HM Treasury Green Book
supplementary guidance as: “a demonstrated, systematic,
tendency for project appraisers to be overly optimistic”
• Observable trends in underestimated project cost and
schedule projections are attributable to OB - these trends
are present in academic research, industry data, and
analysis of NDA projects
• As mitigation, Optimism Bias Adjustments (OBA) can be
applied to project cost & schedule estimates,
compensating for lack of understanding of project risks
early in the lifecycle
• Reference Class Forecasting (RFC) is the technique used
to determine the NDA-specific OBA 1
OBA relationship to other Estimation Methodologies
Reference Class Forecasting on its own does not deliver
better quality estimates. OBA needs to be used alongside;
• More effective Risk Management (threat & opportunity
identification) and differentiation from uncertainties and
ambiguities
• More effective probabilistic analysis / Quantitative Risk
Analysis (QRA)
• The use of better / more accurate volumetric /
Benchmarking data
• Improved Project Management in delivery
• More effective and robust challenge (Assurance)
2
Quantitative Risk Assessment
– example contributory factors
5
Procurement Project
Specific
Client
Specific Environment
External
Influence
• Complexity of Contract
Structure
• Late Contractor
Involvement in Design
• Poor Contractor
Capabilities
• Government Guidelines
• Dispute and Claims
Occurred
• Information management
• Other (specify)
• Design Complexity
• Degree of Innovation
• Environmental Impact
• Other (specify)
• Inadequacy of the
Business Case
• Large Number of
Stakeholders
• Funding Availability
• Project Management
Team
• Poor Project Intelligence
• Other (specify)
• Public Relations
• Site Characteristics
• Permits / Consents /
Approvals
• Other (specify)
• Political
• Economic
• Legislation / Regulations
• Technology
• Other (specify)
Quantitative
Risk Assessment
Uncertainty + Quantified Risk (Impact / Likelihood)
Monte Carlo Analysis
… not an exhaustive list of
threats and opportunities
Methodology at Early Stage Gates
4
Quantitative Risk Assessment
Reference Class Forecasting & OBA
OBC SOC FBC
Base Estimate
Assurance, Benchmarking, Project & Risk Management Estim
ate
Maturity
FEED / Detailed Design Preliminary Design Scoping
Single Concept Short-listed Concepts All Concepts
Single
Concept
Selected
Contract
Award
Then
Execution
HMT Green Book OBA
• HM Treasury Green Book guidelines based on generic project
categories:
4
Literature Data
5
Budzier et al looks at
a reference class of
216 projects to better
understand the
chances of cost
overruns in nuclear
projects
The sample included
194 nuclear new
builds and 22 nuclear
waste storage
projects
This data shows a
20% chance of going
over budget by over
202% Reference: Budzier et al. Quantitative Cost and Schedule Risk Analysis
of Nuclear Waste Storage, Oxford Global Projects, 2018
NDA Data Sample and Analysis
• Data was obtained covering a representative sample of
Sellafield projects including:
• Cost data on 22 projects covering different lifecycle points (14
covering the whole lifecycle)
• Schedule data on 9 projects covering the whole lifecycle
• For each project an initial and final best estimate was
determined; typically this was a P50 or “most likely” value
from a range - these were used to determine the
percentage increase over the project’s lifetime
• This data was used to plot S-curve diagrams showing the
percentage uplift associated with a given acceptable
chance of overrun (shown in the following slides)
6
Sample Data
7
6
2
3
1
2
0
1
2
3
4
5
6
7
8
9
10
<50% 50% to 120% 120% to 190% 190% to 260% 260% to 330%
Number of
Projects
Histogram Illustrating the Change in Cost Estimates for Projects at Sellafield
Results – Entire Project Lifecycle (from SOC)
8
Results – Preliminary Design Gate (from OBC)
9
Results – Detailed Design Gate (from FBC)
10
Results - Schedule Analysis
11
• The same analysis was carried out using data on schedule
estimates. Less data was available here resulting a
smaller sample of 9 projects covering the whole lifecycle.
Overall Results
• The results from the NDA sample data are as follows:
• Compares closely to the results from the larger global
sample presented in Budzier et al, 2018
• … suggests that the larger sample of global nuclear
projects might be representative of projects undertaken
within the NDA group
12
Chance of Overrun: 50% 20%
Cost 76% 195%
Schedule 65% 111%
Chance of Overrun: 50% 20%
Cost 67% 202%
Schedule 40% 104%
At SOC
16
Build estimate using best
available methodology
and best available data
Determine relevant
reference class for
project
Apply reference class
‘uplift’ using P50
confidence to the
estimate (optimism bias
adjustment)
Apply reference class
‘uplift’ using P80
confidence to calculate
an upper estimate Quote AFC / AFS estimate
range based on the base
estimate and optimism bias
adjustment
Estimating Reference
Class1 or OBA2 QRA Report
Baseline
Estimate
Preliminary Cost
/ Schedule
Range
Notes
1. NDA OBA
2. HMT OBA
At OBC & FBC
17
Update estimate in line
with scope
Determine relevant
reference class for
project
Apply reference class
‘uplift’ using P50
confidence to the
estimate
Quote AFC / AFS range
(P50-P80) based on the
QRA
Estimating Reference
Class QRA Report
Baseline
Estimate
P50 and P80 Cost
and Schedule
Perform QRA, taking into
account the project,
programme, wider risk
factors to derive P50.
Compare QRA at P50
with reference class at
P50
Is QRA consistent
with reference class
NO
YES
Conclusions & Next Steps
When a Project Manager proposed an early stage base estimate (at SOC
equivalent, with allowances but no contingency), consider adjusting for
optimism bias as follows:
Cost For P50 add 67-76%
For P80 add 195-202% (triple it)
Schedule For P50 add 40-65%
For P80 add 104-111% (double it)
Going forward:
• a wider sample of global nuclear projects, and NDA estate projects
will be used to derive specific results for the NDA
• further academic research, for example on statistical validity of
reference classes
13
Recap
8
OBC SOC FBC
Application of
RCF only
QRA tested by RCF QRA tested by RCF
Reference Class
S- curves used to
replace the
deterministic HMG
Green Book factors
(upper bounds) -
based on relevant
experience
OBA OBA OBA
££
Probability
Risks and uncertainty Risks and uncertainty
££
Probability
Estimate
Estimate
OBA
Estimate Estimate
QRA QRA