October 16, 2013presentation to the NAOJ Workshop

Processes and Considerationsfor

Planning and Constructing Major Research Infrastructureat the

US National Science Foundation

2

Funds every field of science – mostly as grants to individual researchers at US universities and non-profit organizations

~ $7B annual budget:~$1.3B for Mathematical and Physical Sciences (MPS)

About $260M for Astronomy More than $130M annually to operate existing astronomical observatories Fraction spent on infrastructure increasing due to Atacama Large Millimeter

Array (ALMA) operations and expenditures for mid-scale instrumentation and infrastructure.

Future budget outlook: ~flatPhysics and Astronomy outlook: declining

multi-disciplinary initiatives and new applied research programs compete with core disciplinary research, reducing core programs in recent years

National Science Foundation

3

Some have been spectacular successes:

NSF has a 50 year history of undertaking large research infrastructure projects

Cornell Electron Storage Ring (CESR) -- high-luminosity 6+6 GeV electron-positron collider at Cornell University

> 30 years operation

And some were not: Project Mohole - an ambitious attempt to drill through the Earth's crust into the Mohorovicic discontinuity

Many lessons learned!

Current NSF ConstructionSix large facilities under construction:

Advanced LIGO, Advanced Technology Solar Telescope, Alaska Region Research Vessel, Atacama Large Millimeter Array (with Japan, and nearly complete!), National Ecological Observatory Network, Ocean Observatories Initiative

Two recently completed smaller scale facilities: National Center for Atmospheric Research’s Wyoming Supercomputer

Center, ALVIN Replacement Human Occupied VehicleLarge Synoptic Survey Telescope One may start construction in

FY 2014Depends on Congressional fundingPlanned complete in 2022?

4

5

NSF doesn’t have a long-term “facility roadmap” of future plans NSF doesn’t manage big projects or operate big facilities (except in Antarctica) NSF oversees planning, construction, and operation, but the skilled project

managers and technical staff work for a university or non-profit organization (the “awardee’), who is responsible for project accomplishment

Construction funded through a separate Congressional appropriation The awardee contracts with major industrial firms to design and build the

project. NSF expects awardee to be capable of managing very large (>$100M) subcontracts to industry

Funding to Awardee via “Cooperative Agreement,” not a contract. This means NSF has substantial project involvement. For example: NSF shares project risk with Awardee, necessitates risk management plan NSF can make management changes at Awardee NSF has latitude to change scope and budget, if necessary NSF approves major expenditures and significant changes to plan NSF requires monthly progress reporting of financial and technical status

Some unique aspects of NSF

6

Project planning A new project must have a credible budget estimate, a resource-loaded

schedule, a sound management plan, and a core team capable of accomplishing the project

Assessment and evaluation NSF asks independent experts to evaluate the merits of the proposed science,

and the soundness of the project plan, including the budget estimateNSF prioritization and budget planning

The most important factor for NSF is whether it can afford to operate the new facility after construction is complete

In a flat budget environment, this means that there is a competition between new opportunities and areas of ongoing support – what is the most important science?

The next few slides provide more details about these 3 topics

Undertaking a new project requires coordination of many factors:

7

NSF expects projects to develop planning using sequentially increasing maturity stages: Horizon: Define Quantitative Science Goals and Objectives; derive functional

requirements and top level technical requirements Develop parametric “top down” cost and schedule estimates, based on scaling

historical data Conceptual Design Report Develop “bottom up” cost and schedule estimates, based on detailed engineering

design, site specific factors, input from industry, interconnection of functional elements, estimates of risk and uncertainty, Project Execution Plan Preliminary Design Report

Develop “ready to build” plan, based substantially on supplier estimates, recruit key staff, secure site permits Final Design Report

Conceptual Design is like the outline of a book – with all the structure and chapter headings. The Preliminary Design and Final Design activities fill in the details. Final Design Report = instruction manual for building facility.

Generally similar processes at US Department of Energy and NASA

Planning Process

Preliminary Design

FinalDesign

Construction

Operations

Horizon planning and Conceptual Design

8

Horizon Assessment: NSF internally prioritizes among candidate projects. NSF Director authorizes NSF funding of activities leading to Conceptual Design Report

Conceptual Design Review: Results in rough definition of project scope and coarse estimate (>50% uncertainty) of construction budget. National Science Board (NSB) authorizes significant funds for planning to develop Preliminary Design Report

Preliminary Design Review: Sets “reference baseline” definition of scope, budget, schedule, and risk. If review committee satisfied, and NSB continues to place high priority on the project, request Presidential concurrence (Office of Management and Budget) followed by Congressional appropriation of $

Final Design Review: Assesses whether project is ready to start construction. If review committee satisfied, NSB authorizes project team to start spending construction money

Assessment and evaluation

9

National Science Board takes on the very challenging task of prioritizing new facility projects across disciplinesExtremely hard: telescopes vs ships comparisonOMB/Congress expectation that annual construction budget <

$200M, so resource loaded schedule and annual construction budget projections are critical factors Need high confidence in Total Project Cost estimate, but also well

supported annual budget estimates for each project year!NSF and NSB are reactive: strong, well organized and continuous

community advocacy is very important influence on prioritizingBroader societal impacts are a big factor in decision: number of

users, education and training opportunities, open data accessNSB can and does re-prioritize

Prioritization

10

Cost estimates are inherently biased too low:They tend to omit or overlook necessary scopeThey are usually over-optimistic regarding technical difficulty of

planned workNSF has a “no-overrun policy” – budget shortfalls in

construction arising from known risks must be mitigated within budget, and by de-scoping if necessary. Strong reluctance to go back to Congress and ask for more $But NSF is not crazy – ATST budget augmented by $34M this year

to recognize cost increases due to prolonged legal challenges to Hawaiian site use

Message to communities not to bring an immature project plan and overoptimistic budget request to NSF

Budgeting

LESSONS LEARNED FROM PRIOR NSF PROJECTS

12

In a flat budget environment, the capacity to sustain operation is the principal factor in deciding whether to undertake new construction But operating cost projections are almost always biased low

because they overlook essential scope and underestimate technical difficulty and risk

Perform full life-cycle cost estimation early, and update regularly:Operating cost projections Budgeting for science useTermination and contingent liabilities (partnership

withdrawal, for example)

Full cost accounting

13

Budget Lessons Learned Construction activity requires a big pre-construction investment (5-25% of

total capital cost, and studies have shown that greater preconstruction spending results in more accurate predictions of actual project cost)

Project management is ~10% of Total Project Cost (TPC) From idea to groundbreaking requires ~10-20 years Earned Value Management (EVM) Reporting is one of many important

diagnostic indicators: By the time a project is 10% complete, EVM cost/schedule variances should

be meaningful. Good managers use EVM as a management tool, not just a reporting tool Early cost/schedule problems are leading indicators of bigger problems later

– pay attention to early warnings! Costs to operate industrial strength project management controls software

~1-2% of total construction cost. It’s worth it!

14

Risk adjusted cost is for “known unknowns”Known unknowns = risks known to the cost estimatorsThis includes provisions for factors such as commodity price

volatility, foreign exchange uncertainty, and uncertain start dates due to delays, including those caused by Federal Gov.

Uncertainty in Federal appropriation process and schedule are part of the landscape for Project Management and budgeting, include these factors in the risk-adjusted budget

Budgeting (continued)

15

Awardee’s Project Execution Plans are expected to follow the Project Management Body of Knowledge (PMBOK) to develop a robust Risk Management Plan (5 step process)

Budget and schedule estimates are similar to physical measurements; they have values and uncertainties. Each uncertainty can have random and systematic components

NSF develops its own “Internal Management Plan” to manage its own risk in overseeing the project: Identify risks to agency, especially early indicators of future trouble,

possible remediation “Unknown unknowns” (=risks unknown to the cost estimator) are funding

agency’s responsibility, and are dealt with at program policy level. Maintain flexibility and reserve capacity to adopt alternate courses of action in future planning

Risk management

16

International collaboration may make it possible to undertake new projects too expensive for any single country alone

Coordination with project planning and assessment becomes even more challenging with multiple funding partners!

Partnership agreements must recognize the unique roles and responsibilities of each partner, and NOT treat partners symmetricallyFor example, NSF funds and oversees large projects, but does not

manage or directly operate themUS strongly endorses open researcher access to facilities based on

merit review. Opposition to European “Pay to play” model. Experimental groups do

not contribute to operating costs of US accelerators (ICFA guidelines), telescopes, etc.

International partnerships

17

US open data access policies may create partnership disincentives in some cases: why pay for operations when you can get the data for free?

Uncertain annual appropriation processes are true for many countries, and complicate construction and operation budgeting

Partnership agreements need to promote long term stability International partnership agreements must enable individual partners

to exercise their expected stewardship roles as overseers of sovereign funds

Big projects are inherently part of political dialogue because of the size of projected budgets Projects have foundered when political influence has resulted in

premature project start with incomplete plans (ITER, SSC, DUSEL) and there has been painful re-scoping with others (ALMA, …)

International Coordination

18

Partnership agreements are founded on good will – structure must be carefully planned to align interests to avoid destabilizing collaboration. Include means for prompt dispute resolution!

International partnerships have intrinsic overhead costs that must be recognized, and different partners have different costs. In-kind contributions may have a cost to the other partners.

Robust systems engineering, spanning partner boundaries, is essential Conduct and manage a single project rather than parallel activities Need to synchronize development process when there are other partners or

potential partners Define the appropriate governance models for construction and operation

Extraordinary projects are successful when led by extraordinary people. Detailed policies and agreements don’t compensate for this.

Challenges to international collaboration

19

Annual NSF facility plan – see on-line meeting materials for NSB February board meetings

NSF Large Facilities Manual – available at http://www.nsf.gov/publications/pub_summ.jsp?ods_key=lfm

GAO Cost Assessment Guide – see especially the discussions of cost and contingency estimation, and the many helpful case studies

PMBOK – see especially its discussion of risk management

AACE-I – international organization that describes best practices for cost and risk estimation

Further information