Planning and Oversight of NSF’s Large Facility Projects

Mark ColesDeputy Director for Large Facility Projects

Office of Budget, Finance and Award Management

2

Introduction

• National Academies study “Setting Priorities for Large Research Facility Projects” (Brinkman report) makes specific recommendations for prioritizing, planning, and overseeing the construction and operation of large facilities supported by NSF.

• The report cites the need to make greater resources available for pre-award planning of a project’s workscope, and

• Implement rigorous post-award oversight through periodic external review

• National Science Board and National Science Foundation joint response embraces this recommendation.

Purpose of this presentation is to discuss how this should be implemented and solicit input

3

Summary of the NAS Study Recommendations

• Development of a 10-20 year facility roadmap• Three levels of criteria for ranking:

– Scientific and technical merit– Agency Strategic Criteria– National Criteria

• New starts ranked in annual budget request using clear rationale based on roadmap

• Enhanced project pre-approval planning and budgeting• Greater independent oversight and review needed• Review effectiveness of Deputy Director for Large Facilities in

two years• OSTP should coordinates roadmaps across agencies• NSF Leadership and NSB should pay careful attention to

implementation of proposed reforms

4

NAS Study Findings

• “There is a lack of funding for disciplines to conduct idea-generating and project-ranking activities and, once ideas have some level of approval, a lack of funding for conceptual development, planning, engineering, and design—information needed when judging whether a project is ready for funding in light of its ranking and for preparing a project for funding if it is selected”.

5

NSB defines a new process for approving readiness of a project:

• “Project plans are judged to be construction ready by the Large Facility Deputy”,

• “The budget for construction and for operations costs has been justified to the satisfaction of the Chief Financial Officer”

From: Attachment 5 to NSB-04-92, NSB-04-97

What are the criteria for ascertaining readiness of plans and budget?

6

Readiness certification process needs to be defined

• Goal: NSB approves projects that are well defined and that have thorough cost estimates

• Present situation: – Project baseline, including well scrutinized bottoms-

up contingency estimate, is established after construction.

– Many projects are de-scoped during construction (examples: LIGO, Gemini, IceCube, likely ALMA)

• Development of a project baseline (planning, costing, enabling R&D, PMCS, risk assessment, staffing) requires ~5-10% of total project cost.

7

Recommendation: Develop project baseline prior to approval of construction

award

• Project baseline:– Detailed technical workscope flowdown from science

requirements– Bottoms-up budget estimate– Resource loaded schedule with realistic budget

profile and critical path– Thorough risk assessment and item-by-item

contingency estimate– Robust project execution plan for project

management, and staffing commitments for key work package managers

8

Benefits of defining project baseline prior to construction award

• Development of a robust project baseline reduces uncertainty in project workscope.

• Need to make an earlier commitment of significant development funding to a project, with expectation that construction approval is likely if pre-defined development criteria are satisfied.

• Internal Management Plan (aka “Project Development Plan”) could lay out predefined criteria that must be satisfied. For example:– Completion and successful review of preparatory research– Satisfactory demonstration of enabling technology– Develop of detailed “bottoms-up” cost and contingency

estimates, resource loaded schedule, recruitment of key personnel, Project Management Plan, and other components of Project Baseline, followed by successful external baseline review.

9

More planning resources will:

• Reduce uncertainties in construction and operations budgets• Reduce uncertainty in construction schedule• Reduce likelihood of de-scoping• Thorough pre-construction risk assessment will create a more

robust Project Execution Plan• Create an appropriate framework for cooperation for activities

involving international and inter-agency partnerships as an integral part of in-depth planning.

• Make project oversight during construction more straightforward, add more definition to the proposed workscope and decision points.

• Allow more in-depth consideration of transition to operations and greater certainty in predicted O&M costs.

• Avoid a funding hiatus. Overlap some development activities beyond NSB approval and inclusion in annual NSF budget submission.

10

Other factors to consider

• Greater involvement by NSF OD, NSB, OMB, Congress, in the early stage of a project - prior to entry into Readiness Stage.

• Need to provide visible, but not risk-free, level of commitment to a project during the readiness phase so that a project can recruit engineering and managerial staff and undertake enabling R&D, detailed plans, cost estimates.

• Need to set an expectation that a project will go forward into construction from the readiness phase if it meets the criteria established in its Internal Management Plan (and that if can fall out, or terminate, if unsuccessful)

11

Suggested Implementation

• Prior to entering Readiness phase a project must have:– Scientific merit satisfying the first two NAS criteria – scientific

merit and NSF priority– “Preliminary” Project Plan:

• Top down estimates of technical scope, budget, contingency,• Detailed plan for enabling R&D, planning, technical

demonstrations – “Project Development Plan”• Detailed schedule with identified decision point and plan for

determining construction readiness.• Prioritization of readiness activities with window for construction

approval, to provide latitude for timing of construction approval– Approval by Facilities Panel of Project Development Plan– Approval by MREFC Panel of overall activity, timing, decision

points, etc.

12

Implementation: Requirements to enter Readiness Phase

• Review and approval of:– “Preliminary” technical scope, – Top-down budget and contingency estimates, – Project development plan– Project development schedule, including proposed decision point

window• Presentation and approval to NSF OD and NSB of project for

entry into Readiness phase• Determination by NSF OD and NSB that project meets third

level NAS criteria – national importance• Inform OMB and Congress of:

– Project’s potential construction scope– Development schedule– Window for construction approval (and allow for “tuning” to meet

future budget envelope expectations)

13

While in the Readiness Phase

• Regular status reports to NSF and NSB• Periodic updates to OMB and Congress of project status• External reviews of development milestones as

appropriate• Baseline review to establish construction readiness and

budget readiness• Request to NSF OD and NSB to exit Readiness Phase upon

successful completion of baseline review• Inform OMB and Congress of successful completion of

baseline review and request inclusion of project’s construction budget within “window”

• Projects that fail to meet development milestones may be delayed or terminated, as appropriate.

14

Exiting Readiness Phase

• Construction activity begins• Construction oversight by NSF begins:

– Monthly status reports– Periodic external reviews of construction activity

• Development activities conclude• Periodic updates to inform NSF OD, NSB, OMB,

Congress of project status

15

Summary Impacts on NSF

• Greater commitment of R&RA funds for extensive planning

• Involvement of NSF Director, OMB, Congress from beginning of the intensive planning stage, rather than at the end

• Implementation of formally approved project development plans

• Greater responsibility for evaluation of project technical plans and budget by CFO and Deputy Director for Large Facilities

16

Oversight during construction

• Practices for post-award oversight vary greatly across foundation

• External review by slowly varying panels with a great breadth of relevant experience are used mostly in Physics.

• Other disciplines rely on self-assessment, or narrowly constituted advisory panels from the target research field

How can we promote implementation of periodic external review by a panel of scientific, engineering, management, and technical experts? (As recommended by NAS report)

17

Need for further detailed advice

• NSF’s Facilities Management and Oversight Guide will be revised as part of NSF/NSB response to NAS study

• Other findings from NAS study also present opportunities to improve NSF’s capabilities:– Post award oversight– Full life-cycle cost accounting

• Need for guidance on how to develop and implement these policies, particularly in areas where there has not been a tradition of:– Construction of large facilities– Use of “power tools” for technical and financial reporting– Periodic post-award external review

18

Facilities Subcommittee

• With approval of Business and Operations Committee, propose to establish Facilities Subcommittee as a subsidiary of the Business and Operations Committee to provide specific guidance on these policy issues

• Facilities Subcommittee will report its findings to the full Business and Operations Committee

• Subcommittee will meet twice per year• Plan to have five members of subcommittee,

all with significant experience with management of large scientific facilities

19

Facilities Subcommittee Charter

• The Facilities Subcommittee will examine the business practices the NSF applies to planning and executing major research facility projects, including:– NSF policies and procedures for pre-construction planning

and evaluation of proposed large facility projects,– Non-technical aspects of the oversight practices applied by

NSF during construction of large facilities, and – Practices used to monitor business operations aspects of

the facilities.• The Subcommittee will make recommendations to the

Business and Operations Advisory Committee as to the adequacy of current practices, and will make recommendations for ways that the NSF can improve current practices to enhance their effectiveness.

20

Proposed Facilities Subcommittee Membership

• The following individuals have agreed to serve on the Subcommittee if approved by the Business and Operations Committee:– Tom Kirk (Chair of Subcommittee and member of B&O

Committee)• Associate Laboratory Director of High Energy and Nuclear Physics,

Brookhaven National Laboratory– Jim Yeck

• Project Director, IceCube Project, Univ. of Wisconsin, Madison– Janos Kirz

• Acting Director, Advanced Light Source, Lawrence Berkeley National Laboratory

– Paul Gilbert• Chairman Emeritus, Parsons Brinckerhoff Quade & Douglas, Inc.

– Larry Ramsey• Professor of Astronomy and Astrophysics at Penn State, and Project

Scientist on Hobby-Eberly Large Telescope

21

How the B&O Committee can help

• Recommend business processes for ascertaining readiness of budget and construction plans

• Recommend how best to promote the implementation of periodic external review during construction

• Approve charter and membership of Facilities Subcommittee

Backup materials

23

NSF/NSB Response:

• “The National Academies’ Report properly calls attention to the necessity for considerable pre-approval funding for planning and development when it questions whether there is sufficient NSF support for this "bottom up" process. NSF endorses the Report’s recommendations to provide researchers access to funding sufficient to develop compelling research agendas, to refine and prioritize their facility requirements, and to complete research and development on facility designs and needed technologies. The level and form of funding for planning and development will be reviewed, and an evaluation will be made of how project funds are best invested to attain robust plans and schedules with better cost projections, so that only well-defined and thoroughly-costed projects are brought forward for consideration by the Board.”

24

More NSF/NSB Response:

• “The Director and the Board recognize the need to strengthen oversight of the implementation of large facility projects, which will require increased investments of NSF staff time and travel funds. The Report emphasizes the importance of initial planning and definition of technical scope, budget, and schedule, followed by periodic post-award status reviews held on-site by external experts, with implementation of a transparent process for management of changes to a project’s implementation plan.”

25

And more NSF/NSB response…

• “On the basis of merit-reviewed proposals, NSF will fund these planning and project development efforts.”

• “In accepting the Facility Plan the NSB will concur that each of the Readiness List projects has attained that status by an appropriate process”