OFFICE OF SCIENCE Review of Critical Decision 3b for the Large Liquid Argon Detector for Neutrino Physics (MicroBooNE) Fermi National Accelerator Laboratory

OFFICE OF

SCIENCE

Review of Critical Decision 3b for the

Large Liquid Argon Detector for Neutrino Physics (MicroBooNE)

Fermi National Accelerator LaboratoryFebruary 29 – March 1, 2012

Kurt W. FisherReview Committee Chair

Office of Science, U.S. Department of Energyhttp://www.science.doe.gov/opa/

http://www.science.doe.gov/opa/

OFFICE OF

SCIENCEReview Committee Participants

Department of Energy Subcommittee 1- Cryostat and Cryo-systemKurt W. Fisher, DOE, Chairperson *Joel Fuerst, ANL

Bruce Strauss, DOE/SC

Subcommittee 2- Detector and Electronics *Bill Wisniewski, SLAC

Karen Byrum, ANL

Observers Subcommittee 3- Infrastructure and InstallationAlan Harris, DOE/FSO *Victor Guarino, ANLTed Lavine, DOE/SC Eli Rosenberg, DOE/SC Subcommittee 4- Management, Cost, Schedule Paul Philp, DOE/FSO and ES&H *Jeff Sims, ANL

Gil Gilchriese, LBNLEthan Merrill, DOE/SC

2

OFFICE OF

SCIENCE

3

Department of EnergyOffice of the Secretary

Dr. Steven Chu, Secretary

Deputy Secretary* Daniel B. Poneman

Associate Deputy SecretaryMelvin G. Williams, Jr.

Office of the Under Secretary

for Nuclear Security

Thomas P. D’AgostinoUnder Secretary

For Nuclear Security

*The Deputy Secretary also serves as the Chief Operating Officer.

06 December, 2011

Energy InformationAdministration

Bonneville PowerAdministration

Southwestern PowerAdministration

Southeastern PowerAdministration

Western Area PowerAdministration

Intelligence andCounterintelligence

Assistant Secretary forPolicy and International

Affairs

Assistant Secretary forCongressional and

Intergovernmental Affairs

General Counsel

Chief FinancialOfficer

Chief Human CapitalOfficer

Chief InformationOfficer

Public Affairs

Economic ImpactAnd Diversity

Deputy Administratorfor Defense Programs

Deputy Under Secretaryfor Counter-terrorism

Associate Administratorfor Emergency

Operations

Associate Administratorfor Acquisition &

Project Management

Associate Administratorfor Defense Nuclear

Nonproliferation

Deputy Administratorfor Naval Reactors

Associate Administratorfor Defense Nuclear

Security


for Science

VacantUnder Secretary

for Science

Office of Science

Advanced ScientificComputing Research

Basic Energy Sciences

Biological andEnvironmental Research

Fusion Energy Science

High Energy Physics

Nuclear Physics


VacantUnder Secretary

Associate Administratorfor External Affairs

Workforce DevelopmentFor Teachers/Scientists

Management

Health Safety andSecurity

Hearings and Appeals

Advanced ResearchProjects Agency-Energy

Loans ProgramOffice

American Recovery &Reinvestment Act Office

Federal EnergyRegulatory Commission

Inspector General

Assistant Secretaryfor Environmental

Management

Assistant Secretaryfor

Fossil Energy

Legacy Management

Indian EnergyPolicy and Programs

Assistant Secretaryfor Energy Efficiency

and Renewable Energy

Assistant Secretaryfor Electrical Delivery and

Energy Reliability

Assistant Secretaryfor

Nuclear Energy

Associate Administratorfor Info. Management

& CIO

Associate Administratorfor Management &

Budget

Associate Administratorfor Safety & Health

Technology TransferCoordinator

Chief of Staff

National NuclearSecurity Administration

Hearings and Appeals

4

OFFICE OF

SCIENCEOffice of Science

Chicago Office

Roxanne Purucker

Office of the Director (SC-1)William F. Brinkman

Advanced ScientificComp. Research (SC-21)

Daniel Hitchcock

Workforce Development for Teachers/

Scientists (SC-27)

P. Dehmer (A)Basic Energy

Sciences (SC-22)Harriet Kung

Fusion EnergySciences (SC-24)

Edmund Synakowski

High EnergyPhysics (SC-25)James Siegrist

Biological & Environ. Research (SC-23)

Sharlene Weatherwax

Nuclear Physics(SC-26)

Timothy Hallman (A) Acting

12/2011

Deputy Directorfor Science Programs (SC-2)

Patricia Dehmer

Deputy Directorfor Resource Management (SC-4)

Jeffrey Salmon

Deputy Directorfor Field Operations (SC-3)

Joseph McBrearty

Office of Project

Assessment (SC-28)Daniel

Lehman

Office of Budget (SC-41)

Kathleen Klausing

Office of Scientific and Tech. Info. (SC-44)

Walt Warnick

Office of SC Program Direction (SC-46)

Rebecca Kelley

Office of Grants/ Cont. Support (SC-43)Linda Shariati

Office of Business Policy

& Ops (SC-45)

J. Salmon (A)

SC Communications & Public Affairs

(SC-4)Dolline Hatchett

Ames SOCynthia Baebler

Thomas Jeff. SOJoe Arango

Stanford SOPaul Golan

Pacific NWest SORoger Snyder

Princeton SOMaria Dikeakos

Oak Ridge SOJohnny Moore

Fermi SOMichael Weis

Brookhaven SOMichael Holland

Berkeley SOAundra Richards

Argonne SOJoanna Livengood

SCIntegratedSupportCenter

Office of Lab

Policy & Evaluat.(SC-32)D. Streit

Office of Safety,

Security & Infra.(SC-31)

M. Jones

Human Resources & Admin.(SC-45.3)

Cynthia Mays

Small BusinessInnovationResearch(SC-29)

Manny Oliver

Oak Ridge Office

J. Eschenberg (A)

5

OFFICE OF

SCIENCECharge Questions

1. Final Design: Is the design sufficiently mature so that the project can continue with procurement and fabrication? Baseline Cost and Schedule: Are the current project cost and schedule projections consistent with the approved baseline cost and schedule? Is the contingency adequate for the risks?

2. Management: Evaluate the management structure as to its adequacy to deliver the proposed final design within the baselines as identified in the PEP. Has there been adequate progress on the fabrication activities approved under CD-3a?

3. Has the project responded satisfactorily to the recommendations from the previous independent project review?

6

OFFICE OF

SCIENCEReport Outline/ Writing Assignments

Executive Summary Fisher

1. Introduction Lavine

2. Technical Status (Charge Questions 1, 3)

2.1 Cryostat and Cryo-System Fuerst*/Strauss

2.1.1 Findings

2.1.2 Comments

2.1.3 Recommendations

2.2 Detector Wisniewski*/Byrum

3. Infrastructure and Installation (Charge Questions 1, 2, 3) Guarino*

4. Management , Cost, Schedule and ES&H(Charge Questions 2, 3)

Sims*/Gilchriese/Merrill

*Lead

7

OFFICE OF

SCIENCEExpectations

Present closeout reports in PowerPoint. Forward your sections for each review report

(in MSWord format) to Casey Clark, [email protected],

by March 5, 8:00 a.m. (EDT).

mailto:[email protected]

OFFICE OF

SCIENCE2.1 Cryostat and Cryo-SystemFuerst, ANL/Strauss, DOE/SC

1. Final Design: Is the design sufficiently mature so that the project can continue with procurement and fabrication? Baseline Cost and Schedule: Are the current project cost and schedule projections consistent with the approved baseline cost and schedule? Is the contingency adequate for the risks? Yes.

3. Has the project responded satisfactorily to the recommendations from the previous independent project review? Yes

8

OFFICE OF


9

Findings• LAr pumps, cooldown compressor, and heat exchanger (long-

lead procurements) are on order.• The cryostat bid package is complete.• The LAr fill is not a KPP.• “Piston purge” has proven successful in LAPD and will be used

instead of evacuation for initial cryostat purification.• The welded (vs. flanged) cryostat head design choice has been

made.• A thorough Hazard Analysis Review exists including cryogenics

ODH.

OFFICE OF


10

Comments• There may be additional risk associated with failure of the long-

lead cryogenic rotating machinery in the absence of on-hand spare parts.

• The cryostat bid package should continue to be expedited to enhance the schedule.

• A strategy for LAr removal should be considered.

Recommendations1) By 15MAR2012, consider ordering the recommended spare

parts kit (or a reasonable subset thereof) for the cooldown compressor.

OFFICE OF

SCIENCE

11

3. Infrastructure and InstallationGuarino, ANL

1. Final Design: Is the design sufficiently mature so that the project can continue with procurement and fabrication? YES Baseline Cost and Schedule: Are the current project cost and schedule projections consistent with the approved baseline cost and schedule? Is the contingency adequate for the risks? YES

2. Management: Evaluate the management structure as to its adequacy to deliver the proposed final design within the baselines as identified in the PEP. Has there been adequate progress on the fabrication activities approved under CD-3a? YES

3. Has the project responded satisfactorily to the recommendations from the previous independent project review? YES

OFFICE OF

SCIENCE3. Infrastructure and Installation

Guarino, ANL

12

FindingsThe group has performed a many tests of wire strength and has changed the diameter of the pins and applied tension as a result. A cold test of the wire assembly has been performed. The infrastructure for TPC assembly is well underway.The group has a good plan for TPC installation and cryogenics assemblyThe group has solid QA procedures and processes defined.

CommentsThe TPC assembly manpower relies on university labor and approximately 1FTE of a FNAL technician. For continuity, efficiency, and quality the group should consider using a minimum of 2 dedicated techs throughout the TPC assembly process.The re-welding of the flange and potential damage to the electronics is a concern and some testing might be warranted

RecommendationsNone

OFFICE OF

SCIENCE2.2 Detector and ElectronicsWisniewski, SLAC/Byrum, ANL

13

Detector & Electronics review scope includes PMT (WBS 1.8), TPC (1.4), Front End Electronics (1.5); Electronics Readout (1.11), DAQ / Monitoring & Slow Control (1.9) and some of infrastructure. PMT, TPC and Electronics Readout are in-kind contributions funded by NSF. FEE and DAQ are DOE funded.

The Committee commends the team on continued progress since the last review.

1. Final Design: Is the design sufficiently mature so that the project can continue with procurement and fabrication? Baseline Cost and Schedule: Are the current project cost and schedule projections consistent with the approved baseline cost and schedule? Is the contingency adequate for the risks?

The design is sufficiently mature to continue with procurement and fabrication. Cost performance since the last review has been good, using little contingency and allowing fractional contingency on work remaining to increase. Contingency is adequate to the risks.

OFFICE OF


14

3. Has the project responded satisfactorily to the recommendations from the previous independent project review?

There were no recommendations for the experimental team at the last review. However, there were Comments. The team has addressed these as if they were recommendations, and have made satisfactory response.

Findings: PMT has continued to make progress. Key components are in hand,

with all remaining materials to be purchased soon. Costs are close to estimates. SHV connectors have been found to be unreliable when cold. New

cabling solutions, possibly with no intermediate cold connections, are under consideration.

Optical simulations have been performed and indicate that typically 10 PE can be seen in most of the TPC volume.

All PMTs have been tested in air.

OFFICE OF


15

Findings: PMT:

The PMT LN2 test stand is in now in use. 4 PMTs can be tested simultaneously. Gain vs HV is measured using an LED calibrator.

Cold test procedures passed safety review, allowing unattended cold tests to be performed.

Cold tests are expected to be complete by the start of CY2012Q4 A long full PMT unit test is planned to follow. WLS degradation on the plate has been found to be due to light, not

humidity. Care will be taken to limit exposure to light after plate painting; in particular, the assembly tent lights will be UV filtered.

Fabrication will commence in CY2013Q1, under the current schedule, with PMT unit assembly complete in Q4.

OFFICE OF


16

Findings: TPC:

Work since the last review has been focused on completing drawings for the TPC components, winding machines and tension devices.

The first wire winding machine has been assembled and tested. Teams at Yale and Syracuse have been trained. The first machine is now at Syracuse ready for use. This is about a month later than expected at the time of the last review. Yale’s line will start in a month

The wire carriers have been refined (thicker pins, bigger resistors, smaller caps, simpler connections

A wire catcher system will not be used. Instead, the QA program was expanded. In the process, a cold test of the full-length wire carrier assembly has been successfully performed, and no breakage found up to 2.5kg (tension below yield point).

Milestones were adjusted to reflect sequential release of TPC component part POs, allowing revision of parts for the Cathode. Anode parts are in fab, and field cage tubes out for quote.

OFFICE OF


17

Findings: TPC:

Wire winding is expected to be complete by CY2012Q4, with other parts fab’d on the same time scale, and delivered to Fermilab.

There appears to be adequate manpower to complete the task. Wire breakage is not expected to be a problems after the extensive test program and under the QA program. There is a tension safety factor of 4.5 (RT) to 6 (LN2).

Front end electronics: FEE prototype cycle 1 development was finished and successfully

tested. Development of the cycle 2 prototype, which is the pre-production prototype, is almost complete, with preparations for testing the electronics for a full TPC feed-through in progress. Many boards have already been fab’d and tested toward this goal. A 2nd prototype TPC signal feed-through has been produced, as well as cables, and a power supply candidate ordered.

OFFICE OF


18

Findings: Front end electronics:

The CMOS front end ASIC 3rd version was received in late Oct. 2011. Thirty-eight were warm tested. Eight, mounted on a motherboard, have been cold tested through more than 50 LN2 immersions. The ASIC was found to function with increased gain and lower noise.

A 4th ASIC cycle will be submitted in early March to address fabrication issues: channel-to-channel variations for two chips is +/-4% (higher than expected); defects seen at one edge of the chips; chip level yield of ~60% with single channel yield of 97%. More conservative design rules will be used. Options are being considered to mitigate the cost and schedule impact of this unexpected extra ASIC cycle.

Readout Electronics: The Front End Module was tested with TPC FEE prototypes. ADC data

was received correctly, calibration triggers could be propagated through the system. Baseline noise, cross-talk and linearity were measured.

OFFICE OF


19

Findings: Readout Electronics:

The PMT electronics chain, which operates at a higher frequency, was also tested: the shaper can send data to the PMT ADC, where it was successfully digitized; the FEM can receive that data and generate a trigger. Proto 2 tests are on the horizon.

Production of components has begun, with completion expected February 2013.

DAQ is responsible for assembling events into a file, as well as monitoring and control, run control, calibration sequencing, file management.

Glomation has been selected as the slow monitoring and control hub. PCIe functionality, network, and calibration process were specified.

PCIe testing is being carried out on Fermilab and Nevis test stands. Calibration Hardware Interface code was developed. Event building is largely finished, including MicroBooNE specific fake

data files.

OFFICE OF


20

Findings: DAQ:

Processes for Online monitoring and EPICs issues are understood, with IOCs accessing shared memory.

Relying on experience from PHENIX, ATLAS, NOvA, DoubleChooz and D0.

The D0 Assembly Building test stand is imminent. CD resources have been identified. The consequent rework of the

schedule is in progress, and was unavailable to the Committee. Hardware costs and purchase sequencing are well understood.

OFFICE OF


21

Comments: PMT has continued to make progress. Most components are in hand,

and the schedule of work moving forward is not demanding, but does merit continuing attention.

TPC has been very aggressive in understanding wire issues. Cold tests have been performed on the full length wire system. The stringing tension has been reduced to provide more headroom at room temperatures and at LN2 temperatures. The team has put heavy emphasis on QA. The Committee commends the team on this work. In light of what has been learned in this program, the decision to dispense with the complications of a wire catcher system is reasonable.

OFFICE OF


22

Comments: The TPC group is aware of the consequences of an approximate 4

month shut down if a wire should break once the experiment is taking data.

The current TPC cost projection is consistent with the approved baseline. The group identifies the biggest risk to be one of the wire winding assemblies not working and wire breakage rate being larger than expected. Mitigation in both cases relies on the team working longer and spreading the load between the two groups.

While there appears to be enough manpower available, a detailed schedule utilizing this manpower is missing.

Solid progress continues in Readout Electronics. The full chain prototype readout is an important forward step and its success is welcome.

OFFICE OF


23

Comments: DAQ has made good progress on design. Purchase planning has

received adequate attention. ‘Construction’ is starting. The status of manpower after the negotiations with CD, as well as the consequent schedule changes, was not clear to the committee. The team is encouraged to develop a clearer path forward here.

Front End Electronics has in general made good progress in moving forward, developing the front end boards through to the pre-production stage. The ASIC presents complications in this picture.

Since the 3rd version of the CMOS ASIC would be acceptable, though not preferred, for the needs of the experiment, the suite of tests to fully characterize this version of the ASIC should be carried out on the remaining chips. This will mitigate the risk if the 4th version has serious problems: there will at least be one solution available for production. The schedule consequences on the project’s schedule should be evaluated.

OFFICE OF


24

Recommendations1. Execute the complete suite of tests to fully characterize Version 3 of the

ASIC. 2. Understand the effects of Version 4 of the ASIC on the project

schedule.3. Conduct a design review of the ASIC before submission of the

production run.

4. Approve CD-3b

OFFICE OF

SCIENCE

25

MicroBooNE Project StatusSims, ANL/Gilchriese, LBNL/

Merrill, DOE/SC

PROJECT STATUS as of January 2012Project Type MIE / Line Item / Cooperative AgreementCD-1 Planned: Actual: 7/9/2010CD-2 Planned: Actual: 9/27/2011CD-3a Planned: Actual: 9/27/2011CD-3b Baseline: June 2012 Actual: CD-4 Baseline: Sept 2015 Actual: TPC Percent Complete Planned: 30.8% Actual: 29.3%TPC Cost to Date $4.629M

TPC Committed to Date $4.989M TPC $19.9MTEC $14.76MContingency Cost (w/Mgmt Reserve) $4.286M 38% to goContingency Schedule on CD-4 18 months 41.9%CPI Cumulative 0.99

SPI Cumulative 0.95

OFFICE OF

SCIENCE

26

4. Management, Cost, Schedule and ES&H

Sims, ANL/Gilchriese, LBNL/Merrill, DOE/SC

2. Management: Evaluate the management structure as to its adequacy to deliver the proposed final design within the baselines as identified in the PEP. Has there been adequate progress on the fabrication activities approved under CD-3a? Yes, CD-3a scope appears to be on schedule.

3. Has the project responded satisfactorily to the recommendations from the previous independent project review? Yes. CR-019 for ASIC replan is still in progress

OFFICE OF

SCIENCE

27



4.1 Findings

• The GPP funded facility LARTF is under construction and ahead of schedule with

nearly 6 months of float off the Microboone critical path

• The OPC scope is substantially complete.

• The project presented the change request that is being prepared to revise the logic

and durations related to front end electronics. This change may reduce the current 18

months of project float on CD-4 to 12 months.

• The TEC scope design is complete and the cryostat related items are in the

solicitation phase.

• A detailed analysis of schedule and cost using Monte Carlo was performed which

confirmed the current cost and schedule contingency.

OFFICE OF

SCIENCE

28



4.1 Findings

• Quality Assurance modules with detailed procedures and requirements for each level

2 work package were presented.

• There are 36 active risks that are updated quarterly.

• A Project Hazard Analysis and Environment Safety and Health Plan have been

developed for all assembly and installation activities.

• Project schedule has been updated incorporating all changes through February 2012

• The project schedule has ~400 total remaining activities, ~ 60 activities on the

critical path, and ~100 activities within 30 days of the critical path.

• Critical Path has shifted to Front End Electronics followed by Installation.

• MIE funding in FY12 is $6.0M. The program will convert FY11 carry over

(approximately $3M) to MIE funds to meet planned obligations

• The project has processed ~$100K in changes since CD-2/3a in September 2011

OFFICE OF

SCIENCE

29



4.2 Comments

• The project management team is complete and functioning very well. The team

should be commended on its vigilance in project management processes.

• The management structure and team appears adequate to deliver the baseline

design.

• There has been adequate progress on the activities approved under CD-3a.

• As of January, contingency of about $4M remains for an ETC of about $11M.

• There appears to be adequate float between the forecast project completion and the

CD4 milestone, even after including the delays expected from a fourth round of

front-end ASIC fabrication and testing that is not yet in the baseline schedule.

• QA processes and plans are well developed for a project of this scope

OFFICE OF

SCIENCE

30



4.2 Comments

• Additional resource loading of student and scientific staff is needed, primarily in the

areas of assembly and installation labor.

• CR-19 related to the ASIC front end electronics should be processed promptly.

• A call on remaining contingency could be as high as $0.9M (25 % of current

contingency) for the procurement and testing of front-end ASIC round 4. Almost all

of this results from personnel costs (project management, engineering) maintained

for approximately an additional 6 months to complete the project. With project

management excluded contingency is approximately 30% on remaining work.

• Contingency needs should be re-evaluated after major procurements are placed in

FY2012.

OFFICE OF

SCIENCE

31



4.2 Comments

• A risk review and status effort should be performed about once per month as the

project progresses. The status should use the risk registry, include mitigation steps

from the risk owners and address risks by both timeliness and impact.

• The QA modules should be signed by the L2 and related managers to ensure

concurrence and acceptance.

• The current prediction of completion of the LARTF as much 6 months ahead of the

project critical path is commendable. The project should leverage this float to

prepare the LARTF with MicroBooNE specific needs.

• The assembly and installation L2 managers must be very vigilant to ensure the

visiting team members are properly trained and adhere to Fermilab safety

procedures .

• Develop a plan to utilize potential unused contingency at the end of the project.

OFFICE OF

SCIENCE

32



4.2 Comments

• 25% of project activities are on or near critical path. This should be monitored to

ensure the percentage does not increase.

• The project is cautioned against over-reliance on the risk simulations

OFFICE OF

SCIENCE

33



4.3 Recommendations

1. Update the schedule and cost to include the impact of the run 4 ASIC fabrication

and test cycle and initiate change control actions to make the updated schedule/cost

the basis prior to CD-3b.

2. Sign off on Quality Assurance modules by L2 and related managers should be

completed prior to CD-3b

Documents

OFFICE OF SCIENCE Review of Critical Decision 3b for the Large Liquid Argon Detector for Neutrino Physics (MicroBooNE) Fermi National Accelerator Laboratory