sPHENIX Director’s Review Project Management and Status · 2019. 4. 9. · sPHENIX Project Scope (1) • Three large detectors, with their support services (LV, HV, gas) –Time

sPHENIX Director’s ReviewProject Management and Status

Glenn R. Young

April 9, 2019

BNL

sPHENIX Project Scope (1)• Three large detectors, with their support services (LV, HV, gas)

– Time Projection Chamber (TPC) to track charged particles

– ElectroMagnetic Calorimeter (EMCal) to measure energies and positions of gamma-rays and electrons/positrons

– Hadronic Calorimeter (HCal) to measure energies and positions of charged pions/kaons/protons

• Silicon vertexing detectors (INTT, MVTX) to spot rapid decays and time vertex

• Minimum Bias Detector (MBD, existing) to tag RHIC collisions of interest

• Front-End Electronics (FEE) to process & digitize analog signals from all detectors

• Data-Acquisition (DAQ) to collect & buffer this digital information

• Triggering system to select events of interest

• Timing system to synchronize all the above

2sPHENIX Director’s ReviewApril 9-11, 2019

sPHENIX Project Scope (2)• Superconducting Magnet and support systems

– associated cryogenics

– power supply and quench detection system

– controls

• Cradle/Carriage to support all parts of the experiment

– Internal detector supports

– Platforms, bridge, access items, stairs

• Magnet steel: Barrel Yoke (doubles as Outer HCal absorber) and Pole Tips

• Infrastructure improvements to Building 1008 complex

• Installation of all items

• Commissioning of all items


All Control Account Managers are at L2

4

1.x L2/CAM and L3 WBS Org

sPHENIX Director’s ReviewApril 9-11, 2019

All Control Account Managers are at L2

5

2.x L2/CAM and L3 WBS Org


sPHENIX Project Organization (1)• New detector construction led by University and other Lab groups

– TPC: Stony Brook for main chamber; Wayne State/Vanderbilt/Weizmann for Readout planes

– EMCal: Illinois for Blocks, Michigan for optical sensor testing

– HCal: Iowa State for Sector assembly, Georgia State for scintillator tiles

– INTT: RIKEN

– MVTX: LANL, LBNL, MIT

• Electronics and Triggering also depend on University groups– Calorimeter Digitizers & MBD FEE: Columbia/Nevis Lab

– Triggering: Colorado


sPHENIX Project Organization (2)• BNL Staff for the balance of the effort

– Front-End Electronics for Calorimeters and Time Projection Chamber

– Data Acquisition; Global Trigger; Timing System

– MBD refurbishment

– Detector support services: High/Low Voltage, Gas, Calibration Lasers

– Superconducting Magnet and all support systems

– Cradle/Carriage, Inner Detector Supports

– Magnet Steel: yoke and pole tips

– Infrastructure upgrades

– Installation

• University and all Lab groups all participate in commissioning step

• BNL Organizations include Physics, Collider-Accelerator, Instrumentation, and Superconducting Magnet


sPHENIX Project Management• sPHENIX Project Management at BNL, staffed by BNL

– Includes Project Director, Project Manager, Project Scientist, Project Engineer, Resource Coordinator, ES&H rep, QA rep, Chief Mechanical Engineer, Project Controls Manager and staff, Integration

– Significant experience – core of scientific and engineering group that built PHENIX and operated it for 16 years; project controls group experienced on several other completed DOE-413-type projects

• L2 Managers/CAMs: 2 University-based, 11 BNL staff, 1 LANL staff

• L3 Managers: 11 University-based, 23 BNL Staff, 3 LANL/LBNL staff

• Engineering and design staff largely at BNL– Draws on staff in Physics Dept., Collider-Accelerator Dept., Superconducting Magnet Div.

and Instrumentation Div.

– Major contribution from Electrical Engineering/Design staff at Columbia/Nevis Lab on the Calorimeter Digitizers and MBD FEE


sPHENIX Labor Resources• Engineering and Designer effort for new detectors largely at BNL

• Technician staff for new detector construction largely at Universities

– TPC: Stony Brook for main chamber; Wayne State/Vanderbilt/Weizmann for readout chamber construction

– EMCal: Illinois for Blocks; Michigan for optical sensor testing

– HCal: Iowa State for sector construction; Georgia State for scintillator tile preparation

– All the above efforts make use of many kilo-hours of student labor

– INTT at RIKEN; MVTX at LBNL/LANL/MIT for ladder and carbon-fiber work

– Assembly of heavy items, specifically EMCal Module and Sector construction and HCal sector handling, to be done at BNL by BNL staff

– Testing of assembled units, including testing with final FEE, largely performed by University groups with observation/participation by BNL research staff

– Over 50 person-years of University staff in the labor total9sPHENIX Director’s ReviewApril 9-11, 2019

sPHENIX Labor Resources• Engineering and Design for INTT at RIKEN; Engineering and Design for MVTX at

LBNL, LANL, MIT

• Engineering and Design for new Electronics, DAQ, Triggering and Timing systems largely at BNL, with notable contribution from Columbia/Nevis Lab on Calorimeter Digitizers and MBD FEE

• Electronics, DAQ, Triggering and Timing construction largely depends on commercial vendors for components, boards and assembly thereof

– Testing largely done at BNL by technical staff under engineer’s supervision

– Testing of mating to detector also involves staff that constructed detector

– Columbia/Nevis Labs supplies testing staff for, and performs tests on, all those items which they are supplying to sPHENIX


sPHENIX Labor Resources• Firmware (plenty of that) relies on BNL and University professional and

research staff working in collaboration

• Mechanical and electrical engineering and design centered at BNL for Superconducting magnet, Cradle/Carriage, Magnet steel, Infrastructure upgrades and Installation

• Installation phase is led by BNL staff (Physics, C-AD)

• Cabling and later commissioning work depends heavily upon contributions from University staff as well as staff from other Laboratories: RIKEN/LANL/LBNL


sPHENIX Internal Reviews• All construction, be it detector, electronics, DAQ, trigger, timing or support

services, must undergo a series of reviews

– Design: conceptual, intermediate, final as needed

– Performance evaluation after prototyping steps (1 or more)

– Procurement readiness

– ESHQ check

• A series of reviews of the conceptual designs has been held

– Reports on the Indico sites of the various reviews

– Reviews of detectors, detector electronics and DAQ/trigger/timing have been held

– These outputs informed construction of the Resource Loaded Schedule, in particular prototyping and testing that should be done prior to proceeding with Production effort


sPHENIX Prototyping• All detector construction proceeds through a series of prototypes

• This is a major component of the effort to identify and remove risks associated with building a series of new detectors and electronics and getting it all to work together

– These include in-beam tests (multiple such tests for the calorimeters)

– These include coupling of detectors and Front End Electronics

– These will include prototyping and testing of the one custom Application Specific Integrated Circuit (ASIC) for the Time Projection Chamber (the SAMPA ASIC)

– These include coupling of FEE and Digitizers to elements of the DAQ system

• The DAQ makes use of the existing and long-used Data-Collection Modules-2 from PHENIX

– Timing and Trigger prototyping is underway


sPHENIX QA Plans• All construction, be it detector, electronics, DAQ, trigger, timing or support

services, must create and submit a QA plan

• sPHENIX has created a master QA plan

– C. Gortakowski’s talk, Project Management Breakout Session

• Detector and electronics efforts to submit a Detector-Specific QA Plan

– Address materials qualifications and inspections, procedures to be followed, tests during construction, acceptance testing after construction, qualification and training of personnel, travelers, record-keeping, submission of final records to BNL

– First such plan has been prepared and accepted for EMCal Block production at U. Illinois, as that effort is on the critical path

– Second such plan submitted by Georgia State U for HCal Scintillating Tile testing

– Next up from U Michigan for SiPM testing14sPHENIX Director’s ReviewApril 9-11, 2019

sPHENIX MOAs• The sPHENIX Project plans for effort by personnel from the Physics Department, the

Collider-Accelerator Department, the Superconducting Magnet Division, and the Instrumentation Division at BNL

• Memoranda of Agreement between sPHENIX and these organizations address the activities and personnel involved, and the amounts and duration of the planned effort

• Four of five are complete:

– Physics

– C-AD for Magnet and Carriage/Cradle

– Instrumentation for TPC and DAQ

– SMD for Engineering/FEA

• Fifth between C-AD and sPHENIX for Infrastructure improvements in draft


WBS Schedule Construction• A Resource Loaded Schedule has been developed for sPHENIX

• All tasks have resources, durations & links determined

• MIE (1.x) and Silicon (3.x) Level 2 WBS are largely self-contained and consist of a prototype, preproduction prototype and fabrication sequence. Key predecessor links are to funding authorizations. These WBS typically end on a completion milestone and a link out to Installation (WBS 2.05)

• OHCal (WBS 1.04) necessarily has a dependency on delivery schedule for the Barrel Flux Return Steel (WBS 2.03.03)

• Infrastructure/Facilities (2.x) for SC Magnet (WBS 2.02), Cradle/Carriage and Steel fabrications(WBS 2.03) have major design/engineering/review stages followed by Procurement Readiness Review and bid tenders

– Three tenders are above $1M, with one awarded

– Several tenders above $100K16sPHENIX Director’s ReviewApril 9-11, 2019

WBS Schedule Construction• Critical path for (1.x) is EMCal construction (WBS 1.03) and Calorimeter

Electronics (WBS 1.05) as expected due to:

– complexity and high element count (over 5000) of the EMCal (WBS 1.03)

– number of photo-sensors (over 80,000 SiPMs) to be mounted to circuit cards (WBS 1.05)

– Number of different types of circuit boards which must interoperate for Calorimeter Electronics (WBS 1.05.02)

• Critical path for (2.x) is Cradle/Carriage

– This is central to installation sequence and must be placed first on the existing track system in Bldg. 1008

• All detectors (1.x, 3.x) have hand-offs to Installation (WBS 2.05)


Critical Path(1.x) – Electronics& EMCal

18

Calorimeter Electronics Sector 0 Prototype

Calorimeter Electronics Production

Calorimeter Electronics Sector 1-12 Prototype

= Review/Hold Point

EMCal Production Fabrication/

Assembly/Testing

Schedule Contingency

Critical Path(2.x) – Cradle & Installation

19

Cradle/Carriage Assembly

Cradle/Carriage Production

IHCal and EMCal also Install now


SC Magnet Install

OHCal – 2

OHCal – 1

Platforms

Magnet Infrastr.

Magnet Comm.

Critical Path(2.x) – Cradle & Installation

20

TPC Install

Schedule Contingency


INTT/MVTX Install

MBD Install / INTT&MVTX

Commissioning

Full sPHENIX Commissioning

Beampipe Install

sPHENIX Large Procurements• Four Long Lead Procurements have been identified

– Three are associated with the EMCal and associated electronics production• Scintillating Fibers for EMCal - placed

• Tungsten (W) Powder for EMCal – vendors identified; down-select review soon; “commodity” item

• Silicon PhotoMultipliers (SiPMs) to read out the EMCal - placed

– Fourth involves a non-US vendor which needs to scale up rate of production to meet the sPHENIX schedule• Scintillating Tiles for Outer Hadron Calorimeter (OHCal) – placed

• Prototyping has been done for all four items and specifications set

• Infrastructure/Facilities has three major >$1M procurements

– Barrel Steel – placed ($5.6M) – last sector delivery planned August 2019

– Cradle/Carriage – PRR planned for November 2019

– Helium Interface for cryogenics – PRR planned for November 201921sPHENIX Director’s ReviewApril 9-11, 2019

Sample Basis of Estimate• Sample of Labor estimate sheet for WBS 1.02.01.05 TPC v2 Modules


LaborWBS Activity Description Justification Category Hours TOTAL

1.02.01.05.02 117800 Assemble TPC v2 Module Grid Similar to PHENIX HBD. TECH 80 80

TECH 40

STUD 40

PROF 8

TECH 8

SCI 8

PROF 8

TECH 8

SCI 8

PROF 80

STUD 80

SCI 80

PROF 40

SCI 160

STUD 160

PROF 80

STUD 160

SCI 80

1.02.01.05.03 118600 Frame TPC v2a Module GEMs Similar to PHENIX HBD. TECH 40 40

1.02.01.05.02 117300 Procure TPC v2 Module Frames Vendor purchase TECH 24 24

1.02.01.05.02 117600 Procure TPC v2 Module Grid Parts Vendor purchase TECH 24 24

1.02.01.05.02 117000 Procure TPC v2 Module Strongback SBU shop TECH 16 16

1.02.01.05.03 118300 Procure TPC v2a Module GEMs Similar to EIC R&D from eRD6 TECH 32 32

1.02.01.05.03 118000 Procure TPC v2a Module Padplane Similar to EIC R&D from eRD7 TECH 16 16

TECH 40

STUD 40

TECH 40

STUD 40

TECH 160

STUD 160

Grand Total 1760 1760

1.02.01.05.03 Assemble TPC v2a Module Prototype Similar to PHENIX HBD.

1.02.01.05.02 Design TPC v2 Module Frames Copied from v1 field cage design.

118800

117200

1.02.01.05.02 Design TPC v2 Module Grid

1.02.01.05.02 Design TPC v2 Module Strongback Copied from v1 field cage design.

Copied from v1 field cage design.117500

116900

1.02.01.05.03 Design TPC v2a Module GEMs Copied from v1 field cage design.

1.02.01.05.03 Design TPC v2a Module Padplane Similar to EIC R&D from eRD6

118200

117900

1.02.01.05.03 Test TPC v2a Module Prototype Similar to PHENIX HBD.

1.02.01.05.03 Test TPC v2a Module Framed GEMs Similar to PHENIX HBD.

1.02.01.05.03 Test TPC v2a Module GEMS Similar to PHENIX HBD.

118700

118500

118900

320

80

80

320

80

24

24

240

360

sPHENIX Risk Registry• Risk Registry is developed and populated in continuing consultation with L2,

L3 and engineering staff.

– Risk evaluation includes consideration of dollar cost risk and schedule risk

– Probability assigned after discussion among PM, L2 and L3 managers

• Project plans to update and review Elements on regular basis

– Monthly review is foreseen, coupled with monthly status, cost and schedule reporting


Recommendation Tracking System (RTS)• Some 13 major reviews with reviewers external to sPHENIX held over past 4

years– Subjects covered TPC, EMCal, HCal, Calorimeter Electronics, Trigger, Magnet (multiple),

OHCal Steel, Carriage/Cradle, Cost & Schedule, and Readiness for CD-1/3A

• Three major reviews in past year:– Cost & Schedule February 13-14, 2018

– Director’s CD-1/3A Readiness March 6-8, 2018

– OPA CD-1/3A May 23-25, 2018

– Some 57 recommendations, 0 open, 1 In-Progress (MoA’s), 56 Closed

• Formal Recommendation Tracking System set up– Spreadsheet in DocDB

– Manager assigned to each Recommendation,

– Status, Response & any attachment, Date of Update

• Tracking all Recommendations using this RTS24sPHENIX Director’s ReviewApril 9-11, 2019

Review Charge Element #2• “Is the scope of each project sufficiently defined to support approval of PD-2

and PD-3? Are the specifications, designs and execution plans likely to meet the technical performance requirements? Are the interfaces with other projects properly understood and documented?”

• Comments to Charge Element– All detectors are based on known technologies with some novel configuration

– Revised ASIC for TPC (the SAMPA chip) is a modest alteration of a working device

– TPC, EMCal, and HCal have been through one or more round of prototyping (WBS 1.02, 1.03, 1.04) –results to date are positive – “we could build this design”

• includes tests of operating prototypes in a test beam of particles at FNAL

• gives confidence the designs are mature, and time-to-fabricate/procure the needed parts has been experienced

– Calorimeter Electronics (WBS 1.05) and DAQ & Trigger (WBS 1.06) are based on PHENIX electronics

• built by same groups; WBS 1.05 boards used successfully in test beam run; BOE include detailed Bills of Materials

• has similar level of channel count, complexity, and required performance as PHENIX

– Min Bias Detector (WBS 1.07) is an existing detector being refurbished with new electronics


Review Charge Element #2• “Is the scope of each project sufficiently defined to support approval of PD-2

and PD-3? Are the specifications, designs and execution plans likely to meet the technical performance requirements? Are the interfaces with other projects properly understood and documented?”

• Further Comments to Charge Element– INTT sensing elements employs a familiar design for silicon-strips (WBS 3.01)

– INTT readout electronics re-uses the successful FEE from PHENIX FVTX

– MVTX employs MAPS ladders of proven ALICE design; carbon-fiber supports from proven shop (WBS 3.02)

– INTT and MVTX have tested prototype ladders in-beam

– SC Magnet has been tested to full current; cryo interfaces build on RHIC experience

– Cradle/Carriage inherits from similar designs and follows similar analysis methodology

– Barrel Steel sectors are arriving and have been test-fit with final-sized active OHCal elements

– Interface control documents have been prepared, reviewed and recorded

– Full 3-D models of sPHENIX are being built and examined for installation and fit-up

– Cables, services, LV & bias HV, power routing are defined


Review Charge Element #4• “Is there a capable team in place to effectively manage the two projects as

defined in the approved baselines for each? Does the management team have the resources and management tools necessary for the size and complexity of this overall effort?”

• Comments to Charge Element– Management team is staffed and experienced on projects of similar scale

– L2/CAM and L3 managers identified and at work leading current prototyping/fabrication efforts

– Risk Registry is developed and populated in continuing consultation with L2, L3 and engineering staff

– Plans for Procurement are developed and being used. EVMS tracking of items has commenced.

– Prototyping series for various detectors have been defined and are being executed, generating feedback on resources required and schedule achievable

– Management protocols (design review, ESHQ reviews, procurement readiness reviews, QA plans, MoA, etc.) have been put in place and are being used.


Review Charge Element #4• “Is there a capable team in place to effectively manage the two projects as

defined in the approved baselines for each? Does the management team have the resources and management tools necessary for the size and complexity of this overall effort?”

• Further Comments to Charge Element

– We believe a comprehensive suite of management tools are in place and now being exercised• Reviews

• Prototyping series, initial short series-production (“PreProduction”)

• QA plans, including specific ones for individual detectors

• MoA

• Risk Registry and plans to review/update/retire risks on regular basis

• Recommendation Tracking System

• Resource Loaded Schedule (“P6”)– exercising this to record progress has begun (EVMS)


Review Charge Element #5• “Are the cost and schedule estimates for each project credible and adequately

detailed to establish their respective baselines? Do they include adequate scope, cost and schedule contingency?”

• Comments to Charge Element

– All subsystems have developed a BOE and a RLS for their area

– Activity descriptions are broken down to “$1K/1-day” range, as appropriate

– These have been reviewed and defended internally

– Control-point reviews are built in to the schedule, and require results from prototyping, chain-tests, FEA analyses, and ESHQ questions as appropriate

– Links among the subsystems have been added to develop the overall schedule, in particular the Installation schedule

– Estimate uncertainties are assigned to individual activities and estimates

– Risk Registry is used to assign further cost and schedule contingency29sPHENIX Director’s ReviewApril 9-11, 2019

Review Charge Element #5• “Are the cost and schedule estimates for each project credible and adequately

detailed to establish their respective baselines? Do they include adequate scope, cost and schedule contingency?”

• Further Comments to Charge Element– Four Long Lead Procurements have been identified

– Three are associated with the EMCal and associated electronics production, which form the Critical path

• Scintillating Fibers for EMCal - placed

• Tungsten (W) Powder for EMCal – ready for down-select and procurement

• Silicon PhotoMultipliers (SiPMs) to readout the EMCal - placed

– Fourth involves a non-US vendor which needs to scale up rate of production to meet sPHENIX’s schedule

• Scintillating Tiles for Outer Hadron Calorimeter (OHCal) - placed

– Cost experience with these four suggests no net contingency draw will be needed

– Schedule experience with these four shows the vendor delivery schedules can be accommodated and, as a result, will not drive the Critical Path


Issues and Concerns• Changing shaping time of SAMPA chip (WBS 1.02.05.03) - Schedule

– Revision of working chip, but to speed it up, thus noise/stability a concern

– Any ASIC has yield risk, factory “end-of-life-on-process” risk

• Vendors – Impacts to Schedule, or Cost and Schedule if replace vendor

– Four Long-Lead Procurements are in good shape

– Carriage/Cradle and Helium Interface require bid cycles

• Placing Major Contracts - Schedule

– Factory start-up depends on this, particularly W-powder and Scintillating Fibers (SciFi) for EMCal – good experience to date

– Overall installation critical path requires the two largest remaining orders not yet under contract – Cradle/Carriage and Helium Interface


Summary

• We have the team, plans and procedures in place to build sPHENIX

• We have a RLS that lays out a feasible schedule to do this

• We have a plan that respects the proposed Cost Range

• We have the tools and methods in place to manage this process

• We are proceeding with the Long Lead Procurements

• The Project is ready to proceed to PD-2 and PD-3


Back Up


WBS Activity Lines• WBS is enumerated (usually at Level 4) as a set of specific Activities:

– Resources (labor and/or materials) assigned

– Duration assigned

– Predecessor and successor links made

– Tags attached for e.g. milestone status, funding source, and other fields.

• Estimating and scheduling works from these Activities

• The Activity numbering is typically “S” followed by a 6-digit sequential number

– Each Activity has entries identifying its WBS, manager, estimator, and other identifying markers


Documents

sPHENIX Director’s Review Project Management and Status · 2019. 4. 9. · sPHENIX Project Scope (1) • Three large detectors, with their support services (LV, HV, gas) –Time