Reserves on Schedule to Go (STG) Based on Historical Data

Bob Bitten, Bob Kellogg, Marc Hayhurst, Shirin Eftekharzadeh The Aerospace Corporation

Voleak Roeum NASA Headquarters

2014 NASA Cost Symposium 12-14 August 2014

2

Outline

• Introduction/Background

Methodology

Results from PDR, CDR, SIR, and PSR

Recommendations & Summary

•

•

•

3

Introduction

• NASA Center guidance requires that a mission hold Funded ScheduleReserve (FSR) corresponding to the remaining Schedule to Go (STG) atan increasing rate over the development lifecycle

The rates of required FSR may not be adequate given that many projects have experienced significant schedule growth

This study looks at the project-held FSR and actual schedule growth for a variety of NASA missions at the Preliminary Design Review (PDR), Critical Design Review (CDR), Systems Integration Review (SIR), and Pre-Ship Review (PSR)

The data can then be used as guidance for adjusting project FSR requirements and for establishing additional reserves at the center or HQ level for future missions

•

•

•

NASA Project Lifecycle

• BackgroundProject

Phase

Phase A:

Conceptual

Design

Phase B:

Preliminary

Design

Phase C:

Final Design

&

Fabrication

Phase D:

System

Assembly,

Integration &

Test, Launch &

Checkout

Phase E:

Operations

Milestones

Reviews

KDP-B KDP-C KDP-D KDP-E

Launch

PDR CDR SIR PSR LRD

• Definitions:– KDP = Key Decision Point = Transition from one Phase to the next

– PDR = Preliminary Design Review

– CDR = Critical Design Review

– SIR = Systems Integration Review

– PSR = Pre-Ship Review

– LRD = Launch Readiness Date

4

5

Reserve Tracking Assumptions

• JPL and GSFC Guidelines specify funded schedule reserves in months forPhases C & D

Milestone

PDR (KDP-C)

to

SIR (KDP-D)

to

Ship

to

SIR (KDP-D) Ship Launch

JPL & GSFC Reserve Guidelines Schedule Margin Rate 1 month/year 2 month/year 1 week/month

• Most projects track Funded Schedule Reserve (FSR) in weeks or days– 1 Month = 4 weeks

– 1 Week = 5 working days

Phase Months/Year Weeks/Year Days/Year

KDP-C to KDP-D 1 4 20

KDP-D to Ship 2 8 40

Ship to Launch 3 12 60

• Weeks used in this analysis for consistency across all projects

6

Example Project Lifecycle

Total

• Funded Schedule Reserve (FSR) = No planned activities, but funded schedule• FSR Rate = FSR/(Planned Activity + FSR) = FSR / (P + FSR)

KDP-B KDP-C KDP-D KDP-E

Launch

PDR CDR SIR PSR LRD

Ship

Critical Path

P FSR P P

Planned

Activity

Funded

Schedule

Reserve

Schedule

FSR FSR

Phase ATP to PDR to SIR SIR to PSR PSR to Launch Total

PDR

Example Duration 0.75 2.0 1.0 0.25 4.0 (years)

Guideline FSR Rate 0 4 weeks/year 8 weeks/year 12 weeks/year

Guideline FSR 0 8 8 3 19 (weeks)

7

Outline

• Introduction/Background

• Methodology

• Results from PDR, CDR, SIR, and PSR

• Recommendations & Summary

Large Diversity of Missions Included in Analysis • Data collected from 37 science missions providing a representative mix

ct

R

MissionLaunch

Date ThemeMission

Type Lead Org.

AIM Apr-07 Helio CAT 3 OtherCloudsat Apr-06 Earth CAT 2 JPLDAWN Sep-07 Planetary CAT 2 JPLDeep Impa Jan-05 Planetary CAT 2 JPLEO-1 Nov-00 Earth CAT 2 GSFCGALEX Apr-03 Astro CAT 3 JPLGENESIS Aug-01 Planetary CAT 2 JPLGLAST Jun-08 Astro CAT 2 GSFCGRAIL Sep-11 Planetary CAT 2 JPLIBEX Oct-08 Helio CAT 3 OtherICESAT Jan-03 Earth CAT 2 GSFCKepler Mar-09 Astro CAT 2 JPLLRO Jun-09 Planetary CAT 2 GSFCMER Jul-03 Planetary CAT 1 JPLMESSENGE Aug-04 Planetary CAT 2 APLMRO Aug-05 Planetary CAT 1 JPLNew Horiz Jan-06 Planetary CAT 1 APLOCO Feb-09 Earth CAT 2 JPLPHOENIX Aug-07 Planetary CAT 2 JPL

MissionLaunch

Date ThemeMission

Type Lead Org.

RHESSI Feb-02 Helio CAT 3 OtherSDO Feb-10 Helio CAT 1 GSFCSpitzer Aug-03 Astro CAT 1 JPLStardust Feb-99 Planetary CAT 2 JPLSTEREO Oct-06 Helio CAT 2 GSFCSwift Apr-04 Astro CAT 2 GSFCTerra Dec-99 Earth CAT 1 GSFCWISE Jun-09 Astro CAT 2 JPLGLORY Mar-11 Earth CAT 2 GSFCGPM Feb-14 Earth CAT 2 GSFCIRIS Jun-13 Helio CAT 3 LMJuno Aug-11 Planetary CAT 1 JPLLADEE Sep-13 Planetary CAT 2 ARCLDCM Feb-13 Earth CAT 2 GSFCMAVEN Nov-13 Planetary CAT 3 GSFCMSL Nov-11 Planetary CAT 1 JPLRBSP Aug-12 Helio CAT 2 APLNuSTAR Jun-12 Astro CAT 3 JPL

• Inclu– 15

des mix of Science ThemesPlanetary, 7 Astrophysics, 8 Earth Science and 7 Heliophysics

• Includes mix of different sized missions based on NASA 7120.5E Mission Category– 8 Category 1 (CAT 1) missions with Life Cycle Cost (LCC) > $1B FY12$

– 22 Category 2 (CAT 2) missions with LCC >$250M but < $1B FY12$

– 7 Category 3 (CAT 3) missions with LCC < $250M FY12$

Study Definitions – Project Actual Margin (FSR) = Actual historical margin held by project at milestone

– Actual Required Margin = Project Actual Margin + Schedule Growth = FSR + G

– Actual Schedule Margin Rate =

• Actual Required Margin/(Planned Activity + Project Actual Margin) = (FSR + G) / (P + FSR)

Planned Duration

Actual Duration

P

G

FSR

– External Events: factors resulting increased schedule growth which were outside of the control of

NASA and/or the project and unreasonable to anticipate and plan for

• NASA External: Contractor closes and relocates facility; labor strike; Hurricane delays launch• Project External (Including above): center/headquarters directed launch date move; funding cut;

stand down

– Actual Required Margin less External Events =

• Actual Required Margin – External Event Duration = (FSR + G) - E– Actual Required Margin Rate less External =

• Actual Required Margin less External Events/(Planned Activity + Project Actual Margin)• ((FSR + G) – E) / (P + FSR)

9

Planned Duration

Actual Duration

P

E

FSR

G

10

Outline

• Introduction/Background

• Methodology

• Results from PDR, CDR, SIR, and PSR

• Recommendations & Summary

11

Average All Causes Average NASA External Removed

Average Project External Removed Current FSR Guidelines

Results of Historical Mission Examination FSR Rates - All Missions

13.5

20.5

33.9

42.7

10.6

14.9

28.4

19.2

10.5

14.7

24.3

13.0

4.0 4.0

8.0

12.0

0.0

5.0

10.0

15.0

20.0

25.0

30.0

35.0

40.0

45.0

50.0

PDR to SIR CDR to SIR SIR to PSR PSR to Launch

Wee

ks p

er y

ear

bet

we

en M

ilest

on

e

Schedule Phase

Weeks per year calculations

-Numbers: average weeks per year used between each milestone

-Orange line: unadjusted calculation from the raw schedule data

-Purple line: adjusted calculation removing NASA external events

- Blue line: adjusted calculation removing Project external events

-Black line: current FSR guidelines

12

Average All Causes Average NASA External Removed Average Project External Removed

Current FSR Guidelines Avg. Launched On Schedule

Results of Historical Mission Examination FSR Rates - All Missions

13.5

20.5

33.9

42.7

10.6

14.9

28.4

19.2

10.5

14.7

24.3

13.0

4.0 4.0

8.0

12.0

7.1

12.7 8.812.5

0.0

5.0

10.0

15.0

20.0

25.0

30.0

35.0

40.0

45.0

50.0

PDR to SIR CDR to SIR SIR to PSR PSR to Launch

Wee

ks p

er y

ear

bet

we

en M

ilest

on

e

Schedule Phase

On Schedule Missions

-Green line: average of the 8 missions launched on schedule

-The FSR they budgeted was approximately the actual required FSR

Results of Historical Mission Examination

13.510.6

20.5

14.9

14.7

1

4.0

7.

0.5

4.0

1

12.7

8.0

8.8

PDR to SIR CDR to SIR

Schedu

SIR to PSR PSR to Launch

le Phase

Average Project External RemovedAverage All Causes

Current FSR Guidelines

Average NASA External Removed

Avg. Launched On Schedule

0.0

5.0

10.0

15.0

20.0

25.0

30.0

35.0

40.0

45.0

50.0

Wee

ks p

er y

ear

bet

wee

n M

ilest

on

e 4

192

12.012.5

FSR Rates - All Missions

1 w3.0e

13

PDR and CDR to SIR

-Minimal usage of FSR prior to CDR

-Increase in required FSR at CDR is partly an artifact of non-usage from PDR to CDR

-PDR to SIR: 2 years; 8 wks FSR = 4 wks/yr -CDR to SIR: 1 year; 8 wks FSR = 8 wks/yr

-Most missions had modest schedule growth between PDR and SIR

-Missions that had growth post CDR were mostly attributable to instruments

13.5

33.9

42.7

28.4

19.2

24.3

13.0

8.0

12.08.8

12.5

SIR to PSR PSR to Launch

le Phase

10.6

14.9

10.5

14.7

4.0 4.0

7.1

12.7

0.0

5.0

10.0

15.0

20.0

25.0

30.0

35.0

40.0

45.0

50.0

Wee

ks p

er y

ear

bet

wee

n M

ilest

on

e

PDR to SIR CDR to SIR

Schedu

2d0u.5l

SIR to PSR

-Even with external events removed the I&T phase is the most prominent with largest variance from guidelines

- Expected as the most intense development effort takes place in this phase

-Values near the guidelines were sufficient for missions launched on schedule

-Nearly all missions launched on schedule also had constrained launch windows

Average Project External RemovedAverage All Causes

Current FSR Guidelines

Average NASA External Removed

Avg. Launched On Schedule

Results of Historical Mission Examination FSR Rates - All Missions

14

15

4.000.

SIR to PSR PSR to LaunchPDR to SIR CDR to SIR

Schedule Phase

Average Project External RemovedAverage All Causes

Current FSR Guidelines

Average NASA External Removed

Avg. Launched On Schedule

13.5

20.5

10.6

14.9

10.5

14.7

4.0

7.1

12.7

0

.0

.0

.0

.0

5.

10

15

20

25

30.0

35.0

40.0

45.0

50.0

Wee

ks p

er y

ear

bet

wee

n M

ilest

on

e

19.2

24.3

13.0

8.0

12.08.8

12.5

33.9

42.7

28.4

PSR to Launch

-External events such as launch vehicle

technical delays & range conflicts drive the

needs for FSR from ship to launch

- When these events are removed the

guidelines appear adequate for a nominal

launch processing phase

Results of Historical Mission Examination FSR Rates - All Missions

16

Further Analysis

• Examination of the average can be useful to asses overall trends– However, there is much variation present in the schedule data

– Some missions were affected by “extraordinary” events which could not be planned for

– Data points to the far right can unfairly influence the average

Another approach is to examine a cumulative distribution of the data to identify the most probable outcomes – Distributions were examined for potential FSR guidance consistent with NASA policy

• The 50% confidence level on the curve with Project External events removed issuggested for Project held reserves

• The 70% confidence level on the curve with NASA External events removed issuggested for HQ/Center held reserves

The following slides show the cumulative distributions at the guideline specified milestones (PDR, SIR, PSR) with the 50% and 70% levels highlighted

•

•

Required Schedule Reserve at PDR Guideline: 4 weeks/year

17

4

4

7.213.5

7.1

10.6

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

0 5 10 15 20 25 30 35 40 45 50

Cu

mu

lati

ve P

rob

abili

ty

Weeks Per Year

Required FSR @ PDR

Current FSR Guideline

All Causes

NASA External Removed

Project External Removed

Launched On Schedule

50% w/o Project External

7

70% w/o NASA

External

15

Required Schedule Reserve at SIR Guideline: 8 weeks/year

18

8

8

24.3

33.9

8.8

28.4

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

0 5 10 15 20 25 30 35 40 45 50

Cu

mu

lati

ve P

rob

abili

ty

Weeks Per Year

Required FSR @ SIR

Current FSR Guideline

All Causes

NASA External Removed

Project External Removed

Launched On Schedule

50% w/o Project External

13

70% w/o NASA

External

18

Required Schedule Reserve at PSR Guideline: 12 weeks/year

19

12

12

11.8

42.7

13.4

19.2

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

0 5 10 15 20 25 30 35 40 45 50

Cu

mu

lati

ve P

rob

abili

ty

Weeks Per Year

Required FSR @ PSR

Current FSR Guideline

All Causes

NASA External Removed

Project External Removed

Launched On Schedule12

70% w/o NASA

External

17

50% w/o Project External

20

Outline

• Introduction/Background

• Methodology

• Results from PDR, CDR, SIR, and PSR

• Recommendations & Summary

21

Recommendations

Examination of the cumulative distributions of historical data yielded the following recommended values of FSR given the example project schedule

ATP to PDR PDR to SIR SIR to PSR PSR to Total Total

Launch (Years) (months)

Example Duration (years) 0.75 2.0 1.0 0.25 4.0 48

Phase

Current Guidelines FSR Rate (wks/year) 0 4 8 12 Total Total

(Weeks) (months)

Total FSR (weeks) 0 8 8 3 19 4

Recommended Project FSR at 50% confidence level FSR Rate (wks/year) 0 7 13 12 Total Total

(Weeks) (months)

Total FSR (weeks) 0 14 13 3 30 7

Recommended Center/HQ FSR at 70% confidence level FSR Rate (wks/year) 0 15 18 17 Total Total

(Weeks) (months)

Total FSR (weeks) 0 30 18 4.25 52.25 12

FSR above Proj. (wks) 0 16 5 1.25 22.25 5

•

22

Summary

• This study aimed to assess if the adequacy of the current guidelines forcarrying funded schedule reserve based on schedule-to-go

Historical data from 37 missions were assessed to determine required FSR over the project lifecycles – Average weeks per year assessed for overall trends

– Cumulative distributions examined to establish recommendations for specific

confidence levels

Recommended FSR rates are listed below

•

•Phase PDR to SIR SIR to PSR PSR to Launch

Current Guidelines 4 8 12

Recommended Project FSR

at 50% confidence (weeks per year) 7 13 12

Recommended Additional Center/Headquarters FSR

at 70% confidence (weeks per year) 8 5 5

Recommended Total Project + Center/Headquarters FSR

at 70% confidence (weeks per year) 15 18 17