Mission Operations Cost Estimation Tool (MOCET)

Update

Marc Hayhurst, Brian Wood, Shirin EftekharzadehThe Aerospace Corporation

Cindy Daniels, Lissa Jordin, Washito Sasamoto, Waldo RodriguezNASA Langley Research CenterScience Office for Mission Assessments (SOMA)

NASA Cost Symposium 2016

Glenn Research Center, Cleveland Ohio

24 August 2016

© 2016 The Aerospace Corporation

Aerospace acknowledges NASA sponsorship under Contract NNL11AA01B Task Order NNL16AA49T.

2

Topics

• MOCET Overview & Status

Updates in MOCET v1.1

User Community

Common User Questions

Summary & Next Steps

•

•

•

•

.

4

•

MOCET’s First Year and Beyond

In August of FY15 MOCET was presented at the NASA Cost Symposium 2015

MOCET Excel tool was formally released as v1.0 to the NASA community at end of FY15

− Tool became available on the One NASA Cost Engineering (ONCE) Model Portal

− Approximately 30+ unique users downloads via ONCE to date

− MOCET has been used in Full Mission AO competitions since FY15

MOCET v1.1 was released in March of FY16

− More data from recent missions included in the tool

− More rigorous statistical methods applied to the model

First User’s Workshop was held virtually at the beginning of June FY16

NASA Cost Symposium 2016 Update and demo session

•

•

•

•

• Second User’s Workshop to be held virtually November FY17

More planned updates with new mission data and user’s feedback in FY17

SOMA has initiated the process to release MOCET on the NASA Software Catalog website at

https://software.nasa.gov

– No specific release date is known at this time.

– If you want to be notified when MOCET is available on the NASA Software Catalog website, send an email to

MOCET@aero.org

•

•

NOW

5

What Has Changed in v1.1?• Most CERs have been reformulated to produce better results

– Updated equations and performance comparison in following charts

Overall look and feel of the Excel tool and documentation remains the same

New Mission Data Added

– LADEE, MAVEN, OCO-2, GPM, New Horizons (Pluto Flyby)

More Comprehensive Statistical Process Applied

– Graphical analysis - Selecting independent variables, outlier detection

– Goodness of fit statistics - measures discrepancy between observed and expected values

– Analysis of residuals - Determines if linear regression is appropriate

– Significance testing - Determines if variables are necessary for the equation or if it is over-fitted

– Measures of performance -Test metrics to compare various models against each other

– Cross validation - Detects overfitting, compares regression forms, generalizes performance on an

independent data set, prevents throwing away data

Fewer inputs required from equation reformulation & significance testing

– No more enumerated variables

– Some inputs have changed, but follow similar assumptions

• For example, Number of Instruments was used in v1.0 which was favored for being simple and objective

• This has been replaced by Payload Mass in one CER and Peak Downlink in another for v1.1

• These parameters all attempt to quantify the amount of science/data analysis done in Phase E but those

used in v1.1 do a more effective job

•

•

•

•

6

MOCET Mission Types and Database Overview

Mission/CER Type

Planetary

Earth Science

Explorers

Near Earth Discovery Helio-Astro

Program

Discovery

Mars Scout

Robotic Lunar Exploration

New Frontiers

Mars Exploration

Earth System Science Pathfinder (ESSP)

Earth Systematic Missions (ESM)

Mission of Opportunity (MO)

Small Explorers (SMEX)

Medium Explorers (MIDEX)

Discovery

Solar Terrestrial Probes (STP)

Living With a Star (LWS)

Cosmic Origins

Physics of the Cosmos

Missions

MESSENGER, Dawn, Stardust, Deep Impact, GRAIL, NEAR

Phoenix

LRO, LADEE

Juno (Jupiter Ops), New Horizons (Pluto Flyby), OSIRIS-REx

MRO, Odyssey, MER, MSL, MAVEN

GRACE, CloudSat, CALIPSO, Aquarius, OCO-2, CYGNSS

Aqua, Aura, Terra, Jason-1, OSTM, ICESat, GPM, SMAP, Jason-

Suzaku (ASTRO-E2), TWINS, CINDI

NuSTAR, IRIS, IBEX, AIM, GALEX, RHESSI

THEMIS, Swift, WISE

Genesis, Kepler

STEREO, TIMED, MMS

RBSP, SDO

Spitzer

Fermi, Chandra

3

New Data in version 1.1 Potential New Data for version 1.X

– – –

Data Sources Utilized and Consulted

NASA SAP Business

Warehouse

• Monthly Expenditures

Monthly Flight Project Reviews (FPR)

and Monthly Status Reviews (MSR)

• Monthly Expenditures, Mission Events

& Schedule

NASA Cost Analysis Data

Requirements (CADRe)

• Project development costs used

to assign mission class

7

MOCET v1.1 CERs UpdatesPlanetary

CER NameNew Data New Eq. Form Additional Evaluation No Change Comments

Cruise CER X X XSeparated into two CERs: Checkout and Nominal/Quiescent

Approach/Orbit Insertion/Reduction CER X X X

Orbital Operations CER X X X

Flyby/Encounter CER X X

Approach EDL CER X No new/limited data. Not able to improve

First Landed Month CER X No new/limited data. Not able to improve

Landed Prime Operations CER X No new/limited data. Not able to improve

Data/Sample Analysis & Archiving CER X No new/limited data. Not able to improve

Extended Orbital Ops CER X Not primary focus

Extended Landed Ops CER X Not primary focus

Earth/Near-Earth Orbiting

CER NameNew Data New Eq. Form Additional Evaluation No Change Comments

Earth/Near-Earth Orbiting Checkout CER X X X

Earth Science Prime Operations CER X X X

Explorer Prime Operations CER X X

Near Earth Discovery Helio Astro Prime Operations CER X X

Updated in version 1.1

8

MOCET Cost Estimating Relationships (CERs)Planetary

CER Name

Nominal/Quiescent

Cruise CER

Checkout Cruise CER

Orbital Operations CER

Approach/Orbit

Insertion/Reduction

CER

Flyby/Encounter

Approach EDL

First Landed Month

Landed Prime

Operations

Data/Sample Analysis &

Archiving

Extended Orbital Ops

CER

Extended Landed Ops

CER

Equation Inputs

IL = Is the mission class Large? (0 No, 1 Yes)

IQ = Quiescent Ops (0 No, 1 Yes)

IL = Is the mission class Large? (0 No, 1 Yes)

EN = Does this checkout period require Enhanced Navigation? (0 No, 1

Yes)

PM = Payload Mass of NASA funded instruments

IM = Is Mars the operational environment? (0 No, 1 Yes)

PC = Estimated Preceding Cruise Phase Cost

PC = Estimated Preceding Cruise Phase Cost

PC = Estimated Preceding Cruise Phase Cost

PC = Estimated Preceding Cruise Phase Cost

PC = Estimated Preceding Prime Collection Phase Cost

P = Prime mission average monthly cost,

N = Number of times the mission is being extended or renewed (1 to N)

P = Prime mission average monthly cost,

N = Number of times the mission is being extended or renewed (1 to N)

No.

Inputs

2

2

2

1

1

1

1

1

1

2

2

N

6

7

7

6

5

3

3

3

3

15

11

Adj. R^2

or

r^2

0.87

0.97

0.86

0.99

0.97

0.97

0.97

0.70

0.99

0.94

0.99

SEE (%)

or

SE

(RSS)

0.21

0.17

0.22

0.11

0.10

13.3%

13.1%

31.0%

5.6%

7.2%

3.1%

Y = Average Monthly cost for phase (FY13)

N = Number of Data points

Adj. R^2 = Adjusted Coefficient of Determination

r^2 = Correlation Coefficient

SEE = Standard Error of the Estimate (Zero

Percentage Bias Minimum Percentage Error)

SE (RSS) = Standard Error (Root Sum of Squares)

Updated in version 1.1

8884.1* 0400.1* 1210.1* 7605.0* 7509.0* IMPM *4627.0*0133.03 PC*8836.017 IQIL *7805.0*8581.02 ENIL *2605.2*5666.091 YY PP**((00..209236**llnn((NN))00..61288443))

9

MOCET Cost Estimating Relationships (CERs)Earth Science, Explorers, and Near Earth Discovery Helio Astro

CER Name

Earth Science,

Explorers, and Near

Earth Discovery

Helio Astro

Checkout CER

Earth Science

Prime Operations

CER

Explorer Prime

Operations CER

Near Earth

Discovery Helio

Astro Prime

Operations

Equation Inputs

P = Prime mission operations average monthly cost

PD = Peak Downlink data rate from the spacecraft in megabits per

second

IFL = Is this a Large or Flagship mission class mission? (0 No, 1 Yes)

IMe = Is this a Medium mission class mission? (0 No, 1 Yes)

IMi = Is this a Micro mission class mission? (0 No, 1 Yes)

IA = Is this an Astrophysics mission? (0 No, 1 Yes)

IF = Is this a Flagship mission class mission? (0 No, 1 Yes)

No.

Inputs

1

2

3

2

N

13

13

12

Adj.

R^2

or

r^2

0.87

0.97

0.94

SEE (%)

or

SE

(RSS)

0.59

0.19

0.06

Y = Average Monthly cost for phase (FY13)

N = Number of Data points

Adj. R^2 = Adjusted Coefficient of Determination

r^2 = Correlation Coefficient

SEE = Standard Error of the Estimate (Zero

Percentage Bias Minimum Percentage Error)

SE (RSS) = Standard Error (Root Sum of Squares) 9741.1* IFIL *194.5*5894.0337 IFLPD *5782.1*0083.0204 IAIMiIMe *1204.0*3018.0*3135.0677 Updated in version 1.1

10

Model Performance

• The model was compared against actual mission costs to gauge performance

• Standard Deviation for all missions was found to be 20% (from 25% in v1.0)

• Planetary: 27% (v1.0 = 32%)

• Earth/Near Earth Orbiting: 16% (v1.0 = 22%)

v1.1

v1.0

11

MOCET User Community

• Since last years cost symposium and the release of v1.0 we have worked to engage the user

community

We have also briefed key personnel in the various NASA program offices for awareness of how

projects that fall under their portfolios will be evaluated

– Programs can also use MOCET to size program budgets for Phase E

As of late June 2016, approximately 30+ unique users have downloaded MOCET from ONCE

– A survey conducted by CAD (SID) in March 2016 showed 27 MOCET users

Expect user community to grow with proposal cycles and future release to contractors and

universities via the NASA Software Catalog at https://software.nasa.gov

User workshops are opportunities for users to learn more about MOCET and voice feedback that

can help steer future releases of MOCET

– First ever workshop was held virtually in June of 2016

– Next workshop will be held virtually in November of 2016

A feedback form is available with the tool for users to fill out

– We have already received a number of responses that have helped us make improvements to the tool this FY

•

•

•

•

•

12

Common User Questions

• What does MOCET estimate?

– MOCET estimates the total cost of Phase E

• WBS 1.0 Project Management

• WBS 2.0 System Engineering

• WBS 3.0 Mission Assurance

• WBS 4.0 Science/Data Analysis

• WBS 7.0 Mission Operations

• WBS 9.0 GDS/Data Analysis

• WBS 11.0 Education & Public Outreach

• Also any Phase E costs from WBS 5.0 Payload, 6.0 Spacecraft, 10.0 System I&T

– Only cost removed from MOCET estimates are LV related from KSC (WBS 8.0)

– This also does not include Deep Space Network (DSN) costs

• How was the data normalized?

– MOCET data is monthly data from SAP at the top level

– The only normalization is to remove any launch vehicle costs associated with KSC

– This means any post launch software development is included in MOCET

• The tool also has a Frequently Asked Questions (FAQ) section that should cover most questions

Email: MOCET@aero.org

anytime with questions

13

Summary & Next Steps

• New data has been added and CERs have been reformulated for MOCET v1.1

MOCET v1.1 has seen improvement over v1.0

The look and feel of the Excel tool is generally the same as v1.0

MOCET will continue to be periodically updated with new mission data

The model is currently being used by evaluators and proposers alike

We will continue to engage and grow the user community

•

•

•

•

•

Download MOCET

ONCE Model Portal https://oncedata.msfc.nasa.gov

Soon available external to NASA via https://software.nasa.gov

For More Information

Demo session Wednesday afternoon (see agenda for time and room)

Next user workshop in November 2016

Email: MOCET@aero.org