Embed Size (px)
Citation preview
© 2017 The Aerospace Corporation
Mission Operations Cost
Estimation Tool (MOCET)
FY17 Update
Marc Hayhurst, Brian Wood, Shirin EftekharzadehThe Aerospace Corporation
Cindy Daniels, Lissa Jordin, Washito Sasamoto, Waldo RodriguezNASA Langley Research Center
Science Office for Mission Assessments (SOMA)
August 29th, 2017
2
Topics
• MOCET Overview & Status
• Updates in MOCET v1.2
• Public Release of MOCET
• User Community
• Summary & Future Work
3
MOCET Overview
• The Mission Operations Cost Estimation Tool (MOCET)
– A capability for Phase E estimation jointly developed by The Aerospace Corporation and NASA
Science Office for Mission Assessments (SOMA)
– Based on actual costs of historical missions with emphasis on competed missions
– Constructed by breaking the mission operations cost into the various phases
– Has few subjective inputs
– Estimates total Phase E mission cost
– Implemented entirely in Excel and requires no additional software or tools
– Also includes a user manual which provides additional instruction and background
Mar-34 Oct-34 Apr-35 Nov-35 May-36 Dec-36 Jun-37
Cruise
Approach/EDL
First Landed Month
Landed Prime Operations
Mars 2034 Rover
Mission Schedule
Download from ONCE Model Portal
https://oncedata.msfc.nasa.gov
Now available external to NASA via
https://software.nasa.gov
For more information:
Email: [email protected]
M. R. Hayhurst et al., "Mission Operations Cost Estimation Tool (MOCET)," 2017 IEEE Aerospace Conference, Big Sky, MT, 2017, pp. 1-13.
4
Presentation at NASA Cost Symposium
2015
MOCET v1.0 release
MOCET Timeline
MOCET v1.1 release
MOCET User’s Workshop 1
NASA Cost Symposium
2016 v1.1 Demo
MOCET User’s Workshop 2
MOCET v1.2 release;
IEEE Conf.
Initial approval for MOCET
public release
NASA Cost and Schedule
Symposium 2017
Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec
2015
2016
2017
Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec
Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec
5
MOCET Mission Types Database Overview
Mission/CER Type Program Missions
Planetary
Discovery MESSENGER, Stardust, Deep Impact, GRAIL, NEAR, Dawn (updated Orbital Ops)
Mars Scout Phoenix
Robotic Lunar Exploration LRO, LADEE
New Frontiers New Horizons, Juno (Cruise, Orbit Insertion), OSIRIS-REx
Mars Exploration MRO, Odyssey, MER, MSL, MAVEN (updated Orbital Ops)
Outer Planets Flagship Cassini (Orbit Insertion)
Earth Science
Earth System Science Pathfinder (ESSP) GRACE, CloudSat, CALIPSO, Aquarius, OCO-2, CYGNSS
Earth Systematic Missions (ESM) Aqua, Aura, Terra, Jason-1, OSTM, ICESat, GPM, SMAP, Jason-3
Explorers
Mission of Opportunity (MO) Suazku (ASTRO-E2), TWINS, CINDI
Small Explorers (SMEX) NuSTAR, IRIS, IBEX, AIM, GALEX, RHESSI
Medium Explorers (MIDEX) THEMIS, Swift, WISE
Near Earth Discovery
Helio-Astro
Discovery Genesis, Kepler
Solar Terrestrial Probes (STP) STEREO, TIMED, MMS
Living With a Star (LWS) RBSP, SDO
Cosmic Origins Spitzer
Physics of the Cosmos Fermi, Chandra
Bold = New Data in version 1.2 Potential New Data for version 1.X
– NASA SAP Business
Warehouse
• Monthly Expenditures
– Monthly Flight Project Reviews (FPR)
and Monthly Status Reviews (MSR)
• Monthly Expenditures, Mission Events
& Schedule
– NASA Cost Analysis Data
Requirements (CADRe)
• Project development costs used
to assign mission class
Data Sources Utilized and Consulted
6
MOCET v1.2 CERs Updates
CER Name New
Data
New Eq.
Form
No
Change
Comments
Cruise CER X Minor update with complete Juno Cruise Phase Data
Approach/Orbit Insertion/Reduction CER XJuno incorporated along with Cassini and “Outer Planets”
input option
Orbital Operations CER XUpdated Dawn & MAVEN data incorporated. “IsMars?”
variable determined to be no longer significant.
Flyby/Encounter CER X
Approach EDL CER X
First Landed Month CER X
Landed Prime Operations X
Data/Sample Analysis & Archiving X
Extended Orbital Ops CER X
Extended Landed Ops CER X
CER Name New
Data
New Eq.
Form
No
Change
Comments
Earth/Near-Earth Orbiting Checkout CER X
Earth Science Prime Operations CER X
Explorer Prime Operations CER X
Near Earth Discovery Helio Astro Prime Operations CER X
Planetary
Earth/Near-Earth Orbiting
7
MOCET v1.2 Update
• MOCET CERs and tool were updated to include new Juno data and released as v1.2
– Additional data from prime missions accumulated since v1.1 were also included: Dawn, MAVEN
– The same statistical processes used for v1.1 were applied to this update
– The new v1.2 CERs have been benchmarked against v1.1
– Overall the performance of the v1.2 updates is about the same as v1.1
• This update affected three CERs
– Nominal/Quiescent Cruise CER
• Updated to include the complete Juno Cruise Phase
• Resulted in only a minor change to the coefficients
– Approach/Orbit Insertion/Reduction
• The Juno Jupiter Orbit Insertion phase is now included in this CER
• It was also possible to include the Cassini mission which resulted in better performance of the CER
• Resulted in change in coefficients and the addition of a new variable
– Orbital Operations
• Only four months of Juno operations at Jupiter were available to include into this CER
– At this time recommend waiting until more data is accumulated to include Juno in this CER
• Rates for Dawn and MAVEN were updated with monthly data since the last update.
• An update to the MAVEN payload mass input variable value was also performed per user feedback
• Resulted in updates of coefficients and elimination of one variable due to new data and significance testing
8
Nominal/Quiescent Cruise Prime Phase CER
• CER Function:
• Goodness of Fit
• Database
IQILY *7836.0*8624.02537.1
Variable Name Units Definition
Y Cost FY13 $M
Planetary mission average monthly prime nominal or quiescent cruise
phase cost
IL IsLarge? 0 or 1 Is this mission a large mission class per MOCET definitions?
IQ IsQuiescent? 0 or 1 Is this cruise period a quiescent mode?
Measure Value
Regression Method OLS
Standard Error (RSS) 0.20
Average Percentage Bias 0.0%
Coefficient of Determination (R^2) 0.92
Adjusted R^2 0.87
Pearson's Correlation Sqd (r^2) 0.92
Number of Observations 6
Number of Input Variables 2
1. MESSENGER – Nominal (IL 0, IQ 0) 4. New Horizons – Nominal (IL 1, IQ 0) 7. GRAIL (A/B) – (IL 0, IQ 0)*
2. Dawn - Nominal (IL 0, IQ 0) 5. Stardust - Quiescent (IL 0, IQ 1)** 8. Cassini – (IL 0, IQ 0)***
3. Juno - Nominal (IL 1, IQ 0) 6. New Horizons - Quiescent (IL 1, IQ 1)
Additional Notes• Some missions may have nominal, quiescent, and checkout periods of cruise. Each of these phases should be broken out and estimated separately (e.g. 4 months of cruise with checkout, 12 months of
nominal cruise, 36 months of quiescent ops.)
• *GRAIL was not included in the model due using multiple spacecraft. At this time there is not sufficient data to recommend an adjustment for multiple planetary spacecraft and the model does not support
estimation of planetary multiple spacecraft missions
• **Stardust cruise considered as quiescent during the original prime mission. Limited data was available about the original mission so this assumption is based on description of the NEXT Mission.
• ***Cassini was not included in the CER since it the only example of a flagship class mission currently available
0
0.5
1
1.5
2
2.5
$- $0.50 $1.00 $1.50 $2.00 $2.50
Pre
dic
ted
Actual
9
Approach, Orbit Insertion, Reduction Prime Phase CER
• CER Function:
• Goodness of Fit
• Database
OPPCY *2668.1*9478.05024.0
Variable Name Units Definition
Y Cost FY13 $M Planetary mission average monthly prime orbital operations phase cost
PC Preceding Cruise FY13 $M Average monthly cost of the preceding cruise phase
OP OuterPlanets? 0 or 1 Is the orbit insertion being done at one of the outer planets (beyond Mars)?
Measure Value
Regression Method OLS
Standard Error (RSS) 0.16
Average Percentage Bias 0.0%
Coefficient of Determination (R^2) 0.99
Adjusted R^2 0.99
Pearson's Correlation Sqd (r^2) 0.99
Number of Observations 8
Number of Input Variables 2
Additional Notes• None
1. MESSENGER – MOI (PC $1.53, OP 0) 4. MAVEN – MOI (PC $1.98, OP 0) 7. Juno – JOI (PC $2.03, OP 1)
2. Dawn – Vesta (PC $0.98, OP 0) 5. GRAIL – LOI (PC $1.60, OP 0) 6. Cassini – SOI (PC $6.85, OP 1)
3. Dawn – Ceres (PC $0.97, OP 0) 6. MRO – MOI (PC $5.03, OP 0)
0
1
2
3
4
5
6
7
8
9
$- $2.00 $4.00 $6.00 $8.00 $10.00
Pre
dic
ted
Actual
10
Orbital Operations Prime Phase CER
• CER Function:
• Goodness of Fit
• Database
PMY *0157.0016.1
Variable Name Units Definition
Y Cost FY13 $M Planetary mission average monthly prime orbital operations phase cost
PM Payload Mass* kg > 0 Mass of payload portion for which NASA will fund operations
1. MESSENGER (PM 47.2) 4. MAVEN (PM 67.3) 7. LRO (PM 97.1)
2. Mars Odyssey (PM 44.3) 5. Dawn (PM 21.0) 9.GRAIL (PM 17.7)
3. LADEE (PM 49.6) 6. MRO (PM 134.0) 10. Cassini (PM 337.7)**
Additional Notes• Radio Science experiments generally are part of the S/C telecom system and do not need to be considered when totaling up the payload mass. Only include the mass of instruments for which NASA is
performing operations.
• **GRAIL was not included in the model due to using multiple spacecraft. At this time there is not sufficient data to recommend an adjustment for multiple planetary spacecraft and the model does not support
estimation of planetary multiple spacecraft missions
• ***Cassini was not included in the CER since it the only example of a flagship class mission currently available
0
0.5
1
1.5
2
2.5
3
3.5
4
$- $0.50 $1.00 $1.50 $2.00 $2.50 $3.00 $3.50 $4.00
Pre
dic
ted
Actual
Measure Value
Regression Method GERM ZMPE
Standard Error of the Estimate 11.0%
Average Percentage Bias 0.0%
Pearson's Correlation Sqd (r^2) 0.86
Number of Observations 7
Number of Input Variables 1
11
• The model was compared against actual mission costs to gauge performance
• Standard Deviation for all missions was found to be 20% (v1.1: 20%, v1.0: 25%)
• Planetary: 26% (v1.1: 27%, v1.0: 32%)
• Earth/Near Earth Orbiting: 16% (v1.0: 22%, No changes in v1.2)
Performance of v1.2 is about the same as v1.1
Overall Model Performance
v1.2
v1.1
12
Public Release of MOCET
• MOCET has been approved for public release via
software.nasa.gov
– The direct link can be found here:
https://software.nasa.gov/software/LAR-18894-1
• Now available to U.S. contractors and universities
etc.
• User must create an account on software.nasa.gov
• Fill out the request form
• Signature authority for a business cannot be an
engineer or those with similar titles
– An authorized signature authority is considered
• president of the company
• general counsel
• or contract officer
• Rules may be different depending on type of
institution
– Academic/University
– Individual Contractor etc.
• Contact MOCET help staff at [email protected] with
issues
13
Downloads vs. Time
• To date MOCET has been downloaded from ONCE 106* times since the initial release
• *Includes all downloads from all users including users that may be currently inactive
• Version 1.0 was downloaded 30 times between October 2015 and March 2016
• Version 1.1 has been downloaded 60 times from March 2016 to April 2017
• Version 1.2 has been downloaded 16 times from April 2017 to June 2017
0
20
40
60
80
100
120
Cumulative Downloads
10
0
2
7
3
12
10
7
4
0
3
11
6
7
0
5
1 1
9
3
4
00
2
4
6
8
10
12
14
Downloads by Month
v 1.0 Release v 1.1 Release v 1.0 Release v 1.1 ReleaseNCS 2016 NCS 2016v 1.2 Release v 1.2 Release
14
Users vs. Time
• As of June 2017, 59 unique users have downloaded MOCET from ONCE
• Since the release of v1.0 the number of users has increased steadily
0
10
20
30
40
50
60
70
Cumulative New Users
12
0
2
5
3 3
4 4
1
0
2
3
6
3
0
5
0 0
1
2
3
00
2
4
6
8
10
12
14
New Users by Month
v 1.0 Release v 1.1 Release v 1.0 Release v 1.1 ReleaseNCS 2016 NCS 2016v 1.2 Release v 1.2 Release
15
Summary & Next Steps
• New data has been added and the updated model has been released as MOCET v1.2
• MOCET v1.2 performs about the same as v1.1
• MOCET has been released publicly via software.nasa.gov
• MOCET IEEE paper* has been published for further background on the model
• MOCET will continue to be periodically updated with new mission data
• Model is currently being used by both evaluators and proposers
• We will continue to engage and grow the user community
Download MOCET
ONCE Model Portal https://oncedata.msfc.nasa.gov
Now available external to NASA via https://software.nasa.gov
For More Information
Demo session (see agenda for time and room)
Email: [email protected]
*M. R. Hayhurst et al., "Mission Operations Cost Estimation Tool (MOCET)," 2017 IEEE Aerospace Conference, Big Sky, MT, 2017, pp. 1-13.
