Design and Analysis of Experiments forEuropa Clipper’s Science Sensitivity Model
Amy Braverman1, Manny Uy2, Vineet Yadav1 and Kelli McCoy1
1 Jet Propulsion Laboratory, California Institute of Technology2 Applied Physics Laboratory, Johns Hopkins University
April 11, 2019
1
Outline
I Introduction
I The Europa Clipper mission
I Data acquisition
I Factors affecting data acquisition
I Mission requirements
I The experiment
I Experiment design
I Experiment analysis
I Conclusions
2
Introduction
Goals:
I Design, implement, and analyze a simulation experiment to quantify theprobability of mission success as a function of instrument fault andautonomous recovery rates.
I Mission success = meeting the set of “Level 1" Baseline requirements withprobability at least .95.
I Provide a tool to allow scientists and engineers to study how changes inboth requirements and hardware performance affect mission and Level 1requirement success probability.
3
Introduction
The Europa Clipper spacecraft will carry nine science instruments– thus thereare ten “systems":
System Code System Code
Spacecraft Sc Instrument 5 I5
Instrument 1 I1 Instrument 6 I6
Instrument 2 I2 Instrument 7 I7
Instrument 3 I3 Instrument 8 I8
Instrument 4 I4 Instrument 9 I9
4
Introduction
System Code System Code
Spacecraft Sc Instrument 5 I5
Instrument 1 I1 Instrument 6 I6
Instrument 2 I2 Instrument 7 I7
Instrument 3 I3 Instrument 8 I8
Instrument 4 I4 Instrument 9 I9
Each system’s transient fault behavior is modeled by two different exponentialdistributions, depending on orbital conditions.
Each system’s recovery time (after a fault) is modeled by a (shifted) betadistribution “outside" the flyby region, and as a constant “within" the flyby region(see next slides).
5
Europa Clipper Mission
Jupiter
Europa’sorbit: ∼3.5 days
Closest approach-3 days
Closest approach+2 days
Closestapproach
Closest approach+10 hrs
Closestapproach-10 hrs
High radiationzone: fault ratesare higher
6
Europa Clipper Mission
Europa Clipper’sorbit: ∼14 days;nominal missionis 46 orbits
Jupiter
Europa’sorbit: ∼3.5 days
Closest approach-3 days
Closest approach+2 days
Closestapproach
Closest approach+10 hrs
Closestapproach-10 hrs
High radiationzone: fault ratesare higher
7
Europa Clipper Mission
Europa Clipper’sorbit: ∼14 days;nominal missionis 46 orbits
Jupiter
Europa’sorbit: ∼3.5 days
Closest approach-3 days
Closest approach+2 days
Closestapproach
Closest approach+10 hrs
Closestapproach-10 hrs
High radiationzone: fault ratesare higher
Low-radiationzone: fault ratesare lower
Most importantscience dataacquired in high-radiation zone
8
Europa Clipper Mission
Europa Clipper’sorbit: ∼14 days;nominal missionis 46 orbits
Jupiter
Europa’sorbit: ∼3.5 days
Closest approach-3 days
Closest approach+2 days
Closestapproach
Closest approach+10 hrs
Closestapproach-10 hrs
Most importantscience dataacquired in high-radiation zone
“Outside" portion oforbit: recovery canbe slower
9
Europa Clipper Mission
Europa Clipper’sorbit: ∼14 days;nominal missionis 46 orbits
Jupiter
Europa’sorbit: ∼3.5 days
Closest approach-3 days
Closest approach+2 days
Closestapproach
Closest approach+10 hrs
Closestapproach-10 hrs
Most importantscience dataacquired in high-radiation zone
“Outside" portion oforbit: recovery canbe slower
“Within" portion oforbit: recoverymust be faster
10
Data Acquisition
I Each system will be in either an active or inactive state at every time pointduring the 46-orbit “tour".
I Let Xi(t) be the state of system i , i = 0, 1, 2, . . . , 9, at (continuous) time t ,t ∈ (0,T ), where T is the number of seconds in the 46-day tour (i = 0 forthe spacecraft).
I If system i is active at time t , then Xi(t) = 1. Otherwise Xi(t) = 0.
I If the spacecraft goes down (X0(t) = 0), all other systems go down.
I Ability to acquire high-value science data will depend on the complexsuper-positiion of the ten systems’ states during the period from -10 to +10hours of closest approach. (Caveat: two instruments do take datathroughout the whole orbit, and these are also critical for science.)
11
Data Acquisition
Sc
t=
0
t=
T
| | | |
I1 | | || | | | | | | | |
I2 | |
I3 | | | |
I4 | | | | | |
I5 | |
I6 | | | | | | | | | |
I7 | |
I8 | | | | | |
I9 | | | | | |
12
Factors affecting data acquisition
Sc
t=
0
t=
T
| | | |
I1 | | || | | | | | | | |
I2 | |
I3 | | | |
I4 | | | | | |
I5 | |
I6 | | | | | | | | | |
I7 | |
I8 | | | | | |
I9 | | | | | |
high-value data acquisition/high-radiation periods
Each instrument takes complete, partial, or no data during the critical periods.
13
Factors affecting data acquisition
Sc
t=
0
t=
T
| | | |
I1 | | || | | | | | | | |
I2 | |
I3 | | | |
I4 | | | | | |
I5 | |
I6 | | | | | | | | | |
I7 | |
I8 | | | | | |
I9 | | | | | |
“within" periods
14
Factors affecting data acquisition
Sc
t=
0
t=
T
| | | |
I1 | | || | | | | | | | |
I2 | |
I3 | | | |
I4 | | | | | |
I5 | |
I6 | | | | | | | | | |
I7 | |
I8 | | | | | |
I9 | | | | | |
“outside" periods
Different fault rates and recovery times apply (for each system) depending onwhether EC is in the red, orange, or green period.
15
Mission requirements
I Each instrument takes complete, partial, or no data during the criticalperiods.
I Each instrument has a set of instrument-specific requirements that areeither met or not met depending on the degree to which data are acquiredduring the critical periods.
I A tree-structured graph shows how instrument-level requirements areprogressively aggregated up through intermediate science objectivesthrough to Level 1 requirements.
16
Mission requirements
0.96
0.96
0.99 0.97
0.99 0.99 0.97 0.97
0.99 11 1 1 0.971 1
0.99 0.83 1101 11 1 1 0.970.96 1 11
0.83 0.83 1 11101 11 0.99 0.960.96 11 110.971
0.83 0.83 1 11101 11 0.99 0.960.96 11 110.971
0.9611 0.96110.99 111 11 0.870.970.95 1 0.87 0.970.95 11 1 101 11 1 10.971
1 01 11 11 111 0.961111 0.87 0.970.950.99 111 1 0.9711 111 1 11 11 1 11 1
l1reqid RQ106.316
RQ106.316 1
Atmospheric Composition Space Environment Composition
Atmospheric Composition 1Atmospheric Composition 2
Space Environment Composition 1Space Environment Composition 2
Atmospheric Composition Particulates
Atmospheric Composition PlasmaAtmospheric Composition Simple Volatile Species
Atmospheric Composition Complex Volatile Species
Space Environment Composition Complex Volatile Species
Space Environment Composition ParticulatesSpace Environment Composition Plasma
Space Environment Composition Simple Volatile Species
3260
3261
3259
3365
3366
3465
3450
3453
3454
3456
3562
3660
3661
3659
3766
3857
3866
3963
3964
579 68105 678 511 121317 14 15 1618 97 6 81011 1214 10 151011
UVS.01
8
UVS.01
7
UVS.01
9
UVS.02
1
UVS.03
1
UVS.01
4
UVS.01
6
UVS.02
2
ICE.021
ICE.016
PIM.015
PIM.026
PIM.033
PIM.032
PIM.007
PIM.025
PIM.024
PIM.011
SUD.002
SUD.003
SUD.040
SUD.017
SUD.018
SUD.033
SUD.027
SUD.035
SUD.036
MAS.003
MAS.004
MAS.007
MAS.012
MAS.016
MAS.040
MAS.028
MAS.008
MAS.011
MAS.019
MAS.027
MSTAF Row(AND)
P-STAF cell(AND)
PIES
PIE-group(AND)
Approach(OR)
Approach Group(AND)
Theme(OR)
Theme Group(AND)
L1reqid(OR)
Inst. 1 req’s Inst. 2 req’s Inst. 3 req’s Inst. 6 req’s Inst. 7 req’s Inst. 8 req’s
Level 1 req.
17
The experiment
I Determine maximum allowable fault rates and recovery times such that theset of Level 1 Baseline requirements have success probabilities thatexceed .95.
I Use Monte Carlo to simulate 1000 tours, each with faults generated byexponential distributions with specified parameters, and recovery timesgenerated as described earlier.
I These parameters are factors in the experiment.
I Responses are the probabilities of success for the Level 1 requirementsafter aggregating over the ensemble of 1000 tours.
I Build a response surface that relates factor levels to responses.
18
The experiment
timeline 1
timeline 2
timeline 1000
Y1 = (Y11,Y12, . . . ,Y1,91)′
Y2 = (Y21,Y22, . . . ,Y2,91)′
Y1000 = (Y1000,1,Y1000,2, . . . ,Y1000,91)′
Requirementschecker
Yjk = 1 if timeline j passes req. k ,
Yjk = 0 otherwise.
19
The experiment
Suppose p is the 91-dimensional vector of probabilities,
p =1
1000
1000∑n=1
Xn = (p1, p2, . . . , p91)′,
where pk is the probability of passing the k th (basic/“level 2"/leaf) requirement.
p1 p2 p3 p91
req.001 req.002 req.003 req.091
20
Experiment design
System Factors System Factors
Spacecraft ScTH-F, ScTL-F, ScT-WR Instrument 5 I5TH-F, I5TL-F, I5T-WR
Instrument 1 I1TH-F, I1TL-F, I1T-WR Instrument 6 I6TH-F, I6TL-F, I6T-WR
Instrument 2 I2TH-F, I2TL-F, I2T-WR Instrument 7 I7TH-F, I7TL-F, I7T-WR
Instrument 3 I3TH-F, I3TL-F, I3T-WR Instrument 8 I8TH-F, I8TL-F, I8T-WR
Instrument 4 I4TH-F, I4TL-F, I4T-WR Instrument 9 I9TH-F, I9TL-F, I9T-WR
T = “transient" fault (the only kind here). H/L = high/low radiation. F = fault rate,R = recovery time. W = “within".
All other factor combinations are fixed, and treated deterministically.
Not all factors applicable to all Level 1 requirements.
21
Experiment design
I 30 factors (10 systems, three factors each). Experiment run for two levelsfor each factor (a high value and a low value).
I ∼ 9 Baseline Level 1 requirements.
I Definitive Screening Design (Jones and Nachtsheim, (2011). Journal ofQuality Technology, Vol 43, No. 1.) created in JMP 14 Pro. Requires only arelatively small number of runs (∼ 2× (number of factors)).
I We had sufficient computational power to augment the DSD with additionalspace-filling runs. Total number of runs = 577.
I Design space = 577 points in a 30-dimensional space. Responses = 577probabilities for each Level 1 req.
22
Experiment design
23
Experiment analysis
Fit full response surfaces and main effects only models.
24
Conclusions
I Tom Youmans to discuss substantive conclusions (next talk).
I The Europa Clipper requirements model and experiment provides aquantitative way to relate science outcomes to design choices in buildinginstrument and spacecraft systems.
I The Monte Carlo simulation experiment allows us to interrogate theserelationships, which are too complex to understand analytically.
I The experimental design is crucial to making the Monte Carlo-basedstrategy feasible: it ensures that the limited number of conditions underwhich the experiment can be run, is as informative as possible.
I Personal reflection: JMP is a great tool, but I wish it was programmable!
25
This work was partially performed at the Jet Propulsion Laboratory, CaliforniaInstitute of Technology, under contract with the National Aeronautics and SpaceAdministration.
©2019. Government sponsorship acknowledged.
Contact information: [email protected]
26