Design and Analysis of Experiments for Europa Clipper’s Science … · 2019-06-24 · Europa Clipper’s orbit: ˘14 days; nominal mission is 46 orbits Jupiter Europa’s orbit:

Design and Analysis of Experiments forEuropa Clipper’s Science Sensitivity Model

Amy Braverman1, Manny Uy2, Vineet Yadav1 and Kelli McCoy1

1 Jet Propulsion Laboratory, California Institute of Technology2 Applied Physics Laboratory, Johns Hopkins University

April 11, 2019

1

Outline

I Introduction

I The Europa Clipper mission

I Data acquisition

I Factors affecting data acquisition

I Mission requirements

I The experiment

I Experiment design

I Experiment analysis

I Conclusions

2

Introduction

Goals:

I Design, implement, and analyze a simulation experiment to quantify theprobability of mission success as a function of instrument fault andautonomous recovery rates.

I Mission success = meeting the set of “Level 1" Baseline requirements withprobability at least .95.

I Provide a tool to allow scientists and engineers to study how changes inboth requirements and hardware performance affect mission and Level 1requirement success probability.

3

Introduction

The Europa Clipper spacecraft will carry nine science instruments– thus thereare ten “systems":

System Code System Code

Spacecraft Sc Instrument 5 I5

Instrument 1 I1 Instrument 6 I6




4

Introduction

System Code System Code

Spacecraft Sc Instrument 5 I5





Each system’s transient fault behavior is modeled by two different exponentialdistributions, depending on orbital conditions.

Each system’s recovery time (after a fault) is modeled by a (shifted) betadistribution “outside" the flyby region, and as a constant “within" the flyby region(see next slides).

5

Europa Clipper Mission

Jupiter

Europa’sorbit: ∼3.5 days

Closest approach-3 days

Closest approach+2 days

Closestapproach

Closest approach+10 hrs

Closestapproach-10 hrs

High radiationzone: fault ratesare higher

6


Europa Clipper’sorbit: ∼14 days;nominal missionis 46 orbits

Jupiter




Closestapproach




7



Jupiter




Closestapproach




Low-radiationzone: fault ratesare lower

Most importantscience dataacquired in high-radiation zone

8



Jupiter




Closestapproach




“Outside" portion oforbit: recovery canbe slower

9



Jupiter




Closestapproach




“Outside" portion oforbit: recovery canbe slower

“Within" portion oforbit: recoverymust be faster

10

Data Acquisition

I Each system will be in either an active or inactive state at every time pointduring the 46-orbit “tour".

I Let Xi(t) be the state of system i , i = 0, 1, 2, . . . , 9, at (continuous) time t ,t ∈ (0,T ), where T is the number of seconds in the 46-day tour (i = 0 forthe spacecraft).

I If system i is active at time t , then Xi(t) = 1. Otherwise Xi(t) = 0.

I If the spacecraft goes down (X0(t) = 0), all other systems go down.

I Ability to acquire high-value science data will depend on the complexsuper-positiion of the ten systems’ states during the period from -10 to +10hours of closest approach. (Caveat: two instruments do take datathroughout the whole orbit, and these are also critical for science.)

11

Data Acquisition

Sc

t=

0

t=

T

| | | |

I1 | | || | | | | | | | |

I2 | |

I3 | | | |

I4 | | | | | |

I5 | |

I6 | | | | | | | | | |

I7 | |

I8 | | | | | |

I9 | | | | | |

12

Factors affecting data acquisition

Sc

t=

0

t=

T

| | | |

I1 | | || | | | | | | | |

I2 | |

I3 | | | |

I4 | | | | | |

I5 | |

I6 | | | | | | | | | |

I7 | |

I8 | | | | | |

I9 | | | | | |

high-value data acquisition/high-radiation periods

Each instrument takes complete, partial, or no data during the critical periods.

13


Sc

t=

0

t=

T

| | | |

I1 | | || | | | | | | | |

I2 | |

I3 | | | |

I4 | | | | | |

I5 | |

I6 | | | | | | | | | |

I7 | |

I8 | | | | | |

I9 | | | | | |

“within" periods

14


Sc

t=

0

t=

T

| | | |

I1 | | || | | | | | | | |

I2 | |

I3 | | | |

I4 | | | | | |

I5 | |

I6 | | | | | | | | | |

I7 | |

I8 | | | | | |

I9 | | | | | |

“outside" periods

Different fault rates and recovery times apply (for each system) depending onwhether EC is in the red, orange, or green period.

15

Mission requirements

I Each instrument takes complete, partial, or no data during the criticalperiods.

I Each instrument has a set of instrument-specific requirements that areeither met or not met depending on the degree to which data are acquiredduring the critical periods.

I A tree-structured graph shows how instrument-level requirements areprogressively aggregated up through intermediate science objectivesthrough to Level 1 requirements.

16

Mission requirements

0.96

0.96

0.99 0.97

0.99 0.99 0.97 0.97

0.99 11 1 1 0.971 1

0.99 0.83 1101 11 1 1 0.970.96 1 11

0.83 0.83 1 11101 11 0.99 0.960.96 11 110.971

0.83 0.83 1 11101 11 0.99 0.960.96 11 110.971

0.9611 0.96110.99 111 11 0.870.970.95 1 0.87 0.970.95 11 1 101 11 1 10.971

1 01 11 11 111 0.961111 0.87 0.970.950.99 111 1 0.9711 111 1 11 11 1 11 1

l1reqid RQ106.316

RQ106.316 1

Atmospheric Composition Space Environment Composition

Atmospheric Composition 1Atmospheric Composition 2

Space Environment Composition 1Space Environment Composition 2

Atmospheric Composition Particulates

Atmospheric Composition PlasmaAtmospheric Composition Simple Volatile Species

Atmospheric Composition Complex Volatile Species

Space Environment Composition Complex Volatile Species

Space Environment Composition ParticulatesSpace Environment Composition Plasma

Space Environment Composition Simple Volatile Species

3260

3261

3259

3365

3366

3465

3450

3453

3454

3456

3562

3660

3661

3659

3766

3857

3866

3963

3964

579 68105 678 511 121317 14 15 1618 97 6 81011 1214 10 151011

UVS.01

8

UVS.01

7

UVS.01

9

UVS.02

1

UVS.03

1

UVS.01

4

UVS.01

6

UVS.02

2

ICE.021

ICE.016

PIM.015

PIM.026

PIM.033

PIM.032

PIM.007

PIM.025

PIM.024

PIM.011

SUD.002

SUD.003

SUD.040

SUD.017

SUD.018

SUD.033

SUD.027

SUD.035

SUD.036

MAS.003

MAS.004

MAS.007

MAS.012

MAS.016

MAS.040

MAS.028

MAS.008

MAS.011

MAS.019

MAS.027

MSTAF Row(AND)

P-STAF cell(AND)

PIES

PIE-group(AND)

Approach(OR)

Approach Group(AND)

Theme(OR)

Theme Group(AND)

L1reqid(OR)

Inst. 1 req’s Inst. 2 req’s Inst. 3 req’s Inst. 6 req’s Inst. 7 req’s Inst. 8 req’s

Level 1 req.

17

The experiment

I Determine maximum allowable fault rates and recovery times such that theset of Level 1 Baseline requirements have success probabilities thatexceed .95.

I Use Monte Carlo to simulate 1000 tours, each with faults generated byexponential distributions with specified parameters, and recovery timesgenerated as described earlier.

I These parameters are factors in the experiment.

I Responses are the probabilities of success for the Level 1 requirementsafter aggregating over the ensemble of 1000 tours.

I Build a response surface that relates factor levels to responses.

18

The experiment

timeline 1

timeline 2

timeline 1000

Y1 = (Y11,Y12, . . . ,Y1,91)′

Y2 = (Y21,Y22, . . . ,Y2,91)′

Y1000 = (Y1000,1,Y1000,2, . . . ,Y1000,91)′

Requirementschecker

Yjk = 1 if timeline j passes req. k ,

Yjk = 0 otherwise.

19

The experiment

Suppose p is the 91-dimensional vector of probabilities,

p =1

1000

1000∑n=1

Xn = (p1, p2, . . . , p91)′,

where pk is the probability of passing the k th (basic/“level 2"/leaf) requirement.

p1 p2 p3 p91

req.001 req.002 req.003 req.091

20

Experiment design

System Factors System Factors

Spacecraft ScTH-F, ScTL-F, ScT-WR Instrument 5 I5TH-F, I5TL-F, I5T-WR

Instrument 1 I1TH-F, I1TL-F, I1T-WR Instrument 6 I6TH-F, I6TL-F, I6T-WR




T = “transient" fault (the only kind here). H/L = high/low radiation. F = fault rate,R = recovery time. W = “within".

All other factor combinations are fixed, and treated deterministically.

Not all factors applicable to all Level 1 requirements.

21

Experiment design

I 30 factors (10 systems, three factors each). Experiment run for two levelsfor each factor (a high value and a low value).

I ∼ 9 Baseline Level 1 requirements.

I Definitive Screening Design (Jones and Nachtsheim, (2011). Journal ofQuality Technology, Vol 43, No. 1.) created in JMP 14 Pro. Requires only arelatively small number of runs (∼ 2× (number of factors)).

I We had sufficient computational power to augment the DSD with additionalspace-filling runs. Total number of runs = 577.

I Design space = 577 points in a 30-dimensional space. Responses = 577probabilities for each Level 1 req.

22

Experiment design

23

Experiment analysis

Fit full response surfaces and main effects only models.

24

Conclusions

I Tom Youmans to discuss substantive conclusions (next talk).

I The Europa Clipper requirements model and experiment provides aquantitative way to relate science outcomes to design choices in buildinginstrument and spacecraft systems.

I The Monte Carlo simulation experiment allows us to interrogate theserelationships, which are too complex to understand analytically.

I The experimental design is crucial to making the Monte Carlo-basedstrategy feasible: it ensures that the limited number of conditions underwhich the experiment can be run, is as informative as possible.

I Personal reflection: JMP is a great tool, but I wish it was programmable!

25

This work was partially performed at the Jet Propulsion Laboratory, CaliforniaInstitute of Technology, under contract with the National Aeronautics and SpaceAdministration.

©2019. Government sponsorship acknowledged.

Contact information: [email protected]

26

Documents

Design and Analysis of Experiments for Europa Clipper’s Science … · 2019-06-24 · Europa Clipper’s orbit: ˘14 days; nominal mission is 46 orbits Jupiter Europa’s orbit: