Center for Space Environment Modeling Gábor Tóth Center for Space Environment Modeling University of Michigan Gábor Tóth Center

Center for Space Environment Modeling http://csem.engin.umich.edu

Gábor Tóth

Center for Space Environment ModelingUniversity of Michigan

Gábor Tóth

Center for Space Environment ModelingUniversity of Michigan

The Grand Challenge of Space Weather PredictionThe Grand Challenge of Space Weather Prediction


•Tamas Gombosi, Kenneth Powell

•Ward Manchester, Ilia Roussev

•Darren De Zeeuw, Igor Sokolov

•Aaron Ridley, Kenneth Hansen

•Richard Wolf, Stanislav Sazykin (Rice University)

• József Kóta (Univ. of Arizona)

•Tamas Gombosi, Kenneth Powell

•Ward Manchester, Ilia Roussev

•Darren De Zeeuw, Igor Sokolov

•Aaron Ridley, Kenneth Hansen

•Richard Wolf, Stanislav Sazykin (Rice University)

• József Kóta (Univ. of Arizona)

CollaboratorsCollaborators

GrantsGrants DoD MURI and NASA CT Projects DoD MURI and NASA CT Projects


•What is Space Weather and Why to Predict It?

•Parallel MHD Code: BATSRUS

•Space Weather Modeling Framework (SWMF)

•Some Results

•Concluding Remarks

•What is Space Weather and Why to Predict It?

•Parallel MHD Code: BATSRUS

•Space Weather Modeling Framework (SWMF)

•Some Results

•Concluding Remarks

Outline of TalkOutline of Talk


What Space Weather Means

What Space Weather Means

Conditions on the Sun and in the solar wind, magnetosphere, ionosphere, and thermosphere that can influence the performance and reliability of space-born and ground-based technological systems and can endanger human life or health.

Conditions on the Sun and in the solar wind, magnetosphere, ionosphere, and thermosphere that can influence the performance and reliability of space-born and ground-based technological systems and can endanger human life or health.

Space physics that affects us.Space physics that affects us.


Affects Earth: The AuroraeAffects Earth: The Aurorae


Other Effects of Space Weather

Other Effects of Space Weather


MHD Code: BATSRUSMHD Code: BATSRUS• Block Adaptive Tree Solar-wind Roe Upwind Scheme

•Conservative finite-volume discretization

•Shock-capturing Total Variation Diminishing schemes

•Parallel block-adaptive grid (Cartesian and generalized)

•Explicit and implicit time stepping

•Classical and semi-relativistic MHD equations

•Multi-species chemistry

•Splitting the magnetic field into B0 + B1

•Various methods to control the divergence of B

• Block Adaptive Tree Solar-wind Roe Upwind Scheme

•Conservative finite-volume discretization

•Shock-capturing Total Variation Diminishing schemes

•Parallel block-adaptive grid (Cartesian and generalized)

•Explicit and implicit time stepping

•Classical and semi-relativistic MHD equations

•Multi-species chemistry

•Splitting the magnetic field into B0 + B1

•Various methods to control the divergence of B


MHD Equations in Conservative vs. Non-

Conservative Form

MHD Equations in Conservative vs. Non-

Conservative Form• Conservative form is required for correct jump

conditions across shock waves.

• Energy conservation provides proper amount of Joule heating for reconnection even in ideal MHD.

• Non-conservative pressure equation is preferred for maintaining positivity.

• Hybrid scheme: use pressure equation where possible.

• Conservative form is required for correct jump conditions across shock waves.

• Energy conservation provides proper amount of Joule heating for reconnection even in ideal MHD.

• Non-conservative pressure equation is preferred for maintaining positivity.

• Hybrid scheme: use pressure equation where possible.


Splitting the Magnetic FieldSplitting the Magnetic Field• The magnetic field has huge gradients near the Sun and

Earth:

– Large truncation errors.

– Pressure calculated from total energy can become negative.

– Difficult to maintain boundary conditions.

• Solution: split the magnetic field as B = B0 + B1 where

B0 is a divergence and curl free analytic function.

– Gradients in B1 are small.

– Total energy contains B1 only.

– Boundary condition for B1 is simple.

• The magnetic field has huge gradients near the Sun and Earth:

– Large truncation errors.

– Pressure calculated from total energy can become negative.

– Difficult to maintain boundary conditions.

• Solution: split the magnetic field as B = B0 + B1 where

B0 is a divergence and curl free analytic function.

– Gradients in B1 are small.

– Total energy contains B1 only.

– Boundary condition for B1 is simple.


Vastly Disparate ScalesVastly Disparate Scales•Spatial:

•Resolution needed at Earth: 1/4 RE

•Resolution needed at Sun: 1/32 RS

•Sun-Earth distance: 1AU

• 1 AU = 215 RS = 23,456 RE

•Temporal:

•CME needs 3 days to arrive at Earth.

•Time step is limited to a fraction of a second in some regions.

•Spatial:

•Resolution needed at Earth: 1/4 RE

•Resolution needed at Sun: 1/32 RS

•Sun-Earth distance: 1AU

• 1 AU = 215 RS = 23,456 RE

•Temporal:

•CME needs 3 days to arrive at Earth.

•Time step is limited to a fraction of a second in some regions.


Adaptive Block StructureAdaptive Block Structure

Each block is NxNxNEach block is NxNxN Blocks communicate with neighbors through “ghost”

cells

Blocks communicate with neighbors through “ghost”

cells


Parallel Distribution of the Blocks

Parallel Distribution of the Blocks


Optimized Load Balancing

Optimized Load Balancing


Parallel PerformanceParallel Performance


Why Explicit Time-Stepping May Not Be Good Enough

Why Explicit Time-Stepping May Not Be Good Enough

•Explicit schemes have time step limited by CFL condition: Δt < Δx/fastest wave speed.

•High Alfvén speeds and/or small cells may lead to smaller time steps than required for accuracy.

•The problem is particularly acute near planets with strong magnetic fields.

•Implicit schemes do not have Δt limited by CFL.

•Explicit schemes have time step limited by CFL condition: Δt < Δx/fastest wave speed.

•High Alfvén speeds and/or small cells may lead to smaller time steps than required for accuracy.

•The problem is particularly acute near planets with strong magnetic fields.

•Implicit schemes do not have Δt limited by CFL.


• BDF2 second-order implicit time-stepping scheme requires solution of a large nonlinear system of equations at each time step.

• Newton linearization allows the nonlinear system to be solved by an iterative process in which large linear systems are solved.

• Krylov solvers (GMRES, BiCGSTAB) with preconditioning are robust and efficient for solving large linear systems.

• Schwarz preconditioning allows the process to be done in parallel:

• Each adaptive block preconditions using local data only

• MBILU preconditioner

• BDF2 second-order implicit time-stepping scheme requires solution of a large nonlinear system of equations at each time step.

• Newton linearization allows the nonlinear system to be solved by an iterative process in which large linear systems are solved.

• Krylov solvers (GMRES, BiCGSTAB) with preconditioning are robust and efficient for solving large linear systems.

• Schwarz preconditioning allows the process to be done in parallel:

• Each adaptive block preconditions using local data only

• MBILU preconditioner

Building a Parallel Implicit Solver

Building a Parallel Implicit Solver


•Fully implicit scheme has no CFL limit, but each iteration is expensive (memory and CPU)

•Fully explicit is inexpensive for one iteration, but CFL limit may mean a very small Δt

•Set optimal Δt limited by accuracy requirement:

•Solve blocks with unrestrictive CFL explicitly

•Solve blocks with restrictive CFL implicitly

•Load balance explicit and implicit blocks separately

•Fully implicit scheme has no CFL limit, but each iteration is expensive (memory and CPU)

•Fully explicit is inexpensive for one iteration, but CFL limit may mean a very small Δt

•Set optimal Δt limited by accuracy requirement:

•Solve blocks with unrestrictive CFL explicitly

•Solve blocks with restrictive CFL implicitly

•Load balance explicit and implicit blocks separately

Getting the Best of Both Worlds - Partial Implicit

Getting the Best of Both Worlds - Partial Implicit


Timing Results for Space Weather on Compaq

Timing Results for Space Weather on Compaq


Controlling the Divergence of B

Controlling the Divergence of B• Projection Scheme (Brackbill and Barnes)

• Solve a Poisson equation to remove div B after each time step.

• Expensive on a block adaptive parallel grid.

• 8-Wave Scheme (Powell and Roe)

• Modify MHD equations for non-zero divergence so it is advected.

• Simple and robust but div B is not small. Non-conservative terms.

• Diffusive Control (Dedner et al.)

• Add terms that diffuse the divergence of the field.

• Simple but it may diffuse the solution too.

• Conservative Constrained Transport (Balsara, Dai, Ryu, Tóth)

• Use staggered grid for the magnetic field to conserve div B

• Exact but complicated. Does not allow local time stepping.

• Projection Scheme (Brackbill and Barnes)

• Solve a Poisson equation to remove div B after each time step.

• Expensive on a block adaptive parallel grid.

• 8-Wave Scheme (Powell and Roe)

• Modify MHD equations for non-zero divergence so it is advected.

• Simple and robust but div B is not small. Non-conservative terms.

• Diffusive Control (Dedner et al.)

• Add terms that diffuse the divergence of the field.

• Simple but it may diffuse the solution too.

• Conservative Constrained Transport (Balsara, Dai, Ryu, Tóth)

• Use staggered grid for the magnetic field to conserve div B

• Exact but complicated. Does not allow local time stepping.


Effect of Div B Control Scheme

Effect of Div B Control Scheme


From Codes To Framework

From Codes To Framework

•The Sun-Earth system consists of many different interconnecting domains that are independently modeled.

•Each physics domain model is a separate application, which has its own optimal mathematical and numerical representation.

•Our goal is to integrate models into a flexible software framework.

•The framework incorporates physics models with minimal changes.

•The framework can be extended with new components.

•The performance of a well designed framework can supercede monolithic codes or ad hoc couplings of models.


Physics Domains ID Models

Physics Domains ID Models•Solar Corona SC BATSRUS

•Eruptive Event Generator EE BATSRUS

• Inner Heliosphere IH BATSRUS

•Solar Energetic Particles SP Kóta’s SEP model

•Global Magnetosphere GM BATSRUS

• Inner Magnetosphere IM Rice Convection Model

• Ionosphere Electrodynamics IE Ridley’s potential solver

•Upper Atmosphere UA General Ionosphere Thermosphere Model (GITM)


Space Weather Modeling Framework

Space Weather Modeling Framework


The SWMF ArchitectureThe SWMF Architecture


Parallel Layout and Execution

Parallel Layout and Execution

LAYOUT.in for 20 PE-s SC/IH GM IM/IEID ROOT LAST STRIDE

#COMPONENTMAP

SC 0 9 1

IH 0 9 1

GM 10 17 1

IE 18 19 1

IM 19 19 1

#END


Parallel Field Line Tracing

Parallel Field Line Tracing•Stream line and field line tracing is a common

problem in space physics. Two examples:

•Coupling inner and global magnetosphere models

•Coupling solar energetic particle model with MHD

•Tracing a line is an inherently serial procedure

•Tracing many lines can be parallelized, but

•Vector field may be distributed over many PE-s

•Collecting the vector field onto one PE may be too slow and it requires a lot of memory


Coupling Inner and Global Magnetosphere Models

Coupling Inner and Global Magnetosphere Models

PressureInner magnetosphere model:

needs the field line volumes,

average pressure and density

along field lines connected to

the 2D grid on the ionosphere.

Global magnetosphere model:

needs the pressure correction

along the closed field lines:

€

pMHDn+1 = pMHD

n +Δt

τpRCM − pMHD

n)(


Interpolated Tracing Algorithm

Interpolated Tracing Algorithm

1. Trace lines inside blocks

starting from faces.

2. Interpolate and

communicate mapping.

3. Repeat 2. until the mapping

is obtained for all faces.

4. Trace lines inside blocks

starting from cell centers.

5. Interpolate mapping to

cell centers.


Parallel Algorithm without Interpolation

Parallel Algorithm without Interpolation

PE 1

PE 3

PE 2

PE 4

2b. If not done send to other PE.

1. Find next local field line.

3. Go to 1. unless time to receive.

6. Go to 1. unless all finished.

2. If there is a local field line then

2a. Integrate in local domain.

4. Receive lines from other PE-s.

5. If received line go to 2a.


Interpolated versus No Interpolation

Interpolated versus No Interpolation

0

5

10

15

20

25

30

35

40

45

CPU time [sec]

8 16 30

Number of processors

Interpolated No interpolation


•Set B0 to a magnetogram based potential field.

•Obtain MHD steady state solution.

•Use source terms to model solar wind acceleration and heating so that steady solution matches observed solar wind parameters.

•Perturb this initial state with a “flux rope”.

•Follow CME propagation.

•Let CME hit the Magnetosphere of the Earth.

•Set B0 to a magnetogram based potential field.

•Obtain MHD steady state solution.

•Use source terms to model solar wind acceleration and heating so that steady solution matches observed solar wind parameters.

•Perturb this initial state with a “flux rope”.

•Follow CME propagation.

•Let CME hit the Magnetosphere of the Earth.

Modeling a Coronal Mass Ejection

Modeling a Coronal Mass Ejection


Initial Steady State in the Corona

Initial Steady State in the Corona

• Solar surface is colored with the radial magnetic field.

• Field lines are colored with the velocity.

• Flux rope is shown with white field lines.

• Solar surface is colored with the radial magnetic field.

• Field lines are colored with the velocity.

• Flux rope is shown with white field lines.


Close-up of the Added Flux Rope

Close-up of the Added Flux Rope


Two Hours After Eruption in the Solar Corona

Two Hours After Eruption in the Solar Corona


65 Hours After Eruption in the Inner Heliosphere

65 Hours After Eruption in the Inner Heliosphere


The Zoom Movie

The Zoom Movie


More Detail at EarthMore Detail at Earth

Density and magnetic fieldat shock arrival time

Before shock After shock

South Turning BZ North Turning BZ

Pressure and magnetic field


Ionosphere Electrodynamics

Ionosphere ElectrodynamicsCurrent Potential

• Before shock hits.

• After shock: currents and the resulting electric potential increase.

• Region-2 currents develop.

• Although region-1 currents are strong, the potential decreases due to the shielding effect.


Upper Atmosphere

Upper Atmosphere

• The Hall conductance is calculated by the Upper Atmosphere component and it is used by the Ionosphere Electrodynamics.

• After the shock hits the conductance increases in the polar regions due to the electron precipitation.

• Note that the conductance caused by solar illumination at low latitudes does not change significantly.

Before shock arrival

After shock arrival


Performance of the SWMFPerformance of the SWMF



2003 Halloween Storm Simulation with GM, IM and

IE Components

2003 Halloween Storm Simulation with GM, IM and

IE Components• The magnetosphere during the solar storm associated with an X17 solar eruption.

• Using satellite data for solar wind parameters

•Solar wind speed: 1800 km/s.

• Time: October 29, 0730UT

• Shown are the last closed field lines shaded with the thermal pressure.

• The cut planes are shaded with the values of the electric current density.


GM, IM, IE Run vs. ObservationsGM, IM, IE Run vs. Observations


• The Space Weather Modeling Framework (SWMF) uses sate-of-the-art methods to achieve flexible and efficient coupling and execution of the physics models.

• Missing pieces for space weather prediction:

• Better models for solar wind heating and acceleration;

• Better understanding of CME initiation;

• More observational data to constrain the model;

• Even faster computers and improved algorithms.

• The Space Weather Modeling Framework (SWMF) uses sate-of-the-art methods to achieve flexible and efficient coupling and execution of the physics models.

• Missing pieces for space weather prediction:

• Better models for solar wind heating and acceleration;

• Better understanding of CME initiation;

• More observational data to constrain the model;

• Even faster computers and improved algorithms.

Concluding RemarksConcluding Remarks

Documents

Center for Space Environment Modeling Gábor Tóth Center for Space Environment Modeling University of Michigan Gábor Tóth Center