Sandia is a multiprogram laboratory operated by Sandia Corporation, a Lockheed Martin Company,for the United States Department of Energy’s National Nuclear Security Administration

under contract DE-AC04-94AL85000.

Andy Salinger, Eric PhippsComputer Science Research Institute, Sandia National Laboratories

Albuquerque, NM, USA

Tipping Points in Complex Flows - Numerical Methods for Bifurcation Analysis of Large-Scale Systems

from 31 Oct 2011 through 4 Nov 2011

General-Purpose Software for Large-Scale Bifurcation Analysis

Trilinos: Algorithms and Enabling Technologies for Large-Scale Applications

Two-level design:– Self-contained packages (50+)– Leveraged common tools.

• Version Control• Build System• Test Harness

Nonlinear, Transient & Optimization

SolversLinear &

Eigen Solvers

Geometry, Meshing &

Load Balancing

Framework, Tools &

Interfaces

Discretizations

Scalable Linear

Algebra

http://trilinos.sandia.gov

Objective Package(s)Linear algebra objects Epetra, TpetraKrylov solvers AztecOO, Belos, KomplexILU-type preconditioners AztecOO, IFPACK, ShyLUMultilevel preconditioners ML, CLAPS

Eigenvalue problems AnasaziBlock preconditioners TekoDirect sparse linear solvers Amesos (MUMPS, SuperLU, UMFPack, …)Load Balancing, Graph Algs Zoltan, IsorropiaContinuation/Bifurcation LOCANonlinear system solver NOXTime Integrators/DAEs RythmosOptimization Moocho, Aristos, Dakota (via TriKota)

Automatic Diffferentiation Sacado

Parameter List, Timers, Memory TeuchosUncertainty Quantification Stokhos, Dakota (via Trikota)Abstract interfaces Thyra, EpetraExtNode Kernels on New Architectures Kokkos (Cuda, Threads, OpenMP)

Trilinos Package Summary

LOCA Provides Analysis Capabilities to Large-Scale Applications

Pseudo-Arclength Continuation

Hopf

Bifurcation location and continuation (turning point, pitchfork, and Hopf)

Linear eigen-analysis through Anasazi (Thornquist & Lehoucq)

Periodic orbit tracking (experimental)

Multi-parameter continuation through Multifario (Henderson)

Pitchfork

Why Do We Need Stability Analysis Tools for Large-Scale Applications

• Several powerful continuation and bifurcation analysis tools are available:– AUTO (Doedel et al)– CONTENT (Kuznetsov et al)– MATCONT (Govaerts et al)– PyDSTool (Guckenheimer et al)– …

• Large-scale applications have specific requirements– Massively parallel distributed memory architectures– Complicated parallel data structures and sparse matrices– Application-tuned linear algebra– Limited derivative capabilities

• Tools and algorithms are needed that– Do not change matrix sparsity or increase memory requirements– Agnostic to linear algebra and architecture– Can be incorporated into existing simulation codes (i.e., libraries)

Basic Defining Equations in LOCA

Turning Point

Pitchfork Hopf

Pseudo-Arclength

ODE/DAE Linearization

Shift and Invert

Generalized Cayley Transformation

Bifurcations Discovered Through Eigenvalue Analysis and Spectral Transformations

Eigenvalue problem

• Eigenvalues/vectors approximated via Block Krylov-Schur Iterations (Anasazi – Thornquist & Lehoucq)

• Analogies to time integration can be used to pick transformation parameters (Lehoucq and Salinger, 2001, Burroughs et al, 2004).

NOX: Object-Oriented Nonlinear Solverin Trilinos: Pawlowski et al

ConcreteImplementation

Layer

Linear Algebra

User Interface• Residual• Jacobian

SolverLayer

Methods• Line Search• Trust Region

• Searches• Directions

Linear Algebra

Application Interface

AbstractLayer

• •

• Epetra • LAPACK• User Defined • Thyra

LOCA Builton and around NOX

ConcreteImplementation

Layer

Linear Algebra

User Interface• Residual• Jacobian

SolverLayer

Methods• Line Search• Trust Region

• Searches• Directions

Linear Algebra

Application Interface

AbstractLayer

• Epetra • LAPACK• User Defined • Thyra

StepperLayer

1. Step2. Solve3. Analyze4. Predict5. Stop?

Continuation

Bifurcation

AugmentedEquations

Layers

• Update parameters• Mass matrix

Mix-and-match between• Continuation methods• Predictor modules• Step-size control modules• Bifurcation modules• Nonlinear solvers• Linear solvers/preconditioners

• Eigensolvers• Spectral transformations

• •

Block Elimination Algorithm for Turning Point (fold) Tracking Uses 4 Solves

•Turning Point Bifurcation •Full Newton Algorithm

•Block Elimination Algorithm

Solve 5 bordered systems of equations using QR approach

Then

Modified Turning Point Bordering Algorithm

Given and , let

then

There are constants such that

Standard formulation:

Note for Newton’s method:

3 linear solves per Newton iteration (5 for modified bordering)!For symmetric problems reduces to 2 solves.

Minimally Augmented Turning Point Formulation

MPSalsa, Charon, Albany codes

• Incompressible Navier-Stokes• Heat and Mass Transfer,

Reactions, variable properties• Unstructured Finite Element• Galerkin/Least-Squares: Q1Q1• Analytic Jacobian matrix in

distributed sparse storage• Compute with AD

• Fully Coupled Newton Method• GMRES with ILUT or MultI-Level

Preconditioners

Flow Calculations Performed with Sandia CFD codes and Trilinos solvers

Frank-Kamenetskii Explosion Model(~230K hex elements, ~1.1M unknowns, 128 cores)

Scenario:• Continuous Stirred Tank Reactor• Exothermic Cehmical Reaction• Cooling at WallsThe stirring breaks!?Will natural convection prevent

explosion? Arc-length Continuation

Frank-Kamenetskii Explosion ModelTurning Point Location

• ILU(k) fill factor: 1

• ILU(k) overlap: 2

• Max Krylov space: 1000

• MS Bordering

• Minimally augmented

Frank-Kamenetskii Explosion ModelTurning Point Continuation

Method Continuation Steps

Failed Steps

Nonlinear Iterations

Linear Solves

Linear Iterations

Total Time (hrs)

Moore-Spence Mod. Bordering

49 5 290 2012 472968 11.0

Min. Augmented 38 4 214 810 154027 6.9

“Analysis Beyond Simulation”

LOCA

Natural Convection Instability in 8x1 and 8x1x1 Cavities

Stability Analysis of Impinging Jets Pawlowski, Salinger, Shadid, Mountziaris (2005)

Region II

Region III

Region I

HH

P

Rayleigh-Benard in 5x5x1 cavity with Bifurcation Tracking

Codimension 2 Bifurcationnear Pr=0.0434, Ra=2106

Eigenvectors at Hopf

Hopf

Pitchfork

Pitchfork

Hydromagnetic Rayleigh-Bernard ProblemPawlowski, Shadid

Parameters: • Q ~ B0

2 (Chandresekhar number)• Ra (Rayleigh number)

• Buoyancy driven instability initiates flow at high Ra numbers.• Increased values of Q delay the onset of flow. • Domain: 1x20

Ra (fixed Q)

No flow Recirculations

B0 g

Extended MHD Model in Residual Form

Involution:

Resistive, Extended MHD Equations

Hydro-Magnetic Rayleigh-Bernard Stability: Direct Determination of Linear Stability and Nonlinear

Equilibrium Solutions (Steady State Solves)

• 2 Direct-to-steady-state solves at a given Q• Arnoldi method using Cayley transform to determine

approximation to 2 eigenvalues with largest real part• Simple linear interpolation to estimate Critical Ra*

Temp.

Vx

Vy

By

Bx

Leading Eigenvector at Bifurcation Point, Ra = 1945.78, Q=10

Q=10

Q=0

Design (Two-Parameter) Diagram

Vx

Ra

Q

Ra

Q

No Flow

Buoyancy Driven Flow

• “No flow” does not equal “no-structure” – pressure and magnetic fields must adjust/balance to maintain equilibrium.

• LOCA can perform continuation of bifurcation

Critical Mode is different for various Q values

• Analytic solution is on an infinite domain with two bounding surfaces (top and bottom)

• Multiple modes exist, mostly differentiated by number of cells/wavelength.

• Therefore tracking the same eigenmode does not give the stability curve!!!

• Periodic BCs will not fix this issue.

Mode: 20 Cells: Q=100, Ra=4017

Mode: 26 Cells: Q=100, Ra=3757

Q

Ra

Leading mode is 20 cells

Leading mode is 26 cells

2000

3000

4000

Scaling studies

~20x

Cores Fine MeshLevel 0Unkns.

Intermed.Level 1Unkns.

Intermed.Level 2Unkns.

Coarse Level 3Unkns.

Newton Iters.

Avg. No.Linear Its. /Newton

Total Sim.Time*(min.)

24,000 1.05 billion 23.3M .5M 11.2K 18 86 33

LOCA has impacted several application areas without flow

Phase Transitions in a Confined Fluid [Frink] Super-Conductivity Transitions in Ginzburg-Landau [Schlomer, Vanroose]

TeraHz Resonance in Quantum Tunneling Diode [Kelley]

Pattern Formation in Swift-Hohenberg Eqs [Avitabile, Sanstede]

How do I attribute the successes that LOCA has had?

1.Algorithmic research in large-scale bifurcations2.Science demonstrations3.LOCA is hooked up to Trilinos linear solvers

What broader lesson is there?

10%15%75%

Build software in independent-yet-interoperable components

Nonlinear SolverTime Integration

Optimization

Continuation

Constrained Solves

Sensitivity AnalysisStability Analysis

Analysis Tools

Data Structures

Direct Solvers

Linear Algebra

Preconditioners

Iterative Solvers

Eigen Solver

Matrix Partitioning

DerivativesDerivative Tools

Sensitivities

Build Software in Independent-yet-Interoperable Components

Continuation, Bifurcation, Stability Analysis

Build Software in Independent-yet-Interoperable Components

Nonlinear SolverTime Integration

Optimization

Continuation

Constrained Solves

Sensitivity AnalysisStability Analysis

Analysis Tools

Data Structures

Direct Solvers

Linear Algebra

Preconditioners

Iterative Solvers

Eigen Solver

Matrix Partitioning

DerivativesDerivative Tools

Sensitivities

Transient sensitivity analysis

Build Software in Independent-yet-Interoperable Components

Initial 4-Param Bound 4-Param Free

Nonlinear SolverTime Integration

Optimization

Continuation

Constrained Solves

Sensitivity AnalysisStability Analysis

Analysis Tools

Data Structures

Direct Solvers

Linear Algebra

Preconditioners

Iterative Solvers

Eigen Solver

Matrix Partitioning

DerivativesDerivative Tools

Sensitivities

CVD Reactor Optimization

Element Level FillMaterial Models

Sensitivities

Field ManagerDiscretization Library

Remeshing

UQ Solver

Nonlinear SolverTime Integration

Optimization

Objective Function

Local Fill

Mesh Database

Mesh Tools I/O Management

Input File ParserUtilitiesUQ (non-invasive)

Parameter Studies Solution Control

Mesh I/O

Optimization

Geometry Database

Discretizations

Derivative Tools

Adjoints

UQ / PCEPropagation Constraints

Error Estimates

Continuation

Constrained Solves

Sensitivity AnalysisStability Analysis

V&V, CalibrationParameter List

Feature ExtractionEmbedded Verification

VisualizationPostProcessing

Data Reduction

Adaptivity

Model Reduction

Memory Management

System Models

MultiPhysics Coupling

OUU, ReliabilityComputational Steering

Communicators

MultiCoreParallelization Tools

PartitioningLoad Balancing

Analysis Tools (black-box)

Physics Fill

Composite Physics

Data Structures

Direct Solvers

Linear Algebra

Architecture-Dependent Kernels

Preconditioners

Iterative Solvers

Eigen Solver

System UQ

Analysis Tools (embedded)

Matrix Partitioning

Inline Meshing

MMS Source Terms

Grid TransfersMesh Quality

Mesh Database

Solution Database

Runtime Compiler

Derivatives

Regression Testing

Bug Tracking

Version Control

Software Quality

Porting

Performance TestingCode Coverage

Mailing Lists

Release Process

Unit Testing

Web Pages

Build SystemBackups

“Agile Components”

Neighbor Search / SortData Structures

Particle Code Tools

Field Manager

PDE Assembly isTemplated for AD, PCE

Discretization

Application

Nonlinear ModelNonlinear

TransientStochasticGalerkin

Optimization

Continuation

Solvers w/ Sensitivities

OptimizationUQ

Analysis Tools

IterativeBlock Iterative

Direct

Linear Solvers / Preconditioners

DomainDecomp

MultiLevel

Mesh Database

Exodus File

Hand-Coded:

ProblemDiscretization

PDE TERMS

Main()

ManyCore Node KernelsMulit-CoreAccelerators

Inline Mesher

Application

Linear Solve

Eigensolve

Bifurcation

Albany Code: DemonstratingComponent-based Code Design

QualityImprovement

Load Balancing

SchurComp

Applications in Albany are born with Transformational Analysis Capabilities

LCM: Platform for R&D in mechanics:• Load Stepping, AD of material models

QCAD: Quantum dot design tool.•Optimization of gate voltages

2-Param OptimumInitial Mesh Std Deviation

ThermoElectrostatics: Shape Optimization with Embedded UQ

Summary

• LOCA and Trilinos provide powerful simulation and analysis capabilities

– Continuation, bifurcation, and linear stability analysis– Scalable linear algebra– Optimization, time integration, automatic differentiation,

uncertainty quantification, discretization, …• Missing Capabilities (formerly future work)

– More generic algorithms for bordered matrix solves• Much is hardwired to our Epetra format

– Periodic Orbit tracking beyond initial attempt– Automated initial guess generation for null vectors– Better documentation, examples, error checking, etc.

• Current Passion– Component-based code design with Embedded Analysis in

Mind from the beginning