Upload
others
View
2
Download
0
Embed Size (px)
Citation preview
ORNL is managed by UT-Battelle
for the US Department of Energy
Experiences with Applying Tapenade to a Modern 3D CFD Code
Charlotte Kotas†
Jacob Barhen†
Bob Wilson†
Mike Traweek‡
EuroADFebruary 2017
† ORNL
‡ Office of Naval Research
DISTRIBUTION STATEMENT A. Approved for public release. Distribution is unlimited.
2 Experiences with Tapenade and CFD
Outline
• Motivation
– Model validation using uncertainty and sensitivity analysis
• Application
– Computational fluid dynamics code, REX
• Applying Tapenade to REX
– Work in progress
3 Experiences with Tapenade and CFD
Motivation: Model Validation
• Experimental data contain uncertainties due to measurement bias, precision, and random factors
• Computational model predictions contain uncertainties due to parameters, simplifications in physical equations, and numerical techniques
Given these uncertainties, we want to combine experimental data with computational model responses to determine a best estimate for the independent variables, model parameters, and experimental responses
And use these best estimates to reduce the uncertainties
For details see:
D. G. Cacuci, Sensitivity and Uncertainty Analysis, Chapman & Hall (2003)
J. Barhen et al, “Consistent uncertainty reduction in modeling nonlinear systems”, SIAM Journal on Scientific Computing, 26 (2), 653-665 (2004).
and references therein.
4 Experiences with Tapenade and CFD
Notation• Let 𝒙 be input variables, 𝜽 be
model parameters, 𝒎(𝒙, 𝜽) be the computational model response,𝒆the be experimentally measured response, 𝒓 be the true system response, 𝜺 be experimental errors, and 𝜹 𝒙, 𝜽 be model errors
– 𝒓 = 𝒆 − 𝜺 = 𝒎 𝒙, 𝜽 + 𝜹 𝒙, 𝜽
• Let 𝒂 be a vector collecting 𝒙 and 𝜽 such that 𝒎 𝒙, 𝜽 = 𝒎(𝒂)
• Let 𝑪𝒛 be a covariance matrix computed from standard deviations in the elements of vector 𝒛
• Let ^ denote the “best estimate” of a quantity
Model
𝒙
𝜽
𝒎(𝒙, 𝜽)
5 Experiences with Tapenade and CFD
Sensitivity Analysis and Uncertainty Quantification (1)
• The sensitivities of the system describe the relative importance of individual input values in overall system responses.
• Mathematically, the sensitivity matrix (𝑺) is the derivative of the desired responses 𝑹 = 𝑹(𝒎, 𝒂). We assume that the computation model computes 𝑹 directly, thus,
– 𝑺𝒏𝒊 =𝒅𝑹𝒏
𝒅𝒂𝒊=
𝝏𝑹𝒏
𝝏𝒂𝒊+
𝝏𝑹𝒏
𝝏𝒎𝝂
𝝏𝒎𝝂
𝝏𝒂𝒊=
𝝏𝒎𝒏
𝝏𝒂𝒊
– To first order, 𝒎 = 𝒎0 + 𝑺 𝒂−𝒂𝟎
• The sensitivity matrix is used to propagate uncertainty from the inputs to the model responses (to the first order):
– 𝑪𝒎 = 𝑺𝑪𝒂𝑺𝑻, where 𝑺 is evaluated at 𝒂
– If independent variables, 𝑪𝒂 is diagonal and this becomes the more
familiar 𝜎𝑚2 =
𝜕𝑚
𝜕𝑥
2𝜎𝑥
2 +𝜕𝑚
𝜕𝑦
2
𝜎𝑦2 + ⋯
Tapenade computes
6 Experiences with Tapenade and CFD
Sensitivity Analysis and Uncertainty Quantification (2)
• We can combine the model and experimental responses to estimate the true response and reduce uncertainty.
𝜺 = 𝒆 − 𝒓 = 𝒆 − 𝒎 𝒙, 𝜽 + 𝜹 𝒙, 𝜽
≅ 𝒆 − 𝒎 𝒂 − 𝑺 ෝ𝒂−𝒂
• We create a generalized loss function 𝑄 based on the experimental error and the model constraints:
𝑸 =ො𝒓 − 𝒆ෝ𝒂−𝒂
⋮
𝑇 𝑪𝒆 𝑪𝒑 …
𝑪𝒑𝑇 𝑪𝒂 …
… … …
−1ො𝒓 − 𝒆ෝ𝒂−𝒂
⋮
• ℒ can be minimized for the best estimate of ො𝒓 and ෝ𝒂, and used to compute new covariance (uncertainty) estimates
ℒ = 𝑸 + 𝜆 𝒎 𝒂 − 𝒆 + 𝑺 ෝ𝒂−𝒂
7 Experiences with Tapenade and CFD
Sensitivity Analysis and Uncertainty Quantification (3)
• Reduction of uncertainties:
𝑺 = ቚ𝝏𝒎
𝝏𝒂 𝒂
𝑺ෝ𝒂 = ቤ𝝏𝒎
𝝏𝒂ෝ𝒂
𝑹 = 𝑪𝒆 − 𝑺𝑪𝒑𝑻 −
𝑪𝒑𝑺𝑻 + 𝑺𝑪𝒂𝑺𝑻
𝑪ෝ𝒂 = 𝑪𝒂 − 𝑪𝒑𝑻 − 𝑪𝒂𝑺𝑻 𝑹 −𝟏 𝑪𝒑 − 𝑺𝑪𝒂
𝑪ො𝒓 = 𝑪𝒆 − 𝑪𝒆𝑻 − 𝑪𝒑𝑺𝑻 𝑹 −𝟏 𝑪𝒆 − 𝑺𝑪𝒑
𝑪 ෝ𝒎 = 𝑺ෝ𝒂𝑪ෝ𝒂𝑺ෝ𝒂𝑻
NOGA=Nonlinear Optimization Generalized Adjustments
𝑪𝒂
Ce
Model
+
NOGA
Experiment ො𝒓 ෝ𝒎
𝑪 ෝ𝒎 𝑪ො𝒓
𝑪ෝ𝒂
ෝ𝒂𝒂
Ce
𝑪𝒂
Model
Cm
Experiment
e
m𝒂
8 Experiences with Tapenade and CFD
Tapenade and Uncertainty
• Let Ԧ𝑎 denote the parameters and 𝜎𝑎,𝑖2
denote the uncertainty of the 𝑖 th
parameter. (Assume that the parameters are independent.)
• Let 𝑚 denote the unknowns computed by Rex (e.g. U, force coefficients)
• The uncertainty of a given element of 𝑚 is computed from the derivatives provided by Tapenade through:
𝜎𝑚,𝑖2
=𝜕𝑚𝑖
𝜕 Ԧ𝑎
𝜎𝑎,12
⋯ 0
⋮ ⋱ ⋮
0 ⋯ 𝜎𝑎,𝑛2
𝜕𝑚𝑖
𝜕 Ԧ𝑎
𝑇
Note that 𝜕𝑚𝑖
𝜕𝑎is a row vector; Tapenade is used to compute
𝜕𝑚𝑖
𝜕𝑎𝑗.
• This definition can be expanded to the full covariance matrix of uncertainties as:
𝑪𝒎 = 𝑆𝑪𝒂𝑆𝑇
Where the elements of the Jacobian matrix, 𝑆, are given by:
𝑆𝑖𝑗 =𝜕𝑚𝑖
𝜕𝑎𝑗
• Tapenade computes one row/column of the Jacobian for each pass in
forward/reverse mode
9 Experiences with Tapenade and CFD
Application: REX
• Unsteady, RANS/DES Solver (IIHR: University of Iowa)
– 25-year history of on-going ONR support
– Verified & validated for naval hydrodynamics: resistance/propulsion, seakeeping, maneuvering, design optimization
– Modeling
• Two-equation turbulence and subgrid-scale models
• Free-surface capturing for air-water interface
• Density stratification and multiphase model for bubbly
– Numerical
• Pressure-based, PISO approach for incompressibility constraint
• Upwind convection schemes from second- to fifth-order
• Second-order implicit time differencing
– Capabilities
• Dynamic overset: grid generation and bodies w/relative motion
• 6DOF multi-body calculations with fully-appended UUV/subs/ships with rotating props, rudders etc.
• HPC parallel simulations using Open-MPI
10 Experiences with Tapenade and CFD
CFD: Methodology (1)
• Governing Equations: Reynolds Averaged Navier-Stokes (RANS)
• Mass conservation
• Momentum conservation
¶r
¶t+Ñ× ru( ) = 0
¶
¶truéë ùû +Ñ× ruÄuéë ùû = -Ñp̂ +Ñ×
2
Reeff
Ñ2u
æ
èç
ö
ø÷
1
Reeff
=1
Re+ n
t
Re =r
0U
0L
0
m
nt=
k
w
p̂ = p +2
3rk +
2
3
1
Reeff
Ñ ×u
u, p̂( ); velocity, pressure
k,w( ); turbulent kinetic energy, dissipation
r0,m; density,viscosity
U0, L
0; characteristic velocity, length
11 Experiences with Tapenade and CFD
CFD: Methodology (2)
• Fully turbulent, k-ω, two-equation turbulence model
– Turbulent kinetic energy, k
– Specific turbulent dissipation rate, ω
12 Experiences with Tapenade and CFD
Applying Tapenade to REX
• TAPENADE is an Automatic Differentiation program that operates by source transformation on C and FORTRAN 77/90/95 codes. It was developed at INRIA Sophia-Antipolis, and is currently maintained by Laurent Hascoët and Valerie Pascual.
• As REX is written in (mostly) FORTRAN 95, Tapenade can be applied with minor changes to the source code to bring the code to F95 standards or simplify things for Tapenade.
– “flush(11)” replaced with “call flush(11)”
– Tapenade option –linelength to allow lines longer than F95 standard
– Adding some intermediate variables manually where Tapenade needed help seeing array size. For example, MAXVAL(ABS(A(:))) becomes result1=ABS(A(:)); MAXVAL(result1)
– Replacing DOT_PRODUCT(a(:),b(:)) with SUM(a(:)*b(:))
13 Experiences with Tapenade and CFD
Tapenade “features”
• REX is complex. There are a few places where Tapenade gets confused.
– Wants to initialize at the top level a couple of local array variables
– Leaves in “#” in a couple of places, as in EXTERNAL #FUNCTION_NAME#
– Calls for nonexistent module PACKINGD. (Module PACKING exists, but not PACKINGD.)
– Puts in module variable that is not within local scope in initializing array.
• Original: A(:,:)=0.0, where A is brought into scope through “USE MYMODULE, ONLY: A”
• Adds: Ad(NX,NY)=0.0, but NX and NY are not in scope. “USE MYMODULE, ONLY: NX,NY” added by hand.
14 Experiences with Tapenade and CFD
External library: PETSc
• The PETSc library is designed for parallel, scalable solutions of PDEs (https://www.mcs.anl.gov/petsc/index.html)
– Distributed matrix and vector data structures
– Linear and non-linear solvers
– Written in C with interfaces for C, C++, Fortran, and Python
– Built on the MPI library
• REX uses PETSc to store distributed matrices for use in PETSc’s linear solver routines.
15 Experiences with Tapenade and CFD
Using PETSc (1)
• Tapenade is designed to generate calls to derivative versions of unknown functions if it thinks they are necessary
• For PETSc, setting the value of a vector looks like:
– call VecSetValue(rhsp,nijk,pval,INSERT_VALUES,ierr)
– Which roughly translates to rhsp(nijk)=pval
• But to Tapenade, the arguments look like (integer, integer, real, integer, integer)
– Tapenade generates
call VecSetValue_D(rhsp,nijk,pval,pvald,INSERT_VALUES,ierr)
– Tapenade doesn’t see the rhsp vector data structure as differentiable
• These library calls need to be managed by hand.
– A new vector for rhspd was created to handle the differentials of rhsp
– Replace the Tapenade generated call with calls to set rhsp and rhspd
call VecSetValue(rhsp,nijk,pval,INSERT_VALUES,ierr)
call VecSetValue(rhspd,nijk,pvald,INSERT_VALUES,ierr)
16 Experiences with Tapenade and CFD
Using PETSc (2)
• PETSc’s linear solvers can be handled similarly to the example on Tapenade’s FAQ (https://www-sop.inria.fr/tropics/tapenade/faq.html#linearsolvers) for hand coding the SOLVE_D and SOLVE_B routines
– For linear solvers, hand coding is typically better than letting Tapenade compute the derivatives even when the solver source code is available.
• For forward mode, SOLVE becomes SOLVE_D which solves for 𝑦 and ሶ𝑦– 𝑦 = 𝐴−1𝑏 (as before)
– ሶ𝑦 = 𝐴−1 ሶ𝑏 − ሶ𝐴𝑦 , where ሶ𝑏 and ሶ𝐴 are inputs
17 Experiences with Tapenade and CFD
External library: MPI
• The message passing interface (MPI) library allows different compute nodes to communicate, sharing partial results and status messages.
• Automatic Differentiation adds data dependencies to the original program that must be accounted for in the communication patterns. Plain MPI libraries are not sufficient to handle these dependencies without additional hand coding.
• The Adjoinable MPI library (http://www.mcs.anl.gov/~utke/AdjoinableMPI/AdjoinableMPIDox/index.html) was written to alleviate pressure on code developers by providing an interface between AD tools and a modified MPI library suitable for computing adjoints.
– Used with AD tools like Tapenade, Adol-C, etc
18 Experiences with Tapenade and CFD
Using AMPI (1)
• From a user’s perspective, switching from MPI to AMPI involves code changes
– Changing MPI headers to AMPI headers (#include "mpif.h"⇒#include "ampi/ampif.h“)
– Changing function calls to “MPI_” to “AMPI_” version
• Use “NT” versions outside of AD regions
– Add arguments to AMPI_Recv and AMPI_Send type function to help map communications
• AMPI_TO_RECV, AMPI_TO_IRECV_WAIT, AMPI_FROM_SEND, AMPI_FROM_ISEND_WAIT
• The AMPI website suggests modifying MPI data types (e.g. MPI_REAL ⇒MPI_AREAL), but this does not appear to be necessary for Tapenade.
• Add AMPI include directory to compile step and link with the AMPI libraries
19 Experiences with Tapenade and CFD
Using AMPI (2)
• Initially, several issues between Tapenade and AMPI were apparent.
• Missing bits in the AMPI library
– AMPI library ampiPlainC missing functions AMPI_Bcast and AMPI_Reduce
– AMPI library ampiCommon missing functions for TLS_AMPI_SCATTER, TLS_AMPI_BCAST, TLS_AMPI_REDUCE, TLS_AMPI_ALLREDUCE
• Tapenade not generating correct code
– Tapenade attempts to add mpif.h and ampi/ampif.h, resulting in duplicate definitions
– AMPI_BARRIER calls become calls to AMPI_BARRIER_D rather than TLS_AMPI_BARRIER
– Tapenade generated calls to TLS_AMPI_ALLGATHER, TLS_AMPI_REDUCE, TLS_AMPI_ALLREDUCE, TLS_AMPI_BCAST, TLS_AMPI_SCATTER have wrong format
– Tapenade wants to create derivatives for MPI_INTEGER data types
20 Experiences with Tapenade and CFD
Using AMPI (3)
• Laurent Hascoet has been working with us to be sure that the AMPI library implements the missing functions
• Laurent Hascoet has been working to improve the Tapenade code as well
– We greatly appreciate his help!
• In addition, AMPI_Waitall is not implemented in the FORTRAN version of the AMPI library, as is noted on the AMPI website.
– As a workaround, we can loop over AMPI_WAIT calls
• Currently, the webpage user manual does not provide an example where both the “PlainC” and “Common” libraries are used. It may be necessary to link both libraries for code that contains undifferentiated routines.
– Both libraries have some common functions like AMPI_COMM_SIZE, so I’m not sure this was the original design
21 Experiences with Tapenade and CFD
Validating code
• Tapenade has some features to help validate its output against finite difference estimates.
– Command line options “debugTGT” and “debugADJ”
– Requires support code from Tapenade’s “ADFirstAidKit”
• ADFirstAidKit has both a C and Fortran version of “debugAD”, which are used differently but both require simultaneous two simultaneous runs of the code
– Fortran requires two executables with small changes between them
> ./test1 | ./test2
– C requires environment variables to be set so that the code can determine which phase is currently executing
> export DBAD_PHASE=1 && ./test > out.1.txt &
export DBAD_PHASE=2 && ./test
• Output from C and Fortran version not the same
22 Experiences with Tapenade and CFD
Questions?
Dr. Jacob BarhenCESAR Director and ORNL Fellow
Phone: 865-574-7131
E-mail: [email protected]
1 Bethel Valley Road
Bldg 5600, MS 6015
Oak Ridge, TN 37831-6015
USA
Dr. Charlotte KotasORNL/CESAR member of technical staff
Phone: 865-241-6597
E-mail: [email protected]
Center for Engineering Science Advanced Research (CESAR)Computer Science and Mathematics Division
Oak Ridge National Laboratory