Embed Size (px)
Citation preview
Coping With Simulators That Don’t Always Return
Andrew Warrington Saeid Naderiparizi Frank WoodUniversity of Oxford
University of British [email protected]
University of British [email protected]
Abstract
Deterministic models are approximations ofreality that are easy to interpret and of-ten easier to build than stochastic alterna-tives. Unfortunately, as nature is capri-cious, observational data can never be fullyexplained by deterministic models in prac-tice. Observation and process noise needto be added to adapt deterministic mod-els to behave stochastically, such that theyare capable of explaining and extrapolatingfrom noisy data. We investigate and addresscomputational inefficiencies that arise fromadding process noise to deterministic simula-tors that fail to return for certain inputs; aproperty we describe as “brittle.” We showhow to train a conditional normalizing flow topropose perturbations such that the simula-tor succeeds with high probability, increasingcomputational efficiency.
1 Introduction
In order to compensate for epistemic uncertainty dueto modelling approximations and unmodelled aleatoricuncertainty, deterministic simulators are often “con-verted” to “stochastic” simulators by perturbing thestate at each time step. In practice this allows the sim-ulator to explain the variability observed in real datawithout requiring excessive observation noise. Suchmodels are more resilient to misspecification, are ca-pable of providing uncertainty estimates, and providebetter inferences in general [Møller et al., 2011; Saari-nen et al., 2008; Lv et al., 2008; Pimblott and LaVerne,1990; Renard et al., 2013].
Often, state-independent Gaussian noise with heuris-
Proceedings of the 23rdInternational Conference on Artifi-cial Intelligence and Statistics (AISTATS) 2020, Palermo,Italy. PMLR: Volume 108. Copyright 2020 by the au-thor(s).
tically tuned variance is used to perturb the state [Ad-hikari and Agrawal, 2013; Brockwell and Davis, 2016;Fox, 1997; Reddy and Clinton, 2016; Du and Sam,2006; Allen, 2017; Mbalawata et al., 2013]. However,naively adding noise to the state will, in many ap-plications, render the perturbed input state “invalid,”where invalid states cause the simulator to raise anexception and not return a value [Razavi et al., 2019;Lucas et al., 2013; Sheikholeslami et al., 2019]. We for-mally define failure by extending the possible outputof the simulator to include ⊥ (read as “bottom”) de-noting simulator failure. The principal contribution ofthis paper is a technique for avoiding invalid states bychoosing perturbations that minimize the failure rate.The technique we develop results in a reduction in sim-ulator failures, while maintaining the original model.
Examples of failure modes include ordinary differen-tial equation (ODE) solvers not converging to the re-quired tolerance in the allocated time, or, the per-turbed state entering into an unhandled configura-tion, such as solid bodies intersecting. Establishingthe state-perturbation pairs that cause failure is non-trivial. Hence, the simulation artifact can be sensitiveto seemingly inconsequential alterations to the state– a property we describe as “brittle.” Failures wastecomputational resources and reduce the diversity ofsimulations for a finite sample budget, for instance,when used as the proposal in sequential Monte Carlo.As such, we wish to learn a proposal over perturbationssuch that the simulator exits with high probability, butrenders the joint distribution unchanged.
We proceed by framing sampling from brittle simula-tors as rejection samplers, then seek to eliminate rejec-tions by estimating the state-dependent density overperturbations that do not induce failure. We thendemonstrate that using this learned proposal yieldslower variance results when used in posterior infer-ence with a fixed sample budget, such as pseudo-marginal evidence estimates produced by sequentialMonte Carlo sweeps. Source code for reproductionof figures and results in this paper is available athttps://github.com/plai-group/stdr.
arX
iv:2
003.
1290
8v1
[cs
.LG
] 2
8 M
ar 2
020
Coping With Simulators That Don’t Always Return
2 Background
2.1 Smoothing Deterministic Models
Deterministic simulators are often stochastically per-turbed to increase the diversity of the achievable sim-ulations and to fit data more effectively. The mostwidespread example of this is perturbing linear dy-namical systems with Gaussian noise at each timestep.The design of the system is such that the distribu-tion over state at each point in time is Gaussian dis-tributed. However, the simplistic dynamics of such asystem may be insufficient for simulating more com-plex systems. Examples of such systems are: stochas-tic models of neural dynamics [Fox, 1997; Coutin et al.,2018; Goldwyn and Shea-Brown, 2011; Saarinen et al.,2008], econometrics [Lopes and Tsay, 2011], epidemi-ology [Allen, 2017] and mobile robotics [Thrun et al.,2001; Fallon et al., 2012]. In these examples, the sim-ulator state is perturbed with noise drawn from a dis-tribution and is iterated using the simulator to creatediscrete approximations of the distribution over stateas a function of time.
2.2 Simulator Failure
As simulators become more complex, guaranteeing thesimulator will not fail for perturbed inputs becomesmore difficult, and individual function evaluations be-come more expensive. Lucas et al. [2013] and Edwardset al. [2011] establish the sensitivity of earth sciencemodels to global parameter values by building a dis-criminative classifier for parameters that induce fail-ure. Sheikholeslami et al. [2019] take an alternativeapproach instead treating simulator failure as an im-putation problem, fitting a function regressor to pre-dict the outcome of the failed experiment given theneighboring experiments that successfully terminated.However these methods are limited by the lack of clearprobabilistic interpretation in terms of the originallyspecified joint distribution in time series models, theirability to scale to high dimensions, and their applica-bility to state-space models.
2.3 State-space Inference and ModelSelection
Probabilistic models are ultimately deployed to makeinferences about the world. Hence the goal is to beable to recover distributions over unobserved states,predict future states and learn unknown parametersof the model from data. Posterior state-space infer-ence refers to the task of recovering the distributionpM(x0:T |y1:T ), where x0:T are the latent states, y1:T
are the observed data, and M denotes the model ifmultiple different models are available. Inference in
Algorithm 1 Sequential Monte Carlo
1: procedure SMC(pM(x0), pM(xt|xt−1), y1:T ,pM(yt|xt), N)
2: for n = 1 : N do3: x
(n)0 ∼ pM(x0) . Initialize from prior.
4: LM ← 0 . Track log-evidence5: for t = 1 : T do6: for n = 1 : N do7: x
(n)t ∼ pM
(xt|x(n)
t−1
). Alg 2.
8: w(n)t ← pM
(yt|x(n)
t
). Score particle.
9: for n = 1 : N do . Normalize weights.
10: W(n)t ← w
(n)t /
∑Ni=1 w
(i)t
11: for n = 1 : N do . Apply resampling.
12: a(n)t ∼ Discrete (Wt)
13: x(n)t ← x
(a(n)t
)t
14: LM ← LM + log(
1N
∑Ni=1 w
(i)t
)15: return x
(1:N)0:T , a
(1:N)1:T , LM
16: end procedure
Gaussian perturbed linear dynamical systems can beperformed using techniques such as Kalman smooth-ing [Kalman, 1960], however, the restrictions on suchtechniques limit their applicability to complex simu-lators, and so numerical methods are often used inpractice.
A common method for performing inference in com-plex, simulation based models is sequential MonteCarlo (SMC) [Doucet et al., 2001]. The basic algo-rithm for SMC is shown in Algorithm 1, where pM(x0)is the prior over initial state, pM(xt|xt−1) is the dy-namics model, or simulator, pM(yt|xt) is the likeli-hood, defining the relationship between latent statesand observed data, and N is the number of particlesused. On a high level, SMC produces a discrete ap-proximation of the target distribution by iterating par-ticles through the simulator, and then preferentiallycontinuing those simulations that “explain” the ob-served data well. While a detailed understanding ofparticle filtering is not required, the core observationrequired for this work is that the likelihood of failedsimulations is defined as zero: p(yt|xt = ⊥) := 0, andhence are rejected with certainty.
Posterior inference pipelines often also provide esti-mates of the model evidence, pM(y1:T ). SMC providessuch an estimate, referred to as a pseudo-marginal ev-idence, denoted in Algorithm 1 as LM. This pseudo-marginal evidence is calculated (in log space) as thesum of the expected value of the unnormalized impor-tance weights (Algorithm 1, Lines 8 and 14). Thisevidence can be combined with the prior probability
Andrew Warrington, Saeid Naderiparizi, Frank Wood
of each model via Bayes rule to estimate the poste-rior probability of the model (up to a normalizing con-stant) [MacKay, 2003]. These posteriors can be com-pared to perform Bayesian model selection, where themodel with the highest posterior is selected and used toperform inference. This is often referred to as marginalmaximum a posteriori parameter estimation (or modelselection) [Doucet et al., 2002; Kantas et al., 2015]. Re-cent work investigates model selection using approxi-mate, likelihood-free inference techniques [Papamakar-ios et al., 2019; Lueckmann et al., 2019], however, wedo not consider these methods here, instead focusingon mitigating computational inefficiencies arising di-rectly from simulator failure.
3 Methodology
We consider deterministic models, expressed as simu-lators, describing the time-evolution of a state xt ∈ X ,where we denote application of the simulator iteratingthe state as xt ← f(xt−1). A stochastic, additive per-turbation to state, denoted zt ∈ X , is applied to inducea distribution over states. The distribution from whichthis perturbation is sampled is denoted p(zt|xt−1), al-though, in practice, this distribution is often state in-dependent. The iterated state is then calculated asxt ← f(xt−1 + zt).
However, we consider the case where the simulatorcan fail for “invalid” inputs, denoted by a returnvalue of ⊥. Hence the complete definition of f isf : X → {X ,⊥}. The region of valid inputs is de-noted as XA ⊂ X , and the region of invalid inputs asXR ⊂ X , such that XA t XR = X , where the bound-ary between these regions is unknown. Over the wholesupport, f defines a many-to-one function, as XR mapsto ⊥. However, the algorithm we derive only requiresthat f is one-to-one in the accepted region. This is notuncommon in real simulators, and is satisfied by, forexample, ODE models. We define the random variableAt ∈ {0, 1} to denote whether the state-perturbationpair does not yield simulator failure and is “accepted.”
We define the iteration of perturbed deterministic sim-ulator as a rejection sampler, with a well-defined tar-get distribution (§3.1). We use this definition andthe assumptions on f to show that we can targetthe same distribution by learning the state-conditionaldensity of perturbations, conditioned on acceptance(§3.2). We train an autoregressive flow to fit this den-sity (§3.3), and describe how this can be used in in-ference, highlighting the ease with which it can be in-serted into a particle filter (§3.4). We empirically showthat using this learned proposal distribution in placeof the original proposal improves the performance ofparticle-based state-space inference methods (§4).
Algorithm 2 Iterate brittle simulator, p(xt|xt−1).
1: procedure IterateSimulator(f , xt−1)2: xt ← ⊥3: while xt == ⊥ do4: if qφ is trained then5: zt ∼ qφ(zt|xt−1) . Perturb under qφ.6: else7: zt ∼ p(zt|xt−1) . Perturb under p.
8: xt ← f(xt−1 + zt) . Iterate simulator.
9: return xt10: end procedure
3.1 Brittle Simulators as Rejection Samplers
The naive approach to sampling from the perturbedsystem, shown in Algorithm 2, is to repeatedly samplefrom the proposal distribution and evaluate f until thesimulator successfully exits. This procedure definesAt = I [f(xt−1 + zt) 6= ⊥] , zt ∼ p(zt|xt−1), i.e. suc-cessfully iterated samples are accepted with certainty.This incurs significant wasted computation as the sim-ulator must be called repeatedly, with failed iterationsbeing discarded. The objective of this work is to derivea more efficient sampling mechanism.
We begin by establishing Algorithm 2 as a rejectionsampler, targeting the distribution over successfully it-erated states. This reasoning is illustrated in Figure 1.The behavior of f and the distribution p(zt|xt−1)implicitly define a distribution over successfully iter-ated states. We denote this “target” distribution asp(xt|xt−1) = p(xt|xt−1, At = 1), where the bar indi-cates that the sample was accepted, and hence placesno probability mass on failures. Note there is no bar onp(zt|xt−1), indicating that it is defined before the ac-cept/reject behaviors of f and hence probability massmay be placed on regions that yield failure. The func-tional form of p is unavailable, and the density cannotbe evaluated for any input value.
The existence of p(xt|xt−1) implies the existence ofa second distribution: the distribution over acceptedperturbations, denoted p(zt|xt−1). Note that this dis-tribution is also conditioned on acceptance under thechosen simulator, indicated by the presence of a bar.We assume f is one-to-one in the accepted region, andso the change of variables rule can be applied to di-rectly relate this to p(xt|xt−1). Under our initial al-gorithm for sampling from a brittle simulator we cantherefore write the following identity:
p(zt|xt−1) =
{1Mpp(zt|xt−1), if f(xt−1 + zt) 6= ⊥
0, otherwise
(1)where the normalizing constant Mp is the acceptance
Coping With Simulators That Don’t Always Return
0
1
�2 0 2
zt
0
1
I [f(zt) 6= ?]
Mpp(zt)
p(zt)
Figure 1: Graphical representation of how a brittle de-terministic simulator acts as a rejection sampler, tar-geting p(zt|xt−1). We set xt = 0 for clarity. The sim-ulator, f(zt), returns ⊥ for unknown input regions,shown in green. The proposal over zt is shown in blue.The target distribution, p(zt), shown in orange, is im-plicitly defined as p(zt) = 1
Mpp(zt)I [f(zt) 6= ⊥], where
Mp is the normalizing constant from p, equal to the ac-ceptance rate. Accordingly, the proposal distribution,scaled by Mp, is exactly equal to p(zt) in the acceptedregion. Algorithm 2 therefore implicitly constructs arejection sampler, where the acceptance criterion re-duces to I [f(zt) 6= ⊥], without needing to specify anyadditional scaling constants.
rate under p. (1) indicates accepting with certaintyperturbations that exit successfully can be seen as pro-portionally shifting mass from regions of p where thesimulator fails to regions where it does not. We exploitthis definition to learn an efficient proposal.
3.2 Change of Variable in Brittle Simulator
We now derive how we can learn the proposal distribu-tion, denoted qφ and parameterized by φ, to replace p,such that the acceptance rate under qφ (denoted Mqφ)tends towards unity, minimizing wasted computation.We denote qφ as the proposal we train, which, coupledwith the simulator, implicitly defines a proposal overaccepted samples, denoted qφ.
Expressing this mathematically, we wish to minimizethe distance between joint distribution implicitly spec-ified over accepted iterated states using the a priorispecified proposal distribution, p, and qφ:
φ∗ = arg minφ
Ep(xt−1)
[DKL
[p(xt|xt−1)||qφ(xt|xt−1)
]],
(2)where we select the Kullback-Leibler (KL) divergenceas the metric of distance between distributions. Theouter expectation defines this objective as amortizedacross state space, where we can generate the samplesby directly sampling trajectories from the model [Leet al., 2017; Gershman and Goodman, 2014]. Weuse the forward KL as opposed to the reverse KL,DKL
[qφ(xt|xt−1)||p(xt|xt−1)
], as high-variance RE-
INFORCE estimators must be used to obtain the the
differential with respect to φ of the reverse KL.
Expanding the KL term yields:
φ∗ = arg minφ
Ep(xt−1)Ep(xt|xt−1) [log (w)] , (3)
w =p(xt|xt−1)
qφ(xt|xt−1). (4)
Noting that qφ and p are defined only on acceptedsamples, where f is one-to-one, we can apply a changeof variables defined for qφ as:
qφ(xt|xt−1) = qφ(f−1(xt)|xt−1)
∣∣∣∣df−1(xt)
dxt
∣∣∣∣ , (5)
and likewise for p. This transforms the distributionover xt into a distribution over zt and a Jacobian term:
w =p(f−1(xt)|xt−1)
∣∣∣df−1(xt)dxt
∣∣∣qφ(f−1(xt)|xt−1)
∣∣∣df−1(xt)dxt
∣∣∣ . (6)
taking care to also apply the change of variables in thedistribution we are sampling from in (3). Noting thatthe same Jacobian terms appear in the numerator anddenominator we are able to cancel these:
w =p(f−1(xt)|xt−1)
qφ(f−1(xt)|xt−1). (7)
We can now discard the p term as it is independent ofφ. Noting f−1(xt) = xt−1 + zt we can write (2) as:
φ∗ = argmaxφ
Ep(xt−1)Ep(zt|xt−1)
[log qφ(zt|xt−1)
]. (8)
However, this distribution is defined after rejectionsampling, and can only be defined as in (1):
qφ(zt|xt−1) = qφ(zt|xt−1, At = 1) (9)
=
{1
Mqφqφ(zt|xt−1) if f(xt−1 + zt) 6= ⊥,
0 otherwise,
denoting Mqφ as the acceptance rate under qφ.
However, there is an infinite family of qφ proposalsthat yield p = qφ, each a maximizer of (8) but withdifferent rejection rates. Noting however that there isonly a single qφ that has a rejection rate of zero andrenders qφ = p, and that this distribution also rendersqφ = qφ, we can instead optimize qφ(zt|xt−1):
φ∗ = argmaxφ
Ep(xt−1)Ep(zt|xt−1) [log qφ(zt|xt−1)] ,
(10)with no consideration of rejection behavior under qφ.
One might alternatively try to achieve low rejectionrates by adding a regularization term to (8) penal-izing high Mqφ . However, differentiation of Mqφ is
Andrew Warrington, Saeid Naderiparizi, Frank Wood
Algorithm 3 Training qφ
1: procedure TrainQ(p(x), p(zt|xt−1), K, N , q,φ0, η)
2: for k = 0 : K − 1 do3: for n = 1 : N do4: x
(n)t−1 ∼ p(x) . Sample from prior.
5: z(n)t ∼ p(z(n)t |x(n)
t−1) . Sample noise.
6: Ek ←∏Nn=1 qφk
(z(n)t |x(n)
t−1
)7: Gk ← ∇φkEk . Do backprop.8: φk+1 ← φk + η (Gk) . Apply update.
9: return φK . Return learned parameters.10: end procedure
intractable, meaning direct optimization of (8) is in-tractable.
The objective stated in (10) implicitly rewards qφ dis-tributions that place minimal mass on rejections byplacing as much mass on accepted samples as possi-ble. This expressionn is differentiable with respect to φand so we can maximize this quantity through gradientascent with minibatches drawn from p(xt−1). This ex-pression shows that we can learn the distribution overaccepted xt values by learning the distribution over theaccepted zt, without needing to calculate the Jacobianor inverse of f . Doing so minimizes wasted computa-tion, targets the same overall joint distribution, andretains interpretability by utilizing the simulator.
3.3 Training qφ
To train qφ(zt|xt−1) we first define a method for sam-pling state-perturbation pairs. We initialize the simu-lator to a state sampled from a distribution over initialvalue, and then iterate the perturbed simulator for-ward for some finite period. All state-perturbationpairs sampled during the trajectory are treated asa training example, and, in total, represent a dis-crete approximation to the prior over state for alltime, and accepted state-conditional perturbation, i.e.xt−1 ∼ p(x) and zt ∼ p(zt|xt−1).
We train the conditional density estimator qφ(zt|xt−1)using these samples by maximizing the conditionaldensity of the sampled perturbation under the truedistribution, as in (10), as this minimizes the desiredKL divergence originally stated in (2). Our condi-tional density estimator is fully differentiable and canbe trained through stochastic gradient ascent as shownin Algorithm 3. The details of the chosen architectureis explained in §3.5. The result of this procedure isan artifact that approximates the density over validperturbations conditioned on state, p(zt|xt−1).
𝜖!
𝐱"#$
𝐳"𝜖$ 𝜖%…
…ℎ$ ℎ%
+
𝜙%𝜙$
𝑓AF
Hypernetwork
𝐱"
Conditional AF𝑞& 𝐳"| 𝐱"#$
Figure 2: Diagram visualizing how qφ is structured andused. The previous state is input to the hypernetwork,a series of L single layer neural networks, denoted hl.Each network outputs parameters, denoted φl, for eachof the L layers in the flow conditioned on the state.The flow samples a perturbation as zt ∼ qφ (zt|xt−1),with the internal states of the flow denoted by εl. Thisperturbation is summed with the previous state andpassed through the simulator, f , outputting the iter-ated state, xt.
3.4 Using qφ
Once qφ has been trained, it can be deployed to en-hance posterior inference, by replacing samples fromp(zt|xt−1) with qφ(zt|xt−1). We highlight here theease with which it can be introduced into an SMCsweep. The state is iterated by sampling fromp(xt|xt−1) on Line 7 of Algorithm 1, where this sam-pling procedure is defined in Algorithm 2. Instead ofsampling from p(zt|xt−1) in Algorithm 2, the sampleis drawn from qφ(zt|xt−1), and as such the sample ismore likely to be accepted. This modification requiresonly changing a single function call made inside theimplementation of Algorithm 2.
3.5 Implementation
We parameterize the density qφ using an autoregres-sive flow (AF) [Larochelle and Murray, 2011]. Flowsdefine a parameterized density estimator that can betrained using stochastic gradient descent, and vari-ants have been used in image generation [Kingmaand Dhariwal, 2018], as priors for variational autoen-coders [Kingma et al., 2016], and in likelihood-free in-ference [Papamakarios et al., 2019; Lueckmann et al.,2019].
Specifically, we structure qφ using a masked autore-gressive flow [Papamakarios et al., 2017], with 5single-layer MADE blocks [Germain et al., 2015], andbatch normalization at the input to each intermediateMADE block. The dimensionality of the flow is thenumber of states perturbed in the original model. Weimplement conditioning through the use of a hypernet-work [Ha et al., 2016], which outputs the parameters of
Coping With Simulators That Don’t Always Return
the flow layers given xt−1 as input, as shown in Figure2. The hypernetworks are single-layer neural networksdefined per flow layer. Together, the flow and hyper-network define qφ(zt|xt−1), and can be jointly trainedusing stochastic gradient descent. The networks areimplemented in PyTorch [Paszke et al., 2017] and areoptimized using ADAM [Kingma and Ba, 2014].
4 Experiments
4.1 Toy Problem – Annulus
We first demonstrate our approach on a toy problem.The true generative model of the observed data is aconstant speed circular orbit around the origin in thex-y plane, such that xt = {xt, yt, xt, yt} ∈ R4. Toanalyze this data we use a misspecified model thatonly simulates linear forward motion. To overcomethe model mismatch and fit the observed data, we addGaussian noise to position and velocity. We impose afailure constraint limiting the change in the distanceof the point from the origin to a fixed threshold. Thiscondition mirrors our observation that states in brittlesimulators have large allowable perturbations in par-ticular directions, but very narrow permissible pertur-bations in other directions. The true radius is un-known and so we must amortize over possible radii.
The results of this experiment are shown in Figure 3.The interior of the black dashed lines in Figure 3a in-dicates the permissible x-y perturbation, for the givenposition and zero velocity, where we have centered eachdistribution on the current position for ease of visualinspection. Red contours indicate the original densityp(zt|xt−1), and blue contours indicate the learned den-sity qφ(zt|xt−1). The fraction of the probability massoutside the black dashed region is the expected rejec-tion rate. Figure 3b shows the rejection rate dropsfrom approximately 75% under the original model toapproximately 4% using a trained qφ.
We then use the learned qφ as the perturbation pro-posal in an SMC sweep, where we condition on noisyobservations of the x-y coordinates. As we focus onthe sample efficiency of the sweep, we fix the numberof calls to the simulator in Algorithm 2 to a singlecall, instead of proposing and rejecting until accep-tance. Failed particles are then not resampled (withcertainty) during the resampling. This means thateach iteration of the SMC makes a fixed number ofcalls to the simulator, and hence we can compare algo-rithms under a fixed sample budget. Figure 3c showsthat we recover lower variance evidence approxima-tions for a fixed sample budget by using qφ insteadof p. A paired t-test evaluating the difference in vari-ance returns a p-value of less than 0.0001, indicating a
−1.25−1.00−0.75
0.8
1.0
1.2
−0.25 0.00 0.25
1.8
2.0
2.2
0.75 1.00 1.25
0.8
1.0
1.2
−2.25−2.00−1.75
−0.2
0.0
0.2
−0.25 0.00 0.25
−0.2
0.0
0.2
1.75 2.00 2.25
−0.2
0.0
0.2
−1.25−1.00−0.75
−1.2
−1.0
−0.8
−0.25 0.00 0.25
−2.2
−2.0
−1.8
0.75 1.00 1.25
−1.2
−1.0
−0.8
(a)
0 100 200
Training epoch
0.0
0.5
1.0F
ailu
rera
te
p
q
(b)
p qφ0
10
20
SD
[p(y
)]
(c)
Figure 3: Results for the annulus problem introducedin Section 4.1, where the acceptable region of perturba-tions is inside the black dashed band. 3a shows in bluethe learned state-dependent proposal distribution overvelocity (for a state at rest) is the well-approximatingthe original proposal (shown in red) inside the accept-able region, with minimal mall in the invalid region,all but eliminating rejection as shown in 3b. 3c showsthe reduction in the variance of the evidence by usingqφ. We compute the variance using 100 independentSMC sweeps, each using 100 particles, and compareacross 100 datasets.
strong statistical difference between the performanceunder p and qφ, confirming that using qφ increases thefidelity of inference for a fixed sample budget.
4.2 Bouncing Balls
Our second example uses a simulator of balls bouncingelastically, as shown in Figure 4a. We model the posi-tion and velocity of each ball, such that the dimension-ality of the state vector, xt, is four times the numberof balls. We add a small amount of Gaussian noiseat each iteration to the position and velocity of eachball. This perturbation induces the possibility thattwo balls overlap, or, a ball intersects with the wall,
Andrew Warrington, Saeid Naderiparizi, Frank Wood
t = 1 t = 4 t = 7 t = 10
(a)
4 5 6
24
25
26
13 14 15
24
25
26
24 25 26
24
25
26
4 5 6
14
15
16
13 14 15
14
15
16
24 25 26
14
15
16
4 5 6
4
5
6
13 14 15
4
5
6
24 25 26
4
5
6
(b)
0 20
0
10
20
30
0 20
0
10
20
30
0.0
0.2
0.4
0.6
0.8
1.0
p qφ
(c)
Figure 4: Results of the bouncing balls experimentintroduced in Section 4.2, with two radius five, unitmass balls in an enclosure of size 30. 4a shows an ex-ample trajectory of the system. 4b shows, in red, theproposal distribution over the perturbation to the po-sition of the first ball specified in the model, and thelearned proposal in blue. The edge of the permissibleregion of the enclosure is shown as a black dashed line.The second ball is fixed at [25, 15], and the inducedinvalid region shaded green. The flow has learned todeflect away from the disallowed regions. 4c shows therejection rate as a function of the position of the firstball, with the second ball in the position shown. Thetrained proposal (right) has all but eliminated rejec-tion in the permissible space compared to the a-priorispecified proposal (left). The rejection rate under p ishigh in the interior as the second ball may also leavethe enclosure, whereas qφ has practically eliminatedrejection by jointly proposing perturbations.
representing an invalid physical configuration and re-sults in simulator failure. We note that here, we areconditioning on the state of all balls simultaneously,and proposing the perturbation to the state jointly.
Figure 4b shows the distribution over position pertur-bation of a single ball, conditioned on the other ballbeing stationary. Blue contours show the estimateddistribution over accepted perturbations learned byautoregressive flow. Figure 4c shows the rejection rateunder p and qφ as a function of the position of thefirst ball, with the second ball fixed in the positionshown, showing that rejection has been all but elimi-nated. We again see a reduction in the variance of theevidence approximation computed by a particle filterwhen using qφ instead of p (figure in the supplemen-tary materials).
4.3 MuJoCo
We now apply our method to the popular robotics sim-ulator MuJoCo [Todorov et al., 2012], specifically us-ing the built-in example “tosser,” where a capsule is“tossed” by an actuator into a bucket, shown in Fig-ure 5a. Tosser displays “choatic” aspects, as minorchanges in the position of the object results in largechanges in the trajectories achieved by the simulator.
MuJoCo allows some overlap between the objects tosimulate contact dynamics. This is an example ofmodel misspecification borne out of the requirementsof reasonably writing a simulator. We therefore placea hard limit on the amount objects are allowed to over-lap. This is an example of a user-specified constraintthat requires the simulator to be run to evaluate. Weadd Gaussian distributed noise to the position and ve-locity of the capsule.
Figure 5 shows the results of this experiment. Thecapsule is mostly in free space resulting in an averagerejection rate under p of 10%. Figure 5b shows thatthe autoregressive flow learns a proposal with a lowerrejection rate, reaching 3% rejection. However theserejections are concentrated in the critical regions ofstate-space, where chaotic behavior occurs, and so thisreduction yields an large reduction in the variance ofthe evidence approximation, as shown in Figure 5c.
We conclude this example by evaluating our methodon hypothesis testing using pseudo-marginal evidenceestimates. The results for this are shown in Figure5d. We test 5 different hypothesis of the mass of thecapsule. Using p results in higher variance evidenceapproximations than when qφ is used. Additionally,under p the wrong model is selected (2 instead of 3),although with low significance (p = 0.125), while us-ing qφ selects the correct hypothesis with p = 0.0127.For this experiment we note that qφ was trained on
Coping With Simulators That Don’t Always Return
(a)
0 50 100
Training epoch
0.0
0.1
0.2
Fai
lure
rate p q
(b)
p qφ0
50
100
SD
[p(y
)]
(c)
1 2 3 4 51200
1300
p(y
|M)
p
1 2 3 4 5
q�
0.0 0.2 0.4 0.6 0.8 1.0
Hypothesis, M
0.0
0.5
1.0
(d)
Figure 5: Results of the “tosser” experiment intro-duced in Section 4.3. 5a shows the evolution of stateover time. 5b shows the AF we learn markedly re-duces the number of rejections. 5c shows the results ofperforming SMC using the a priori specified proposaland our learned autoregressive flow. The autoregres-sive flow attains a much lower variance estimate, witha p-value of less than 0.0001 in a paired t-test, indi-cating a strong statistical difference in performance.5d shows the results of performing hypothesis testing,where hypothesis 3 is correct, under a uniform priorover hypothesis. The incorrect hypothesis is selectedusing p, while using qφ the correct hypothesis is se-lected, with a statistically significant confidence.
a single value of mass, and that this “training mass”was different to the “testing mass.” We believe thiscontributes to the increased variance in hypothesis 1,which is very light compared to the training mass.Training a qφ with a further level of amortization overdifferent mass values would further increase the fidelityof the model selection. This is intimately linked withthe larger project of jointly learning the model, and sowe defer investigation to future works.
4.4 Neuroscience Simulator
We conclude by applying our algorithm to a simulatorfor the widely studied Caenorhabditis elegans round-worm. WormSim, presented by Boyle et al. [2012], isa simulator of the locomotion of the worm, using a510 dimensional state representation. We apply per-turbations to a 98 dimensional subspace defining thephysical position of the worm, while conditioning on
10−1010−810−610−4
SD [p(zt|xt−1)]
0
1
Fai
lure
rate
(a)
0 75 150
Training epoch
0.0
0.5
1.0
Fai
lure
rate
p q
(b)
Figure 6: Results from the WormSim example intro-duced in Section 4.4. 6a shows the rate at which thesimulator fails increases sharply as a function of thestandard deviation of the applied perturbation. 6bshows the reduction in rejections during training.
the full 510 dimensional state vector. The expectedrate of failure increases sharply as a function of thescale of the perturbation applied, as shown in Figure6a, as the integrator used in WormSim is unable tointegrate highly perturbed states.
The rejection rate during training is shown in Figure6b. We are able to learn an autoregressive flow withlower rejection rates, reaching approximately 53% re-jection, when p has approximately 75% rejection. Al-though the rejection rate is higher than ultimately de-sired, we include this example as a demonstration ofhow rejections occur in simulators through integratorfailure. We believe larger flows with regularized pa-rameters can reduce the rejection rate further.
5 Conclusion
In this paper we have tackled reducing simulator fail-ures caused by naively perturbing the input state. Weachieve this by showing that stochastically perturbedsimulators define a rejection sampler with a well de-fined target distribution and learning a conditionalautoregressive flow to estimate the state-dependentproposal distribution conditioned on acceptance. Wethen show that using this learned proposal reducesthe variance of inference results, with applications forBayesian model selection. We believe this work hasreadily transferable practical contributions to not justthe machine learning community, but the wider scien-tific community where such naively modified simula-tion platforms are widely deployed. As part of the ex-periments we present, we identify an extension: intro-ducing an additional level of amortization over staticsimulation parameters. This extension builds towardsour larger research vision of building toolchains for effi-cient inference and learning in brittle simulators. Fur-ther development will facilitate efficient gradient-basedmodel learning in these brittle simulators.
Andrew Warrington, Saeid Naderiparizi, Frank Wood
6 Acknowledgements
Andrew Warrington is supported under the ShilstonScholarship, awarded by Keble College and the De-partment of Engineering Science, University of Oxford.Saeid Naderiparizi and Frank Wood are supported un-der the Natural Sciences and Engineering ResearchCouncil of Canada (NSERC), the Canada CIFAR AIChairs Program, Compute Canada, Intel, CompositesResearch Network (CRN) and DARPA under its D3Mand LWLL programs.
References
R. Adhikari and R. K. Agrawal. An introductorystudy on time series modeling and forecasting. arXivpreprint arXiv:1302.6613, 2013.
L. J. Allen. A primer on stochastic epidemic models:Formulation, numerical simulation, and analysis. In-fectious Disease Modelling, 2(2):128–142, 2017.
J. H. Boyle, S. Berri, and N. Cohen. Gait modulationin c. elegans: an integrated neuromechanical model.Frontiers in computational neuroscience, 6:10, 2012.
P. J. Brockwell and R. A. Davis. Introduction to timeseries and forecasting. springer, 2016.
L. Coutin, J.-M. Guglielmi, and N. Marie. On a frac-tional stochastic hodgkin–huxley model. Interna-tional Journal of Biomathematics, 11(05):1850061,2018.
A. Doucet, N. De Freitas, and N. Gordon. An in-troduction to sequential monte carlo methods. InSequential Monte Carlo methods in practice, pages3–14. Springer, 2001.
A. Doucet, S. J. Godsill, and C. P. Robert. Marginalmaximum a posteriori estimation using markovchain monte carlo. Statistics and Computing, 12(1):77–84, 2002.
N. H. Du and V. H. Sam. Dynamics of a stochas-tic lotka–volterra model perturbed by white noise.Journal of mathematical analysis and applications,324(1):82–97, 2006.
N. R. Edwards, D. Cameron, and J. Rougier. Precal-ibrating an intermediate complexity climate model.Climate dynamics, 37(7-8):1469–1482, 2011.
M. F. Fallon, H. Johannsson, and J. J. Leonard. Effi-cient scene simulation for robust monte carlo local-ization using an rgb-d camera. In 2012 IEEE in-ternational conference on robotics and automation,pages 1663–1670. IEEE, 2012.
R. F. Fox. Stochastic versions of the hodgkin-huxleyequations. Biophysical journal, 72(5):2068–2074,1997.
M. Germain, K. Gregor, I. Murray, and H. Larochelle.Made: Masked autoencoder for distribution esti-mation. In International Conference on MachineLearning, pages 881–889, 2015.
S. Gershman and N. Goodman. Amortized inference inprobabilistic reasoning. In Proceedings of the annualmeeting of the cognitive science society, volume 36,2014.
J. H. Goldwyn and E. Shea-Brown. The what andwhere of adding channel noise to the hodgkin-huxleyequations. PLOS Computational Biology, 7(11):1–9,11 2011.
D. Ha, A. Dai, and Q. V. Le. Hypernetworks. arXivpreprint arXiv:1609.09106, 2016.
R. Kalman. A new approach to linear filtering andprediction problems. Transactions of the ASME–Journal of Basic Engineering, 82(Series D):35–45,1960.
N. Kantas, A. Doucet, S. S. Singh, J. Maciejowski,N. Chopin, et al. On particle methods for parameterestimation in state-space models. Statistical science,30(3):328–351, 2015.
D. P. Kingma and J. Ba. Adam: A method for stochas-tic optimization. arXiv preprint arXiv:1412.6980,2014.
D. P. Kingma and P. Dhariwal. Glow: Generativeflow with invertible 1x1 convolutions. In Advancesin Neural Information Processing Systems, pages10215–10224, 2018.
D. P. Kingma, T. Salimans, R. Jozefowicz, X. Chen,I. Sutskever, and M. Welling. Improved variationalinference with inverse autoregressive flow. In Ad-vances in neural information processing systems,pages 4743–4751, 2016.
H. Larochelle and I. Murray. The neural autoregressivedistribution estimator. In Proceedings of the Four-teenth International Conference on Artificial Intel-ligence and Statistics, pages 29–37, 2011.
T. A. Le, A. G. Baydin, and F. Wood. Inference com-pilation and universal probabilistic programming.In Proceedings of the 20th International Conferenceon Artificial Intelligence and Statistics (AISTATS),volume 54 of Proceedings of Machine Learning Re-search, pages 1338–1348, Fort Lauderdale, FL, USA,2017. PMLR.
H. F. Lopes and R. S. Tsay. Particle filters andbayesian inference in financial econometrics. Journalof Forecasting, 30(1):168–209, 2011. doi: 10.1002/for.1195.
D. Lucas, R. Klein, J. Tannahill, D. Ivanova, S. Bran-don, D. Domyancic, and Y. Zhang. Failure analy-sis of parameter-induced simulation crashes in cli-
Coping With Simulators That Don’t Always Return
mate models. Geoscientific Model Development, 6(4):1157–1171, 2013.
J.-M. Lueckmann, G. Bassetto, T. Karaletsos, andJ. H. Macke. Likelihood-free inference with emu-lator networks. In Symposium on Advances in Ap-proximate Bayesian Inference, pages 32–53, 2019.
Q. Lv, M. K. Schneider, and J. W. Pitchford. Individ-ualism in plant populations: Using stochastic differ-ential equations to model individual neighbourhood-dependent plant growth. Theoretical Population Bi-ology, 74(1):74 – 83, 2008. ISSN 0040-5809. doi:https://doi.org/10.1016/j.tpb.2008.05.003.
D. J. MacKay. Information theory, inference andlearning algorithms. Cambridge university press,2003.
I. S. Mbalawata, S. Sarkka, and H. Haario. Parameterestimation in stochastic differential equations withmarkov chain monte carlo and non-linear kalman fil-tering. Computational Statistics, 28(3):1195–1223,2013.
J. K. Møller, H. Madsen, and J. Carstensen. Parameterestimation in a simple stochastic differential equa-tion for phytoplankton modelling. Ecological mod-elling, 222(11):1793–1799, 2011.
G. Papamakarios, T. Pavlakou, and I. Murray. Maskedautoregressive flow for density estimation. In Ad-vances in Neural Information Processing Systems,pages 2338–2347, 2017.
G. Papamakarios, D. Sterratt, and I. Murray. Sequen-tial neural likelihood: Fast likelihood-free inferencewith autoregressive flows. In The 22nd InternationalConference on Artificial Intelligence and Statistics,pages 837–848, 2019.
A. Paszke, S. Gross, S. Chintala, G. Chanan, E. Yang,Z. DeVito, Z. Lin, A. Desmaison, L. Antiga, andA. Lerer. Automatic differentiation in PyTorch. InNIPS Autodiff Workshop, 2017.
S. M. Pimblott and J. A. LaVerne. Comparison ofstochastic and deterministic methods for modelingspur kinetics. Radiation Research, 122(1):12–23,1990. ISSN 00337587, 19385404.
S. Razavi, R. Sheikholeslami, H. V. Gupta, andA. Haghnegahdar. Vars-tool: A toolbox for com-prehensive, efficient, and robust sensitivity and un-certainty analysis. Environmental Modelling & Soft-ware, 112:95 – 107, 2019. ISSN 1364-8152.
K. Reddy and V. Clinton. Simulating stock prices us-ing geometric brownian motion: Evidence from aus-tralian companies. Australasian Accounting, Busi-ness and Finance Journal, 10(3):23–47, 2016.
P. Renard, A. Alcolea, and D. Gingsbourger. Stochas-tic versus deterministic approaches. In Environmen-
tal Modelling: Finding Simplicity in Complexity,Second Edition (eds J. Wainwright and M. Mulli-gan), pages 133–149. Wiley Online Library, 2013.
A. Saarinen, M.-L. Linne, and O. Yli-Harja. Stochasticdifferential equation model for cerebellar granule cellexcitability. PLOS Computational Biology, 4(2):1–11, 02 2008. doi: 10.1371/journal.pcbi.1000004.
R. Sheikholeslami, S. Razavi, and A. Haghnegahdar.What do we do with model simulation crashes? rec-ommendations for global sensitivity analysis of earthand environmental systems models. GeoscientificModel Development Discussions, 2019:1–32, 2019.doi: 10.5194/gmd-2019-17.
S. Thrun, D. Fox, W. Burgard, and F. Dellaert. Ro-bust monte carlo localization for mobile robots. Ar-tificial intelligence, 128(1-2):99–141, 2001.
E. Todorov, T. Erez, and Y. Tassa. Mujoco: A physicsengine for model-based control. In 2012 IEEE/RSJInternational Conference on Intelligent Robots andSystems, pages 5026–5033. IEEE, 2012.
