Structure of the workshop - Jeremias Knoblauch · Structure of the workshop 1.Theoretical foundations & The Rule of Three (10:00{11:00) 1.1Basics: losses & divergences 1.2Bayesian

Structure of the workshop

1. Theoretical foundations & The Rule of Three (10:00–11:00)1.1 Basics: losses & divergences

1.2 Bayesian inference: updating view vs. optimization view

1.3 The rule of three: special cases, modularity & axiomatic foundations

1.4 Q&A/Discussion

2. Optimality of VI, F-VI suboptimality & GVI’s motivation (11:30–12:30)2.1 VI interpretations: discrepancy-minimization vs constrained optimization

2.2 VI optimality & sub-optimality of F-VI

2.3 GVI as modular and explicit alternative to F-VI

2.4 GVI use cases

2.5 GVI’s lower bound interpretation

2.6 Q&A/Discussion

3. GVI Applications (14:00 – 15:00)3.1 Robust Bayesian On-line Changepoint Detection

3.2 Bayesian Neural Networks

3.3 Deep Gaussian Processes

3.4 Other work

3.5 Q&A/Discussion

4. Chalk talk: Consistency & Concentration rates for GVI (15:30 – 16:30)4.1 The role of Γ-convergence

4.2 GVI procedures as ε-optimizers

4.3 Central results

4.4 Q&A/Discussion1 / 47 GVI

Part 2

2 / 47 GVI

Part 2: Variational Inference (VI) Optimality & Generalized

Variational Inference (GVI)

1. VI: Three views

1.1 VI: The lower-bound optimization view

1.2 VI: The discrepancy-minimization view

1.3 VI: The constrained-optimization view

2. Motivating GVI

2.1 VI optimality

2.2 F-VI sub-optimality

2.3 GVI: New modular posteriors via P(`n,D,Q)

3. GVI: use cases

3.1 Robustness to model misspecification

3.2 Prior robustness & Adjusting marginal variances

4. GVI: A lower-bound interpretation

4.1 VI: lower-bound optimization interpretation

4.2 GVI: generalized lower-bound optimization interpretation

5. Discussion

3 / 47 GVI

1 Notation & color code

Notation:

(i) Θ = parameter space, θ ∈ Θ = parameter value

(ii) q, π are densities on Θ, i.e. q, π : Θ→ R+

π = prior (i.e., known before data is seen)

q = posterior (i.e., known after data is seen)

(iii) P(Θ) = set of all probability measures on Θ

(iv) Q = parameterized subset of P(Θ), i.e. Q = variational family

(v) xiiid∼ g(xi ); p(xi |θ) = likelihood model indexed by θ

Color code:

• P(`n,D,Π) = loss, Divergence, (sub)space of P(Θ)

• Standard Variational Inference (VI)

• Variational Inference based on F -discrepancy (F-VI)

• Generalized Variational Inference (GVI)

4 / 47 GVI

1 VI: Three views

Purpose of section 1: Give three interpretations of VI.

(1) Lower-bound on the log evidence (model selection view)

(2) Discrepancy-minimization (F-VI view)

(3) Constrained optimization (GVI view)

5 / 47 GVI

1.1 VI: The lower-bound optimization view

Derivation 1: VI maximizes Evidence Lower bound (ELBO):

log p(x1:n) = log

(∫Θ

p(x1:n|θ)π(θ)dθ

)= log

(∫Θ

p(x1:n|θ)π(θ)q(θ|κ)

q(θ|κ)dθ

)CoM= log

(Eq(θ|κ)

[p(x1:n|θ)π(θ)

q(θ|κ)

])JI≥ Eq(θ|κ)

[log

(p(x1:n|θ)π(θ)

q(θ|κ)

)]= Eq(θ|κ) [log (p(x1:n|θ)π(θ))]− Eq(θ|κ) [log (q(θ|κ))]

= ELBO(q)

Recipe:

(1) Apply a change of measure (CoM) with parameterized q(θ|κ) ∈ Q(2) Lower-bound log-evidence log p(x1:n) with Jensen’s Inequality (JI)

(3) Maximizing ELBO = Minimizing ’information loss’ due to CoM

Note: Interpretation from model selection (comparing log p(x1:n|m))6 / 47 GVI


Derivation 2: VI minimizes KLD between q and p

ELBO(q) = Eq(θ|κ) [log (p(x1:n|θ)π(θ))]− Eq(θ|κ) [log (q(θ|κ))]

Note 1: arg maxq∈Q ELBO(q) = arg maxq∈Q [ELBO(q) + C ] for any C .

Note 2: Picking C = − log p(x1:n), easy to show that

ELBO(q)− log p(x1:n) = −KLD (q(θ|κ)||p(x1:n|θ)π(θ)/p(x1:n))

= −KLD (q(θ|κ)||p(θ|x1:n))

Note 3: So maximizing ELBO = minimizing KLD between q & target p!

7 / 47 GVI


Discrepancy-minimization view: VI = approximation minimizing the

KLD to p(θ|x1:n). (Inspiration for F-VI methods)

[From Variational Inference: Foundations and Innovations (Blei, 2019)]8 / 47 GVI

1.2 VI: discrepancy-minimization view inspires F-VI

F-Variational inference (F-VI): VI based on discrepancy F 6= KLD

(locally) solving

q∗ = arg minq∈Q

F (q‖p) (1)

for p = standard Bayesian posterior, e.g.

F = Renyi’s α-divergence (Li and Turner, 2016; Saha et al., 2017)

F = χ-divergence (Dieng et al., 2017)

F = operators (Ranganath et al., 2016)

F= scaled AB-divergence (Regli and Silva, 2018)

F = Wasserstein distance (Ambrogioni et al., 2018)

F = local reverse KLD (= Expectation Propagation!)

. . .

9 / 47 GVI

1.3 VI: The constrained optimization view

Recall: Minimizing KLD = Maximizing ELBO = Minimizing −ELBO

−ELBO(q) = −Eq(θ|κ) [log (p(x1:n|θ)π(θ))] + Eq(θ|κ) [log (q(θ|κ))]

= −Eq(θ|κ)

[log

(n∏

i=1

p(xi |θ)

)]+ Eq(θ|κ)

[log

(q(θ|κ)

π(θ)

)]

= Eq(θ|κ)

[n∑

i=1

− log p(xi |θ)︸︷︷︸=`n(θ,x1:n)

]+ KLD (q(θ|κ)||π(θ))

Observation:

arg minq∈Q

−ELBO(q)!

= P(`n,KLD,Q) (2)

Conclusion: VI solves problem specified via Rule of Three P(`n,KLD,Q)

10 / 47 GVI


Recall: Minimizing KLD = Maximizing ELBO = Minimizing −ELBO

−ELBO(q) = −Eq(θ|κ) [log (p(x1:n|θ)π(θ))] + Eq(θ|κ) [log (q(θ|κ))]

= −Eq(θ|κ)

[log

(n∏

i=1

p(xi |θ)

)]+ Eq(θ|κ)

[log

(q(θ|κ)

π(θ)

)]

= Eq(θ|κ)

[n∑

i=1

− log p(xi |θ)︸︷︷︸=`n(θ,x1:n)

]+ KLD (q(θ|κ)||π(θ))

Observation:

arg minq∈Q

−ELBO(q)!

= P(`n,KLD,Q) (3)

Conclusion: VI solves problem specified via Rule of Three P(`n,KLD,Q)

11 / 47 GVI


Key message: VI = Q-constrained version of exact Bayesian inference

P( n,D, ( ))p P( n,D, )

q

Figure 1 – Left: Unconstrained Bayesian inference. Right: Q-constrained

Bayesian inference (=VI)

12 / 47 GVI

Summary: Three views on VI

(i) Maximize (approximate) log p(x1:n) using q ∈ Q=⇒ Lower bound view

(ii) Minimize F = KLD discrepancy of q to p

=⇒ F-VI view

(iii) Solve the Q-constrained version of original Bayes problem

=⇒ GVI view

13 / 47 GVI

2 Motivating GVI

Purpose of section 2: Relating VI, F-VI & GVI

(1) VI optimality

(2) F-VI sub-optimality

(3) GVI: the best of both worlds

14 / 47 GVI

2.1 VI optimality

So VI is the Q-constrained version of original Bayes?!

=⇒ I.e., it is the best you can do in Q!

Theorem 1 (VI optimality)

For exact and coherent Bayesian posteriors solving P(`n,KLD,P(Θ))

and a fixed variational family Q, standard VI produces the uniquely

optimal Q-constrained approximation to P(`n,KLD,P(Θ)) Having

decided on approximating the Bayesian posterior with some q ∈Q, VI

provides the uniquely optimal solution.

Proof: Many ways to show it, e.g. by contradiction: Suppose there

exists a q′ ∈ Q s.t. q′ yields better posterior for original Bayes problem

than than qVI =⇒ contradicts the fact that qVI solves P(`n,KLD,Q).

15 / 47 GVI


Q: If VI is optimal, what about F-VI procedures with F 6= KLD?

=⇒ For fixed Q, F-VI is sub-optimal relative to P(`n,KLD,P(Θ))!

Proof: Same proof applies: Suppose there exists a qF-VI ∈ Q s.t. qF-VI

yields better posterior for original Bayes problem than qVI =⇒ contradicts

the fact that qVI solves P(`n,KLD,Q).

Surprising Conclusion: For approximating original Bayesian inference

objective as best as one can, F-VI is always worse than standard VI.16 / 47 GVI


Consequences of F-VI-suboptimality:

(1) If F 6= KLD, F-VI violates Axioms 1–4 (in Part 1).

(2) F-VI conflates `n and D (i.e., modularity of P(`n,D,Π) lost).

(3) Thm: F-VI gives worse Q-constrained posterior than standard VI

(relative to the standard Bayesian problem P(`n,KLD,P(Θ)))

Objection! F-VI can produce better posteriors than VI in practice!

17 / 47 GVI


Seeming contradiction:

(1) VI is the best approximation to the standard Bayesian posterior

(2) F-VI often outperforms VI (e.g., on test scores)

Resolution: F-VI implicitly targets better Bayesian inference problem

than P(`n,KLD,P(Θ)) – but this problem is non-standard!

Q: Why not design these non-standard targets explicitly?

(i) Inference should be optimal relative to something (i.e. `n,D & Q)

(ii) Ingredients of non-standard problem should be interpretable

=⇒ Generalized Variational Inference (GVI)

P( n,D, ( ))p P( n,D, )

q

Figure 2 – Left: Unconstrained inference. Right: Q-constrained inference

(=VI)

18 / 47 GVI


Definition 1 (GVI)

Any Bayesian inference method solvingP(`n,D,Q) with admissible

choices `n, D and Q is a Generalized Variational Inference (GVI) method

satisfying Axioms 1 – 4.

GVI = combining advantages of VI and F-VI:

(1) Like VI: Has form P(`n,D,Q)!

(i) satisfies Axioms 1 – 4 (in Part 1);

(ii) provably interpretable modularity (loss, uncertainty quantifier,

admissible posteriors) (see Part 1)

(2) Like F-VI: Targets non-standard posteriors! BUT:

(i) without conflating `n and D

(ii) with explicit rather than implicit changes .

19 / 47 GVI


Illustration: F-VI aims for D, but changes `n. GVI doesn’t.

-0.5 0.0 0.5 1.0

0.0

0.5

1.0

1.5

2.0

2.5

3.0

µ1

Density

Exact Posterior

VI

F-VI,F = D(0.5)

ARGVI, α = 0.25

MLE

Figure 3 – Exact, VI, F-VI (F = D(0.5)AR ) and P(`n, D

(α)AR ,Q) based GVI marginals of the location

in a 2 component mixture model. Respecting `n, VI and GVI provide uncertainty quantification

around the most likely value θn via D. In contrast, F-VI implicitly changes the loss and has a mode

at the locally most unlikely value of θ.20 / 47 GVI

Summary: Motivating GVI

(1) VI = P(`n,KLD,Q) is optimal for the original Bayesian

problem P(`n,KLD,P(Θ))

(2) F-VI = sub-optimal approximation of P(`n,KLD,P(Θ)),

but can implicitly target non-standard posteriors

(3) GVI = P(`n,D,Q): combines the best of both worlds by

leveraging the novel constrained-optimization view

21 / 47 GVI

3 GVI: What does it do?

Purpose of part 4: Exploring three use cases of GVI

(1) Robust alternatives to `(θ, xi) = − log(p(xi |θ))

(2) Prior-robust uncertainty quantification via D

(3) Adjusting marginal variances via D

22 / 47 GVI

3.1 GVI: The losses I/III

GVI modularity: The loss `n

Q1: Why use `n(θ, x) =∑n

i=1− log(p(xi |θ))?

A: Assuming that the true data-generating mechanism is x ∼ g ,

arg minθ

n∑i=1

− log(p(xi |θ)) ≈ arg minθ

Eg [− log(p(x |θ))]

= arg minθ

Eg [− log (p(x |θ)) + log(g(x))] = arg minθ

KLD(g‖p(·|θ))

Interpretation: − log (p(xi |θ)) = targeting KLD-minimizing p(·|θ)

Q2: Are there other LD(p(xi |θ)) for divergence D?

A: Yes! (e.g. Jewson et al., 2018; Futami et al., 2017; Ghosh and Basu,

2016; Hooker and Vidyashankar, 2014)

23 / 47 GVI

3.1 GVI: The losses II/III

Q3: Why use other LD(p(xi |θ))?

A: Robustness (for D = a robust divergence) [log/KLD non-robust!]

Robustness recipe: α/β/γ-divergences using generalized log functions

E.g.: β indexes β-divergence (D(β)B ) via

logβ(x) =1

(β − 1)β

[βxβ−1 − (β − 1)xβ

]D

(β)B (g ||p(·|θ)) = Eg

[logβ(p(x |θ))− logβ(g(x))

]Note 1: D

(β)B → KLD as β → 1!

Note 2: Admits D(β)B -targeting loss as

Lβp (θ, xi ) = − 1

β − 1p(xi |θ)β−1 +

Ip,β(θ)

β, Ip,c(θ) =

∫p(x |θ)cdx

24 / 47 GVI

3.1 GVI: The losses III/III

-5 0 5 10 15

0.0

0.1

0.2

0.3

0.4

x

Density

(1− ε)N(0, 1)εN(8, 1)VI,− log(p(x; θ))GVI,Lβp (x, θ)

Contamination

0 2 4 6 8 10

0.0

0.2

0.4

0.6

0.8

1.0

1.2

1.4

Standard Deviations from the Posterior MeanIn

fluen

ce/D

ensi

ty

− log(p(x; θ))

Lβp (x, θ), β = 1.05

Lβp (x, θ), β = 1.1

Lβp (x, θ), β = 1.25

N(0,1)

Figure 4 – Left: Robustness against model misspecification. Depicted are

posterior predictives under ε = 5% outlier contamination using VI and

P(∑n

i=1 Lβp (θ, xi ),KLD,Q), β = 1.5. Right: From Knoblauch et al. (2018).

Influence of xi on exact posteriors for different losses.

25 / 47 GVI

3.1 GVI: Choosing loss hyperparameters I/II

Q: What do losses for β/γ-divergences look like?

Lβp (θ, xi ) = − 1

β − 1p(xi |θ)β−1 +

Ip,β(θ)

β

Lγp (θ, xi ) = − 1

γ − 1p(xi |θ)γ−1 γ

Ip,γ(θ)γ−1γ

Ip,c(θ) =

∫p(x |θ)cdx

where Ip,c(θ) =∫p(x |θ)cdx .

Note 1: Lγp (θ, xi ) multiplicative & always < 0 → store as log!

Note 2: Conditional independence 6= additive for Lβp (θ, xi ),Lγ

p (θ, xi )

Q: For losses based on β- or γ-divergence, how to pick β/γ?

Note 3: Good choices for β/γ depend on dimensionality of d

Note 4: In practice, usually best to choose β/γ = 1 + ε for some small ε

26 / 47 GVI

3.1 GVI: Choosing loss hyperparameters II/II

Q: Any way of choosing hyperparameters?

A: Very much unsolved problem, proposals so far:

• Cross-validation (Futami et al., 2017)

• Via points of highest influence (Knoblauch et al., 2018)

• on-line updates using loss-minimization (Knoblauch et al., 2018)

0 2 4SD

0.0

0.1

0.2

0.3

0.4

0.5

0.6

Influ

ence

/Den

sity

N(0, 1)KLD

0 2 4SD

N(0, 1)= 0.05

0 2 4SD

N(0, 1)= 0.2

0 2 4SD

N(0, 1)= 0.25

Figure 5 – Illustration of the initialization procedure using points of highest

influence logic, from left to right.27 / 47 GVI

3.2 GVI: Uncertainty Quantification I/III

GVI modularity: The uncertainty quantifier D

Q: Which VI drawbacks can be addressed via D?

A: Any uncertainty quantification properties, e.g.

• Over-concentration ( = underestimating marginal variances)

• Sensitivity to badly specified priors

• . . .

28 / 47 GVI

3.2 GVI: Uncertainty Quantification II/III

Example 1: GVI can fix over-concentrated posteriors

0.0 0.5 1.0 1.5 2.0

010

20

3040

50

α/β/γ

Divergence

D(β)

B

D(γ)

G

D(α)

A

KLD

-1 0 1 2 3 4 5

0.0

0.2

0.4

0.6

0.8

1.0

1.2

θ1

Density

Exact PosteriorVIGVI, α = 0.5GVI, α = 0.025MLE

Figure 6 – Left: Magnitude of the penalty incurred by D(q||π) for different

uncertainty quantifiers D and fixed densities π, q. Right: Using D(α)AR with

different choices of α to “customize” uncertainty.29 / 47 GVI

3.2 GVI: Uncertainty Quantification III/III

Example 2: Avoiding prior sensitivity

-1 0 1 2 3 4 5

0.0

0.5

1.0

1.5

1wKLD, w = 1 , VI

θ1

Density

π(θ1) = N(3, 22σ2)π(θ1) = N(−5, 22σ2)π(θ1) = N(−20, 22σ2)π(θ1) = N(−50, 22σ2)MLE

-1 0 1 2 3 4 5

0.0

0.5

1.0

1.5

D(α)

AR, α = 0.5 , GVI

θ1Density


Figure 7 – Prior sensitivity with VI (left) vs. prior robustness with GVI (right).

Priors are more badly specified for darker shades.

30 / 47 GVI

Summary: GVI, three use cases

Summary: some GVI applications include

(1) Robustness to model misspecification ( = adapting `n)

(2) “Customized” marginal variances ( = adapting D)

(3) Prior robustness ( = adapting D)

31 / 47 GVI

4 GVI: A lower-bound interpretation

Purpose of section 4: Use theory of generalized Bayesian

posteriors for lower-bound interpretation of GVI

(1) Deriving a generalized lower bound

(2) VI = special case holding without slack

(3) GVI: Some interpretable bounds with D = D(α)AR , D

(γ)G , D

(β)B

32 / 47 GVI

4.1 VI lower bound interpretation

Recall: VI also interpretable as optimizing a lower bound on log evidence!

log p(x1:n)JI≥ Eq(θ|κ)

[log

(p(x1:n|θ)π(θ)

q(θ|κ)

)]= ELBO(q)

Recall: We also saw that VI = P(`n,D,Q) minimizes

−ELBO(q) = − log p(x1:n) + KLD(q(θ|κ)||p(θ|x1:n)) (4)

Thus: max. ELBO = min. KLD = max. lower-bound on log p(x1:n)

Q: Can we extend this logic to GVI?

A: Yes, but we need to introduce a few things first . . .

33 / 47 GVI

4.1 VI lower bound interpretation

Introduce: Generalized Bayes posterior (Bissiri et al., 2016):

P(`n,D,P(Θ))

q∗`n(θ) ∝ π(θ) exp {−`n(θ, x1:n)}Introduce: Generalized evidence (corresponding to q∗`n )

p[`n](x1:n) =

∫Θ

q∗`n(θ)dθ

Introduce: Functions LD , ED , TD depending on D:

LD ,ED :R → R (loss and evidence maps)

TD :Q → R (approximate target map)

For VI: EKLD(x) = LKLD(x) = x , TD(q) = KLD(q||q∗`n) recovers:

−ELBO(q) = EKLD(− log p[LKLD(`n)](x1:n)

)+ TKLD(q)

For GVI: With D based on generalized log, one recovers

L(q) ≥ ED(− log p[LD (`n)](x1:n)

)+ TD(q)

34 / 47 GVI

4.2 GVI lower bound interpretation

For VI: EKLD(x) = LKLD(x) = x , TD(q) = KLD(q||q∗`n) recovers:

−ELBO(q) = EKLD(− log p[LKLD(`n)](x1:n)

)+ TKLD(q)


L(q) ≥ ED(− log p[LD (`n)](x1:n)︸︷︷︸

generalized negative log evidence;

LD maps `n into a new loss

)+ TD(q)︸︷︷︸

Approximate target

Note: VI = special case which holds with equality

=⇒ For VI, approximate target TKLD(q) = exact target!

35 / 47 GVI



L(q) ≥ ED(− log p[LD (`n)](x1:n)︸︷︷︸

generalized negative log evidence;

LD maps `n into a new loss

)+ TD(q)︸︷︷︸

Approximate target

Example: Similar results/decompositions for D(α)AR , D

(β)B , D

(γ)G .

Renyi’s α-divergence (D(α)AR ) for α > 1 gives

ED(α)AR (z) = 1

α · z ,

LD(α)AR (`n(θ, x1:n)) = α · `n(θ, x1:n),

TD

(α)AR

(q) = 1αKLD(q||q∗

LD(α)AR (`n)

),

so putting it together one finds that for D = D(α)AR with α > 1,

L(q) ≥ − 1

αlog pα`n(x1:n) +

1

αKLD(q||q∗α`n)

Note: Above is 1α -scaled version of the ELBO with the loss α`n!

=⇒ approximately minimizes log pα`n(x1:n) of α-power posterior in Q36 / 47 GVI


Q: Two different problems, same objective?!

`n-loss GVI with D=D(α)AR :

L(q) ≥ − 1

αlog pα`n(x1:n) +

1

αKLD(q||q∗α`n)

α-Power `n-loss VI:

1

αELBO(q) = − 1

αlog pα`n(x1:n) +

1

αKLD(q||q∗α`n)

No! Slack term!

1

αELBO(q)

!= . . .

L(q)!

= Slack(q)− 1

αlog pα`n(x1:n) +

1

αKLD(q||q∗α`n)

37 / 47 GVI


Q: What does the slack term represent? =⇒ Prior robustness!!!

-1 0 1 2 3 4 5

0.0

0.5

1.0

1.5

1wKLD, w = 0.75 , GVI

θ1

Den

sity


-1 0 1 2 3 4 5

0.0

0.5

1.0

1.5

1wKLD, w = 0.5 , GVI

θ1

Den

sity


-1 0 1 2 3 4 5

0.0

0.5

1.0

1.5

D(α)

AR, α = 0.75 , GVI

θ1

Den

sity


-1 0 1 2 3 4 5

0.0

0.5

1.0

1.5

D(α)

AR, α = 0.5 , GVI

θ1

Den

sity


Figure 8 – Power Bayes VI not prior-robust – but GVI is!38 / 47 GVI

Summary: GVI, a lower-bound interpretation

(1) One can bound GVI objectives via a generalized log

evidence term ED(− log p[LD(`n)](x1:n)

)plus an

approximate target term TD(q)

(2) Idea: Minimizing GVI objective = approximately

minimizing target term TD(q)

(3) Finding 1: This is simlar to approximatly minimizing VI

objective of a power posterior

(4) Finding 2: The slack term (i.e. what makes the bound

approximate) accounts for the prior robustness

39 / 47 GVI

Discussion / Q&A

If you have questions that you think are stupid, I am sure they are

not – here are some questions to compare against that people

actually asked on Reddit:

• When did 9/11 happen?

• Can you actually lose weight by rubbing your stomach?

• What happens if you paint your teeth white with nail polish?

• Is it okay to boil headphones?

• . . .

40 / 47 GVI

Main References i

Ambrogioni, L., Guclu, U., Gucluturk, Y., Hinne, M., van Gerven, M. A. J., and Maris, E. (2018). Wasserstein variational inference. InBengio, S., Wallach, H., Larochelle, H., Grauman, K., Cesa-Bianchi, N., and Garnett, R., editors, Advances in Neural InformationProcessing Systems 31, pages 2478–2487. Curran Associates, Inc.

Bissiri, P. G., Holmes, C. C., and Walker, S. G. (2016). A general framework for updating belief distributions. Journal of the RoyalStatistical Society: Series B (Statistical Methodology), 78(5):1103–1130.

Dieng, A. B., Tran, D., Ranganath, R., Paisley, J., and Blei, D. (2017). Variational inference via χ upper bound minimization. InAdvances in Neural Information Processing Systems, pages 2732–2741.

Futami, F., Sato, I., and Sugiyama, M. (2017). Variational inference based on robust divergences. arXiv preprint arXiv:1710.06595.

Ghosh, A. and Basu, A. (2016). Robust bayes estimation using the density power divergence. Annals of the Institute of StatisticalMathematics, 68(2):413–437.

Hooker, G. and Vidyashankar, A. N. (2014). Bayesian model robustness via disparities. Test, 23(3):556–584.

Jewson, J., Smith, J., and Holmes, C. (2018). Principles of Bayesian inference using general divergence criteria. Entropy, 20(6):442.

Knoblauch, J., Jewson, J., and Damoulas, T. (2018). Doubly robust Bayesian inference for non-stationary streaming data usingβ-divergences. In Advances in Neural Information Processing Systems (NeurIPS), pages 64–75.

Li, Y. and Turner, R. E. (2016). Renyi divergence variational inference. In Advances in Neural Information Processing Systems, pages1073–1081.

Ranganath, R., Tran, D., Altosaar, J., and Blei, D. (2016). Operator variational inference. In Advances in Neural Information ProcessingSystems, pages 496–504.

Regli, J.-B. and Silva, R. (2018). Alpha-beta divergence for variational inference. arXiv preprint arXiv:1805.01045.

Saha, A., Bharath, K., and Kurtek, S. (2017). A geometric variational approach to bayesian inference. arXiv preprint arXiv:1707.09714.

41 / 47 GVI

Appendix: Choosing D for conservative marginals I/II

0 1 2 3 4 5

0.0

0.5

1.0

1.5

D(α)

AR

θ1

Den

sity

Exact PosteriorGVI, α = 1.25VIGVI, α = 0.5GVI, α = 0.025MLE

0 1 2 3 4 5

0.0

0.5

1.0

1.5

D(β)

B

θ1

Den

sity

Exact PosteriorGVI, β = 1.5VIGVI, β = 0.75GVI, β = 0.5MLE

0 1 2 3 4 5

0.0

0.5

1.0

1.5

D(γ)

G

θ1

Den

sity

Exact PosteriorGVI, γ = 1.5VIGVI, γ = 0.75GVI, γ = 0.15MLE

0 1 2 3 4 5

0.0

0.5

1.0

1.5

1wKLD

θ1

Den

sity

Exact PosteriorGVI, w = 2VIGVI, w = 0.5GVI, w = 0.125MLE

Figure 9 – Marginal VI and GVI posterior for a Bayesian linear model under the D(α)AR , D

(β)B , D

(γ)G

and 1wKLD uncertainty quantifier for different values of the divergence hyperparameters.

42 / 47 GVI

Appendix: Choosing D for conservative marginals II/II

0 1 2 3 4 5

0.0

0.5

1.0

1.5

D(α)

A

θ1

Density

PosteriorGVI, α = 1.25VIGVI, α = 0.95GVI, α = 0.5GVI, α = 0.01MLE

Figure 10 – Marginal VI and GVI posterior for a Bayesian linear model under

the D(α)A uncertainty quantifier. The boundedness of the D

(α)A causes GVI to

severely over-concentrate if α is not carefully specified.

43 / 47 GVI

Appendix: Choosing D for prior robustness I/IV

-1 0 1 2 3 4 5

0.0

0.5

1.0

1.5

1wKLD, w = 1.25 , GVI

θ1

Density


-1 0 1 2 3 4 5

0.0

0.5

1.0

1.5

1wKLD, w = 1 , VI

θ1

Density


-1 0 1 2 3 4 5

0.0

0.5

1.0

1.5

1wKLD, w = 0.75 , GVI

θ1

Density


-1 0 1 2 3 4 5

0.0

0.5

1.0

1.5

1wKLD, w = 0.5 , GVI

θ1

Density



different priors, using D = 1

wKLD as the uncertainty quantifier.

44 / 47 GVI

Appendix: Choosing D for prior robustness II/IV

-1 0 1 2 3 4 5

0.0

0.5

1.0

1.5

D(α)

AR, α = 1.25 , GVI

θ1

Density


-1 0 1 2 3 4 5

0.0

0.5

1.0

1.5

D(α)

AR, α = 1 , VI

θ1

Density


-1 0 1 2 3 4 5

0.0

0.5

1.0

1.5

D(α)

AR, α = 0.75 , GVI

θ1

Density


-1 0 1 2 3 4 5

0.0

0.5

1.0

1.5

D(α)

AR, α = 0.5 , GVI

θ1

Density



different priors, using D = D(α)AR as the uncertainty quantifier.

45 / 47 GVI

Appendix: Choosing D for prior robustness III/IV

-1 0 1 2 3 4 5

0.0

0.5

1.0

1.5

D(β)

B , β = 1.25 , GVI

θ1

Density


-1 0 1 2 3 4 5

0.0

0.5

1.0

1.5

D(β)

B , β = 1 , VI

θ1

Density


-1 0 1 2 3 4 5

0.0

0.5

1.0

1.5

D(β)

B , β = 0.9 , GVI

θ1

Density


-1 0 1 2 3 4 5

0.0

0.5

1.0

1.5

D(β)

B , β = 0.75 , GVI

θ1

Density



different priors, using D = D(β)B as the uncertainty quantifier.

46 / 47 GVI

Appendix: Choosing D for prior robustness IV/IV

-1 0 1 2 3 4 5

0.0

0.5

1.0

1.5

D(γ)

G , γ = 1.25 , GVI

θ1

Density


-1 0 1 2 3 4 5

0.0

0.5

1.0

1.5

D(γ)

G , γ = 1 , VI

θ1

Density


-1 0 1 2 3 4 5

0.0

0.5

1.0

1.5

D(γ)

G , γ = 0.75 , GVI

θ1

Density


-1 0 1 2 3 4 5

0.0

0.5

1.0

1.5

D(γ)

G , γ = 0.5 , GVI

θ1

Density



different priors, using D = D(γ)G as the uncertainty quantifier.

47 / 47 GVI

Documents

Structure of the workshop - Jeremias Knoblauch · Structure of the workshop 1.Theoretical foundations & The Rule of Three (10:00{11:00) 1.1Basics: losses & divergences 1.2Bayesian