Vanilla rao blackwellisation

Vanilla Rao–Blackwellisation of

Metropolis–Hastings algorithms

Christian P. Robert

Universite Paris-Dauphine, IuF, and CRESTJoint works with Randal Douc, Pierre Jacob and Murray Smith

LGM2012, Trondheim, May 30, 2012

1 / 32

Main themes

1 Rao–Blackwellisation on MCMC

2 Can be performed in any Hastings Metropolis algorithm

3 Asymptotically more efficient than usual MCMC with acontrolled additional computing

4 Takes advantage of parallel capacities at a very basic level(GPUs)

2 / 32

Main themes

2 / 32

Main themes

2 / 32

Main themes

2 / 32

Metropolis Hastings revisitedRao–Blackwellisation

Rao-Blackwellisation (2)

Metropolis Hastings algorithm

1 We wish to approximate

∫h(x)π(x)dx∫π(x)dx

h(x)π(x)dx

2 π(x) is known but not∫π(x)dx.

3 Approximate I with δ = 1n

∑nt=1 h(x

(t)) where (x(t)) is a Markov

chain with limiting distribution π.

4 Convergence obtained from Law of Large Numbers or CLT forMarkov chains.

3 / 32

h(x)π(x)dx

∑nt=1 h(x

3 / 32

h(x)π(x)dx

∑nt=1 h(x

3 / 32

h(x)π(x)dx

∑nt=1 h(x

3 / 32

Metropolis Hasting Algorithm

Suppose that x(t) is drawn.

1 Simulate yt ∼ q(·|x(t)).

2 Set x(t+1) = yt with probability

α(x(t), yt) = min

1,π(yt)

π(x(t))

q(x(t)|yt)q(yt|x(t))

Otherwise, set x(t+1) = x(t) .

3 α is such that the detailed balance equation is satisfied: ⊲ π is thestationary distribution of (x(t)).

◮ The accepted candidates are simulated with the rejection algorithm.

4 / 32

α(x(t), yt) = min

1,π(yt)

π(x(t))

3 α is such that the detailed balance equation is satisfied: ⊲ π is thestationary distribution of (x(t)).

4 / 32

α(x(t), yt) = min

1,π(yt)

π(x(t))

3 α is such that the detailed balance equation is satisfied:

π(x)q(y|x)α(x, y) = π(y)q(x|y)α(y, x).

⊲ π is the stationary distribution of (x(t)).

4 / 32

α(x(t), yt) = min

1,π(yt)

π(x(t))

3 α is such that the detailed balance equation is satisfied:

π(x)q(y|x)α(x, y) = π(y)q(x|y)α(y, x).

⊲ π is the stationary distribution of (x(t)).

4 / 32

Some properties of the HM algorithm

1 Alternative representation of the estimator δ is

h(x(t)) =1

nih(zi) ,

zi’s are the accepted yj ’s,Mn is the number of accepted yj ’s till time n,ni is the number of times zi appears in the sequence (x(t))t.

5 / 32

q(·|zi) =α(zi, ·) q(·|zi)

p(zi)≤ q(·|zi)

p(zi),

where p(zi) =∫α(zi, y) q(y|zi)dy. To simulate from q(·|zi):

1 Propose a candidate y ∼ q(·|zi)2 Accept with probability

q(y|zi)/(

q(y|zi)p(zi)

= α(zi, y)

Otherwise, reject it and starts again.

◮ this is the transition of the HM algorithm.The transition kernel qadmits π as a stationary distribution:

π(x)q(y|x) =

6 / 32

q(·|zi) =α(zi, ·) q(·|zi)

p(zi)≤ q(·|zi)

p(zi),

q(y|zi)/(

q(y|zi)p(zi)

= α(zi, y)

π(x)q(y|x) =

6 / 32

q(·|zi) =α(zi, ·) q(·|zi)

p(zi)≤ q(·|zi)

p(zi),

q(y|zi)/(

q(y|zi)p(zi)

= α(zi, y)

π(x)q(y|x) =

6 / 32

q(·|zi) =α(zi, ·) q(·|zi)

p(zi)≤ q(·|zi)

p(zi),

q(y|zi)/(

q(y|zi)p(zi)

= α(zi, y)

π(x)q(y|x) = π(x)p(x)∫π(u)p(u)du

︸︷︷︸

α(x, y)q(y|x)p(x)

︸︷︷︸

q(y|x)

6 / 32

q(·|zi) =α(zi, ·) q(·|zi)

p(zi)≤ q(·|zi)

p(zi),

q(y|zi)/(

q(y|zi)p(zi)

= α(zi, y)

π(x)q(y|x) = π(x)α(x, y)q(y|x)∫π(u)p(u)du

6 / 32

q(·|zi) =α(zi, ·) q(·|zi)

p(zi)≤ q(·|zi)

p(zi),

q(y|zi)/(

q(y|zi)p(zi)

= α(zi, y)

π(x)q(y|x) = π(y)α(y, x)q(x|y)∫π(u)p(u)du

6 / 32

q(·|zi) =α(zi, ·) q(·|zi)

p(zi)≤ q(·|zi)

p(zi),

q(y|zi)/(

q(y|zi)p(zi)

= α(zi, y)

π(x)q(y|x) = π(y)q(x|y) ,

6 / 32

Lemma (Douc & X., AoS, 2011)

The sequence (zi, ni) satisfies

1 (zi, ni)i is a Markov chain;

2 zi+1 and ni are independent given zi;

3 ni is distributed as a geometric random variable with probabilityparameter

p(zi) :=

α(zi, y) q(y|zi) dy ; (1)

4 (zi)i is a Markov chain with transition kernel Q(z, dy) = q(y|z)dyand stationary distribution π such that

q(·|z) ∝ α(z, ·) q(·|z) and π(·) ∝ π(·)p(·) .

7 / 32

p(zi) :=

q(·|z) ∝ α(z, ·) q(·|z) and π(·) ∝ π(·)p(·) .

7 / 32

p(zi) :=

q(·|z) ∝ α(z, ·) q(·|z) and π(·) ∝ π(·)p(·) .

7 / 32

p(zi) :=

q(·|z) ∝ α(z, ·) q(·|z) and π(·) ∝ π(·)p(·) .

7 / 32

Old bottle, new wine [or vice-versa]

zi−1

8 / 32

zi−1 zi

ni−1

8 / 32

zi−1 zi zi+1

ni−1 ni

8 / 32

zi−1 zi zi+1

ni−1 ni

h(x(t)) =1

nih(zi) .

8 / 32

zi−1 zi zi+1

ni−1 ni

h(x(t)) =1

nih(zi) .

8 / 32

Formal importance samplingVariance reductionAsymptotic resultsIllustrations

Importance sampling perspective

1 A natural idea:

δ∗ =1

p(zi),

9 / 32

1 A natural idea:

δ∗ ≃

p(zi)∑Mn

π(zi)

π(zi)h(zi)

π(zi)

9 / 32

1 A natural idea:

δ∗ ≃

p(zi)∑Mn

π(zi)

π(zi)h(zi)

π(zi)

2 But p not available in closed form.

9 / 32

1 A natural idea:

δ∗ ≃

p(zi)∑Mn

π(zi)

π(zi)h(zi)

π(zi)

2 But p not available in closed form.

3 The geometric ni is the replacement, an obvious solution that isused in the original Metropolis–Hastings estimate sinceE[ni] = 1/p(zi).

9 / 32

The Bernoulli factory

The crude estimate of 1/p(zi),

ni = 1 +∞∑

ℓ≤j

I {uℓ ≥ α(zi, yℓ)} ,

can be improved:

If (yj)j is an iid sequence with distribution q(y|zi), the quantity

ξi = 1 +

∞∑

ℓ≤j

{1− α(zi, yℓ)}

is an unbiased estimator of 1/p(zi) which variance, conditional on zi, is

lower than the conditional variance of ni, {1− p(zi)}/p2(zi).

10 / 32

Rao-Blackwellised, for sure?

ξi = 1 +∞∑

ℓ≤j

{1− α(zi, yℓ)}

1 Infinite sum but finite with at least positive probability:

α(x(t), yt) = min

1,π(yt)

π(x(t))

For example: take a symmetric random walk as a proposal.

2 What if we wish to be sure that the sum is finite?

Finite horizon k version:

ξki = 1 +

∞∑

1≤ℓ≤k∧j

{1− α(zi, yj)}∏

k+1≤ℓ≤j

I {uℓ ≥ α(zi, yℓ)}

11 / 32

Rao-Blackwellised, for sure?

ξi = 1 +∞∑

ℓ≤j

{1− α(zi, yℓ)}

1 Infinite sum but finite with at least positive probability:

α(x(t), yt) = min

1,π(yt)

π(x(t))

For example: take a symmetric random walk as a proposal.

2 What if we wish to be sure that the sum is finite?

Finite horizon k version:

ξki = 1 +

∞∑

1≤ℓ≤k∧j

{1− α(zi, yj)}∏

k+1≤ℓ≤j

11 / 32

Variance improvement

Proposition (Douc & X., AoS, 2011)

If (yj)j is an iid sequence with distribution q(y|zi) and (uj)j is an iiduniform sequence, for any k ≥ 0, the quantity

ξki = 1 +

∞∑

1≤ℓ≤k∧j

{1− α(zi, yj)}∏

k+1≤ℓ≤j

is an unbiased estimator of 1/p(zi) with an almost sure finite number ofterms.

12 / 32

ξki = 1 +

∞∑

1≤ℓ≤k∧j

{1− α(zi, yj)}∏

k+1≤ℓ≤j

is an unbiased estimator of 1/p(zi) with an almost sure finite number of

terms. Moreover, for k ≥ 1,

=1− p(zi)

p2(zi)− 1− (1− 2p(zi) + r(zi))

2p(zi)− r(zi)

2− p(zi)

p2(zi)

(p(zi)− r(zi)) ,

where p(zi) :=∫

α(zi, y) q(y|zi) dy. and r(zi) :=∫

α2(zi, y) q(y|zi) dy.

12 / 32

ξki = 1 +

∞∑

1≤ℓ≤k∧j

{1− α(zi, yj)}∏

k+1≤ℓ≤j

is an unbiased estimator of 1/p(zi) with an almost sure finite number of

terms. Therefore, we have

∣∣∣ zi

= V [ni| zi] .

12 / 32

zi−1

ξki = 1 +∞∑

1≤ℓ≤k∧j

{1− α(zi, yj)}∏

k+1≤ℓ≤j

13 / 32

zi−1 zi

ξki−1

not indep

ξki = 1 +∞∑

1≤ℓ≤k∧j

{1− α(zi, yj)}∏

k+1≤ℓ≤j

13 / 32

zi−1 zi zi+1

ξki−1 ξki

not indep

ξki = 1 +∞∑

1≤ℓ≤k∧j

{1− α(zi, yj)}∏

k+1≤ℓ≤j

13 / 32

zi−1 zi zi+1

ξki−1 ξki

not indep

δkM =

∑Mi=1 ξ

ki h(zi)

∑Mi=1 ξ

13 / 32

zi−1 zi zi+1

ξki−1 ξki

not indep

δkM =

∑Mi=1 ξ

ki h(zi)

∑Mi=1 ξ

13 / 32

δkM =

∑Mi=1 ξ

ki h(zi)

∑Mi=1 ξ

For any positive function ϕ, we denote Cϕ = {h; |h/ϕ|∞ <∞}.

14 / 32

δkM =

∑Mi=1 ξ

ki h(zi)

∑Mi=1 ξ

For any positive function ϕ, we denote Cϕ = {h; |h/ϕ|∞ <∞}. Assumethat there exists a positive function ϕ ≥ 1 such that

∀h ∈ Cϕ,∑Mi=1

h(zi)/p(zi)∑Mi=1

1/p(zi)

P−→ π(h)

Theorem (Douc & X., AoS, 2011)

Under the assumption that π(p) > 0, the following convergence property holds:

i) If h is in Cϕ, then

δkMP−→M→∞ π(h) (◮Consistency)

14 / 32

δkM =

∑Mi=1 ξ

ki h(zi)

∑Mi=1 ξ

For any positive function ϕ, we denote Cϕ = {h; |h/ϕ|∞ <∞}.Assume that there exists a positive function ψ such that

∀h ∈ Cψ ,√M

∑Mi=1

h(zi)/p(zi)∑Mi=1

1/p(zi)− π(h)

L−→ N (0,Γ(h))

Under the assumption that π(p) > 0, the following convergence property holds:

ii) If, in addition, h2/p ∈ Cϕ and h ∈ Cψ , then

√M(δkM − π(h))

L−→M→∞ N (0, Vk[h− π(h)]) , (◮Clt)

where Vk(h) := π(p)∫

π(dz)V[

h2(z)p(z) + Γ(h) .

14 / 32

We will need some additional assumptions. Assume a maximal inequalityfor the Markov chain (zi)i: there exists a measurable function ζ such thatfor any starting point x,

∀h ∈ Cζ , Px

∣∣∣∣∣∣

sup0≤i≤n

[h(zi)− π(h)]

∣∣∣∣∣∣

≤ nCh(x)

Assume that h is such that h/p ∈ Cζ and {Ch/p, h2/p2} ⊂ Cφ. Assume

moreover that

(δ0M − π(h)

) L−→ N (0, V0[h− π(h)]) .

Then, for any starting point x,

(∑nt=1 h(x

n− π(h)

n→+∞−→ N (0, V0[h− π(h)]) ,

where Mn is defined by15 / 32

∀h ∈ Cζ , Px

∣∣∣∣∣∣

sup0≤i≤n

[h(zi)− π(h)]

∣∣∣∣∣∣

≤ nCh(x)

Moreover, assume that ∃φ ≥ 1 such that for any starting point x,

∀h ∈ Cφ, Qn(x, h)P−→ π(h) = π(ph)/π(p) ,

moreover that

(δ0M − π(h)

) L−→ N (0, V0[h− π(h)]) .

√(∑n

h(x(t)))

n→+∞15 / 32

∀h ∈ Cζ , Px

∣∣∣∣∣∣

sup0≤i≤n

[h(zi)− π(h)]

∣∣∣∣∣∣

≤ nCh(x)

Moreover, assume that ∃φ ≥ 1 such that for any starting point x,

moreover that

(δ0M − π(h)

) L−→ N (0, V0[h− π(h)]) .

√(∑n

h(x(t)))

n→+∞15 / 32

∀h ∈ Cζ , Px

∣∣∣∣∣∣

sup0≤i≤n

[h(zi)− π(h)]

∣∣∣∣∣∣

≤ nCh(x)

moreover that

(δ0M − π(h)

) L−→ N (0, V0[h− π(h)]) .

(∑nt=1 h(x

n− π(h)

n→+∞−→ N (0, V0[h− π(h)]) ,

where Mn is defined by15 / 32

moreover that

(δ0M − π(h)

) L−→ N (0, V0[h− π(h)]) .

(∑nt=1 h(x

n− π(h)

n→+∞−→ N (0, V0[h− π(h)]) ,

where Mn is defined by

ξ0i ≤ n <

Mn+1∑

ξ0i .

15 / 32

Variance gain (1)

h(x) x x2 IX>0 p(x)τ = .1 0.971 0.953 0.957 0.207τ = 2 0.965 0.942 0.875 0.861τ = 5 0.913 0.982 0.785 0.826τ = 7 0.899 0.982 0.768 0.820

Ratios of the empirical variances of δ∞ and δ estimating E[h(X)]:100 MCMC iterations over 103 replications of a random walk Gaussianproposal with scale τ .

16 / 32

Illustration (1)

Figure: Overlay of the variations of 250 iid realisations of the estimatesδ (gold) and δ∞ (grey) of E[X] = 0 for 1000 iterations, along with the90% interquantile range for the estimates δ (brown) and δ∞ (pink), inthe setting of a random walk Gaussian proposal with scale τ = 10.

17 / 32

Extra computational effort

median mean q.8 q.9 timeτ = .25 0.0 8.85 4.9 13 4.2τ = .50 0.0 6.76 4 11 2.25τ = 1.0 0.25 6.15 4 10 2.5τ = 2.0 0.20 5.90 3.5 8.5 4.5

Additional computing effort due: median and mean numbers of additionaliterations, 80% and 90% quantiles for the additional iterations, and ratioof the average R computing times obtained over 105 simulations

18 / 32

Illustration (2)

Figure: Overlay of the variations of 500 iid realisations of the estimatesδ (deep grey), δ∞ (medium grey) and of the importance sampling version(light grey) of E[X] = 10 when X ∼ Exp(.1) for 100 iterations, alongwith the 90% interquantile ranges (same colour code), in the setting ofan independent exponential proposal with scale µ = 0.02. 19 / 32

Independent caseGeneral MH algorithms

Integrating out white noise [C+X, 96]

In Casella+X. (1996), averaging over possible past and future histories(by integrating out uniforms) to improve weights of accepted values

20 / 32

In Casella+X. (1996), averaging over possible past and future histories(by integrating out uniforms) to improve weights of accepted valuesGiven the whole sequence of proposed values yt ∼ µ(yt), averaging overuniforms is possible: starting with y1, we can compute

E[h(Xt)|y1, . . . , yT ] =1

ϕth(yt)

through a recurrence relation:

20 / 32

In Casella+X. (1996), averaging over possible past and future histories(by integrating out uniforms) to improve weights of accepted values

ϕ(i)t = δt

with δ0 = 1 , δt =

t−1∑

δjξj(t−1)ρjt

and ξtt = 1 , ξtj =

(1− ρtu)

occurence survivals of the yt’s, associated with Metropolis–Hastings ratio

ωt = π(yt)/µ(yt) , ρtu = ωu/ωt ∧ 1 .

20 / 32

Integrating out white noise (2)

Extension to generic M-H feasible (C+X, 96)Potentialy large variance improvement but cost of O(T 2)...

Possible recovery of efficiency thanks to parallelisation:Moving from (ǫ1, . . . , ǫp) towards...

(ǫ(1), . . . , ǫ(p))

by averaging over ”all” possible orders

21 / 32

(ǫ(1), . . . , ǫ(p))

21 / 32

(ǫ(1), . . . , ǫ(p))

21 / 32

Case of the independent Metropolis–Hastings algorithm

Starting at time t with p processors and a pool of p proposed values,

(y1, . . . , yp)

use processors to examine in parallel p different “histories”

22 / 32

Case of the independent Metropolis–Hastings algorithm

Starting at time t with p processors and a pool of p proposed values,

(y1, . . . , yp)

use processors to examine in parallel p different “histories”

22 / 32

Improvement

The standard estimator τ1 of Eπ [h(X)]

τ1(xt, y1:p) =1

h(xt+k)

is necessarily dominated by the average

τ2(xt, y1:p) =1

nkh(yk)

where y0 = xt and n0 is the number of times xt is repeated.

23 / 32

Further Rao-Blackwellisation

E.g., use of the Metropolis–Hastings weights wj : j being the index suchthat xt+i−1 = yj , update of the weights at each time t+ i:

wj = wj + 1− ρ(xt+i−1, yi)

wi = wi + ρ(xt+i−1, yi)

resulting into a more stable estimator

τ3(xt, y1:p) =1

wkh(yk)

E.g., Casella+X. (1996)

τ4(xt, y1:p) =1

ϕkh(yk)

24 / 32

Further Rao-Blackwellisation

E.g., use of the Metropolis–Hastings weights wj : j being the index suchthat xt+i−1 = yj , update of the weights at each time t+ i:

wj = wj + 1− ρ(xt+i−1, yi)

wi = wi + ρ(xt+i−1, yi)

resulting into a more stable estimator

τ3(xt, y1:p) =1

wkh(yk)

E.g., Casella+X. (1996)

τ4(xt, y1:p) =1

ϕkh(yk)

24 / 32

Markovian continuity

The Markov validity of the chain is not jeopardised! The chain continues

by picking one sequence at random and taking the corresponding x(j)t+p as

starting point of the next parallel block.

25 / 32

Markovian continuity

The Markov validity of the chain is not jeopardised! The chain continues

by picking one sequence at random and taking the corresponding x(j)t+p as

starting point of the next parallel block.

25 / 32

Impact of Rao-Blackwellisations

Comparison of

τ1 basic IMH estimator of Eπ [h(X)],

τ2 improving by averaging over permutations of proposed values andusing p times more uniforms

τ3 improving upon τ2 by basic Rao-Blackwell argument,

τ4 improving upon τ2 by integrating out ancillary uniforms, at a costof O(p2).

26 / 32

Illustration

Variations of estimates based on RB and standard versions of parallelchains and on a standard MCMC chain for the mean and variance of thetarget N (0, 1) distribution (based on 10, 000 independent replicas).

27 / 32

Illustration

27 / 32

Illustration

27 / 32

Illustration

27 / 32

Impact of the order

Parallelisation allows for the partial integration of the uniforms

What about the permutation order?Comparison of

τ2N with no permutation,

τ2C with circular permutations,

τ2R with random permutations,

τ2H with half-random permutations,

τ2S with stratified permutations,

28 / 32

Impact of the order

What about the permutation order?

28 / 32

Impact of the order

28 / 32

Impact of the order

28 / 32

Impact of the order

28 / 32

Importance target

Comparison with the ultimate importance sampling

29 / 32

Importance target

29 / 32

Importance target

29 / 32

Importance target

29 / 32

Extension to the general case

Same principle can be applied to any Markov update: if

xt+1 = Ψ(xt, ǫt)

then generate(ǫ1, . . . , ǫp)

in advance and distribute to the p processors in different permutationordersPlus use of Douc & X’s (2011) Rao–Blackwellisation ξki

30 / 32

Extension to the general case

Same principle can be applied to any Markov update: if

xt+1 = Ψ(xt, ǫt)

then generate(ǫ1, . . . , ǫp)

in advance and distribute to the p processors in different permutationordersPlus use of Douc & X’s (2011) Rao–Blackwellisation ξki

30 / 32

Implementation

Similar run of p parallel chains (x(j)t+i), use of averages

τ2(x(1:p)1:p ) =

nkh(x(j)t+k)

and selection of new starting value at random at time t+ p:

31 / 32

Implementation

Similar run of p parallel chains (x(j)t+i), use of averages

τ2(x(1:p)1:p ) =

nkh(x(j)t+k)

and selection of new starting value at random at time t+ p:

31 / 32

Illustration

Variations of estimates based on RB and standard versions of parallelchains and on a standard MCMC chain for the mean and variance of thetarget distribution (based on p = 64 parallel processors, 50 blocs of pMCMC steps and 500 independent replicas).

RB par org

32 / 32

Illustration

Variations of estimates based on RB and standard versions of parallelchains and on a standard MCMC chain for the mean and variance of thetarget distribution (based on p = 64 parallel processors, 50 blocs of pMCMC steps and 500 independent replicas).

RB par org

32 / 32

Vanilla rao blackwellisation

Education

Gelato - Napoli Food and Wines · 2016. 11. 23. · Elenka Vanilla is made with Vanilla bean sticks from Madagascar. Vanilla is a classic elenka Specialty flavour. Vanilla is used

Vanilla Chemistry

Vanilla Ice

Vanilla extract

Vanilla Specialty Crop

Faber, Michel. “Vanilla Bright Like Eminem.” Vanilla ...pioneer.netserv.chula.ac.th/~tpuckpan/Faber, Michel-vanillabright.pdf · Vanilla Bright like Eminem and his family are

Rapid Infusion. - JB Prince Company · 4 g · 0.2 oz Bourbon vanilla bean Preparation: Scrape out the bourbon vanilla bean. Put the vanilla bean, vanilla extract and vodka into a

Slides Set 11 (part a)dechter/courses/ics-276/spring...Slides Set 11 (part a): Rina Dechter slides11a 828X 2019 Sampling Techniques for Probabilistic ... 5.Rao‐Blackwellisation,

Vanilla Ice Cream.pdf

Vanilla Vanilla Chocolate Vanilla Red Velvet Chocolate ... · Gluten Free Available Vegan Available Vanilla Lemon Vanilla Coconut Chocolate Strawberry Banana Chocolate Chocolate Peanut

Vanilla Report Reunion

1999 Model Owners Manual - my-sport.spb.rumy-sport.spb.ru/manual_1/1999-shox-manual.pdf · 1999 Model Owners Manual Air Vanilla Air Vanilla FLOAT Air Vanilla FLOAT-R Air Vanilla FLOAT-RC

Vanilla JS*

Vanilla Cande Wedding Brochure - Vanilla Catering & Events

VANILLA - assets.adm.com

This Month's Autoship: French Vanilla...This Month's Autoship: French Vanilla Table of Contents French Vanilla: A creamy confection with a strong vanilla scent and popular year-round

Vanilla Fry

Red Vanilla 2009

Vanilla Plus Directory

Vanilla Funk