R package 'bayesImageS': a case study in Bayesian computation using Rcpp and OpenMP

R packages Potts model Bayesian computation Conclusion

bayesImageS: a case study in Bayesiancomputation using Rcpp and OpenMP

Matt Moores

OxWaSP mini-symposiumDecember 2, 2016


Outline

1 R packagesPerformance of BLAS

2 Potts model

3 Bayesian computationChequerboard Gibbs samplerPseudolikelihoodThermodynamic integrationExchange algorithmApproximate Bayesian Computation (ABC)


Why write an R package?

PortabilityTest bed for new statistical methodsBuild on existing codeResearch impactKudos

Hadley Wickham (2015) R packages


Why C++?

Most statistical algorithms are iterativeMarkov chain Monte CarloScalability for large datasets

RcppOpenMPEigen or Armadillo

Dirk Eddelbuettel (2013) Seamless R and C++ integration with Rcpp


Inline

One function at a time:�l i b r a r y ( i n l i n e )

sum_ logs ← cxx func t i on ( s igna tu re ( log_vec = "numeric" ) , p l ug in = "RcppArmadillo" , body=’arma::vec log_prob = Rcpp::as<arma::vec>(log_vec);double suml = 0.0;double maxl = log_prob.max();for (unsigned i=0; i < log_prob.n_elem; i++){

if (arma::is_finite(log_prob(i)))suml += exp(log_prob(i) - maxl);

}return Rcpp::wrap(log(suml) + maxl);

’ )


Package Skeleton

Create a new R package:package.skeleton("myPackage", path=".")

Specific skeletons for each C++ library:

Rcpp.package.skeleton("myRcppPackage")

RcppArmadillo.package.skeleton("MyArmadilloPackage")

RcppEigen.package.skeleton("MyEigenPackage")


Annotations

Rcpp wrappers generated automatically:compileAttributes("myRcppPackage")R package documentation generated automatically:roxygenize("myRcppPackage")�

/ /’ Compute the effective sample size (ESS) of the particles./ /’/ /’ The ESS is a ‘‘rule of thumb’’ for assessing the degeneracy of/ /’’ the importance distribution:/ /’ \ deqn {ESS = \ f r a c { ( \ sum_ { q=1}^Q w_q ) ^ 2 } { \ sum_ { q=1}^Q w_q ^ 2 } }/ /’/ /’’ @param log_weights logarithms of the importance weights of each particle./ /’’ @return the effective sample size, a scalar between 0 and Q/ /’’ @references/ /’’ Liu, JS (2001) "Monte Carlo Strategies in Scientific Computing." Springer’/ / [ [ Rcpp : : export ] ]double ef fec t iveSampleSize ( NumericVector log_weights ){

double sum_wt = sum_ logs ( log_weights ) ;double sum_sq = sum_ logs ( log_weights + log_weights ) ;double res = exp (sum_wt + sum_wt − sum_sq ) ;i f ( s td : : i s f i n i t e ( res ) ) return res ;else return 0;

}


Common Problems

Rcpp parameters are passed by reference (not copied):Can rely on R for garbage collectionMemory allocation is slowerCan crash R (and Rstudio (and your OS))R is not thread safe

Cannot call any R functions (even indirectly)within parallel code!

Drew Schmidt (@wrathematics, 2015) Parallelism, R, and OpenMP

@wrathematics


Performance

A simple performance comparison:�l i b r a r y ( RcppEigen )l i b r a r y ( RcppArmadil lo )l i b r a r y (RcppGSL)l i b r a r y ( rbenchmark )

source ( system . f i l e ("examples" , "lmBenchmark.R" ,package ="RcppEigen" ) )

Fitting a linear regression model(by default, n=100000 rows × p=40 columns,with 20 repeat measurements)


Results with default R BLAS

Table: lm benchmark for Mac OS X with reference R BLAS

test relative elapsed user.self sys.self

3 LDLt 1.000 3.611 3.448 0.1647 QR 1.356 4.896 4.801 0.0778 LLt 1.421 5.130 5.007 0.123

11 gpuLm.fit 2.634 9.510 8.332 0.3951 lm.fit 3.760 13.576 13.550 0.0256 SymmEig 5.475 19.770 19.634 0.1262 PivQR 5.517 19.923 19.760 0.1629 arma 20.416 73.723 73.459 0.2554 GESDD 24.815 89.607 89.429 0.175

10 GSL 198.975 718.500 718.157 0.2725 SVD 201.717 728.401 727.958 0.423

matrix dimensions:1650 × 875


Accelerate Umbrella Framework

Table: lm benchmark for Mac OS X with vecLib

test relative elapsed user.self sys.self

3 LDLt 1.000 3.682 3.517 0.1657 QR 1.330 4.898 4.809 0.0888 LLt 1.413 5.204 5.081 0.1211 lm.fit 1.832 6.745 6.702 0.0249 arma 2.396 8.821 20.572 1.229

11 gpuLm.fit 2.655 9.776 8.757 0.3444 GESDD 3.313 12.199 22.120 0.8846 SymmEig 5.405 19.902 19.775 0.1272 PivQR 5.506 20.273 20.095 0.159

10 GSL 195.244 718.889 718.565 0.2735 SVD 198.514 730.930 730.403 0.495

Iacus, Urbanek, Goedman & Ripley (2016) R for Mac OS X FAQ, §10.5


Nested Models

β ← 0

Hidden MRF

θ : scaleD > 1 : dimension∂i : neighbourhoodψ : noise param.

p (yi | zi, ψ)p(zi | z\i, θ

)p (θ | z)p(ψ) p(θ)

y, z, θ, ψ Hidden Ising/Potts

z ∈ {1 . . . k}θ = β : inversetemperature

p(zi | z\i, β

)p (β | z) p(β)

Mixture of Gaussians

λ = {λ1, . . . , λk} :∑k

j=1 λj = 1ψ =

{µ,σ2} : noise param.

p(yi | zi, µj, σ

2j

)p (λ | z) p(λ)


Hidden Markov Random Field

Joint distribution of observed pixel intensities y = {yi}ni=1

and latent labels z = {zi}ni=1:

p(y, z|µ,σ2, β) = p(y|µ,σ2, z)p(z|β) (1)

Additive Gaussian noise:

yi|zi= j iid∼ N(µj, σ

2j)

(2)

Potts model:

π(zi|z\i, β) =exp {β

∑i∼` δ(zi, z`)}∑k

j=1 exp {β∑

i∼` δ(j, z`)}(3)

Potts (1952) Proceedings of the Cambridge Philosophical Society 48(1)


Inverse Temperature


Doubly-intractable likelihood

p(β|z) ∝ C(β)−1π(β) exp {β S(z)} (4)

The normalising constant has computational complexity O(nkn):

C(β) =∑z∈Z

exp {β S(z)} (5)

S(z) is the sufficient statistic of the Potts model:

S(z) =∑

i∼`∈Lδ(zi, z`) (6)

where L is the set of all unique neighbour pairs.


Chequerboard Gibbs

A 2D or 3D regular lattice with first-order neighbourhood ∂i:

◦ ◦ ◦ ◦ ◦◦ ◦ • ◦ ◦◦ • × • ◦◦ ◦ • ◦ ◦◦ ◦ ◦ ◦ ◦

can be partitioned into 2 blocks:

• ◦ • ◦ • ◦ • ◦ • ◦◦ • ◦ • ◦ • ◦ • ◦ •• ◦ • ◦ • ◦ • ◦ • ◦◦ • ◦ • ◦ • ◦ • ◦ •• ◦ • ◦ • ◦ • ◦ • ◦◦ • ◦ • ◦ • ◦ • ◦ •• ◦ • ◦ • ◦ • ◦ • ◦◦ • ◦ • ◦ • ◦ • ◦ •• ◦ • ◦ • ◦ • ◦ • ◦◦ • ◦ • ◦ • ◦ • ◦ •

so that z◦ are conditionally independent, given z•Roberts & Sahu (1997) JRSS B 59(2): 291–317Winkler (2nd ed., 2003) Image analysis, random fields and MCMC methods


Chequerboard Gibbs II

Algorithm 1 Chequerboard sampling for z1: for all blocks b do2: for all pixels i ∈ b do3: for all labels j ∈ 1 . . . k do4: Compute λj ← p(yi | zi = j)π(zi = j | zi∼`, β)5: end for6: Draw zi ∼ Multinomial(λ1, . . . , λk)7: end for8: end for


Gibbs sampler in C++�void gibbsLabels ( const arma : : umat & neigh , const std : : vector <arma : : uvec> & blocks ,

arma : : umat & z , arma : : umat & a l l oc , const double beta ,const arma : : mat & l og_ x f i e l d )

{const Rcpp : : NumericVector randU = Rcpp : : r u n i f ( neigh . n_rows ) ;for ( unsigned b=0; b < blocks . s ize ( ) ; b++){

const arma : : uvec block = blocks [ b ] ;arma : : vec log_prob ( z . n_ co ls ) ;

#pragma omp p a r a l l e l for pr ivate ( log_prob )for ( unsigned i =0; i < b lock . s ize ( ) ; i ++){

for ( unsigned j =0; j < z . n_ co ls ; j ++){

unsigned sum_neigh = 0;for ( unsigned k =0; k < neigh . n_ co ls ; k++){

sum_neigh += z ( neigh ( b lock [ i ] , k ) , j ) ;}log_prob [ j ] = log_ x f i e l d ( b lock [ i ] , j ) + beta∗sum_neigh ;

}double t o t a l _ l l i k e = sum_ logs ( log_prob ) ;double cumProb = 0 . 0 ;z . row ( b lock [ i ] ) . zeros ( ) ;for ( unsigned j =0; j < log_prob . n_elem ; j ++){

cumProb += exp ( log_prob [ j ] − t o t a l _ l l i k e ) ;i f ( randU [ b lock [ i ] ] < cumProb ){

z ( b lock [ i ] , j ) = 1 ;a l l o c ( b lock [ i ] , j ) += 1 ;break ;

}}

}}

}


Pseudolikelihood (PL)

Algorithm 2 Metropolis-Hastings with PL1: Draw proposal β′ ∼ q(β′|β◦)2: Approximate p(β′|z) and p(β◦|z) using equation (7):

p̂PL(β|z) ≈n∏

i=1

exp{β∑

i∼` δ(zi, z`)}∑kj=1 exp{β

∑i∼` δ(j, z`)}

(7)

3: Calculate the M-H ratio ρ = p̂PL(β′|z)π(β′)q(β◦|β′)p̂PL(β◦|z)π(β◦)q(β′|β◦)

4: Draw u ∼ Uniform[0, 1]5: if u < min(1, ρ) then6: β ← β′

7: else8: β ← β◦

9: end ifRydén & Titterington (1998) JCGS 7(2): 194–211


Pseudolikelihood in C++

�double pseudol ike ( const arma : : mat & ne , const arma : : uvec & e , const double b ,

const unsigned n , const unsigned k ){

double num = 0 . 0 ;double denom = 0 . 0 ;

#pragma omp p a r a l l e l for reduc t ion ( + :num, denom)for ( unsigned i =0; i < n ; i ++){

num=num+ne ( e [ i ] , i ) ;double tdenom =0.0 ;for ( unsigned j =0; j < k ; j ++){

tdenom=tdenom+exp ( b∗ne ( j , i ) ) ;}denom=denom+log ( tdenom ) ;

}return b∗num−denom ;

}


Approximation Error

PL for n = 12, k = 3 in comparison to the exact likelihoodcalculated using a brute force method:

0 1 2 3 4

68

10

12

14

16

β

µ

exact

pseudolikelihood

(a) Expectation

0 1 2 3 4

0.0

0.5

1.0

1.5

2.0

2.5

β

σ

exact

pseudolikelihood

(b) Standard deviation


Thermodynamic Integration (TI)

Path sampling identity:

log{C(β◦)C(β′)

}=

∫ β◦

β′E z|β [S(z)] dβ (8)

0.0 0.5 1.0 1.5 2.0

50

00

00

10

00

00

01

50

00

00

20

00

00

0

ϕ

S(x

)

Gelman & Meng (1998) Stat. Sci. 13(2): 163–185.


TI algorithm

Algorithm 3 Random walk Metropolis with TI1: Draw random walk proposal β′ ∼ q(β′|β◦)2: Estimate S(z|β◦) and S(z|β′) by interpolation3: Evaluate the definite integral in equation (8)4: Calculate the log M-H acceptance ratio:

log{ρ} = log{C(β◦)C(β′)

}+ (β′ − β◦)S(z) (9)


8: else9: β ← β◦

10: end if


TI in C++

�unsigned pathBeta ( const arma : : umat & neigh , const s td : : vector <arma : : uvec> & blocks ,

const arma : : mat & path , const arma : : umat & z , double & beta ,const double p r i o r _beta [ 2 ] , const double bw)

{double bprime = rwmh( beta , bw, p r i o r _beta ) ; / / t runca ted Gaussian

/ / approximate log (Z ( bprime ) / Z( beta ) )double l og_ r a t i o = quadrature ( bprime , beta , path )

+ ( bprime−beta ) ∗ sum_ i den t ( z , neigh , b locks ) ;

/ / accept / r e j e c ti f ( u n i f _ rand ( ) < exp ( log_ r a t i o ) ){

beta = bprime ;return 1;

}return 0;

}


Approximate Exchange Algorithm (AEA)

Algorithm 4 AEA1: Draw random walk proposal β′ ∼ q(β′|β◦)2: Generate w|β′ by sampling from eq. (3)3: Calculate the M-H acceptance ratio according to eq. (4):

ρ =π(β′) exp {β′S(z)} C(β◦)π(β◦) exp {β◦S(z)} C(β′)

exp {β◦S(w)} C(β′)exp {β′S(w)} C(β◦)

(10)


7: else8: β ← β◦

9: end if

Murray, Ghahramani & MacKay (2006) Proc. 22nd Conf. UAI, 359–366


AEA in C++�unsigned exchangeBeta ( const arma : : umat & neigh , const std : : vector <arma : : uvec> & blocks ,

const arma : : uvec & s l i c e , const arma : : umat & z , double & beta ,const double p r i o r _beta [ 2 ] , const unsigned aux , const bool useSW,const bool swapAux , const double bw)

{double bprime = rwmh( beta , bw, p r i o r _beta ) ;arma : : umat a l l o c = arma : : zeros <arma : : umat >( z . n_rows−1, z . n_ co ls ) ;arma : : umat w;i f ( swapAux ) w = z ;else w = randomIndices ( z . n_rows−1, z . n_ co ls ) ;for ( unsigned i =0; i <aux ; i ++){

i f (useSW){

swLabelsNoData ( neigh , blocks , bprime , w. n_cols , w, a l l o c ) ;}else{

gibbsLabelsNoData ( neigh , blocks , w, a l l oc , bprime ) ;}

}double sum_z = sum_ i den t ( z , neigh , b locks ) ;double sum_w = sum_ i den t (w, neigh , b locks ) ;double l og_ r a t i o = ( bprime−beta )∗sum_z + ( beta−bprime )∗sum_w;/ / accept / r e j e c ti f ( u n i f _ rand ( ) < exp ( log_ r a t i o ) ){


}return 0;

}


Approximate Bayesian Computation

Algorithm 5 ABC rejection sampler1: Draw independent proposal β′ ∼ π(β)2: Generate w|β′ by sampling from eq. (3)3: if ‖S(w)− S(z)‖ < ε then4: β ← β′

5: else6: β ← β◦

7: end if

Grelaud, Robert, Marin, Rodolphe & Taly (2009) Bayesian Analysis 4(2)Marin & Robert (2014) Bayesian Essentials with R §8.3


ABC with Metropolis-Hastings

Algorithm 6 ABC-MCMC1: Draw proposal β′ ∼ q(β′|β◦)2: Generate w|β′ by sampling from eq. (3)3: Draw u ∼ Uniform[0, 1]4: if u < π(β′)q(β◦|β′)

π(β◦)q(β′|β◦) and ‖S(w)− S(z)‖ < ε then5: β ← β′

6: else7: β ← β◦

8: end if

Marjoram, Molitor, Plagnol & Tavaré (2003) PNAS 100(26): 15324–28


ABC-MCMC in C++�unsigned abcBeta ( const arma : : umat & neigh , const std : : vector <arma : : uvec> & blocks ,

const arma : : umat & z , double & beta , const double p r i o r _beta [ 2 ] ,const unsigned aux , const bool useSW, const bool swapAux ,const double bw, const double eps i l on )

{double bprime = rwmh( beta , bw, p r i o r _beta ) ;arma : : umat a l l o c = arma : : zeros <arma : : umat >( z . n_rows−1, z . n_ co ls ) ;arma : : umat w;i f ( swapAux ) w = z ;else w = randomIndices ( z . n_rows−1, z . n_ co ls ) ;for ( unsigned i =0; i <aux ; i ++){

i f (useSW){

swLabelsNoData ( neigh , blocks , bprime , w. n_cols , w, a l l o c ) ;}else{

gibbsLabelsNoData ( neigh , blocks , w, a l l oc , bprime ) ;}

}double sum_z = sum_ i den t ( z , neigh , b locks ) ;double sum_w = sum_ i den t (w, neigh , b locks ) ;double de l t a = fabs (sum_w − sum_z ) ;i f ( de l t a < eps i l on ){


}return 0;

}


Summary

bayesImageS supports methods for updating the latent labels:Chequerboard Gibbs sampling (Winkler 2003)Swendsen-Wang (1987)

and also methods for updating the inverse temperature β:Pseudolikelihood (Rydén & Titterington 1998)Thermodynamic integration (Gelman & Meng 1998)Exchange algorithm (Murray, Ghahramani & MacKay 2006)Approximate Bayesian computation (Grelaud et al. 2009)Sequential Monte Carlo (ABC-SMC) with pre-computation(Del Moral, Doucet & Jasra 2012; Moores et al. 2015)

Appendix

Acknowledgements

Principal supervisor: Kerrie MengersenAssociate supervisor: Fiona Harden

Radiation Oncology Mater Centre (ROMC), Queensland Health:

Cathy Hargrave

Mike Poulsen

Tim Deegan

Other co-authors:

Christopher Drovandi (QUT)

Anthony N. Pettitt (QUT)

Christian Robert(University of Warwick & Université Paris Dauphine)

PyMCMC team at QUT:

Clair Alston

Christopher Strickland

Appendix

For Further Reading I

D. EddelbuettelSeamless R and C++ integration with RcppSpringer-Verlag, 2013.

H. WickhamR packagesO’Reilly, 2015.

D. Bates & D. EddelbuettelFast and elegant numerical linear algebra using the RcppEigenpackage.J. Stat. Soft. 52(5): 1–24, 2013.

D. Eddelbuettel & C. SandersonRcppArmadillo: Accelerating R with high-performance C++ linearalgebra.Comput. Stat. Data Anal. 71: 1054–63, 2014.

Appendix

For Further Reading II

M. Moores & K. MengersenbayesImageS: Bayesian methods for image segmentation using ahidden Potts model.R package version 0.3-3https://CRAN.R-project.org/package=bayesImageS

M. Moores, A. N. Pettitt & K. MengersenScalable Bayesian inference for the inverse temperature of a hiddenPotts model.arXiv:1503.08066 [stat.CO], 2015.

M. Moores, C. C. Drovandi, K. Mengersen & C. P. RobertPre-processing for approximate Bayesian computation in imageanalysis.Statistics & Computing 25(1): 23–33, 2015.

M. Moores & K. MengersenBayesian approaches to spatial inference: modelling and computationalchallenges and solutions.In Proc. 33rd Int. Wkshp MaxEnt, AIP Conf. Proc. 1636: 112–117, 2014.

https://CRAN.R-project.org/package=bayesImageS

arXiv:1503.08066

Appendix

For Further Reading III

G. WinklerImage analysis, random fields and Markov chain Monte Carlo methods2nd ed., Springer-Verlag, 2003.

J.-M. Marin & C. P. RobertBayesian Essentials with RSpringer-Verlag, 2014.

G. O. Roberts & S. K. SahuUpdating Schemes, Correlation Structure, Blocking andParameterization for the Gibbs SamplerJ. R. Stat. Soc. Ser. B 59(2): 291–317, 1997.

T. Rydén & D. M. TitteringtonComputational Bayesian Analysis of Hidden Markov Models.J. Comput. Graph. Stat., 7(2): 194–211, 1998.

Appendix

For Further Reading IV

A. Gelman & X.-L. MengSimulating normalizing constants: from importance sampling to bridgesampling to path sampling.Statist. Sci., 13(2): 163–185, 1998.

I. Murray, Z. Ghahramani & D. J. C. MacKayMCMC for Doubly-intractable Distributions.Proc. 22nd Conf. UAI, 359–366, 2006.

A. Grelaud, C. P. Robert, J.-M. Marin, F. Rodolphe & J.-F. TalyABC likelihood-free methods for model choice in Gibbs random fields.Bayesian Analysis 4(2): 317–336, 2009.

R. H. Swendsen & J.-S. WangNonuniversal critical dynamics in Monte Carlo simulations.Physical Review Letters, 58: 86–8, 1987.

Science

R package 'bayesImageS': a case study in Bayesian computation using Rcpp and OpenMP