WinBUGS: a · PDF fileOverview WinBUGS: a tutorial Anastasia Lykou1 and Ioannis Ntzoufras2,∗ The reinvention of Markov chain Monte Carlo (MCMC) methods and their

Overview

WinBUGS: a tutorialAnastasia Lykou1 and Ioannis Ntzoufras2,∗

The reinvention of Markov chain Monte Carlo (MCMC) methods and theirimplementation within the Bayesian framework in the early 1990s has establishedthe Bayesian approach as one of the standard methods within the appliedquantitative sciences. Their extensive use in complex real life problems has leadto the increased demand for a friendly and easily accessible software, whichimplements Bayesian models by exploiting the possibilities provided by MCMCalgorithms. WinBUGS is the software that covers this increased need. It is theWindows version of BUGS (Bayesian inference using Gibbs sampling) packageappeared in the mid-1990s. It is a free and a relatively easy tool that estimates theposterior distribution of any parameter of interest in complicated Bayesian models.In this article, we present an overview of the basic features of WinBUGS, includinginformation for the model and prior specification, the code and its compilation, andthe analysis and the interpretation of the MCMC output. Some simple examplesand the Bayesian implementation of the Lasso are illustrated in detail. 2011 JohnWiley & Sons, Inc. WIREs Comp Stat 2011 3 385–396 DOI: 10.1002/wics.176

HISTORICAL BACKGROUND

The BUGS (Bayesian inference using Gibbssampling) project was introduced in 1989 by the

research group of David Spiegelhalter in the MRCBiostatistics Unit as a software that uses Markov chainMonte Carlo (MCMC) methods in complex Bayesianmethods. BUGS was initially a software for DOSand Linux operating systems limited in specializedalgorithms. The project started expanding after itmoved to the Imperial College in 1996 headed byNicky Best, and the first version of WinBUGS forWindows became available in 1997. In the followingyears, WinBUGS was improved and extended byconsidering more complicated model structures. Laterwhen Jon Wakefield and Dave Lunn joined the group,WinBUGS capabilities improved impressively. Thelatest version of WinBUGS (1.4.3) was developedjointly by Imperial College and the School of Medicineat St Mary’s, London. More details can be found inRef 1 and in the web page of OpenBUGS (http://www.openbugs.info).

WinBUGS has become widely popular over thelast years as it can estimate the posterior distributions

∗Correspondence to: [email protected] of Mathematics and Statistics, Lancaster University,Lancaster, UK2Department of Statistics, Athens University of Economics andBusiness, Athens, Greece

DOI: 10.1002/wics.176

of the parameters of interest in a variety of modelsusing MCMC. It only requires to specify the modelcode in which the model likelihood and the priordistribution are defined. The programming syntaxand requirements are similar to the ones in Rand Splus. For those who are not familiar withprogramming, there is also DOODLE, a graphicalinterface in which we can define our model viadrawing the corresponding directed acyclic graph(DAG). Moreover, WinBUGS is freely availableand this is one of the reasons that contributedin its growing popularity. A detailed manual,2 aswell as three volumes of examples3–5 that provideimportant help and insight to the WinBUGS user,is available in the BUGS project website (www.mrc-bsu.cam.ac.uk). A comprehensive introductionin Bayesian modeling using WinBUGS is also offeredby Ntzoufras,6 in which emphasis is given on modelbuilding, implementation using WinBUGS, and theinterpretation and analysis of the posterior results.

The purpose of this article is to provide a com-prehensive short tutorial by summarizing the mostimportant features of WinBUGS. It provides informa-tion on how to code and compile a model through twosimple examples and a more detailed one that illus-trates the implementation of Bayesian Lasso method inWinBUGS. The remainder of this article is organizedas follows. Section ‘Getting Started with Winbugs’contains a brief description of the menu bar anddetailed illustration of the model specification, the

Volume 3, September/October 2011 2011 John Wiley & Sons, Inc. 385

http://www.openbugs.info

http://www.openbugs.info

www.mrc-bsu.cam.ac.uk

www.mrc-bsu.cam.ac.uk

Overview wires.wiley.com/compstats

data and the initial values definition using two simpleexamples. Section ‘Running an MCMC in Winbugsand Obtaining Posterior Summaries’ provides a con-densed list of the actions needed to run the MCMCalgorithm for a model and how to obtain and analyzethe corresponding output. An example of implement-ing the Bayesian Lasso in WinBUGS can be found inthe Section ‘An Illustrating Example—ImplementingBayesian Lasso in Winbugs’. The article closes witha short discussion concerning the future potentials ofWinBUGS followed by a short conclusion.

GETTING STARTED WITH WINBUGS

The Menu BarThe latest version of WinBUGS (1.4.3) as wellas installation instructions and the free key forunrestricted use can be found on the soft-ware’s website: www.mrc-bsu.cam.ac.uk/bugs/winbugs/contents.shtml. Once the installation processhas been completed, WinBUGS can be assessed bydouble-clicking its shortcut. All main operations areavailable in a menu bar, which is similar to the onesfound at any windows-based program.

The basic operations in the menu bar arethe File, Window, and Help menus. The Filemenu handles the usual file actions that areavailable in any windows-based software (e.g., Open,Close, Save, Print), the Window managesactive windows in WinBUGS and the Help providesaccess to the detailed manual and examples ofWinBUGS which are very useful for the user.

The Tools, Edit, Attributes, and Textmenus refer to editing facilities of the documents inWinBUGS. To be more specific, the Editmenu allowsthe user to do the usual editing actions of any word-processing software (e.g., Copy, Cut, Paste),while the Tools menu manages more specializedactions available for WinBUGS compound doc-uments (insert dates, encode, and decodealgorithms). The WinBUGS user can change thecolor, the font type, and size of the text in a com-pound document using the Attributes menu, whilehe/she can find or replace a text, insert a ruler, a para-graph, or blank spaces (amongst other operations) inthe Text menu.

The most substantial WinBUGS operations areincluded in the Model and Inference menus.They include all the commands and actions relatedto running the MCMC algorithm and analyzing itsoutput in order to obtain posterior results.

The Specification tool, under the Modelmenu, is used to compile the model and initializethe MCMC algorithm; the Update tool generates

random observations from the posterior distributionand the Monitor Met tool checks the acceptancerate of the Metropolis-Hasting algorithm (in casesit is used). The last set of generated values can besaved using the Save State tool. This set of valuescan be used as initial values in cases that we wantto rerun the MCMC algorithm from the point that aprevious MCMC run stopped. The Inference menuprovides information about the posterior distributionof the model parameters. This information is basedon the analysis of the MCMC output. Under thismenu, the most frequently used tool is the Samplestool, which provides basic descriptive summaries andgraphical representation for specific attributes of theMCMC output and the corresponding estimated pos-terior distribution.

The Info and Options menus are offeringsome auxiliary tools for MCMC analysis. From theformer menu, we can open or clear a log window(where WinBUGS results and figures are printed) orwe can extract the current parameter values usingthe Node info tool. From the latter menu, we canchange some options concerning the output, the block-ing and the generation algorithm used to update eachparameter.

Finally, the Map and the Doodle menus areoffering more specialized tools for the user. The Mapmenu corresponds to the GeoBUGS add-in moduleand it can be used for spatial modeling and mapping.The Doodle menu is used to construct the DAGthat describes the conditional dependencies of theBayesian model we wish to fit. This tool is essentialfor those who are not familiar with programming asthe model code can be generated from this graphicalrepresentation of the model. Nevertheless, it assumesvery good knowledge and understanding of Bayesianmodels and how they can be represented in terms ofconditional distributions and hierarchies.

How to Code a ModelWinBUGS uses its own type of input and output filesthat are called compound documents and are savedwith the odc suffix. The first step in WinBUGS isto specify the model, which includes the likelihoodfunction for the observed sample and the priorinformation for the parameters. The model code iswritten in a compound file within the syntax

model { ...... } .There are three categories for the model

parameters (or nodes): the constant, the stochastic,or random and the logical. The constant nodes arefixed values while the stochastic are random variables,such as the data and the model parameters. Thestochastic nodes follow a distribution, which can be

386 2011 John Wiley & Sons, Inc. Volume 3, September/October 2011

WIREs Computational Statistics WinBUGS

specified by using commands similar with the onesused in R and Splus packages. The logical nodesare mathematical expressions of other (constant orstochastic) components.

Example 1Let’s assume that a part of a Bayesian model includesa normally distributed variable X ∼ Normal(µ, σ 2 =1/τ ), with known mean µ = 0 and unknownprecision τ . If the uncertainty about the precisionis expressed by considering the Gamma priordistribution τ ∼ Gamma(0.01, 0.01), this can beexpressed in WinBUGS as follows:

model{mu <- 0a<-0.01b<-0.01X ~ dnorm( mu, tau )tau ~ dgamma( a, b )sigma2 <- 1/tau # sigma2:

variance of the normaldistribution

}

In the above syntax, the sign # is used to addcomments, ~ is used to specify that a randomnode follows a distribution and <- is used todefine assignments for constant and logical nodes.Nodes mu, a, b are constants here, X and tauare stochastic components and sigma2 is a logicalcomponent. The commands dnorm( mu, tau) anddgamma(a,b) are used to specify the normal (withmean mu and variance 1/tau) and the gamma (withmean a/b) distributions respectively. A list with allthe distributions available in WinBUGS can be foundat the software’s manual2 and in Ref 6, p. 90–91.Each component/node must be uniquely defined in theWinBUGS syntax.

Vectors, matrices, and arrays can be representedin WinBUGS in a similar way as in R/Splus pack-ages. In the example above, the random variable X isa scalar. If X is a n-dimensional random vector this isdenoted in WinBUGS by X[]. Syntax X[i] refers tothe i-th element of vector X for i = 1, . . . , n, whereas,X[i:j] extracts a vector with components the i-thup to the j-th element of X. If M is a matrix this isspecified in WinBUGS by M[]. If M is a n × p matrixthen M[i,j] corresponds the mij element of M for anyi ∈ {1, . . . , n} and j ∈ {1, . . . , p}. The entire i-th row canbe extracted by typing M[i,]while the j-th column byM[,j]. The sub-matrix that contains all elements mijwith i1 ≤ i ≤ i2 and j1 ≤ j ≤ j2 can be extracted usingthe syntax M[i1:i2,j1:j2]. A three-dimensionalarray A is denoted by A[ , , ] and the item

A[i,j,k] correspond to the Aijk element. Arraysof lower dimensions or parts of the initial array canbe derived using syntax similar to the one used for thematrices; further details can be found in Section 3.3.2of Ref 6.

Calculations among vectors, matrices, andarrays cannot be performed directly in WinBUGS.The following calculation xTy between the vectorsx and y can be performed using the functioninprod( x[], y[] ) and the multiplicationbetween two matrices A and B of dimension n × kand k × m can be performed using the inprod for allthe rows and columns. Thus, the use of the functionfor is required to define a loop among the rows andthe columns, and the multiplication is performed asfollows:

for (i in 1:n){for (j in 1:m){

C[i,j] <- inprod(A[i,],B[,j])

}}

A set of built-in functions are available withinWinBUGS including arithmetic functions such as theabsolute value (abs), the exponential (exp), thenatural logarithm (log), the square root (sqrt), andthe statistical functions such as the sum (sum) thestandard deviation (sd) and the rank (rank). A listwith the functions that can be used in WinBUGS canbe found on the software’s manual.2

Model SpecificationSuppose that n realizations are available for theresponse variable y, which follows a distribution withparameter vector θ . Assume that the parameter θ isrelated with the explanatory variables X1, X2, . . . , Xpthrough a link function h and a parameter vector β.The prior information for β is expressed by a knowndistribution. This model is specified in WinBUGS withthe following syntax.

# Likelihoodfor (i in 1:n){

y[i] ~ distribution.name(theta)}# Link functiontheta <- [function of beta

and X’s]# prior distributionbeta ~ distribution.name( ... )

If the parameters β, θ have dimension higher than one,a for loop is needed to specify them.



Example 2Consider a normal linear regression model withthe response vector y = (y1, . . . , yn)T following thenormal distribution and xi1, xi2 the realizationsof 2 explanatory variables X1, X2 for i = 1, . . . , nindividuals. The Bayesian model is formulated as

Yi ∼ N(µi, σ 2) for i = 1, . . . , p

and the parameter µi is a linear combination of theexplanatory variables,

µi = β0 + β1x1i + β2x2i.

The Normal distribution is chosen to express the priorinformation for β and the Gamma distribution for theprecision τ = σ−2. Hence,

βi ∼ Normal(0, 104) for i = 0, 1, 2,

τ ∼ Gamma(0.01, 0.01).

The syntax of the above regression modelfollows.

model{# definition of the likelihood

functionfor (i in 1:n){

y[i] ~ dnorm( mu[i], tau )mu[i] <- beta0 + beta1*

x1[i] + beta2 * x2[i]}# prior distributionsbeta0 ~ dnorm( 0.0 , 1.0E-4 )beta1 ~ dnorm( 0.0 , 1.0E-4 )beta2 ~ dnorm( 0.0 , 1.0E-4 )tau ~ dgamma( 0.01, 0.01 )# other parameters of interests2 <- 1/taus <- sqrt(s2)

}

This syntax can be written in a more general wayin order to cover the case with p explanatory variables.This can be achieved by introducing the parametervector β = (β0, β1, . . . , βp)T and the n × (p + 1) datamatrix X = (1n, X1, . . . , Xp), where 1n is a vectorof length n with all elements equal to one and X j(j = 1, . . . , p) is the vector of length n, with elementsthe observed values of covariate Xj. The syntax canbe now written as follows:

model{# definition of the likelihood

function

for (i in 1:n){y[i] ~ dnorm( mu[i], tau )mu[i] <- inprod(X[i,]) ,

beta[] )}# prior distributionsfor (j in 1:p+1){

beta[j] ~ dnorm( 0.0 ,1.0E-4 )

}tau ~ dgamma( 0.01, 0.01 )# other parameters of interests2 <- 1/taus <- sqrt(s2)

}

Data and Initial Value SpecificationTo complete the model specification in WinBUGS, weneed to insert the data and provide some initial valuesfor the MCMC algorithm.

The initial values are used as a starting pointin the iterative procedure of the MCMC algorithm.Initial values are needed for all the model parameters,i.e., stochastic components of the model that are notdata and for which prior distributions are imposed.The option of random generation of a part or allthe initial values is available in WinBUGS but it isadvised to be avoided as it may lead to bad startingpoints entailing slow mixing of the algorithm or evennumerical errors such as overflows.

The syntax used to initialize values for the modelin Example 2 is

# INITSlist( beta=0, beta1=0, beta2=0,

tau=1 )

The data can be written using the list syntaxwhich is similar to the corresponding one used inR/Splus. For example, the data of the Example 2with 2 explanatory variables of n = 5 observationscan be specified as follows:

# DATAlist(n=5,

x1=c(1,2,3,4,5),x2=c(23,57,59,14,36)

)

Alternatively, the data can be specified in matrixform as follows:

# DATAlist(n=5, p=2,

X = structure(.Data=c(1,23,2,



57,3,59,4,14,5,36).Dim=c(5,2))

Both codes above define the same data matrix but withdifferent types of nodes. The second code is preferablewhen the number of variables involved is large. Itis also advised to use R and the command dput todirectly import the data in WinBUGS with this format;see Section 3.4.6.2 in Ref 6 for more details.

Alternatively, the data can be represented in asimpler, rectangular form. The names of the vectornodes (followed by []) must be stated in the first rowfollowed by the variable values for each observation ineach line. The data must conclude with the commandEND followed by a blank line. All variables—vectornodes must have the same length.

DATAy[] x1[] x2[] x3[]20 23 1 1419 6 0 2332 18 1 1222 20 1 12... ... ... ...END(Blank line)

RUNNING AN MCMC IN WINBUGSAND OBTAINING POSTERIORSUMMARIES

Once the model, the data, and the initial values havebeen specified we can compile and run the MCMCalgorithm. The procedure to be followed is describedbelow.

1. Open the Specification tool under theModel menu and highlight the word model.WinBUGS checks the syntax of the model byclicking the check model button. A messageappears in the bottom left-hand corner ofthe window indicating whether the model issyntactically correct or not. If an error messageappears, the model code must be corrected andrevised and then the model syntax must bechecked again (note that the cursor indicatesthe location of the error in the code).

2. If the model is syntactically correct, the loaddata box becomes accessible. Highlight theword list, if the data are presented in thelist form, or the first line of the rectangularform and press the load data button. Themessage data loaded should appear. If morethan one dataset is available we repeat the aboveprocedure until all the data are loaded.

3. If steps 1 and 2 are successful, then the model iscompiled by clicking the corresponding button.The message model compiled should appear.If an error message is produced, then the modelcode must be corrected and the procedure mustbe started from step 1 again.

4. Once the model is compiled successfully, theload inits button becomes active. High-light the word list that contains the ini-tial values of the stochastic components andclick the load inits button. The buttongen inits can alternatively be used to gen-erate random initial values for the parame-ters. If the message this chain containsuninitialized variables appears, thensome stochastic nodes do not have initial val-ues. In such cases gen inits can be alsoused. If the message model is initializedappears in the left bottom bar of WinBUGS thenthe MCMC is ready to start generating randomvalues.

5. Open the Update tool under the Model menuand specify the number of iteration in theburn-in period in the box entitled updates.The number in the box refresh shows howoften the results are refreshed, thin defines thelag between stored iterations and iterationshows the current number of iterations of theMCMC algorithm. Insert the desired numbersand click update. The iteration counter willstart changing until the required number ofiterations is reached.

6. Open the Sample Monitor tool under theInference menu and select the parameters forwhich we wish to infer about. Write the name ofthe parameters in the node box and press set.This should be repeated for all the parametersthat we wish to monitor. Return to the updatetool and select the number of iterations that wewish to generate and then press the updatesbutton. When the counter reaches the requirednumber of iterations then an MCMC samplefrom the posterior distribution is available forthe monitored variables.

Analysis of the MCMC output and inferenceconcerning the posterior distribution can be obtainedby the Sample Monitor tool. Summaries of theposterior distributions can be derived by typing thename of the parameter of interest in the node box (orusing the pull down menu) and pressing the statsbutton. The posterior summaries of all monitoredparameters can be extracted if we use the asterisk



(∗) in the node box. A density plot of the marginalposterior distribution of each node can be obtainedusing the density button.

The remaining buttons can be used to obtaina basic analysis of the MCMC output by checkingthe convergence of the chain (history), the MonteCarlo (MC) error (stats) and the autocorrelation(auto cor). The thinning interval of observationsused to derive the summaries can be controlled by thecorresponding box. A detailed diagnostic analysis ofthe MCMC output can be conducted in R, using theCODA7 and BOA8 packages. The observations of theposterior distributions of the WinBUGS output canbe obtained in a compatible format using the optioncoda.

AN ILLUSTRATING EXAMPLE—IMPLEMENTING BAYESIAN LASSOIN WINBUGS

The Lasso9 is a shrinkage and variable selectionmethod for normal linear regression models. Itsestimates are obtained by imposing the L1 normpenalty on the linear regression coefficients. Owing tothe shape of the L1 norm, the regression coefficientsare shrunk toward zero and some of them are setexactly equal to zero.

The Lasso estimates have an apparent Bayesianperspective as they correspond to the posteriormode of a Bayesian normal linear regression modelusing a double-exponential prior distribution for theregression coefficients. The corresponding model isdescribed by the following formulation:

Yi|β, τ ∼ Normal(µi, τ−1), for i = 1, . . . , n

µ = Xβ with µ = (µ1, µ2, . . . , µn)T

βj ∼ DE(

0,1τλ

), for j = 1, . . . , p,

τ ∼ Gamma(a, d),

where τ = 1/σ 2 is the precision of the Normalregression model and λ is the shrinkage parameterwhich controls the prior variance given by 2/(τλ)2.Different values of λ correspond to different levelsof shrinkage. Small values of λ correspond to largevariance introducing low information in the Bayesianmodel and essentially not imposing any shrinkageon the model parameters. On the other hand, largevalues of λ correspond to strong prior that β is closeto zero resulting to high levels of shrinkage. To ensurethat λ has the same meaning for all covariate effects,we assume, without loss of generality, that all thevariables have mean zero and variance equal to one.

ExampleWe illustrate the Bayesian Lasso regression using asimulated data set that consists of n = 50 observationsand p = 5 covariates generated from independentstandardized normal distributions and the responsefrom

Yi ∼ Normal(Xi3 + Xi4, 2.52), for i = 1, . . . , 50.

The Bayesian Lasso is performed on this dataset fordifferent values of λ by considering 2, 000 updatesafter discarding additional 1,000 iterations as burn-inperiod.

In the Bayesian implementation, we standardizeboth the response and the covariates. For this reason,the constant parameter is not included in the modelfor the standardized variables as it is natural to setit equal to zero. All the variables are standardizedwithin the WinBUGS code (see lines 2–14 in thecode of Algorithm 1). Note that WinBUGS allowsto define stochastic nodes twice only in the case oftransforming response variables as in this example.The original regression parameters are calculated inWinBUGS using simple logical nodes as

β0 = y − Sy

p∑j=1

bjxj

βj = Sy

Sxj

bj

σ 2 = S2yσ

2z

where σ 2z is the error variance, bj the coefficients of

the regression model using the standardized variables,y and xj the sample means, and S2

y and S2xj

the samplevariances for Y and Xj respectively.

The code for the above-defined Bayesian Lassomodel is given in Algorithm 1. The model wasinitialized by setting all standardized coefficients equalto zero and the corresponding precision equal to one.Following the arguments in Ref 10, we choose thevalue of λ = 0.067.

We follow the procedure described inSection ‘Running an MCMC in WinBUGS andObtaining Posterior Summaries’ to obtain summariesfor the posterior distribution. The posterior sum-maries of the parameters of the Lasso regressioncoefficients and the variance of the regression modelare displayed in Figure 1 for the standardized data andin Figure 2 for the original data. The posterior meansand medians of the coefficients of X1 and X2 arevery low indicating that these are the least importantvariables. As expected, the variables used to generate



ALGORITHM 1 | WinBUGS Code for the Bayesian Lasso Model.

the response (X3, X4) have the highest posterior mea-sures, whereas the last covariate (X5) has moderatemeasures. Similar conclusions are drawn from theFigure 3, which gives the densities of the posteriorsummaries of the Lasso coefficients. The results hereprovide some evidence about the important variables,although the variable selection problem is not directlyaddressed. Moreover, we observe that the posteriormeans of β are similar to the ordinary least square esti-mates

[β = (0.16, −0.19, 0.067, 1.19, 1.67, −1.11)T

]concluding that our prior was essentially noninforma-tive implementing minor (or no) shrinkage on themodel parameters. For illustration, we also rerunthe model with λ = 2 resulting to a posterior withsummaries given in Figure 4. For this value of λ, theposterior means are shrunk by 42% for β1 and from5 to 20% for the rest of the coefficients. Differences

of the coefficients obtained using the two values of λ

are depicted in the box plots of Figure 5. The mostnoteworthy change is observed for β5, for which zerolies outside the 95% posterior credible interval in thefirst run and inside this interval in the second run.

We can extend the aforementioned model inorder to incorporate the variable selection procedurein our formulation. This can be achieved byintroducing a vector of binary indicators γ thathighlights which variables are included in the model(with γj = 1) or not (with γj = 0) as in Refs 11 and 12.This formulation was proposed by Lykou andNtzoufras10 and is described below.

Yi|β, τ , γ ∼ Normal(µi, τ−1), for i = 1, 2, . . . , n

µ = Xβ(γ ) with β(γ ) = (β(γ )1 , β(γ )

2 , . . . , β(γ )p )T



1_node_stats_standardized

node mean sd median 97.5% start sampleMC error 2.5%b[1]b[2]b[3]b[4]b[5]sz

100110011001100110011001

200020002000200020002000

0.1520.20920.56110.7774−0.030140.8078

−0.053360.022790.36660.5757−0.23120.6557

−0.2448−0.17110.17570.3859−0.42310.5464

0.0027830.002280.0023240.0021230.0020690.001564

0.10210.096050.097150.098210.10060.06534

−0.05050.021720.36590.5782−0.22930.6609

FIGURE 1 | Posterior summaries of the Lasso regression parameters using standardized data (λ = 0.067).

2_node_stats_full

node mean sd median 97.5% start sampleMC error 2.5%beta0beta[1]beta[2]beta[3]beta[4]beta[5]sigma

−0.1563−0.9156−0.56990.57141.113−2.0351.817

2000200020002000200020002000

1001100110011001100110011001

0.47950.56860.69661.8242.243−0.1452.686

0.1688−0.19960.075891.1921.661−1.1122.18

0.00320.010410.0075930.0075570.0061260.0099540.0052

0.16220.3820.31990.31580.28340.48370.2173

0.1655−0.18890.072331.191.668−1.1032.197

FIGURE 2 | Posterior summaries of the Lasso regression parameters for the unstandardized data (λ = 0.067).

3_densities

3.02.01.00.0

beta0 sample: 2000

−0.5 0.50.0

1.51.00.50.0

beta[1] sample: 2000

−2.0 1.00.0−1.0

1.51.00.50.0


0.0 2.01.0

1.00.750.5

0.250.0


−3.0 0.0−1.0−2.0

1.51.00.50.0


−0.2 0.0 1.0−1.0

1.51.00.50.0


0.0 2.01.0

FIGURE 3 | Posterior densities of the Lasso regression coefficients for the unstandardized data (λ = 0.067).



β(γ )j = γjβj

βj|τ ∼ DE(0,

1τλ

), for j = 1, . . . , p,

τ ∼ Gamma(a, d),

γj ∼ Bernoulli(πj)

where Bernoulli(π ) is the Bernoulli distribution withsuccess probability π . The actual model parametersβ(γ ) are denoted with a vector of length p withelements β

(γ )j = γj × βj (for j = 1, . . . , p) which are

constrained to zero if Xj is not included in the modelformulation.

In order to extend the model code of Algorithm 1according to the above-mentioned formulation weneed to

1. substitute line 18 with the following syntax

mu[i] <- inprod(Z.X[i,],bgamma[])

in order to substitute βj by γjbj,

4_stats_lambda2

b[1]b[2]b[3]b[4]b[5]beta[1]beta[2]beta[3]beta[4]beta[5]beta0sigmasz

0.15820.21110.53920.76330.0060720.59180.7031.7532.2020.029210.54442.930.8811

2000200020002000200020002000200020002000200020002000

1001100110011001100110011001100110011001100110011001

−0.02290.012550.32780.541−0.1829−0.085680.041791.0661.561−0.880.20792.3850.7174

−0.2226−0.16410.12670.3387−0.4028−0.8328−0.54660.41190.9774−1.938−0.1311.9990.6013

0.0026710.0023570.0023850.0021940.0028060.0099920.0078510.0077520.0063310.01350.0036060.0065070.001957

0.096580.091010.10330.10670.10630.36130.30310.33590.30780.51150.1670.23940.07201

−0.02910.017120.32920.5424−0.1876−0.10890.057021.071.565−0.90250.20772.4080.7242

node mean sd median 97.5% start sampleMC error 2.5%

Standardized data: b [ j ] = bj ; sz = szUnstandardized data: beta = b0 (constant term); beta [ j ] = bj ; sigma = s

FIGURE 4 | Posterior summaries of the Lasso regression parameters for standardized and unstandardized data (λ = 2).

5b_boxplot_lambda067

box plot: b

1.0

0.5

0.0

−0.5

[1][2]

[3]

[4]

[5]

5_boxplot_lambda2

box plot: b

1.0

0.5

0.0

−0.5

[1][2]

[3]

[4]

[5]

(a) (b)

FIGURE 5 | Posterior box plots describing the 95% credible intervals of the regression coefficients using the standardized data (obtained byinserting node b in the node box of Compare tool inside the Inference menu.). (a) λ = 0.067; (b) λ = 2.



2. substitute line 33 by

beta0<- meany - sdy *inprod( bgamma[], meansd[] )

in order to calculate correctly the constantparameter for any given model,

3. add the following syntax

for (j in 1:p){# b x gamma for the

standardized databgamma[j] <- b[j] *

gamma[j]# prior for gammagamma[j]~dbern(0.5)# beta*gamma for the

unstandardized databetagamma[j] <- beta[j] *

gamma[j]}

in order to define bγj = γjbj, β

γj = γjβj and the

prior for each γj.

Posterior summaries for β(γ ) are given inFigure 6 for λ = 0.067. We clearly observe thatthe first two coefficients are shrunk toward zero(the posterior means shrunk by 99.6 and 96%respectively). Also the posterior distribution of β5was dramatically pushed toward zero with theposterior mean shrunk by 83.7%. On the otherhand, the posterior means for the coefficients of

X3 and X4 remained almost unchanged (shrunk by8.6 and 0.06% respectively). The posterior inclusionprobabilities for each covariate can be estimated fromthe posterior means of each γj as given in Figure 7.From the results, we clearly observe that covariates X3and X4 must be included in the model having posteriorprobabilities 0.9 and 1.0, respectively. CovariatesX1 and X2 have very low posterior probabilities(∼ 1.8%), while covariate X5 is included in the modelwith posterior probability of 17%.

Figure 8 displays the densities of the posteriordistributions of β(γ ). The posterior distributions ofthe coefficients for the non-important variables havehigh spikes at zero except for covariate X4, whichis the most important one in this illustration andits posterior distribution is well placed away fromzero. The posterior distribution of β3 is bimodal,with one mode at zero and one at a positive value.However, the posterior median of the distributionand the posterior mean of the corresponding inclusionprobability (given by the mean of γ3 and it is equalto 0.90) indicate that this variable should also beincluded in the model.

DISCUSSION: A FUTURE FULLOF PROMISESOver the last years WinBUGS gained a tremendouspopularity within the scientific community. WinBUGShas been used in a wide range of practical problemshaving different scientific backgrounds. It has beenthe object of interest in various short courses andworkshops. More and more universities includeBayesian data analysis courses using WinBUGS intheir syllabus; see ‘BUGS resources online’ link in the

6_stats_gamma

beta0betagamma[1]betagamma[2]betagamma[3]betagamma[4]betagamma[5]sigmabgamma[1]bgamma[2]bgamma[3]bgamma[4]bgamma[5]sz

0.70270.00.01.8442.2510.02.980.00.00.56720.780.00.8963

10000100001000010000100001000010000100001000010000100001000010000

1001100110011001100110011001100110011001100110011001

0.20410.00.01.1551.6690.02.3670.00.00.35520.57830.00.7118

−0.082110.00.00.01.076−1.5411.9410.00.00.00.373−0.32040.5837

0.0090465.839E-45.232E-40.021720.0032310.012750.007031.561E-41.571E-40.0066810.001120.0026510.002114

0.19970.055310.046390.47840.29780.43920.2640.014790.013930.14720.10320.091310.07941

0.23247.3E-40.0026181.0871.669−0.18042.391.951E-47.862E-40.33430.5783−0.03750.7188

node mean sd median 97.5% start sampleMC error 2.5%

Standardized data: bgamma [ j ] = gj × bj ; sz = szUnstandardized data: beta = b0 (constant term); betagamma [ j ] = gj × bj ; sigma = s

FIGURE 6 | Posterior summaries of the Lasso regression coefficients with variable selection.



6_stats_gamma2

node mean sd median 97.5% start sampleMC error 2.5%gamma[1]gamma[2]gamma[3]gamma[4]gamma[5]

0.01810.01780.90161.00.173

0.00.00.01.00.0

0.00.01.01.00.0

0.00.01.01.01.0

1000010000100001000010000

10011001100110011001

0.0019830.0019880.017821.0E-120.01213

0.13330.13220.29790.00.3782

FIGURE 7 | Posterior summaries the indicator parameters included in the Bayesian Lasso model.

7_densities_gamma

3.02.01.00.0

beta0 sample: 10000

−0.5 0.50.0

150.0100.050.00.0

betagamma[1] sample: 10000

−1.0 1.00.0

1.51.00.50.0


−1.0 1.0 2.00.0

15.010.05.00.0


−3.0 −1.0 0.0−2.0

150.0100.050.00.0

betagamma[2] sampe: 10000

−1.0 1.00.0

1.51.00.50.0

betagamma[4] sampe: 10000

0.0 2.01.0

FIGURE 8 | Posterior densities for model parameters β (γ ).

WinBUGS website for a coherent list of such activitiesand courses.

The popularity of WinBUGS has motivatedvarious expansions and made WinBUGS applicable ina wider range of disciplines, such as social, actuarialscience, population genetics, and archaeology.GeoBUGS developed by the team in the ImperialCollege at St Mary’s Hospital can be used to fitspatial models and produce a range of maps as output.PKBUGS has been developed by Dave Lunn to fitpharmacokinetic models. The WinBUGS developmentinterface (WBDev)13 allows the users to define theirown distributions and functions. This was originallydesigned for social scientists but it has been used byother researchers as well. Lunn has also developedthe WinBUGS Differential Interface (WBDiff) whichallows the use of complex functions via arbitrary

systems of ordinary differential equations and theWinBUGS jump interface to perform variable selectionthrough the Reversible Jump MCMC.14,15

WinBUGS’s popularity has also been extendeddue to its compatibility with other statistical packagesand software. WinBUGS runs from GenStat, which isa software for bioscience. It can also be called via Rusing the R2WinBUGS R-library on ComprehensiveR Archive Network (CRAN) site. There are alsocodes that can be used to call WinBUGS throughStata,16 SAS and a code for Excel that does notrequire any knowledge of WinBUGS. MATBUGS isa Matlab interface for WinBUGS, which has beenextended to run in Linux systems. Details about it canbe found on www.mrc-bsu.cam.ac.uk/bugs/winbugs/remote14.shtml.



Note that WinBUGS was now stabilized in ver-sion 1.4.3 and it will not be developed any more.Instead, the interest of the group is now turned onthe development of OpenBUGS (www.openbugs.info)which is an open source version of WinBUGS withadditional features and contributions. This projectstarted in 2004 when Andrew Thomas moved fromLondon to Helsinki and now is stable and reliablepackage running under Windows, Linux, and MACoperating systems. The possibility that researchers willbe able to contribute and improve an already popularand successful software leaves high expectations forthe future.

CONCLUSION

This article summarizes the basic concepts required toperform Bayesian analysis using the WinBUGS. It pro-vides information on model specification and coding,

algorithm implementation and output interpretationalong with some toy examples, and a more detailedillustration that demonstrates the implementation ofBayesian Lasso in WinBUGS. It can be used as abrief introduction to WinBUGS for researchers withor without statistical background.

WinBUGS is a useful computational tool that fitscomplicated Bayesian models using MCMC methods.It is widely popular due to its numerous extensions andapplications in various scientific fields. It is relativelystraightforward to use with syntax similar to the onein R and Splus. Alternatively, DOODLE interface canbe used to specify the structure of the model througha graphical representation. The recent development ofOpenBUGS (the open source version of the program)creates high expectations for the future where anyresearcher will be able to contribute to the develop-ment and the improvement of this popular software.

REFERENCES1. Lunn D, Spiegelhalter D, Thomas A, Best N. The bugs

project: Evolution, critique and future directions. StatMed 2009, 28:3049–3082.

2. Spiegelhalter D, Thomas A, Best N, Lunn D. WinBUGSUser Manual, Version 1.4. MRC Biostatistics Unit,Institute of Public Health and Department of Epidemi-ology and Public Health, Imperial College School ofMedicine, UK, 2003. Available at: http://www.mrc-bsu.cam.ac.uk/bugs/winbugs/contents.shtml. (AccessedMay 4, 2011).

3. Spiegelhalter D, Thomas A, Best N, Lunn D. WinBUGSExamples, vol 1. MRC Biostatistics Unit, Institute ofPublic Health and Department of Epidemiology andPublic Health, Imperial College School of Medicine, UK,2003. Available at: http://www.mrc-bsu.cam.ac.uk/bugs/winbugs/contents.shtml. (Accessed May 4, 2011).

4. Spiegelhalter D, Thomas A, Best N, Lunn D. WinBUGSExamples, vol 2. MRC Biostatistics Unit, Institute ofPublic Health and Department of Epidemiology andPublic Health, Imperial College School of Medicine, UK,2003. Available at: http://www.mrc-bsu.cam.ac.uk/bugs/winbugs/contents.shtml. (Accessed May 4, 2011).

5. Spiegelhalter D, Thomas A, Best N, Lunn D. WinBUGSExamples, vol 3. MRC Biostatistics Unit, Institute ofPublic Health and Department of Epidemiology andPublic Health, Imperial College School of Medicine,UK, 2003. Available at: http://www.mrc-bsu.cam.ac.uk/bugs.

6. Ntzoufras I. Bayesian Modeling Using WinBugs. NewYork: John Wiley & Sons; 2009.

7. Best N, Cowles MK, Vines K. CODA: ConvergenceDiagnostics and Output Analysis Software for GibbsSampling Output, Version 0.30. Cambridge: MRC Bio-statistics Unit, Institute of Public Health; 1996.

8. Smith BJ. Bayesian Output Analysis Program (BOA),Version 1.1.5 User’s Manual, Technical Report. Depart-ment of Public Health, The University of Iowa,2005. Available at: http://www.mrc-bsu.cam.ac.uk/bugs/winbugs/contents.shtml. (Accessed May 4, 2011).

9. Tibshirani R. Regression shrinkage and selection via thelasso. J R Stat Soc [SerB] 1996, 58:267–288.

10. Lykou A, Ntzoufras I. ‘‘On Bayesian Lasso VariableSelection and the specification of the shrinkage parame-ter’’, Technical Report, Athens University of Economicsand Business, 2011.

11. Kuo L, Mallick B. Variable selection for regressionmodels. Sankhya, (B) 1998, 60:65–81.

12. Dellaportas P, Forster J, Ntzoufras I. On Bayesian modeland variable selection using MCMC. Stat Comput 2002,12:27–36.

13. Lunn D. WinBUGS Development Interface (WBDev).ISBA Bull 2003, 3:10–11.

14. Lunn DJ, Whittaker JC, Best N. A Bayesian toolkitfor genetic association studies. Genet Epidemiol 2006,30:231–247.

15. Lunn DJ, Best N, Whittaker J. Generic reversible jumpMCMC using graphical models. Stat Comput 2009,19:395–408.

16. Thompson JT, Palmer T, Moreno S. Bayesian analysisin Stata using WinBUGS. Stata J 2006, 6:530–549.


www.openbugs.info

http://www.mrc-bsu.cam.ac.uk/bugs/winbugs/contents.shtml.






http://www.mrc-bsu.cam.ac.uk/bugs

http://www.mrc-bsu.cam.ac.uk/bugs



Documents

WinBUGS: a · PDF fileOverview WinBUGS: a tutorial Anastasia Lykou1 and Ioannis Ntzoufras2,∗ The reinvention of Markov chain Monte Carlo (MCMC) methods and their