Chapter 14 Computing: McCulloch, CE, Searle SR and Neuhaus, JM 2008. Generalized, Linear and Mixed Models. John Wiley & Sons, New York. Second Edition

Review of LM, GLM, LMM, GLMMNumerical Integration for Solving GLMM

GLMM in R

Chapter 14: ComputingGeneralized, Linear, and Mixed Models

Charles E. McCulloch, Shayle R. Searle, John M. Neuhaus

Gota Morota

May 4, 2010

Gota Morota Chapter 14: Computing


GLMM in R

Outline

1 Review of LM, GLM, LMM, GLMM

2 Numerical Integration for Solving GLMM

3 GLMM in R



GLMM in R

Linear Model and Linear Mixed Model

LM: Solving the MLE with a least squares for fixed effects andsimple formula for estimating residual variance.

B = (X′X)−1X′y

σ2e =

RSSN − p

LMM Solving the MLE with a least squares for the fixed effectsand random effects. Solving the MLE with a (RE)ML for thevariance components.

B = (X′V−1X)X′V−1y

u = DZ′V−1(y − XB)

Variance Components = (RE)ML coupled with iterative methods

where D is Var(u), V is Var(y) = ZDZ′ + R



GLMM in R

Generalized Linear Model and Generalized Linear MixedModel

GLM: Fixed effects are estimated by solving the MLE(nonlinear) with an iterative reweighted least squares such asFisher Scoring.

Bm+1 = Bm + (X′WX)−1X′W∆(y − u)

where ui = g−1(x′i B), g(ui) = x′i B, ∆ = {gu(ui)}, W = {wi},wi = [υ(ui)g2

u(ui)]−1

GLMM Requires high dimensional integration to evaluate andmaximizing the likelihood cannot be computed explicitly(hence not able to solve iteratively like GLM)

L =∫ ∏

i

fYi |u(yi |u)fU(u)du



GLMM in R

Outline



3 GLMM in R



GLMM in R

Numerical Integration

Numerical Integration

Method for numerically approximating the value of a definiteintegral ∫ b

af (x)dx

Numerical integration for one-dimensional integrals

Rectangle rule

Trapezoidal rule

Simpson’s rule



GLMM in R

Trapezoidal Rule

∫ b

af (x)dx =

b − a2n

(f (x0) + 2f (x1) + 2f (x2) · · · + 2f (xn−1) + f (xn))

where

xk = a + kb − a

nfor k = 0, 1, · · · , n

Figure 1: From Wikipedia http://en.wikipedia.org/wiki/Trapezoidal rule



GLMM in R

Evaluation of the Integrals

There are various methods to do this:

Approximating the integralGauss-Hermite QuadratureLaplace ApproximationAdaptive Gauss-Hermite Quadrature

Approximating the dataPenalized Quasi-Likelihood



GLMM in R

Gauss-Hermite Quadrature I

yij |u ∼ indep. fYij |U(yij |u)

fYij |U(yij |u) = exp([yijγij − b(γij)]/γ2 − c(yij , γ))

E[yij |u] = µij

g(µij) = x′ijB + ui , ui ∼ i.i.d. N(0, σ2u)

L =∫ ∏

i,j

fYij |Ui (yij |ui)fUi (ui)dui

=∏

i

∫ ∞−∞

e∑

j [yijγij−b(γij )]γ2−∑

j c(yij ,γ) e−u2i /(2σ

2u)√

2πσ2u

dui

=∏

i

∫ ∞−∞

hi(ui)e−u2

i /(2σ2u)√

2πσ2u

dui

⇓



GLMM in R

Gauss-Hermite Quadrature II

It be can seen that the likelihood is the product of one-dimensionalintegrals of the form: ∫ ∞

−∞

h(u)e−u2/(2σ2

u)√2πσ2

u

du

changing u to√

2σuυ gives:∫ ∞−∞

h(√

2σuυ)e−υ

2

√π

dυ ≡∫ ∞−∞

h∗(υ)e−υ2dυ

where h∗(·) ≡ h(√

2σu·)/√π



GLMM in R

Gauss-Hermite Quadrature III

Gauss-Hermite quadrature approximates the integral as aweighted sum: ∫ ∞

−∞

h∗(υ)e−υ2dυ �

d∑k=1

h∗(xk )wk

where wk is the weights, and the evaluation points, xk , aredesigned to provide an accurate approximation in the case whereh∗(·) is a polynomial.



GLMM in R

Constants for Gauss-Hermite Quadrature

Table 1: xk and wk for d = 3

xk wk

d = 3 -1.22474487 0.295408980 1.18163590

1.22474487 0.29540898

Formula for xk and wk

xk = ith zero of Hn(x)

wk =2n−1n!

√π

n2[Hn−1(xk )]2

where Hn(x) is the Hermite polynomial of degree n.Gota Morota Chapter 14: Computing


GLMM in R

Gauss-Hermite Quadrature IV

Example

Approximation of integral using 3-point quadrature:∫ ∞−∞

(1 + x2)e−x2dx � (1 + [−1.22474]2)(0.29541)

+ (1 + 02)(1.18164) + (1 + 1.224742)(0.29541)

= 2.65868



GLMM in R

Other Approximation Methods

Laplace Approximation1 find a peak xpeak of the given integrand f (x) by taking a

derivative2 apply a second-order Taylor series expansion around this peak3 calculate the variance σ = 1/f ′′(xpeak )4 approximate the f (x) ∼ N(xk , σ

2)

Adaptive Gauss-Hermite Quadrature1 apply centralization of the f (x) about zero or standardization

Penalize-Quasi Likelihood1 approximate the likelihood itself



GLMM in R

Outline



3 GLMM in R



GLMM in R

GLMM in R

Table 2: GLMM in R

Package Random Effect Computing MethodglmmPQL (1) MASS intercept/coef PQL 1

glmmML (2) glmmML intercept Laplace/AGQ 2

glmer (3) lme4 intercept/coef Laplace/AGQ 2

MCMCglmm (4) MCMCglmm intercept/coef MCMC1 Approximation to the likelihood2 Numerical Integration

Numerical Integration & approximation to the likelihood

Can be used in Likelihood-based methods (1-3) or bayesianapproach (4) for obtaining the unknown parameters.



GLMM in R

Simulation

0.5 1.0 1.5 2.0

AGQ

0.5 1.0 1.5 2.0

Laplace

0.2 0.4 0.6 0.8 1.0

PQL

pi =1

1 + exp(−(4 + xi + γi))γ ∼ N(0, 32)



GLMM in R

Summary

AGQ: produces greater accuracy in the evaluation of thelog-likelihood

Laplace: special case of AGQ

PQL: typically ends up in biased estimates

Difficulty dealing with more complicated models⇓

MCMC?


Technology

Chapter 14 Computing: McCulloch, CE, Searle SR and Neuhaus, JM 2008. Generalized, Linear and Mixed Models. John Wiley & Sons, New York. Second Edition