Upload
others
View
5
Download
0
Embed Size (px)
Citation preview
Random and mixed effects
ANOVA
STAT 526
Professor Olga Vitek
January 27, 2011
Reading: KNNL Ch. 25
Faraway Ch. 8
2
One-way random effects
ANOVA
2-1
Random Effects vs Fixed
Effects
• Consider factor with numerous levels
• Want to draw inference on population of levels
• Not concerned with any specific levels
• Example of difference (1=fixed, 2=random)
1. Compare reading ability of 10 2nd grade classesin NY
2. Compare variability among all 2nd grade classesin NY
1. Select a = 10 specific classes of interest.Randomly choose n students from each class-room.
2. Randomly choose a = 10 classes from largenumber of classes. Randomly choose n studentsfrom each classroom.
• Inference broader in random effects case
• Levels chosen randomly → inference on population
2-2
Example: KNNL p.1036
• Interested in studying the variability in therating of job applicants
– Variability among applicants
– Variability among personnel officers
• Y is the job applicant rating
• Factor: officer/interviewer (r = 5)
• Interviewers selected at random from pop-
ulation of personnel officers
• Twenty applicants randomly and equally
assigned (n = 4) to officers
2-3
Means Plot
X <- read.table("CH25TA01.txt", sep="", as.is=TRUE,header=FALSE)
dimnames(X)[[2]] <- c("rating", "officer", "replicate")
pdf("ch25t01.pdf", width=6)plot(X$officer, X$rating, xlab="Oficer is", ylab="Rating",
cex=2, cex.lab=2)lines(1:5, with(X, tapply(rating, officer, mean)))legend("top", lty=1, "mean", cex=2)dev.off()
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●●
1 2 3 4 5
5060
7080
90
Oficer id
Rat
ing
mean
2-4
Data for One-way Random
Effects Model
• Y is the response variable
• Factor with levels i = 1,2, ..., r
• Yij is the jth observation from cell i
• Consider j = 1,2, ..., n
2-5
Random Effects Model
• Cell means model
Yij = µi + εij,
µiiid∼ N(µ, σ2
µ), εijiid∼ N(0, σ2)
- µi and εij are independent
• Factor effects model
Yij = µ+ τi + εij,
τiiid∼ N(0, σ2
τ ), εijiid∼ N(0, σ2)
- τi and εij are independent
• Called Model II in KNNL
2-6
Equivalent Specification in
Terms of Conditional
Distributions
• Cell means model
µiiid∼ N(µ, σ2
µ)
Yij|µiind∼ N(µi, σ
2)
• Factor effects model
τiiid∼ N(0, σ2
µ)
Yij|τiind∼ N(µ+ τi, σ
2)
• Similarity to Bayesian methods:- parameters µi have a distribution
• Difference from Bayesian methods:-µ, σ2 and σ2
µ do not have a distribution
2-7
Implications of the Random
Effects Model
• There are TWO variance parameters
• Cell means are random variables, not pa-rameters
• Yij ∼ N(µ, σ2µ + σ2)
• The observations are not independent:Cov(Yij, Yi′j′)=0 but Cov(Yij, Yij′)=σ2
µ
• E.g. if r = 2 and n = 2,
Var
Y11
Y12
Y21
Y22
=
σ2µ + σ2 σ2 0 0σ2 σ2
µ + σ2 0 00 0 σ2
µ + σ2 σ2
0 0 σ2 σ2µ + σ2
2-8
ANOVA Table
Source DF SS EMS
Trt r − 1r∑
i=1n(Yi· − Y··)2 σ2 + nσ2
µ
Error nr − rr∑
i=1
n∑j=1
(Yij − Yi·)2 σ2
Total nr − 1
• H0 : σ2µ = 0 vs Ha : σ2
µ > 0
• F0 = MS(Trt)MSE =
H0∼ Fr−1,nr−r
• Conclusion pertains to entire population
• Reject H0:
– The expected ratings of the population ofofficers has a non-zero variance
– The company need to improve consistencybetween the interviewers
2-9
Inference About E{Yij}
• E{Yij} = µ·
• E{Yij} = Y··
• V ar{Y··} =σ2µr + σ2
rn =nσ2
µ+σ2
rn
• V ar{Y··} = MS(Trt)rn
• Testing: t = Y··−µH0·√
MS(Trt)/rn
H0∼ Studentr−1
• CI: Y·· ± t(1− α/2, r − 1)√MS(Trt)/rn
2-10
Intraclass Correlation
Coefficient
• percentage of total variation due to factor
σ2µ
σ2µ + σ2
=σ2µ
σ2Y
• correlation between two observations withthe same i (e.g. between two evaluationsof a same officer)
ρIC =Cov(Yij, Yik)√
Var(Yij)Var(Yik)=
σ2µ
σ2Y
• Small intraclass correlation= little variation among officers
• Large intraclass correlation= little variation among applicants
2-11
Confidence Interval forσ2µ
σ2µ+σ2
• MS(Trt) and MSE are independent r. v.
MS(Trt)nσ2
µ+σ2 / MSEσ2 ∼ F (r − 1, rn− r)
P{Fα/2, r−1, rn−r ≤ MS(Trt)MSE
· σ2
nσ2µ+σ2 ≤ F1−α/2, r−1, rn−r} = 1− α
• Solve forσ2µ
σ2µ+σ2:
L
L + 1≤
σ2µ
σ2 + σ2µ
≤U
U + 1
where
L =1
n
(MS(Trt)
MSE F1−α/2, r−1, rn−r− 1
)U =
1
n
(MS(Trt)
MSE Fα/2, r−1, rn−r− 1
)
• If low limit of the CI is negative, set to zero
2-12
Confidence Interval for σ2
• σ2 = MSE
• Known distribution of a function of MSE
r(n−1)MSEσ2 ∼ χ2(rn− r)
P{χ2α/2, rn−r ≤
r(n−1)MSEσ2 ≤ χ2
1−α/2, rn−r} = 1− α
• Solve for σ2:
r(n− 1)MSE
χ21−α/2, rn−r
≤ σ2 ≤r(n− 1)MSE
χ2α/2, rn−r
• Replace nr with nT in unbalanced designs
2-13
Point Estimate for σ2µ
• E{MS(Trt)} = σ2 + nσ2µ
• E{MSE} = σ2
• σ2µ = E{MS(Trt)}−E{MSE}
n
=
(1
n
)MS(Trt) +
(−
1
n
)MSE
• σ2µ = MS(Trt)−MSE
n
=
(1
n
)MS(Trt) +
(−
1
n
)MSE
• σ2µ is estimated by a linear combination of indepen-
dent MS
• Can adjust denominator in unbalanced experiments
• σ2µ can be negative
2-14
Confidence Interval for σ2µ
• σ2µ = c1E{MS1}+ . . .+ chE{MSh} = L
• σ2µ = c1MS1 + . . .+ chMSh = L
• dfLL
approx∼ χ2(df)
P{χ2α/2, df ≤
dfLL≤ χ2
1−α/2, df}approx
= 1− α
• Solve for L:
df L
χ21−α/2, df
≤ L ≤df L
χ2α/2, df
• Satterwaite approximation of df:
df =(c1MS1 + . . .+ chMSh)2
(c1MS1)2
df1+ . . .+ (chMSh)2
dfh
• Use the nearest integer for the df
2-15
Job Applicants Example:
ANOVA-based estimation
• Use aov in R; could also use lm or glm
# --------------fit fixed-effects ANOVA-------------X$officerF <- factor(X$officer)fit1 <- aov(rating ~ officerF, data=X)
> summary(fit1)Df Sum Sq Mean Sq F value Pr(>F)
officerF 4 1579.70 394.93 5.389 0.006803 **Residuals 15 1099.25 73.28---Signif. codes: 0 *** 0.001 ** 0.01 * 0.05 . 0.1 1
> coef(fit1)(Intercept) officerF2 officerF3 officerF4 officerF5
75.00 -4.50 -20.25 4.75 2.25
# ---Calculate estimate of sigma2_mu from the ANOVA table---> table(X$officerF)1 2 3 4 54 4 4 4 4
> sigma2MuHat <- (394.93 - 73.28)/4> sigma2MuHat[1] 80.4125
2-16
Pros and Cons of
ANOVA-based Inference
• Advantages
– Explicit formulae
– Clear insight into the mechanism
• Disadvantages
– σµ can be negative
– For unbalanced designs and in presence of multi-ple factors, ANOVA decomposition is not unique(depends on the order of the factors), and SSare not orthogonal. Therefore the inferencedoes not hold.
– Formulae become more complex with more fac-tors
• Alternative solution:
– Maximum likelihood or restricted maximum like-lihood
2-17
Maximum Likelihood
Estimation
• Define yi = [yi1, yi2, . . . , yini]′ and Vi the
ith block of the variance matrix V
• Assumptions: yi ∼MVN(µ1ni,Vi)
• Likelihood:
L =r∏
i=1
(2π)−ni/2|Vi|−1/2exp
[−
1
2(yi − µ1n1)
′V−1i (yi − µ1n1)
]
• Solutions maximizing log-likelihoodin balanced case:
– µ = Y··
– σ2 = MSE
– σ2µ = 1
n
[(1− 1
r
)MS(Trt)−MSE
]2-18
Problems With MLE
• σ2µ is biased towards smaller values
E{σ2µ} =
1
n
[(1−
1
r
)EMS(Trt)− EMSE
]=
(1−
1
r
)σ2µ − σ2/rn 6= σ2
µ
• σ2µ can still be negative
• Solutions in the restricted parameter space:
σ2(MLE)µ =
{σ2µ, if σ2
µ ≥ 00, otherwise
;
σ2(MLE) =
{σ2, if σ2
µ ≥ 0SSTrn , otherwise
2-19
Interviewers Example: ML
• Use lmer in R; could also use lme (older)
library(lme4)fit2 <- lmer(rating ~ 1 + (1|officerF), data=X, REML=FALSE)
• The data are grouped by ’officer’• The random effect is constant within each group
> summary(fit2)Linear mixed model fit by maximum likelihoodFormula: rating ~ 1 + (1 | officerF)
Data: XAIC BIC logLik deviance REMLdev
156.0 158.9 -74.98 150.0 145.3Random effects:Groups Name Variance Std.Dev.officerF (Intercept) 60.664 7.7887Residual 73.283 8.5606
Number of obs: 20, groups: officerF, 5
Fixed effects:Estimate Std. Error t value
(Intercept) 71.450 3.974 17.98
• Due to the bias, the ML-based estimate of varianceis smaller than the ANOVA-based estimate.
• This can result in too much optimism
2-20
Restricted/Residual ML
(REML)
• Apply ML to linear combinations of y K′y
– K selected such that the distribution of K′y doesnot involve µ (or, more generally, any fixed ef-fects)
– Estimates of variance components are invariantto fixed effects
– Implicitly takes into account the df for fixed ef-fects
• Simple example:
– Suppose Yi ∼ N(µ, σ2), i = 1, . . . , n
– Define Y· =n∑i=1
Yi/n, Syy =n∑i=1
(Yi − Y·)2
– σ2ML = Syy/n, (biased)
– σ2REML = Syy/(n− 1), (unbiased)
• Estimate fixed effects as a second step
2-21
REML in 1-Way Random
Effects ANOVA
• Factor effects model:
– The part of the likelihood that does not involvefixed effects is the part that does not involve µ
L(µ, σ2, σ2µ | Y) = L( µ | Y·· ) · L( σ2, σ2
µ | SS(Trt), SSE )
– Use the second product in the likelihood as thelikelihood for REML
• Solutions in balanced case:
– µ = Y··, σ2 = MSE, σ2µ = 1
n[MS(Trt)−MSE]
• Solutions maximizing log-likelihood in therestricted parameter space:
σ2(REML)µ =
{σ2µ, if σ2
µ ≥ 00, otherwise
; same as ANOVA-based
σ2(REML) =
{σ2, if σ2
µ ≥ 0SSTrn−1
, otherwise
2-22
Interviewers Example:
REML
fit3 <- lmer(rating ~ 1 + (1|officerF), data=X, REML=TRUE)
> summary(fit3)Linear mixed model fit by REMLFormula: rating ~ 1 + (1 | officerF)
Data: XAIC BIC logLik deviance REMLdev
151.2 154.2 -72.62 150.0 145.2
Random effects:Groups Name Variance Std.Dev.officerF (Intercept) 80.410 8.9672Residual 73.283 8.5606
Number of obs: 20, groups: officerF, 5
Fixed effects:Estimate Std. Error t value
(Intercept) 71.450 4.443 16.08
• In balanced designs, σ2µ is similar to the
ANOVA-based estimation
2-23
Predicting Random Effects
• Specification withconditional distributions:
Yij|τiind∼ N(µ+ τi, σ
2); τiiid∼ N(0, σ2
µ)
– Can be of interest to predict τi given the data
– Best predictor is E{τi | Y}
• In 1-way ANOVA with random effects:E{τi | Y} = E{τi | Yi·}
= E{τi}+ cov(τi, Yi·)[var(Yi·)
]−1 (Yi· − E{Yi·}
)= 0 + σ2
µ ·1
σ2µ + σ2/ni
·(Yi· − µ
)E{τi | y} =
σ2µ
σ2µ + σ2/ni
·(Yi· − µ
)• In balanced experiments:E{τi | Y} =
σ2µ
σ2µ + σ2/n
·(Yi· − Y··
)Random effects
τi =(Yi· − Y··
)Fixed effects
- shrinkage of the estimation of random effects ascompared to fixed effects
2-24
Extracting Parameters and
Testing in R
• Extracting values of parameters
fixef(fit3) # predicted fixed effectsvcov(fit3) # variance-covariance of fixed effectsranef(fit3) # predicted random effectsfitted(fit3) # yHat = fixedEffectsHat + randomEffectsHat
• Likelihood Ratio Test H0 : σ2µ = 0 vs Ha : σ2
µ 6= 0
• REML-based likelihoods are not comparable for mod-els that differ in random effects
• Use ML-based estimation
• The test is approximate since based on biased ML
fit4 <- lm(rating ~ 1, data=X, REML=FALSE)
> pchisq( as.numeric(2*( logLik(fit2) - logLik(fit4))) ,1, lower=FALSE)
[1] 0.02919295
• Conclusion: at α = 0.05, there is a significantvariation between the officers
• Weaker evidence than with ANOVA-based F test
2-25
Two-way random effects
ANOVA
(Model II in KNNL)
2-26
Example: KNNL 25.15
• Interested in the fuel efficiency (mpg)
• Response: mpg
– Yijk is the kth observed mpg from driver i andcar j, with k = 1,2, ..., nij
• Two random factors
– Factor A: Driver, levels i = 1,2, ..., a
– Factor B: Car (same model), levels j = 1,2, ..., b
– Each driver drove each car twice over same 40mile course
- balanced design
• Scientific question:
– How much of the overall variability is due todriver and/or car?
2-27
Means PlotX <- read.table("CH25PR15.txt", sep="", as.is=TRUE,
header=FALSE)dimnames(X)[[2]] <- c("mpg", "driver", "car", "replicate")
pdf("ch25pr15.pdf", width=6)plot(X$driver, X$mpg, xlab="Driver id", ylab="Miles/gallon")
m <- with(X, tapply(mpg, list(car, driver), mean))for (i in 1:5) { lines(1:4, m[i,], col=i) }
legend("topright", lty=1, "car", cex=1.5)dev.off()
●●
●
●
●●
●●
●●
●
●
●●
●●
●●
●●
●
●
●●
●●
●●
●●
●●
●●
●
●
●
●●●
1.0 1.5 2.0 2.5 3.0 3.5 4.0
2628
3032
3436
Driver id
Mile
s/ga
llon
car
2-28
Random Factor Effects
Model
• In mathematical notation:
Yijk = µ+ αi + βj + (αβ)ij + εijk
i = 1, . . . , a, j = 1, . . . , b, k = 1, . . . , n
– µ - grand mean (or another reference)
– αi: random deviation of the ith level of factor
A from the reference; αiiid∼ N(0, σ2
α)
– βj: random deviation of the jth level of factor
B from the reference; βjiid∼ N(0, σ2
β)
– (αβ)ij: the joint random effect of ith level of A
and jth level of B; (αβ)ijiid∼ N(0, σ2
αβ)
– εijk: random error; εijkiid∼ N(0, σ2)
– All random terms are independent
2-29
Covariance Structure
• There are for variance parameters
– Covariances:
Cov(Yijk, Yijk) = σ2 + σ2α + σ2
β + σ2αβ
Cov(Yijk, Yijk′) = σ2α + σ2
β + σ2αβ
Cov(Yijk, Yij′k) = σ2α
Cov(Yijk, Yi′jk) = σ2β
Cov(Yijk, Yi′j′k) = 0
• Yijk are not independent
• Questions
– Percentage of total variation due to each factor
– Percentage of cell means variation(i.e., ignoring error variance)
– Pairwise comparisons between levels of factorsare not appropriate
2-30
ANOVA Table
Source DF EMS
A a− 1 σ2 + nσ2αβ + bnσ2
α
B b− 1 σ2 + nσ2αβ + anσ2
β
AB (a− 1)(b− 1) σ2 + nσ2αβ
Error ab(n− 1) σ2
Total ab− 1
• Parameter estimates using mean squares
σ2 = MSE
σ2αβ = (MSAB−MSE)/n
σ2β = (MSB−MSAB)/an
σ2α = (MSA−MSAB)/bn
• Estimates can be negative
• Same procedures for CI, and same adjust-ments for df, as in the one-way randomeffects ANOVA
2-31
ANOVA-Based Hypothesis
Tests
Three tests of variance, no hierarchy
• H0AB: σ2αβ = 0 vs H1AB: σ2
αβ > 0
F = MS(AB)MSE
H0∼ F(a−1)(b−1), ab(n−1)
• H0A: σ2α = 0 vs H1A: σ2
α > 0
F = MS(A)MS(AB)
H0∼ F(a−1), (a−1)(b−1)
• H0B: σ2β = 0 vs H1B: σ2
β > 0
F = MS(B)MS(AB)
H0∼ F(b−1), (a−1)(b−1)
2-32
Car Example:
ANOVA-Based Approach
#--------Fit the fixed-effects 2-way ANOVA---------X$driverF <- factor(X$driver)X$carF <- factor(X$car)fit1 <- aov(mpg ~ driverF*carF, data=X)
#---------------Summary of fit--------------------summary(fit1)
Df Sum Sq Mean Sq F value Pr(>F)driverF 3 280.285 93.428 531.60 < 2.2e-16 ***carF 4 94.714 23.678 134.73 3.664e-14 ***driverF:carF 12 2.447 0.204 1.16 0.3715Residuals 20 3.515 0.176
#---------Estimate variance components-------------> sigma2ABHat <- (0.204 - 0.176)/2[1] 0.014
> sigma2DriverHat <- (93.428 - 0.204)/(2*5)[1] 9.3224
> sigma2CarHat <- (23.678 - 0.204)/(2*4)[1] 2.93425
• Only the F test for the interaction is used directly
2-33
Car Example: ML-Based
Approach
#--------------REML estimation------------------library(lme4)fit2 <- lmer(mpg ~ 1 + (1|driverF)+(1|carF)+(1|driverF:carF),
data=X, REML=TRUE)
#---------Summary for random effects--------------# Similar to ANOVA for variance of main effectsGroups Name Variance Std.Dev.driverF:carF (Intercept) 1.6379e-14 1.2798e-07carF (Intercept) 2.9365e+00 1.7136e+00driverF (Intercept) 9.3242e+00 3.0536e+00Residual 1.8630e-01 4.3162e-01
#----------------ML estimation------------------fit3 <- lmer(mpg ~ 1 + (1|driverF)+(1|carF)+(1|driverF:carF),
data=X, REML=FALSE)
#---------Summary for random effects--------------# Similar to ANOVA for variance of residuals and interactionGroups Name Variance Std.Dev.driverF:carF (Intercept) 0.014080 0.11866carF (Intercept) 2.772125 1.66497driverF (Intercept) 7.413777 2.72283Residual 0.175750 0.41923
2-34
Car Example: REML in SAS
proc mixed data=a1;class car driver;model mpg=;random car driver car*driver / vcorr;
run;
Covariance Parameter EstimatesCov Parm Estimate Alpha Lower Uppercar 2.9343 0.05 1.0464 24.9038driver 9.3224 0.05 2.9864 130.79car*driver 0.01406 0.05 0.001345 3.592E17Residual 0.1757 0.05 0.1029 0.3665
• SAS implementation produces REML estimates thatare identical to ANOVA
• This is due to differences in the implementation ofREML between SAS and R
2-35
Car Example: ML
Additive model
#--------------REML estimation------------------library(lme4)fit4 <- lmer(mpg ~ 1 + (1|driverF)+(1|carF), data=X,
REML=TRUE)
#---------Summary for random effects--------------# Same as REML with interactionRandom effects:Groups Name Variance Std.Dev.carF (Intercept) 2.93651 1.71362driverF (Intercept) 9.32419 3.05355Residual 0.18630 0.43162
#----------------ML estimation------------------fit5 <- lmer(mpg ~ 1 + (1|driverF)+(1|carF), data=X,
REML=FALSE)
#---------Summary for random effects--------------# Smaller estimates of variationGroups Name Variance Std.Dev.carF (Intercept) 2.77436 1.66564driverF (Intercept) 7.41579 2.72319Residual 0.18631 0.43163
2-36
Two-way mixed effects
ANOVA
(Model III in KNNL)
2-37
Model formulation
Yijk = µ+ αi + βj + (αβ)ij + εijk
i = 1, . . . , a, j = 1, . . . , b, k = 1, . . . , n
• µ is the overall mean
• αi is the fixed effect of the i-th level of factor A,∑i αi = 0
• βj is the random effect of the j-th level of factor
B, βjiid∼ N (0, σ2
β)
• (αβ)ij is the joint random effect of the i-th level
of factor A and the j-th level of factor B, (αβ)ijiid∼
N (0, σ2αβ)
• εijk is the random error εijkiid∼ N (0, σ2)
• All random terms are independent
2-38
ANOVA Table
Source Type DF EMS
A F a− 1 σ2 + nσ2αβ + bn
a−1
a∑i=1
α2i
B R b− 1 σ2 + nσ2αβ + anσ2
β
AB R (a− 1)(b− 1) σ2 + nσ2αβ
Error R ab(n− 1) σ2
Total ab− 1
• Parameter estimates using mean squares
σ2 = MSE
σ2αβ = (MSAB−MSE)/n
σ2β = (MSB−MSAB)/an
• Estimates can be negative
• Same procedures for CI, and same adjust-ments for df, as in the one-way randomeffects ANOVA
2-39
ANOVA-Based Hypothesis
Tests
Two tests of variance, one test of fixed effects.In balanced designs:
• H0AB: σ2αβ = 0 vs H1AB: σ2
αβ > 0
F = MS(AB)MSE
H0∼ F(a−1)(b−1), ab(n−1)
• H0B: σ2β = 0 vs H1B: σ2
β > 0
F = MS(B)MS(AB)
H0∼ F(b−1), (a−1)(b−1)
• H0A: αi = 0, all i vs H1A: αi 6= 0, some i
F = MS(A)MS(AB)
H0∼ F(a−1), (a−1)(b−1)
2-40
Balanced Designs: Variance of
Linear Combinations of Means
• Assume balanced design,i.e. nij = n for all i and j
• Suppose we are interested in combinationsof expected values
L = c1(µ+ α1) + . . .+ ca(µ+ αa)
• The inbiased estimator L is
L = c1Y1··+ . . .+ caYa··
• And it’s variance is
V ar{L} =
(nσ2
αβ + σ2
nb
)(a∑
i=1
c2i
)+
(σ2β
b
)(a∑
i=1
ci
)2
2-41
Degrees of Freedom for Linear
Combination of Means
• Suppose the estimate of V ar{L} is
V ar{L} = p1MS1 + . . .+ pkMSk
where
– p1, . . . , pk are nonnegative constraints and
– MS1, . . . ,MSk are the mean squares inthe ANOVA table with degrees of free-dom ν1, . . . , νk
• Then the degrees of freedom associatedwith V ar{L}
ν =(p1MS1 + . . .+ pkMSk)2
p21MS2
1ν1
+ . . .+ p2kMS2
kνk
2-42
Careful: Restricted Mixed
Model Formulation in KNNL
Yijk = µ+ αi + βj + (αβ)ij + εijk
i = 1, . . . , a, j = 1, . . . , b, k = 1, . . . , n
• µ is the overall mean
• αi is the fixed effect of the i-th level of A,∑i αi = 0
• βj is the random effect of the j-th level of B,
βjiid∼ N (0, σ2
β)
• (αβ)ij is the joint random effect of the i-th level ofA and the j-th level of B
(αβ)ijiid∼ N (0, a−1
aσ2αβ),
a∑i=1
(αβ)ij = 0
• εijk is the random error εijkiid∼ N (0, σ2)
• The random terms are not independent:
Cov( (αβ)ij, (αβ)i′j ) = −1aσ2αβ
2-43
Restricted Mixed Model
Formulation: ANOVA Table
Source Type DF EMS
A F a− 1 σ2 + nσ2αβ + bn
a−1
a∑i=1
α2i
B R b− 1 σ2 + anσ2β
AB R (a− 1)(b− 1) σ2 + nσ2αβ
Error R ab(n− 1) σ2
Total ab− 1
• Different EMS for random effect
• Different test statistic to test the randomeffect
– H0B: σ2β = 0 vs H1B: σ2
β > 0
F = MS(B)MSE
H0∼ Fb−1, ab(n−1)
• We will use the non-restricted formulation
2-44
Linear Mixed Models:
General Formulation
• In notation of Faraway:
y = Xβ + Zγ + ε, where
γ ∼ N (0, σ2D) and
ε ∼ N (0, σ2I)
• Or, equivalently:
y|γ ∼ N (Xβ + Zγ, σ2I), where
γ ∼ N (0, σ2D)
• This implies the marginal distribution of y:
y ∼ N (Xβ, σ2(I + ZDZ′))
• Then the likelihood of the data is1
2πn/2|σ2V|1/2e− 1
2σ2(y−Xβ)′V(y−Xβ)
where V = I + ZDZ′
2-45
REML
• Using the general formulation
y ∼ N (Xβ, σ2V), define matrix K where
K′X = 0, therefore
K′y = N (0,K′VK)
• Find estimates of variance parameters on
the transformed scale K′y
• Find estimates of fixed effects assuming
that variance is known
2-46
Likelihood Ratio Test
• Compare a full and a reduced model: test
statistic
2{logL(Full|y)− logL(reduced|y)} H0∼ χ2df
where df is the difference in the number of param-
eters in the model
• Testing fixed effects in a mixed effects model
H0 : αi = 0 for all i, Ha αi 6= 0 for some i
– REML are not comparable (different scale)
– Use ML
– Approximate p-values (too small)
– Alternative: use bootstrap
– Alternative(1) estimate variances from the full model(2) use weighted fixed effects model
as if variances are known
2-47
Likelihood Ratio Test
• Testing random effects in a mixed effects
model
H0 : σ2 = 0, Ha σ2 6= 0 for some variance
– The value under H0 is on the limit of the pa-rameter space
– Approximate test (p-values too large)
– Alternative: use bootstrap
2-48
Predicting Random Effects
• Estimator: expected value of the random
effect conditional on the data
E{β} = DZ′V−1(y −Xβ)
• Basic property:
var(β) = var(E{β|y}) + E{var(β|y)}= var(predictor) + positive value
• Therefore
var(E{β|y} ≤ var(β)
−→ E{β} is a shrinkage estimator
2-49