STAT3914 { Applied Statistics · STAT3914 { Applied Statistics Lecturer Dr. John T. Ormerod School...

Semester 1, 2012 (Last adjustments: September 5, 2012)

Lecture Notes

STAT3914 – Applied Statistics

Lecturer

Dr. John T. Ormerod

School of Mathematics & Statistics F07

University of Sydney

(w) 02 9351 5883

(e) john.ormerod (at) sydney.edu.au

STAT3914 – Outline

2 Multivariate Normal Distribution

2 Point Estimates and Confidence Intervals for MVN

2 Wishart Distriubtion

2 Derivation of Hotelling’s T2

2 The Expecation Maximization Algorithm

2 Missing Data Analysis

STAT3914: Lecture 3 2

The standard univariate normal distribution

2 A random variable (RV) Z has a standard normal distribution N(0, 1), if Z has

density

ϕ(z) =1√2π

e−12z

2, −∞ < z <∞.

2 Z has moment generating function

E(etZ) =

∫ ∞−∞

etz · (2π)−12e−

∫ ∞−∞

(2π)−12 e−

12(z−t)

2 It follows that E[Z] = 0 and Var[Z] = 1.

The standard multivariate normal distribution

2 Let U = (U1, U2, . . . , Up)T be a p-vector of NID(0, 1) RVs.

2 The joint density of U is given byp∏i=1

1√2πe−u

2i /2 = (2π)−p/2 exp

2 Clearly,

E[U] = 0

where 0 is the 0 vector in Rp.

2 Similarly,

Cov[U] = E[UUT ] = I

where I = Ip is the p× p identity matrix.

2 We say U has a standard multivariate normal distribution which we denote

U ∼ Np(0, I).

Non-standard univariate normal distribution

2 Given a RV Z ∼ N(0, 1) we can generate a non-standard normal with mean

µ ∈ R and standard deviation σ > 0 by defining X = µ + σZ.

2 The density of X can be readily recovered by the formula for the density of

transformed RVs.

2 Let ψ(z) = µ + σz, then X = ψ(Z) and ψ−1(x) = (x− µ)/σ so

fX(x) = fZ(ψ−1(x))

∣∣∣∣ ddxψ−1(x)

∣∣∣∣= fZ

(x− µσ

) ∣∣∣∣1σ∣∣∣∣

2πσ2exp

−(x− µ)2

Non-standard univariate normal distribution

2 The MGF of X is given by

E[etX ] = etµE[etσZ]

= exptµ + 1

2σ2t2.

2 An analogous transformation creates the general multivariate normal random

vector, i.e. if z ∼ N(0, 1) then

x = µ + σz ∼ N(µ, σ2).

Non-standard multivariate normal distribution

2 Let U ∼ Np(0, I) and define

X = ψ(U) ≡ µ + AU

where µ ∈ Rp, and A is a p× p non-singular matrix.

2 Clearly,

E[X] = µ

and its covariance is given by

Cov[X] = E[(X− µ) (X− µ)T

[AUUTAT

≡ Σ.

Non-standard multivariate normal distribution

Suppose AAT = Σ then:

2 A is sort of a “square-root” of Σ (if Σ is symmetric then A = Σ12) and

|Σ| = |AAT | = |A| · |AT | = |A|2 > 0.

2 Claim: If U ∼ Np(0, I) and X = ψ(U) ≡ µ+ AU then X has a non-singular

multivariate normal distribution which we will denote

X ∼ Np(µ,Σ)

and X has density

fX(X) = (2π)−p/2|Σ|−12 exp

2(x− µ)TΣ−1(x− µ).

2 Proof: Similarly to the univariate case with ψ−1(X) = A−1(X− µ)

fX(X) = fU

(ψ−1(X)

) ∣∣Jψ−1∣∣= (2π)−p/2 |Jψ|−1 exp

[A−1(X− µ)

]T [A−1(X− µ)

]STAT3914: Lecture 3 8

2 The Jacobian of a linear transformation is the determinant of its matrix so

|Jψ| =

∣∣∣∣∣∣∣∂ψ1∂u1· · · ∂ψ1

∂up... . . . ...

∂ψp∂u1· · · ∂ψp

∣∣∣∣∣∣∣ = |A| = |Σ|12 .

2 The exponent simplifies as follows[A−1(X− µ)

]T [A−1(X− µ)

]= (X− µ)T (A−1)T (A−1)(X− µ)

= (X− µ)TΣ−1(X− µ)

as (A−1)T (A−1) = (AAT )−1 = Σ−1.

2 Thus, X has density

fX(X) = (2π)−p/2|Σ|−12 exp

2(x− µ)TΣ−1(x− µ).

The MGF of the multivariate normal distribution

2 Claim: If X ∼ Np(µ,Σ) then the moment generating function (MGF) of X is

given by:

MX(s) = expsTµ + 1

2sTΣs

2 Proof: For s ∈ Rp we have

MX(s) = E[esTX]

= E[exp(sTµ + sTAU)]

= exp(sTµ) · E[eaTU], where aT = sTA

= exp(sTµ) · exp12a

since aTU ∼ N(0, aTa)

= expsTµ + 1

2sTΣs

2 This expression for the MGF is well defined even when Σ is singular. While it

would be awkward to define the multivariate normal distribution using an MGF

it does suggest the following definition...

The (general) multivariate normal distribution

2 Definition. A p-dimensional random vector X has a multivariate normal dis-

tribution if for every a ∈ Rp the linear combination aTX is univariate normal.

2 Claim. X is a p-dimensional multivariate normal random vector if and only if

its MGF can be expressed as

MX(s) = expsTµ + 1

2sTΣs

2 Aside: if MX is as above then by differentiating the MGF it follows that EX = µ

and that Cov(X) = Σ.

The (general) multivariate normal distribution

2 Proof:. Suppose the above expression for the MGF holds. Then, the MGF of

Y = aTX is given by

MY (t) = E[etaTX]

= MX(ta)

taTµ +

2t2aTΣa

so that

aTX ∼ N(aTµ, aT Σa)

by the uniqueness theorem for MGFs.

Conversely, if Ys = sTX is univariate normal for all s ∈ Rp then

E[eYs] = expE[Ys] + 1

2Var[Ys].

However, E[Ys] = sTE[X] = sTµ and

Var[Ys] = E[sTX− sTµ

] (sTX− sTµ

)T= sT

[E (X− µ) (X− µ)T

= sTCov(X)s

Therefore, with Σ = Cov(X)

MX(s) = E[eYs] = expsTµ + 1

2sTΣs

Marginal Distributions

2 If X ∼ Np(µ,Σ) then for all a ∈ Rp,

aTX ∼ N(aTµ, aTΣa).

2 So by choosing ei = (0, · · · , 0, 1, 0, · · · , 0) with 1 in the ith position we have

Xi ∼ N(µi,Σii).

2 Thus all marginal distributions are normal.

2 The converse is not true (see exercises) .

2 Moreover, if X1 = (X1, . . . , Xr)T consists of the first r coordinates of X, then

X1 is an r-dimensional normal RV (why?)

X1 ∼ Nr(µ1,Σ11),

where [µ1]i = [µ]i and [Σ11]i,j = [Σ]i,j for i, j = 1, . . . , r.

2 Claim. If Σ is diagonal then Xi are independent.

2 Proof.

Block/partitioned matrices

2 Suppose X is a p-dimensional random vector.

2 Let X1 = (X1, . . . , Xr)T be the first r coordinates of X and let X2 = (Xr+1, . . . , Xp)

be its last q = p− r coordinates so that

2 Let µi = E[Xi] (i = 1, 2) then

µ ≡ E[X] =

)2 Similarly, with

[Σ]ij ≡ Cov(Xi,Xj) ≡ E[XiXTj ]− E[Xi]E[Xj]

(so that Σ11 is r × r and Σ12 is r × q, etc.)

Var(X) ≡ Σ =

(Σ11 Σ12

Σ21 Σ22

Degenerate multivariate normal distribution

2 Suppose X is a multivariate normal with mean µ and a singular covariance

matrix Σ with rank r < p.

2 As Σ is symmetric it has (p − r) eigenvalues equal to 0 and there exists an

orthogonal matrix P such that

PTΣP =

)≡ D

D = diag(λ1, · · · , λr), with λ1 ≥ λ2 ≥ · · · ≥ λr > 0.

and the columns of P are the eigenvectors of Σ

2 Let Y = PTX and denote by Y1 = (Y1, Y2, . . . , Yr)T and Y2 = (Yr+1, . . . , Yp)

and similarly for s = (s1, s2) and for γ = PTµ = (γ1,γ2).

2 Then Y has MGF

E[esT1 Y1+sT2 Y2] = E[esTY]

= E[esTPTX]

= expsTPTµ + 1

2sTPTΣPs

sTγ + 1

sT1 γ1 +

2sT1 Ds1

· exp

sT2 γ2

2 The RHS is a product of two MGFs, one of a distribution that is the constant

≡ γ2, and the other is Nr(γ1,D).

2 By the uniqueness of the MGF, Y2 ≡ γ2 and Y1 ∼ Nr(γ1,D).

2 Note that the Y1, . . . , Yr are independent. Does this ring a bell? Principal

Component Analysis. . .

2 It also follows that if X ∼ N(µ,Σ) with Σ of rank r then there exists an

invertible linear transformation G such that X = GW + µ where

Wr+1 = . . . = Wp = 0 and (W1, · · · ,Wr)T ∼ Nr(0, I).

see exercises.

Block independence

2 Theorem 1. X1 and X2 are independent if and only if Σ12 = 0.

2 Proof. If X1 and X2 are independent then X1 and X2 are element-wise uncor-

related and so Σ12 = 0.

Conversely, if Σ12 = 0 then Σ21 = 0 and if s = (s1, s2)T (with s1 ∈ Rr) we

sTΣs = sT1 Σ11s1 + sT2 Σ22s2.

Therefore, X has MGF

MX(s) = expsTµ + 1

2sTΣs

sT1µ1 + 1

2sT1 Σ11s1

· exp

sT2µ2 + 1

2sT2 Σ22s2

and so (why?) X1 and X2 are independent with

X1 ∼ Nr(µ1,Σ11) and X2 ∼ Ns(µ2,Σ22).

Detour into the linear algebra of block matrices

(A11 A12

A21 A22

)be an invertible p×p matrix (where A11 is r×r and A22 is s×s with r+s = p).

2 It is common to denote the inverse of A by

A−1 =

(A11 A12

A21 A22

)(where A11 is r × r and A22 is s× s).

2 By the definition of a matrix inverse we have the relations(A11 A12

A21 A22

)(A11 A12

A21 A22

2 In particular,A21A11 + A22A21 = 0

A21A12 + A22A22 = Is.

2 Multiply the first equation from the right by A−111 A12 and subtract from the

second equation to find that

A22(A22 −A21A−111 A12) = Is =⇒ A22 = (A22 −A21A

−111 A12)

2 The same kind of algebra yields (exercise)

A12 = −A−111 A12A22

A21 = −A22A21A−111

−A12A21 = A11A11 − Ir

A22 = (A22 −A21A−111 A12)

A11 = (A11 −A12A−122 A21)

Linear Predictor (Estimator)

2 X = (X1, X2,XT3 )T is a RV.

2 A linear predictor (estimator) of X1 given X3 is of the form bTX3, where

b ∈ Rp−2 (ignore X2 for now).

2 We seek the best such linear predictor:

bo ≡ argminb

E[(X1 − bTX3)

2 Assume µ = 0 (the results hold for any µ) and let

f (b) ≡ E[(X1 − bTX3)2]

= E[(X1 − bTX3)(X1 − bTX3)

= Σ11 − bTΣ31 −Σ13b + bTΣ33b

= Σ11 − 2bTΣ31 + bTΣ33b

2 Use calculus or algebra to minimize f .

2 We want to minimize (with respect to b)

f (b) = Σ11 − 2bTΣ31 + bTΣ33b.

2 Clearly, ∇b(−2bTΣ31) = −2Σ31 ∈ Rp−2.

2 Since bTAb =∑

i,j aijbibj it follows that for a symmetric A that

∇b(bTAb) = 2Ab.

2 Therefore,

∇f (b) = −2Σ31 + 2Σ33b.

2 Thus, the unique stationary point is attained at

b0 = Σ−133 Σ31.

2 Which is a minimum (as Σ33 is positive definite).

2 Algebra: complete the quadratic form

f (b) = Σ11 − bTΣ31 −Σ13b + bTΣ33b

= Σ11 −Σ13Σ−133 Σ31 + (b−Σ−133 Σ31)

TΣ33(b−Σ−133 Σ31)

≥ Σ11 −Σ131Σ−133 Σ31

with equality if and only if b = Σ−133 Σ31.

2 Thus, the best linear estimator (predictor) of X1 given X3 is

PX3(X1) = Σ13Σ−133 X3

Note that this is a normal RV.

Similarly, PX3(X2) = Σ23Σ−133 X3.

2 This is also known as the projection of X1 (X2) on the subspace spanned by

Conditional Distributions

)∼ Np(µ,Σ)

where Σ is non-singular, X1 = (X1, . . . , Xr)T and X2 = (Xr+1, . . . , Xp)

X1 ∼ Nr(µ1,Σ11)

X2 ∼ Ns(µ2,Σ22)

where s = p− r.

2 What is the conditional distribution of X2 given that X1 = x1?

2 Recall that for the bivariate normal (X1, X2) we defined

fX2|X1(x2|x1) =

fX1,X2(x1, x2)

fX1(x1)

2 We saw that for Xi ∼ N(0, 1), Cov(X1, X2) = ρ and

X2 |X1 = x1 ∼ N(ρx1, 1− ρ2)

2 We can pursue the obvious generalization here (still assuming µ = 0):

log fX2|X1(X2|X1)

= logfX(X)

fX1(X1)

= C − 12

1 ,XT2 )

(Σ11 Σ12

Σ21 Σ22

)−XT

1 Σ−111 X1

]= C − 1

(Σ11 −Σ−111

)X1 + XT

2 Σ21X1

+XT1 Σ12X2 + XT

2 Σ22X2

where C = C(Σ, r) is a constant and

Σ−1 =

(Σ11 Σ12

Σ21 Σ22

From the previous slide

log fX2|X1(X2|X1)

= C − 12

(Σ11 −Σ−111

)X1 + XT

2 Σ21X1 + XT1 Σ12X2 + XT

2 Σ22X2

2 Using the block inverse identities we obtain

Σ11 −Σ−111 = (Σ11Σ11 − I)Σ−111 = −Σ12Σ21Σ−111 = Σ−111 Σ12Σ

22Σ21Σ−111

Σ21 = −Σ22Σ21Σ−111

Σ12 = −Σ−111 Σ12Σ22

into the above expression we have

(Σ11 −Σ−111

)X1 + XT

2 Σ21X1 + XT1 Σ12X2 + XT

2 Σ22X2

= XT1 Σ−111 Σ12Σ

22Σ21Σ−111 X1 −XT

2 Σ22Σ21Σ−111 X1

−XT1 Σ−111 Σ12Σ

22X2 + XT2 Σ22X2

=(X2 −Σ21Σ

−111 X1

)TΣ22

(X2 −Σ21Σ

−111 X1

)2 Therefore,

log fX2|X1(X2|X1) = C − 1

[(X2 −Σ21Σ

−111 X1

)TΣ22

(X2 −Σ21Σ

−111 X1

2 As a function of X2 this is a density of a multivariate normal with mean

Σ21Σ−111 X1 and covariance matrix(

Σ22)−1

= Σ22 −Σ21Σ−111 Σ12.

2 We essentially proved:

Theorem 2. If X ∼ Np(µ,Σ) then the conditional distribution of X2 given

X1 = X is

N(µ2 + Σ21Σ

−111 (X− µ1),Σ22 −Σ21Σ

−111 Σ12

2 Rather than routinely generalizing the result to include the case µ 6= 0 we give

a different proof for the general case.

2 Recall that if (X1, X2) is bivariate normal with Xi ∼ N(0, 1) and correlation ρ,

then we can de-correlate X2 from X1 as follows:

Let Y2 = X2 − ρX1, then

Cov(Y2, X1) = Cov(X2, X1)− ρCov(X1, X1) = 0.

2 So X2 = Y2 + ρX1 where Y2 ∼ N(0, 1− ρ2) is independent of X1 (?)

2 Therefore, conditioning on X1 = x1,

X2 ∼ N(ρx1, 1− ρ2).

2 Next we generalize this to the multivariate normal case.

2 Let Q = Σ21Σ−111 and let

Y2 = X2 −QX1.

Note that QX1 is the best linear predictor of X2 given X1

2 Y2 and X1 are uncorrelated since

Cov(Y2,X1) = Cov(X2,X1)− Cov(QX1,X1)

= Σ21 −QΣ11

2 Then (X1

)is a multivariate normal RV (why?)

2 It follows that Y2 and X1 are independent and

Y2 ∼ N(µ2 −Qµ1,Σ22 −Σ21Σ−111 Σ12),

Y2 = X2 −QX1 and Q = Σ21Σ−111

Cov(Y2) = Cov(X2 −QX1,X2 −QX1)

= Σ22 −Σ21QT −QΣ12 + QΣ11Q

= Σ22 −Σ21Σ−111 Σ12.

2 Finally, X2 = Y2 + QX1 so the conditional distribution of X2 given X1 = X is

N(µ2 −Qµ1 + QX,Σ22 −Σ21Σ

−111 Σ12

)2 Notation: Σ2·1 = Σ22 −Σ21Σ

−111 Σ12 =

(Σ22)−1

Partial Correlation

2 The best linear estimator of X1 given X3 is

PX3(X1) = Σ13Σ−133 X3.

2 The partial correlation between X1 and X2 given X3 is

ρ12·34...p ≡ Cor[X1 − PX3(X1), X2 − PX3(X2)]

2 Theorem: Let

Σ−1 =

Σ11 Σ12 Σ13

Σ21 Σ22 Σ23

Σ31 Σ32 Σ33

and let d = Σ11Σ22 − Σ12Σ21 then

ρ12·34...p = − Σ12

√Σ11Σ22

2 Proof:Covariance is bilinear so

Cov[X1 − PX3(X1), X2 − PX3(X2)]

= Cov(X1, X2)− Cov(X1, PX3(X2))

−Cov(PX3(X1), X2) + Cov(PX3(X1), PX3(X2))

= Σ12 − Cov(X1,X3)Σ−133 ΣT

−Σ13Σ−133 Cov(X3, X2) + Σ13Σ

−133 Cov(X3,X3)Σ

−133 ΣT

= Σ12 −Σ13Σ−133 Σ32.

2 Similarly,

Cov [X1 − PX3(X1)]

= Σ11 −Σ13Σ−133 ΣT

13 −Σ13Σ−133 Σ31 + Σ13Σ

−133 Σ33Σ

−133 ΣT

= Σ11 −Σ13Σ−133 Σ31.

2 Hence,

ρ12·34...p =Σ12 −Σ13Σ

−133 Σ32√

(Σ11 −Σ13Σ−133 Σ31)(Σ22 −Σ23Σ

−133 Σ32)

2 By our block inverse identities,

(Σ22 −Σ12

−Σ21 Σ11

(Σ11 Σ12

Σ21 Σ22

)−1=

(Σ11 Σ12

Σ21 Σ22

)Σ−133

(Σ31 Σ32

2 Therefore,

ρ12·34...p =Σ12 −Σ13Σ

−133 Σ32√

(Σ11 −Σ13Σ−133 Σ31)(Σ22 −Σ23Σ

−133 Σ32)

= − Σ12/d√(Σ22/d)(Σ11/d)

= − Σ12

√Σ11Σ22

2 Our definition of partial correlation and analysis do not require X to be multi-

variate normal.

2 However, if X is multivariate normal then

ρ12·3...p = CorF (X1, X2),

where the correlation is with respect to the distribution F = F(X1,X2)|X3, the

joint conditional distribution of X1 and X2 given X3.

Maximum Likelihood Estimates for µ and Σ

2 X1, . . . ,Xn are independent Np(µ0,Σ0) random vectors (here µ0 and Σ0 are

the true mean and covariance respectively).

2 Theorem: The maximum likelihood estimators for µ and Σ are:

µ = X and Σ = 1nS

n∑i=1

(Xi −X)(Xi −X)T .

2 Proof: Assuming |Σ| > 0 the likelihood of (µ,Σ) is

L(µ,Σ) =

n∏i=1

(2π)−p/2|Σ|−1/2 exp−1

2(Xi − µ)TΣ−1(Xi − µ)

2 Recall: if A is m× n and B is n×m then tr(AB) = tr(BA).

Hence,

(Xi − µ)TΣ−1(Xi − µ) = tr[(Xi − µ)TΣ−1(Xi − µ)

[Σ−1(Xi − µ)(Xi − µ)T

]Thus,

L(µ,Σ) = (2π)−np/2|Σ|−n/2 exp

[Σ−1

n∑i=1

(Xi − µ)(Xi − µ)T

From the previous slide,

L(µ,Σ) = (2π)−np/2|Σ|−n/2 exp

[Σ−1

n∑i=1

(Xi − µ)(Xi − µ)T

2 Relying on the identityn∑i=1

(Xi − µ)(Xi − µ)T =

n∑i=1

(Xi −X)(Xi −X)T + n(X− µ)(X− µ)T

where X = n−1∑n

i=1 Xi, we have

L(µ,Σ) = (2π)−np/2|Σ|−n/2 exp

−12tr

[Σ−1

(n(X− µ)(X− µ)T

n∑i=1

(Xi −X)(Xi −X)T

2 From the previous slide

L(µ,Σ) = (2π)−np/2|Σ|−n/2 exp

−12tr

[Σ−1

(n(X− µ)(X− µ)T

n∑i=1

(Xi −X)(Xi −X)T

2 Maximizing over L(µ,Σ) (with respect to) µ is easy:

−12tr[Σ−1

((X− µ)(X− µ)T

)]= −1

2(X− µ)TΣ−1(X− µ) ≤ 0

with equality if and only if µ = X.

2 ThenmaxµL(µ,Σ) = L(X,Σ)

= (2π)−np/2|Σ|−n/2 exp−1

2tr(Σ−1S

)≡ LP (Σ)

where LP (Σ) is sometimes referred to as the profile likelihood.

2 Next note that

argmaxΣ LP (Σ) = argmaxΣ logLP (Σ)= argmaxΣ

2 log(2π)− n2 log |Σ| − 1

2tr(Σ−1S

)since log(·) is monotonic.

2 In order to maximize the log profile likelihood we note that it can be shown that

∂ log |Σ|∂Σij

[Σ−1

∂Σij

[Σ−1Eij

]where Eij is a zero matrix except for 1 in the (i, j)th entry. Next, it can be

shown that∂Σ−1

∂Σij= −Σ−1

∂ΣijΣ−1 = −Σ−1EijΣ

Hence,∂ logLP (Σ)

∂Σij= −n

2tr[Σ−1Eij

2tr(Σ−1EijΣ

−1S)

= 12tr[Σ−1

(SΣ−1 − nI

Setting the above to 0 for all (i, j) we obtain the solution

Σ = 1nS.

Biasedness of MLE Estimators

Firstly, it is easy to shot that the MLE for µ is unbiased since,

E[X] =1

n∑i=1

and has covariance

Cov(X) =1

n∑i=1

Cov[Xi]

n∑i=1

However, the MLE for Σ is biased since

E[n−1S] = n−1E

[n∑i=1

(Xi −X)(Xi −X)T

= n−1E

[n∑i=1

XiXTi − nXX

]= n−1E

[n(Σ + µµT

)− n

nΣ + µµT

)]= n−1

Hence,1

n−1S

is an unbiased estimator of Σ.

Sampling Distributions

Theorem: If Σ is a consistent estimator of Σ then, by virtue of the central limit

theorem √nΣ

−1/2(X− µ)

converges to Np(0, I) in distribution.

The sampling distribution for S is much more complicated and involve the Wishart

distribution (which we will cover later on).

Results on Quadratic Forms

2 Definition: A square matrix A is called idempotent if A2 = A.

2 Theorem: A symmetrix (symmetric matrix) A is idempotent if and only if all

its eigenvalues are in 0, 1.

2 Proof: Suppose that the eigenvalue descoposition of A is UΛUT then

A2 = UΛUTUΛU = UΛ2U

If A is idempotent then

UΛ2U = UΛU⇒ λ2i = λi

for 1 ≤ i ≤ p. Hence, if A is idempotent then λi ∈ 0, 1 for 1 ≤ i ≤ p.

If λi ∈ 0, 1 for 1 ≤ i ≤ p then λ2i = λi for 1 ≤ i ≤ p and

A2 = UΛ2U = UΛU = A.

2 Let X ∼ Np(0, I) and let C be a symmetric square matrix of rank r > 0.

2 Theorem 3: The random variable

XTCX ∼ χ2r

if and only if C is idempotent.

2 Proof: We can diagonalize: C = UDUT where

U is an orthogonal matrix and

D = diag(λ1, . . . , λr, 0, . . . , 0︸︷︷︸p−r

where λi ≥ λi+1 for i = 1, . . . , r − 1 (and λi 6= 0).

2 Let Y = UTX so that Y ∼ Np(0, I) and note that

W ≡ XTCX

= XTUDUTX

= YTDY

r∑i=1

λiY2i .

2 If C is idempotent then λi = 1 for 1 ≤ i ≤ r and W ∼ χ2r since Y 2

i are i.i.d.

2 Conversely, assume W ∼ χ2r then its MGF is given by

MW (t) = (1− 2t)−r/2 for t < 12.

Since Y 2i are i.i.d. χ2

1, for λit < 1/2 for all i we also have

MW (t) =

r∏i=1

MλiY2i(t)

r∏i=1

MY 2i(λit)

r∏i=1

(1− 2λit)−1/2.

2 These two domains agree if and only if λ1 = 1 and λr > 0 so this has to be the

2 Therefore, MW (t) = Mχ2r(t) for t < 1

2 only if

r∏i=1

(1− 2λit) = [MW (t)]−2

= [Mχ2(r)(t)]−2

r∏i=1

(1− 2t).

2 From the unique polynomial factorization, the equality holds if and only if λi = 1

for i = 1, . . . , r.

2 Theorem 4 [Craig’s Theorem]: If A and B are symmetric, non-negative

definite matrices and X ∼ Np(0, I) then

XTAX and XTBX

are independent if and only if AB = 0.

2 Proof: Assume AB = 0 then

BA = (AB)T = 0 so A and B commute.

Simultaneous diagonalization: there exist an orthogonal U and diagonal ma-

trices DA and DB such that UTAU = DA and UTBU = DB.

Next, AB = UDAUTUDBUT = UDADBUT so that AB = 0 implies

(WLOG) that

DA = diag(λ1, . . . , λr, 0, . . . , 0) and DB = diag(0, . . . , 0︸︷︷︸r

, λr+1, . . . , λp)

(for some r).

It follows that with Y ≡ UTX

XTAX = YTUTAUY =

r∑i=1

λiY2i and XTBX =

p∑i=r+1

λiY2i

As Y ∼ Np(0, I) the latter two are obviously independent random variables.

2 Conversely, suppose the quadratic forms are independent

Let U be an orthogonal matrix such that

UTAU = DA = diag(λ1, . . . , λr,0(p−r)).

Let B∗ = UTBU and let Y = UTX, then

XTAX = YTUTAUY =

r∑i=1

λiY2i

XTBX =∑i

b∗ijYiYj

Independence implies that

E[XTAX]E[XTBX] = E[XTAXXTBX]

Therefore,

[r∑i=1

λiY2i

b∗ijYiYj

( r∑i=1

λiY2i

b∗kjYkYj

But Y1, . . . , Yp are NID(0, 1) with E[Y 2i ] = 1, E[Y 3

i ] = 0 and E[Y 4i ] = 3.

Therefore, (r∑i=1

b∗jj

r∑i=1

λib∗ii +

∑i 6=j

λib∗jj.

Thus,∑r

i=1 λib∗ii = 0.

But λib∗ii ≥ 0 for all i (why?) and λi > 0 for i = i, . . . , r so b∗ii = 0 for

i = 1, . . . , r.

Since B∗ ≥ 0 it follows that any principal submatrix is non-negative definite

as well , hence b∗ij = 0 if i or j are in 1, . . . , r. Thus,

B∗ =

0 B∗22

)hence DAB∗ = 0 and therefore

AB = AUUTB = UDAB∗UT = 0.

The Wishart Distribution

2 Let X = (X1, . . . ,Xn)T be n× p matrix with rows Xi ∼ NIDp(0,Σ)

2 The p×p matrix M ≡ XTX has a Wishart distribution Wp(Σ, n) whose density

is given by

C−1p,n|Σ|−n/2|M|(n−p−1)/2 exp−1

2tr[Σ−1M

Cp,n ≡ 2np/2πp(p−1)/4p∏i=1

Γ((n + 1− i)/2).

We also need n (called the degrees of freedom) to satisfy n > p − 1 and the

Wishart distribution parameter Σ (called the scale matrix) is assumed to be

positive definite.

2 Note that X =∑n

i=1 eiXTi where ei is a vector of length n whose elements are

0 except for the ith element which is equal to 1. Therefore,

n∑j=1

n∑i=1

=∑i,j

XjδijXTi

n∑i=1

where δij is a scalar equal to 1 if i = j and 0 if i 6= j.

2 Hence, for known µ = 0 the MLE/sample covariance nΣ ∼ Wp(Σ, n).

2 If p = 1 then W1(Σ, n) = W1(σ2, n) = σ2χ2

2 E[M] = E(∑n

i=1 XiXTi

)= nΣ.

2 Var(Mij) = n(Σ2ij + ΣiiΣjj).

2 Theorem 5: If M ∼ Wp(Σ, n) and B is a p× q matrix then

BTMB ∼ Wq(BTΣB, n).

2 Proof: Let Y = XB, then Y = (Y1, . . . ,Yn)T is an n × q matrix with

independent rows denoted YTi = XT

i B, or Yi = BTXi ∼ Nq(0,BTΣB).

It follows thatBTMB = BTXTXB

∼ Wq(BTΣB, n).

2 Corollary: The principal submatrices of M have Wishart distributions.

2 Theorem 6: If M ∼ Wp(Σ, n) and a is any fixed p-vector such that aTΣa > 0

thenaTMa

aTΣa∼ χ2

2 Proof: Firstly, from Theorem 5 we have

aTMa ∼ W1(aTΣa, n)

and we have already argued that

W1(aTΣa, n) ∼ (aTΣa)χ2

2 Corollary:

Mii ∼ Σ2iiχ

2 The converse to Theorem 6 is not true.

2 Theorem 7: If

M1 ∼ Wp(Σ, n1) and M2 ∼ Wp(Σ, n2)

independently of M1 then

M1 + M2 ∼ Wp(Σ, n1 + n2).

2 Proof: Write Mi = XTi Xi where Xi has ni independent rows drawn from a

Np(0,Σ) distribution. Then

M1 + M2 = XT1 X1 + XT

= XTX where X

2 Recall that the sample covariance matrix is defined as 1nS where

n∑i=1

(Xi −X)(Xi −X)T .

2 The Wishart distribution describes the distribution of S.

2 Theorem 8: If X is an n × p matrix with rows independent Np(µ,Σ) then

S ∼ Wp(Σ, n− 1) independently of X ∼ Np(µ, n−1Σ).

2 Proof: Recall that

n∑i=1

(Xi −X)(Xi −X)T

n∑i=1

XiXTi − nXX

= XTX− nXXT.

Let P be an n× n orthogonal matrix with first row(1√n, . . . ,

Let Y = PX so Y1 =√nX where

Y = (Y1, . . . ,Yn)T .

S = YTPPTY −Y1YT1

= YTY −Y1YT1

n∑i=2

YiYTi .

2 Exercise: WLOG µ = 0.

2 It follows from the following lemma that Yi ∼ NIDp(0,Σ) and therefore

S ∼ Wp(Σ, n− 1) independently of 1√nY1 = X.

2 Lemma: Let X be an n × p matrix with rows NIDp(0,Σ) and let U be an

n × n orthogonal matrix. Define Y = UX, and let YTi be the rows of Y

(i = 1, . . . , n). Then Yi ∼ NIDp(0,Σ).

2 Proof: We have E(Y) = UE(X) = 0np. Note that Yi = YTei and therefore

E[YiYTj ] = E[XTUTeie

Tj UX]

= E[XTuiuTj X]

[(n∑k=1

(n∑l=1

∑k,l

XkuikujlXTl

=∑k,l

uikujlE[XkXTl ]

uikujkΣ

= δijΣ,

where uTi the ith row of U.

Wishart Distribution (cont.)

2 Theorem 9: Suppose X has n rows which are NIDp(0,Σ) then XTAX ∼Wp(Σ, r) if and only if A is idempotent and of rank r.

2 Proof: If XTAX ∼ Wp(Σ, r) then by Theorem 6 for any a ∈ Rp with aTΣa >

0 we haveaTXTAXa

aTΣa∼ χ2

Y =Xa√aTΣa

then Y ∼ N(0n, In) and YTAY ∼ χ2r so A is idempotent of rank r by

Theorem 3.

2 Conversely, if A is idempotent of rank r then A = UDUT , where U is an

n× n orthogonal matrix and D = diag(1, . . . , 1︸︷︷︸r

, 0, . . . , 0︸︷︷︸n−r

2 Let Y = UTX so by the lemma Y has n rows YTi ∼ NIDp(0,Σ) and

XTAX = YTDY

n∑j=1

(n∑i=1

n∑i,j=1

(eTj Dei

r∑i,j=1

YjδijYTi

r∑i=1

YiYTi ∼ Wp(Σ, r).

2 Theorem 10: If X has n rows which are NIDp(0,Σ) then for any symmetric,

non-negative definite n× n matrices A and B the random values XTAX and

XTBX are independent if and only if AB = 0.

2 Proof: Assume XTAX and XTBX are independent. Then

Choose a such that aTΣa > 0 and let

Y =Xa√aTΣa

∗ Y ∼ Nn(0, I) and

∗ YTAY and YTBY are independent.

Thus by Craig’s Theorem we have AB = 0.

2 Conversely, assume AB = 0. Then

There exists an n× n orthogonal matrix P such that

PTAP = DA ≡ diag(α1, . . . , αr︸︷︷︸r

, 0 . . . 0︸︷︷︸n−r

PTBP = DB ≡ diag(0, . . . , 0︸︷︷︸r

, βr+1, . . . , βn︸︷︷︸n−r

Let Y = PTX, then by the last lemma again Y has n rows YTi ∼

NIDp(0,Σ) and

XTAX = YTDAY

n∑j=1

(n∑i=1

n∑i,j=1

(eTj DAei

αiYiYTi .

Similarly, XTBX =∑n

r+1 βiYiYTi .

The latter two are obviously independent.

2 Theorem 11: Suppose X has n rows which are NIDp(0,Σ) with Σ positive

definite. LetM = XTX

(M11 M12

M21 M22

where M11 is an r × r matrix (with r < n). Let

M2·1 ≡M22 −M21M−111 M12.

M2·1 ∼ Wq(Σ2·1, n− r)

independently of (M11,M12) where q = p− r and

Σ2·1 = Σ22 −Σ21Σ−111 Σ12.

2 Recall that

Σ2·1 is the conditional covariance of

X(2)i ≡ (Xi,r+1, . . . , Xip)

T given X(1)i ≡ (Xi1, . . . , Xir)

This is also the covariance of

Yi ≡ X(2)i −QX

QX(1)i ≡

(Σ21Σ

−111

is the best linear predictor of X(2)i given X

(1)i :

Yi ∼ NIDq(0q,Σ2·1)

and independent of X(1)i .

2 Hence, had we known this decomposition we could have estimated Σ2·1 as

YTY ∼ Wq(Σ2·1, n) (note the degrees of freedom is n rather than n− r).

2 But we typically don’t know Σ...

2 Proof (of Theorem 11).

Write X = (X1,X2) where X1 is n× r and X2 is n× q (with r + q = p),

thenM = XTX

)(X1,X2).

It can be shown that as Σ11 is positive definite and as n > r, M11 is positive

definite almost surely.

Thus, we can define

M2·1 = XT2 X2 −XT

2 X1M−111 XT

= XT2 PX2,

where P = I−X1M−111 XT

if P = I −X1M−111 XT

1 it is easy to verify directly that P is idempotent (a

projection on Span(X1)⊥) and its rank is (n− r) as

tr(P) = n− tr[X1M−111 XT

= n− r.

Also, XT1 P = PX1 = 0 so with

X2·1 ≡ X2 −X1Σ−111 Σ12

andM2·1 = XT

= XT2·1PX2·1.

But X2·1 is the component of X2 that is independent of X1 and in particular

we saw that the n rows of X2·1 ∼ NIDq(0q,Σ2·1).

It follows from Theorem 9 that, conditioned on X1, M2·1 ∼ Wq(Σ2·1, n−r).

This distribution does not depend on X1, therefore it is also the unconditional

distribution of M2·1.

Moreover, it follows that M2·1 is independent of X1 and hence of M11 =

XT1 X1.

Furthermore, P(I−P) = P−P = 0 so using the same rationale as in the

proof of Theorem 10 we see that given X1,

M2·1 = XT2·1PX2·1 and XT

1 (I−P)X2·1

are independent.

XT1 (I−P)X2·1 = XT

1 (X1M−111 XT

1 )(X2 −X1Σ−111 Σ12)

= M12 −M11Σ−111 Σ12.

Therefore, given X1, M2·1 is independent of

M12 = XT1 (I−P)X2·1 + M11Σ

−111 Σ12

Since M2·1 is also independent of X1 it is independent of (X1,M12) (why?)

It follows that M2·1 is independent of (M11,M12) (why?)

Linear Independence Lemma

2 Linear Independence Lemma: Suppose that Xi ∼ NIDp(Σ, 0) for i =

1, . . . , n ≤ p and that Σ is positive definite. Then X1, . . . ,Xn are almost

surely linearly independent.

2 Proof of lemma.

Firsly, without loss of generality we can assume that Σ = Ip. (exercise)

The proof is by induction where the case n = 1 is trivial since X1 ∼ Np(I,0)

implies that P (X1 = 0) = 0.

Now, assume that the lemma holds for n < p. In this case

∗ Either Xn+1 /∈ Span〈X1, . . . ,Xn〉 almost surely and we are done,

∗ or, there exists a set A ⊂ Ω with P (A) > 0 and random variables αi such

Xn+1(ω) =

n∑i=1

αi(ω)Xi(ω) and X1(ω), . . . ,Xn(ω)

are linearly independent for all ω ∈ A.

The latter identity can be thought of as p equations in n unknowns: αi(ω). Since X1(ω), . . . ,Xn(ω) are linearly independent, then without loss of gen-

erality the αi(ω) are uniquely determined from

X1(ω), . . . ,Xn(ω)

and Xn+1,k(ω) for k = 1, . . . , n.

Xn+1,n+1(ω) =

n∑i=1

αi(ω)Xi,n+1(ω)

and it follows that on the set A, Xn+1,n+1 can be determined from

X1(ω), . . . ,Xn(ω)

and Xn+1,k(ω) for k = 1, . . . , n.

This contradicts the independence assumption.

2 Corollary: If M ∼ Wp(Σ, n) with Σ positive definte and n ≥ p then |M| > 0

almost surely (exercise).

2 Theorem 12. If M ∼ Wp(Σ, n) with Σ positive definite and n ≥ p then

(a) For any fixed a 6= 0 ∈ Rp,

aTΣ−1a

aTM−1a∼ χ2

n−p+1.

In particular Σii/M ii ∼ χ2n−p+1.

(b) Mii is independent of all elements of M except Mii.

2 Proof:

Recall that M22 = M−12·1 and similarly Σ22 = Σ−12·1.

Therefore, from Theorem 11, with r = p− 1 we have

(Mpp)−1 = M2·1∼ Wp−r(Σ2·1, n− r)∼ (Σpp)−1χ2

n−p+1.

Moreover, M pp is independent of (M11,M12), i.e., M pp is independent of all

elements of M except Mpp

This proves the theorem for a = ep and hence for a = ei (why?)

For a general a 6= 0 ∈ Rp, let A be a non-singular matrix with last column

a, that is Aep = a. Then

aTΣ−1aaTM−1a

=eTpATΣ−1Aep

eTpATM−1Aep

=eTpΣ−1A ep

eTpM−1A ep

where ΣA = A−1Σ(A−1)T and MA = A−1M(A−1)T .

The proof is complete since by Theorem 5, MA ∼ Wp(ΣA, n).

2 Theorem 13. If M ∼ Wp(Σ, n) where n ≥ p then

|M| = |Σ| ·p∏i=1

where Ui are independent random variables, Ui ∼ χ2n−i+1.

2 Proof: Without loss of generality we may assume that Σ is positive definite

(exercise). We use induction on p.

For the case p = 1 we have

M ∼ σ2χ2n.

(M11 M12

M21 M22

)where M11 is a (p− 1)× (p− 1) matrix.

Recall that

M22 = [M−1]pp = |M11|/|M|

(similarly, Σ22 = |Σ11|/|Σ|).

Therefore,

|M| =|M11|M 22

|Σ||Σ11|

|M11|.

But from Theorem 12,

Up ≡Σ22

[Σ−1]pp[M−1]pp

∼ χ2n−p+1

and is independent of M11.

We complete the proof by noting that M11 ∼ Wp−1(Σ11, n).

Hence, by the inductive hypothesis

|M11| = |Σ11|p−1∏i=1

where Ui ∼ χ2n−i+1 are independent of one another and obviously they depend

only on M11 and hence independent of Up.

2 Corollary: If X has n rows which are NIDp(µ,Σ) with Σ positive definite

and S = XTX− nXXT

Σkk/Skk ∼ χ2n−p.

aTΣ−1a

aTS−1a∼ χ2

for any fixed a 6= 0

If S11 is an r × r submatrix of S, then

S11 ∼ Wr(Σ11, n− 1)

independently ofS2·1 = S22 − S21S

−111 S12

∼ Wp−r(Σ2·1, n− r − 1).

Hotelling’s T 2

2 Definition. Let Z ∼ Np(0, I) independently of M ∼ Wp(I, n) with n ≥ p

nZTM−1Z ∼ T 2(p, n)

where T 2(p, n) is Hotelling’s T 2 distribution.

2 Theorem 14: If Y ∼ Np(µ,Σ) and M ∼ Wp(Σ, n) independently of Y with

Σ positive definite and n ≥ p then

n(Y − µ)TM−1(Y − µ) ∼ T 2(p, n).

2 Proof:

Let Z = Σ−1/2(Y − µ) and let MΣ = Σ−1/2MΣ−1/2.

Then Z ∼ Np(0, I), MΣ ∼ Wp(I, n) and

n(Y − µ)TM−1(Y − µ) = nZTM−1Σ Z.

2 Theorem 15:

T 2(p, n) ∼ np

n− p + 1Fp,n−p+1

2 Proof:

Let Z ∼ Np(0, I) independently of M ∼ Wp(I, n).

By Theorem 12, for a 6= 0, conditioned on Z = a

D ≡ ZTZ

ZTM−1Z∼ χ2

n−p+1

This conditional distribution of D does not depend on a hence it is also the

unconditional distribution of D.

For the same reason, D is independent of Z and therefore also of R ≡ZTZ ∼ χ2

It follows thatnZTM−1Z

D∼ p

n− p + 1Fp,n−p+1.

2 Corollary 1: If X is the mean of a sample of size n ≥ p drawn from a Np(µ,Σ)

population with Σ positive definite, and if (n − 1)−1S is the unbiased sample

covariance matrix then

n(X− µ)T(

n− 1S

)−1(X− µ) ∼ (n− 1)p

n− pFp,n−p.

2 Proof:

It suffices to show that the LHS has a T 2(p, n− 1) distribution.

Let Y =√nX and let S = XTX− nXX

Theorem 8 states: S ∼ Wp(Σ, n− 1) independently of Y ∼ Np(√nµ,Σ).

Therefore by Theorem 14,

(n− 1)(Y −√nµ)TS−1(Y −

√nµ) ∼ T 2(p, n− 1)

Finally,

n(X−µ)T(

n− 1S

)−1(X−µ) = (n− 1)(Y−

√nµ)TS−1(Y−

√nµ).

STAT3914 { Applied Statistics · STAT3914 { Applied Statistics Lecturer Dr. John T. Ormerod School...

Documents

Applied Statistics Master of Professional Studies 2017 …Applied Statistics Master of Professional Studies ... Bechtel Marine Propulsion ... When Did MPS Applied Statistics Graduates

Applied statistics lecture_2

Applied Statistics - MIT

Applied statistics

Applied Statistics 2009

Applied Statistics 3

MATHEMATICS & APPLIED STATISTICS - UOWweb/@unia/documents/...MATHEMATICS & APPLIED STATISTICS ... science, engineering, agriculture and industry. A major in Applied Statistics equips

Applied Statistics Chapter17

MATH602: APPLIED STATISTICS

Applied Statistics En

Introduction to Applied Statistics

Applied Statistics IV

Applied Statistics II

S.Y.B.A. Applied Statistics and Statistics Special

Applied statistics lecture_5

MSc APPLIED STATISTICS - Birkbeck, University of London · MSc APPLIED STATISTICS (MAS) MSc APPLIED STATISTICS WITH MEDICAL APPLICATIONS (MASMA) MSc APPLIED STATISTICS AND OPERATIONAL

Working Papers in Econometrics and Applied Statistics · Working Papers in Econometrics and Applied Statistics ... Working Papers in Econometrics and Applied Statistics ... and Maddala

Bum2413 - Applied Statistics

Applied Multivariate Statistics

MSc APPLIED STATISTICS€¦ · MSc APPLIED STATISTICS (MAS) MSc APPLIED STATISTICS WITH MEDICAL APPLICATIONS (MASMA) MSc APPLIED STATISTICS AND OPERATIONAL RESEARCH (MASOR) MSc APPLIED