MATH2901 Final Revision · 2020. 8. 13. · MATH2901 Final Revision Part II: Statistical Inference Presenter: Henry Lam Based o Rui Tong’s 2018 Slides UNSW Society of Statistics/UNSW

MATH2901 Final RevisionPart II: Statistical Inference

Presenter: Henry LamBased off Rui Tong’s 2018 Slides

UNSW Society of Statistics/UNSW Mathematics Society

13 August 2020

Presenter: Henry Lam, Based off Rui Tong’s 2018 Slides (UNSW Society of Statistics/UNSW Mathematics Society)MATH2901 Final Revision 13 August 2020 1 / 66

Table of Contents

1 Study DesignSamples

2 Statistical InferenceEstimators and their Properties

Properties of any estimatorDesirable features of estimators

More work with ConvergenceConfidence IntervalsDistributions arising from N (0, 1)

Working with S = SX

Common EstimatorsHypothesis tests

3 Coolio Doolio


Table of Contents





Working with S = SX


3 Coolio Doolio


Random Sample

Definition

A random sample (with size n) is a set of n independent, identicallydistributed random variables:

X1, . . . ,Xn

Note:

x1, . . . , xn is the ’observed data’, whereas X1, . . . ,Xn is the ’theoreticaldata’.

Whenever we are working with statistics, we work with the assumptionthat we have ’theoretical data’ first.


Statistics

Definition (Statistic)

For a random sample X1, . . . ,Xn, a statistic is just a function of the sample.

Example (Common Statistics)

Sample mean: X = 1n

∑ni=1 Xi

Sample variance: S2 = 1n−1

∑ni=1(Xi − X )2

Sample median: X( n2 )


The Sample Mean

Theorem (Properties of the Sample Mean)

Let X1, . . . ,Xn be a random sample, with mean µ and variance σ2. Then,

E[X ] = µ Var(X ) =σ2

n


The Sample Mean

Example

Prove that Var(X ) = σ2

n as stated just now.

Var(X)

= Var

(1

n

n∑i=1

Xi

)

=1

n2Var

(n∑

i=1

Xi

)

=1

n2

n∑i=1

Var(Xi ) (indep.)

=1

n2· nσ2 =

σ2

n


Table of Contents





Working with S = SX


3 Coolio Doolio


Estimators

Let X1, . . . ,Xn be a random sample with model {fX (x ; θ) : θ ∈ Θ}.

Definition (Estimator)

An estimator for the parameter θ, denoted θ, is just a real valued functionof the random sample.

θ = θ(X1, . . . ,Xn)

Meaning, fundamentally it’s just a statistic (and so it is a random variable).


Estimators

Basically, we want to narrow our focus to useful estimators.

To do this, we’ll be looking at a figure that measures how well the estimatorperforms on ’average’, the MSE.


Bias

Remember that θ is a parameter, so it’s constant. Whereas θ is anestimator, which is a r.v.

Definition: Bias

Given an estimator θ for θ, its bias is

Bias(θ) = E[θ]− θ.

The estimator is ’unbiased’ if Bias(θ) = 0.

Significance

’Bias’ is essentially a measure by how far off on average an estimator isfrom the true value.


Bias

Example

Let X1, . . . ,X7 be a random Poisson(λ) sample, and consider the estimator

λ =1

28

7∑i=1

i Xi =X1 + 2X2 + · · ·+ 7X7

28

for λ. Is this estimator unbiased?

We compute:

E[λ] = E[X1 + 2X2 + · · ·+ 7X7

28

]

=1

28E[X1 + 2X2 + · · ·+ 7X7]

=1

28(E[X1] + 2E[X2] + · · ·+ 7E[X7])

=1

28(λ+ 2λ+ · · ·+ 7λ)

=1

28× 28λ = λ


Bias

Example

Let X1, . . . ,X7 be a random Poisson(λ) sample, and consider the estimator

λ =1

28

7∑i=1

i Xi =X1 + 2X2 + · · ·+ 7X7

28

for λ. Is this estimator unbiased?

We compute:

E[λ] = E[X1 + 2X2 + · · ·+ 7X7

28

]=

1

28E[X1 + 2X2 + · · ·+ 7X7]

=1

28(E[X1] + 2E[X2] + · · ·+ 7E[X7])

=1

28(λ+ 2λ+ · · ·+ 7λ)

=1

28× 28λ = λ


Bias

We compute:

E[λ] = E[X1 + 2X2 + · · ·+ 7X7

28

]=

1

28E[X1 + 2X2 + · · ·+ 7X7]

=1

28(E[X1] + 2E[X2] + · · ·+ 7E[X7])

=1

28(λ+ 2λ+ · · ·+ 7λ)

=1

28× 28λ = λ

Hence Bias(λ) = λ− λ = 0 and thus it is unbiased.


Standard Error

Definition (Standard Error)

Se(θ) =

√Var(θ)

Significance

This measures how sensitive the estimator is when estimating the parameter,similar to the concept of variance of random variables. Recall that anestimator is just a random variable!


Standard Error

Example

For the earlier example λ = X1+2X2+···+7X728 , find Se(λ).

Var(λ) = Var

(X1 + 2X2 + · · ·+ 7X7

28

)=

1

282(Var(X1) + 4 Var(X2) + · · ·+ 49 Var(X7)) (indep.)

=1

282× 140λ =

5

28λ.

Therefore Se(λ) =√

5λ28 .


Estimated Standard Error

As seen in the previous slide, this value actually depends on the parameteritself, which is unknown. Naively we’ll just plug in our estimator in theparameter’s place (a later theorem justifies this - at least asymptotically).This is actually something we’ll be doing a lot in this course!

Definition: Estimated Standard Error

Se(θ) = Se(θ) |θ=θ


Estimated Standard Error

As seen in the previous slide, this value actually depends on the parameteritself, which is unknown. Naively we’ll just plug in our estimator in theparameter’s place (a later theorem justifies this - at least asymptotically).This is actually something we’ll be doing a lot in this course!

Example

For the earlier example λ = X1+2X2+···+7X728 , find Se(λ) and Se(λ).

From earlier we had,

Se(λ) =

√5λ

28

and so we have:

Se(λ) =

√5λ

28.


Mean Squared Error

The previous two concepts: bias and standard error come from thefollowing measure:

Definition: Mean Squared Error

Given an estimator θ for θ, its mean squared error is

MSE(θ) = E[(θ − θ)2].

Theorem: MSE Formula

MSE(θ) = [Bias(θ)]2 + Var(θ).


Proof: MSE formula

MSE(θ) = E[(θ − θ)2]

= E[(

(θ−E[θ]) + (E[θ]− θ))2]

= E[(θ − E[θ])2

]+ E

[(E[θ]− θ)2

]+ 2E

[(θ − E[θ])(E[θ]− θ)

]from expanding the perfect square. Note that E[(θ − E[θ])] = Var(θ) bydefinition, and

E[(E[θ]− θ)

]= E[Bias(θ)2] = Bias(θ)2.

(Q: Why was I allowed to take off the expected value brackets?)


Proof: MSE formula

As for the leftover bit:

2E[(θ − E[θ])(E[θ]− θ)

]= 2

(E[θ]− θ

)E[θ − E[θ]

]...but

E[θ − E[θ]

]= E[θ]− E[θ] = 0.

Make sure to remember all your properties of the expected value!


Mean Squared Error

Example

For the earlier example λ = X1+2X2+···+7X728 , find MSE(λ).

MSE(λ) = Var(λ) + Bias(λ)2 =5λ

28+ 02 =

5λ

28.


”Better” Estimators

Significance of MSE

Demonstrates a trade-off between the variance and the bias.

Better estimators in the MSE sense

Between two estimators θ1 and θ2, θ1 is better (w.r.t. MSE), at somespecific value of θ, if

MSE(θ1) < MSE(θ2)



Example

Let λ1 be the estimator that we found earlier, with MSE(λ1) = 5λ28 . Now let

λ2 = X . For what values of λ is λ2 better than λ1?

We can compute:

Bias(λ2) = 0

Var(λ2) =λ

7

∴ MSE(λ2) =λ

7



Example

Let λ1 be the estimator that we found earlier, with MSE(λ1) = 5λ28 . Now let

λ2 = X . For what values of λ is λ2 better than λ1?

MSE(λ2) =λ

7

Solving MSE(λ2) < MSE(λ1) gives

λ

7<

5λ

28=⇒ λ > 0.


Application - Sample Proportion

Theorem (Properties of the Sample Mean)

Let X1, . . . ,Xn be a random sample from the Ber(p) distribution. Then thesample proportion p = No. of successes

No. of trials satisfies:

E[p] = p

Var(p) =p(1− p)

n

Se(p) =

√p(1− p)

n


Consistency

A sequence of random variables X1, . . . ,Xn converges in probability to X ,

i.e. XnP→ X , if ∀ε > 0,

limn→∞

P(|Xn − X | > ε) = 0.

Definition: Consistent Estimator

θn is a consistent estimator for θ if it converges in probability to θ. i.e.

θnP→ θ

Slight issue

Generally, this can be quite difficult to show. Luckily for us, we actuallyhave another approach to show convergence in probability!


Consistency

A sequence of random variables X1, . . . ,Xn converges in probability to X ,

i.e. XnP→ X , if ∀ε > 0,

limn→∞

P(|Xn − X | > ε) = 0.

Definition: Consistent Estimator

θn is a consistent estimator for θ if it converges in probability to θ. i.e.

θnP→ θ

Slight issue

Generally, this can be quite difficult to show. Luckily for us, we actuallyhave another approach to show convergence in probability!


Verifying that an estimator is consistent

Theorem: Sufficient criteria for consistency

Iflimn→∞

MSE(θn) = 0

then θn is a consistent estimator for θ.

Quick example: Consider the mean proportion X = 1n

∑ni=1 Xi for µ. Then

MSE(θn) = Var(θn) + Bias(θn)2 =σ2

n+ 02 =

σ2

n.

Clearly limn→∞MSE(θn) = 0 so the sample mean is a consistent estimatorfor µ.


Asymptotic Normality

A sequence of random variables X1, . . . ,Xn converges in distribution to X ,

i.e. XnD→ X , if

limn→∞

FXn(x)→ FX (x).

Definition: Asymptotically Normal Estimator

θn is an asymptotically normal estimator for θ if

θn − θSe(θ)

D→ N (0, 1).


Convergence Theorems

Central Limit Theorem

For a random sample X1, . . . ,Xn with mean µ and finite variance σ,

X − µσ√n

D→ N (0, 1)

Slutsky’s Theorem

Suppose we have two sequences of random variables (or random samples)with:

XnD→ X Yn

P→ c

where c is a constant. Then,

Xn + YnD→ X + c XnYn

D→ cX


The Delta Method

Theorem (Provided on formula sheet!!)

Let θ1, θ2, . . . be a sequence of estimators (or a sequence of randomvariables) of θ such that

θn − θσ√n

D→ N (0, 1).

Then, for any function g that is differentiable at θ, with g ′(θ) 6= 0,

g(θn)− g(θ)

g ′(θ) σ√n

D→ N (0, 1)


Delta Method Example

2015 FE Q2c)

Suppose {Xi}ni=1 are i.i.d. Ber(p) random variables and define Yn = X n.What is the asymptotic distribution of Yn(1− Yn)?



2015 FE Q2c)


Finding the distribution of Yn(1− Yn) may be difficult to do directly, butwe note that Yn = Xn. Thus by the CLT we have:

Yn − p√p(1−p)

n

D→ N (0, 1).



2015 FE Q2c)


Now using the delta method with transform of g(x) = x(1− x) on Yn withθ = p. g ′(x) = 1− 2x .

Yn(1− Yn)− p(1− p)

(1− 2p)√

p(1−p)n

D→ N (0, 1).

Rearranging everything yields:

Yn(1− Yn)− p(1− p)D→ N

(0, (1− 2p)2 p(1− p)

n

)

Yn(1− Yn)D→ N

(p(1− p), (1− 2p)2 p(1− p)

n

).



2015 FE Q2c)



Yn(1− Yn)− p(1− p)

(1− 2p)√

p(1−p)n

D→ N (0, 1).


Yn(1− Yn)− p(1− p)D→ N

(0, (1− 2p)2 p(1− p)

n

)

Yn(1− Yn)D→ N

(p(1− p), (1− 2p)2 p(1− p)

n

).



2015 FE Q2c)



Yn(1− Yn)− p(1− p)

(1− 2p)√

p(1−p)n

D→ N (0, 1).


Yn(1− Yn)− p(1− p)D→ N

(0, (1− 2p)2 p(1− p)

n

)Yn(1− Yn)

D→ N(p(1− p), (1− 2p)2 p(1− p)

n

).



2015 FE Q2c)


We have found the asymptotic distribution! But we must also check thecondition that: g ′(p) 6= 0⇔ p 6= 0.5. Hence, we have:

Yn(1− Yn)D→ N

(p(1− p), (1− 2p)2 p(1− p)

n

)for p ∈ (0, 1), p 6= 0.5.


Confidence Intervals (Generic Definition)

In a confidence interval, we put the parameter in the middle, instead of therandom variable. We want to predict where the true parameter lies in,not the estimator!

Definition: Confidence Interval

For a random sample X1, . . . ,Xn with parameter θ, if

P(L < θ < U) = 1− α

for some statistics (estimators) L and U, then a 100(1− α)% confidenceinterval for θ is

(L,U)

The method to find these CI’s mostly depends on the distribution of yourrandom variables.


Exact Confidence Intervals

Example (2012 FE Q4)

Consider a sequence of i.i.d. {Xi}ni=1 ∼ Unif(0, θ), from which we utilisethe following estimator X(n) = max{X1,X2, . . . ,Xn}.a) Find the density of X(n).

b) Find an exact 95% confidence interval for θ.





P(X(n) ≤ x) = P(∩ni=1Xi ≤ x)

=n∏

i=1

P(Xi ≤ x) (indep.)

= (P(X1 ≤ x))n (identically dist.)

=

(x

θ

)n

for x ∈ (0, θ).Presenter: Henry Lam, Based off Rui Tong’s 2018 Slides (UNSW Society of Statistics/UNSW Mathematics Society)MATH2901 Final Revision 13 August 2020 24 / 66




Hence the PDF of X(n) is:

fX(n)(x) =

d

dxP(X(n) ≤ x)

=d

dx

(xθ

)n=

n

θnxn−1 (x ∈ (0, θ))




Consider a sequence of i.i.d. {Xi}ni=1 ∼ Unif(0, θ), from which we utilisethe following estimator X(n) = max{X1,X2, . . . ,Xn}.b) Find an exact 95% confidence interval for θ.

We desire a 95% exact confidence interval for θ. In this case, theinterpretation of θ is simply the maximum.





We desire a 95% exact confidence interval for θ. In this case, theinterpretation of θ is simply the maximum.

P(X(n) > a0.05) = 0.95

1−

(a0.05

θ

)n

= 0.95

a0.05 = 0.051/nθ.





Subbing that back in and re-arranging yields:

P(X(n) > 0.051/nθ) = 0.95

P(θ < 201/nX(n)) = 0.95

P(0 < θ < 201/nX(n)) = 0.95

Hence an exact 95% confidence interval for θ would be:(0, 20

1nX(n)

).


Exact Confidence Intervals from Normality

Just like in pretty much every scenario, when dealing with confidenceintervals of normally distributed random variables, we have some very niceproperties. These allow us to build fairly easy confidence intervals.


Chi-Squared distribution

Definition (Chi-Squared distribution)

A random variable X follows a χ2ν distribution if for x ≥ 0,

fX (x) =e−

x2 x

ν2−1

2ν2 Γ(ν2

)Lemma (Chi-Squared as a ’special’ distribution)

X ∼ χ2ν ⇐⇒ X ∼ Gamma

(ν2, 2)

Significance of ν: It is the number of degrees of freedom you have.(MATH2831/2931)


Chi-Squared distribution

Theorem (Origin of Chi-Squared)

If Z ∼ N (0, 1), then Z 2 ∼ χ21.

Lemma (Sum of Chi-Squared is Chi-Squared)

Let X1 ∼ χ2ν1

, ..., Xn ∼ χ2νn be independently distributed. Then their sum

satisfies:n∑

i=1

Xi = X1 + · · ·+ Xn ∼ χ2ν1+···+νn


Revision: Probability

Example (Course pack)

If we have independent standard normal random variables Zi , find theprobability that

∑6i=1 Z

2i > 16.81.

Z 2i ∼ χ2

1 for all i , so

n∑i=1

Z 2i ∼ χ2

6 .

P

(6∑

i=1

Z 2i > 16.81

)= pchisq(16.81,df=6,lower.tail=FALSE) ≈ 0.01


Student’s t distribution

Definition (t-distribution)

A random variable T follows a tν distribution if for t ∈ R,

fT (t) =Γ(ν+1

2

)√πνΓ

(ν2

) (1 +t2

ν

)− ν+12

Significance of ν: It is the number of degrees of freedom you have.(MATH2831/2931)


Student’s t distribution

Theorem (Origin of t)

If Z ∼ N (0, 1) and Q ∼ χ2ν , where Z and Q are independent, then

Z√Q/ν∼ tν

Theorem (Convergence of t)

As ν →∞, tν → N (0, 1)

Example (Density of tν is an even function)

Just like the density of normal distributions, fT (−t) = fT (t).


Sample Variance

Definition (Sample Variance)

For a random sample X1, . . . ,Xn, the sample variance is

S2X =

1

n − 1

n∑i=1

(Xi − X

)2

where X is the sample mean.

Key property

S2 is an unbiased estimator for σ2.


Sample Variance and Distributions

Theorem (Distribution of Sample Variance)

Suppose that X1, . . . ,Xn are i.i.d. random samples from the N (µ, σ2)distribution. Then,

(n − 1)S2

σ2∼ χ2

n−1.

Dealing with two unknown variables simultaneously can be quite challenging,and so the following is a way to only deal with a single unknown.

Theorem (S2 to replace σ2)

Suppose that X1, . . . ,Xn are i.i.d. random samples from the N (µ, σ2)distribution. Then,

X − µS/√n∼ tn−1.

These are exact! Not approximations!


Trap

Only ever use these if you know your original sample came from a normaldistribution (or something that resembles it really well)!


Notation

Recall that zα represents the α-th quantile of Z ∼ N (0, 1).

Notation (t-value)

tn−1,α represents the α-th quantile of T ∼ tn−1, i.e. it satisfies

P(T < tn−1,α) = α


Normal Samples: Exact CI

Corollary (Exact CI for normal samples)

Suppose X1, . . . ,Xn are from a N (µ, σ2) sample. If we know what σ2 is, a100(1− α) % confidence interval for µ is(

X − z1−α2

σ√n,X + z1−α

2

σ√n

)If we don’t know what σ2 is, then using the estimator S2,(

X − tn−1,1−α2

S√n,X + tn−1,1−α

2

S√n

)These can derived on the spot as well.


Confidence Intervals

Example

The following data is taken from a normal random sample:

1.1633974 0.2623631 -2.0633406

By considering X , find a 95% confidence interval for its mean µ.

The sample mean is X = −0.2125267, and the sample variance is

S2 =1

3− 1

((1.1633974 + 0.2125267)2

+ (0.2523621 + 0.2125267)2

+ (−2.0633406 + 0.2125267)2

)= 2.7721


Confidence Intervals

Example

The following data is taken from a normal random sample:

1.1633974 0.2623631 -2.0633406

By considering X , find a 95% confidence interval for its mean µ.

For the mean, t2,0.975 = qt(0.975,df=2) = 4.302653.

Therefore a 95% confidence interval is(−0.2125267− 4.302653

√2.7721√

3,−0.2125267 + 4.302653

√2.7721√

3

)

i.e. (-4.34, 3.92)


Approximate CI’s via Asymptotic Normality

Notation (z-value)

zα represents the α-th quantile of Z ∼ N (0, 1), i.e it satisfies

P(Z < zα) = α

Corollary (Approximate CI)

For a random sample X1, . . . ,Xn with parameter θ, if θn is a consistent andasymptotically normal estimator of θ, then(

θn − z1−α2

Se(θn), θn + z1−α2

Se(θn))

is a 100(1− α)% confidence interval for θ.



”Example” (Setting α = 0.05)

For a random sample X1, . . . ,Xn with parameter θ, if θn is a consistent andasymptotically normal estimator of θ, then(

θn − z0.975 Se(θn), θn + z0.975 Se(θn))

is a 95% confidence interval for θ.



Example (Adapted from Tutorial)

Consider a random sample X1, . . . ,Xn from the Poisson(λ) distribution.Take λ = X , i.e. use the sample mean as an estimator. Find a 95%approximate confidence interval for λ.

As the estimator is the sample mean, we can utilise the CLT for very large n.Recalling that Var(Xi ) = λ and since our estimator is the sample mean,

Var(λ) = Var(X ) =λ

n

so therefore

Se(λ) =

√λ

n

This is actually Se(λ)! (which is fine as we are dealing with itasymptotically.)





By CLT, we have:λ− λ√

λn

D→ N (0, 1)

Therefore

P

z0.025 <λ− λ√

λn

< z0.975

= 0.95



Note that z0.025 = −z0.975 . Rearrange to make λ the subject:

−z0.975 <λ− λ√

λn

< z0.975

−z0.975

√λ

n< λ− λ < z0.975

√λ

n

−z0.975

√λ

n< λ− λ < z0.975

√λ

n

λ− z0.975

√λ

n< λ < λ+ z0.975

√λ

n

Be very careful going from line 2 to line 3!





Therefore we can rewrite:

P

λ− z0.975

√λ

n< λ < λ+ z0.975

√λ

n

= 0.95

so a 95% confidence interval isλ− z0.975

√λ

n, λ+ z0.975

√λ

n

(Then just sub everything in.)


Follow-up question

Example (contd. from Tutorial)

For the confidence interval above, suppose that for a sample size of 30 theobserved values are:

8 2 5 5 8 6 7 2 4 8 4 2 8 4 5 3 3 6 8 3 6 5 5 4 6 3 7 5 1 5

Under these observed values, what is the relevant confidence interval?

From the calculator, the mean of this data is 14830 , so subbing X = 148

30 andn = 30 gives(

148/30− 1.96×√

148/30

30, 148/30 + 1.96×

√148/30

30

)

which is approximately (4.1385, 5.7281)


Behaviour of the approximate CI

The confidence interval becomes smaller when we increase n, i.e. add moresamples!

A 99% confidence interval is wider than a 95% confidence interval. Why?


Method of Moments

Compares theoretical value with estimated values to get parameterestimates.

Suppose we need to estimate k parameters: θ1, . . . , θk .

Definition (Method of Moments Estimator)

Consider the system of equations

E[X ] =1

n

n∑i=1

Xi , E[X 2] =1

n

n∑i=1

X 2i , . . . E[X k ] =

1

n

n∑i=1

X ki .

The method of moments estimator is the solution to this system ofequations.

The method of moments estimate is the observed value of the estimator.This is found by replacing Xi with xi .


Method of Moments - Other way around

Compares theoretical value with estimated values to get parameterestimates.

Suppose we need to estimate k parameters: θ1, . . . , θk .

Definition (Method of Moments Estimate)

Consider the system of equations

E[X ] =1

n

n∑i=1

xi , E[X 2] =1

n

n∑i=1

x2i , . . . E[X k ] =

1

n

n∑i=1

xki .

The method of moments estimate is the solution to this system ofequations.

The method of moments estimator is the original estimator in question.This is found by replacing xi with Xi .


Method of Moments

Example (2901 Assignment, 2017)

Let θ be a parameter satisfying θ > −1. Let X1, . . . ,Xn be i.i.d. randomvariables with PDF

fXi(θ) = (θ + 1)xθ, 0 < x < 1

for i = 1, . . . , n. Find the method of moments estimator for θ.

How many parameters to estimate?


Method of Moments



fXi(θ) = (θ + 1)xθ, 0 < x < 1


Only 1 parameter, therefore we only need one equation:

E[X ] =1

n

n∑i=1

xi .


Method of Moments



fXi(θ) = (θ + 1)xθ, 0 < x < 1


E[X ] =

∫ 1

0x(θ + 1)xθ dx

=

∫ 1

0(θ + 1)xθ+1 dx

=θ + 1

θ + 2


Method of Moments



fXi(θ) = (θ + 1)xθ, 0 < x < 1


So we solve:

θ + 1

θ + 2= x

θ + 1 = x θ + 2x

θ − x θ = 2x − 1

θ =2x − 1

1− x


Method of Moments



fXi(θ) = (θ + 1)xθ, 0 < x < 1


Therefore the method of moments estimator is

θ =2X − 1

1− X


Properties of the Method of Moments Estimator

The estimator is

Consistent

Under ’nice’ conditions, asymptotically normal


Likelihood function

The idea here is that we ’punish’ models that have very unlikelyobservations heavily (low density values) and look for a general ’good’ fit.

Likelihood Function

For observations x1, . . . , xn in a random sample, the likelihood function is

L(θ; x) =n∏

i=1

f (xi ; θ)

Log-likelihood function

For observations x1, . . . , xn in a random sample, the log-likelihood functionis

`(θ; x) = lnL(θ; x) =n∑

i=1

ln[f (xi ; θ)]


Maximum Likelihood Estimator (MLE)

Definition (Maximum Likelihood Estimator)

θMLE is the MLE of θ that maximises the likelihood function L(θ; x).

Theorem (Computation of the MLE)

θMLE also maximises the log-likelihood function `(θ)



Example

Consider the following sequence of i.i.d. LogNormal(0, σ2) randomvariables, {Xi}ni=1 with PDF

fXi(xi ;σ) =

1

xiσ√

2πe−

(ln xi )2

2σ2 , x > 0.

Find σMLE.

`(σ; x) =n∑

i=1

ln

[1

xiσ√

2πe−

(ln xi )2

2σ2

]

= −n lnσ −n∑

i=1

(ln xi )2

2σ2+ C



Example


fXi(xi ;σ) =

1

xiσ√

2πe−

(ln xi )2

2σ2 , x > 0.

Find σMLE.

`(σ; x) =n∑

i=1

ln

[1

xiσ√

2πe−

(ln xi )2

2σ2

]

= −n lnσ −n∑

i=1

(ln xi )2

2σ2+ C



Example


fXi(xi ;σ) =

1

xiσ√

2πe−

(ln xi )2

2σ2 , x > 0.

Find σMLE.

We want to find the maximum, so we differentiate w.r.t. σ and find σ suchthat the derative is 0:

`′(σ; x) = −n

σ+

n∑i=1

(ln xi )2

σ3

`′(σ; x) = −n

σ+

n∑i=1

(ln xi )2

σ3= 0

σ2 =1

n

n∑i=1

(ln xi )2 Are we done?



Example


fXi(xi ;σ) =

1

xiσ√

2πe−

(ln xi )2

2σ2 , x > 0.

Find σMLE.

We want to find the maximum, so we differentiate w.r.t. σ and find σ suchthat the derative is 0:

`′(σ; x) = −n

σ+

n∑i=1

(ln xi )2

σ3= 0

σ2 =1

n

n∑i=1

(ln xi )2 Are we done?



Example


fXi(xi ;σ) =

1

xiσ√

2πe−

(ln xi )2

2σ2 , x > 0.

Find σMLE.

We have to check that is indeed a maximum!

`′′(σ; x) =n

σ2−

n∑i=1

3(ln xi )2

σ4

`′′(σ; x) =n

σ2−

n∑i=1

3(ln xi )2

σ4=

1

σ2(n − 3n) = −

2n

σ2< 0.



Example


fXi(xi ;σ) =

1

xiσ√

2πe−

(ln xi )2

2σ2 , x > 0.

Find σMLE.


`′′(σ; x) =n

σ2−

n∑i=1

3(ln xi )2

σ4

`′′(σ; x) =n

σ2−

n∑i=1

3(ln xi )2

σ4

=1

σ2(n − 3n) = −

2n

σ2< 0.



Example


fXi(xi ;σ) =

1

xiσ√

2πe−

(ln xi )2

2σ2 , x > 0.

Find σMLE.


`′′(σ; x) =n

σ2−

n∑i=1

3(ln xi )2

σ4

`′′(σ; x) =n

σ2−

n∑i=1

3(ln xi )2

σ4=

1

σ2(n − 3n) = −

2n

σ2< 0.



Example


fXi(xi ;σ) =

1

xiσ√

2πe−

(ln xi )2

2σ2 , x > 0.

Find σMLE.

Thus we have that this is indeed a maximum stationary point at

σ =

√1

n

∑ni=1(ln xi )2, i.e.

σMLE =

√√√√1

n

n∑i=1

(lnXi )2.


Properties of the MLE

Equivariant: g(θMLE ) is also the MLE of g(θ)

Asymptotically normal

Consistent (in this course)

*Asymptotically optimal


The Fisher Information

Definition

Let `(θ) be the log-likelihood function of a random sample. The Fisherscore is just its defined as:

Sn(θ) = `′(θ; x).

Definition

The Fisher information is defined as

In(θ) = −E[`′′(θ; x)]

where we swap out xi for Xi .

Theorem (Alternate definition of Fisher Information)

In(θ) = E[(`′(θ; x))2]



Example

For the earlier example, with `′(σ;X ) = − nσ +

∑ni=1

(lnXi )2

σ3 , what is itsFisher information?

−E[`′′(σ;X )] = −E

[n

σ2−

n∑i=1

3(lnXi )2

σ4

]

= −n

σ2+

3n

σ4E[(lnX1)2]

Recalling that X1 ∼ LogNormal(0, σ2) and so we have: lnX1 ∼ N (0, σ2).

= −n

σ2+

3n

σ4(Var(lnX1) + E[lnX1]2)

= −n

σ2+

3n

σ4(σ2 + 02) =

2n

σ2.



Example

For the earlier example, with `′(σ;X ) = − nσ +

∑ni=1

(lnXi )2

σ3 , what is itsFisher information?

−E[`′′(σ;X )] = −E

[n

σ2−

n∑i=1

3(lnXi )2

σ4

]

= −n

σ2+

3n

σ4E[(lnX1)2]

Recalling that X1 ∼ LogNormal(0, σ2) and so we have: lnX1 ∼ N (0, σ2).

= −n

σ2+

3n

σ4(Var(lnX1) + E[lnX1]2)

= −n

σ2+

3n

σ4(σ2 + 02) =

2n

σ2.



Example

Find In(θ) if you’re told that E[Xi ] = θ and

`′(θ) = e−θ + θ

n∑i=1

xi .

The second derivative, with xi replaced by Xi , is

`′′(θ) = −e−θ +n∑

i=1

Xi

so its Fisher information is

In(θ) = E

[e−θ −

n∑i=1

Xi

]

= e−θ −n∑

i=1

E[Xi ] (Why?)

= e−θ −n∑

i=1

θ

= e−θ − nθ




`′′(θ) = −e−θ +n∑

i=1

Xi


In(θ) = E

[e−θ −

n∑i=1

Xi

]

= e−θ −n∑

i=1

E[Xi ] (Why?)

= e−θ −n∑

i=1

θ

= e−θ − nθ




`′′(θ) = −e−θ +n∑

i=1

Xi


In(θ) = E

[e−θ −

n∑i=1

Xi

]

= e−θ −n∑

i=1

E[Xi ] (Why?)

= e−θ −n∑

i=1

θ

= e−θ − nθ


Variance and Standard Error of the MLE

Theorem (Estimation for the Standard Error)

Given θMLE ,

In(θ) Var(θMLE )P→ 1

Therefore

Se(θMLE ) ≈ 1√In(θ)


Variance and Standard Error of the MLE

Example

For the earlier example, with In(σ) = 2nσ2 , estimate Se(σMLE )

Se(σMLE ) ≈ σ√2n


Approximate CI’s: A remark

We can just replace Se(θ) with1√In(θ)

if θ is the MLE .


Asymptotic Optimality: A remark

What it means in English:

If the MLE is asymptotically normal, then the variance of θMLE is less thanthe variance of any other estimator for θ


Hypothesis tests

The basic idea behind Hypothesis testing is that we are essentiallyperforming a proof by contradiction with some level of uncertaintyinvolved.

The initial assumption is known as the null hypothesis and we check it’sprobability based on the chosen alternate hypothesis.


The hypotheses

Definition (Null Hypothesis, Alternate Hypothesis)

In the null hypothesis H0, we claim that our parameter θ takes a particularvalue, say θ0. This is usually the simpler assumption to work under.

In the alternate hypothesis H1, we claim some kind of differentdependencies.

The 2901 alternate hypotheses:

H1 : θ 6= θ0

H1 : θ > θ0

H1 : θ < θ0


p-values

Definition (p-value)

The p value tells you how much evidence there is against the nullhypothesis (depends on alternative hypothesis choice).

The smaller the p-value, the more evidence against the null hypothesis thereis (unlikely for initial assumption to be true).

We compare the p-value to a pre-determined acceptance level, α and rejectthe null if it is lower than this pre-determined α.


Set-up of a Hypothesis Test (Using p-value)

1 State the null and alternate hypotheses

2 State the test statistic, and its distribution if we assume H0 is true

3 Find the observed value of the test statistic

4 Compute the corresponding p-value

5 Draw a conclusion


Exact Test Statistics (Normal samples, known variance)

Suppose we know what the variance σ2 is. We test H0 : µ = µθ.

The null distribution is Z ∼ N (0, 1) .

H1 : Test statistic p-value p-value

µ 6= µ0X − µ0

σ/√n

P (|Z | > |observed value|) 2P (Z > |obs. value|)

µ > µ0 As above P (Z > observed value)

µ < µ0 As above P (Z < observed value)


Exact Test Statistics (Normal samples, unknown variance)

Suppose we estimate the variance σ2 via S2. We test H0 : µ = µ0.

The null distribution is T ∼ tn−1 .

H1 : Test statistic p-value p-value

µ 6= µ0X − µ0

S/√n

P (|T | > |observed value|) 2P (T > |obs. value|)

µ > µ0 As above P (T > observed value)

µ < µ0 As above P (T < observed value)


Exact Tests (Example)

Example 2011 FE Q1

Consider the following table which denotes the number of majorearthquakes each year since 2016. We want to see whether there have beenmore earthquakes over these past few years.

Year 2016 2017 2018 2019

# of major earthquakes 17 12 17 21

Long-term records suggest that we should expect an average of 16.25earthquakes per year. We assume that this data is normally distributed.



Example 2011 FE Q1

Year 2016 2017 2018 2019


We assume that this data is normally distributed.

Step 1: State the hypotheses.

H0 : µ = 16.25 v.s. H1 : µ > 16.25



Example 2011 FE Q1

Year 2016 2017 2018 2019



Step 2: Test statistic under assumption of H0.Using the exact test, we have the following statistic:

T =X − µ0

S/√

4

which follows the following distribution under the null,

T =X − 16.25

S/√

4∼ t3.



Example 2011 FE Q1

Year 2016 2017 2018 2019



Step 3: Find the observed value of the statistic.

x = 16.75

s = 3.68556.

Thus the observed statistic is:

T =16.75− 16.25

3.68556/√

4= 0.27133.



Example 2011 FE Q1

Year 2016 2017 2018 2019



Step 4: Compute the corresponding p-value.

p-value = P

(X − µ0

S/√

4> 0.27133

)= P(T > 0.27133)

= pt(0.27133, df = 3, lower.tail = FALSE)

= 0.40187 > 0.05.

Step 5: Draw a conclusionCan’t refute initial assumption, i.e. fail to reject the null hypothesis under asignificance level of 0.05.



Example 2011 FE Q1

Year 2016 2017 2018 2019



Step 4: Compute the corresponding p-value.

p-value = P

(X − µ0

S/√

4> 0.27133

)= P(T > 0.27133)

= pt(0.27133, df = 3, lower.tail = FALSE)

= 0.40187 > 0.05.

Step 5: Draw a conclusionCan’t refute initial assumption, i.e. fail to reject the null hypothesis under asignificance level of 0.05.


Rejection Region

Definition (α-level)

The α-level, sets a standard upon which we reject H0.

Definition (Rejection region)

Under an α-level, we reject H0 if our observed value lies in the relevantrejection region.


Test Statistics in Exact tests (Normal samples)

Suppose we know what the variance σ2 is. We test H0 : µ = µ0.

The null distribution is Z ∼ N (0, 1) .

H1 : Test statistic Rejection region

µ 6= µ0X − µ0

σ/√n

{|observed value| > z1−α

2

}µ > µ0 As above {observed value > z1−α}µ < µ0 As above {observed value < −z1−α}


Test Statistics in Exact tests (Normal samples)

Suppose we estimate the variance σ2 via S2. We test H0 : µ = µ0.

The null distribution is T ∼ tn−1 .

H1 : Test statistic Rejection region

µ 6= µ0X − µ0

S/√n

{|observed value| > tn−1, 1−α

2

}µ > µ0 As above {observed value > tn−1, 1−α}µ < µ0 As above {observed value < −tn−1, 1−α}


Set-up of a Hypothesis Test (Using Rejection regions)

1 State the null and alternate hypotheses

2 State the test statistic with its distribution if we assume H0 is true,and the α-value

3 Determine the relevant rejection region

4 Find the observed value of the test statistic

5 Draw a conclusion


Earlier Example

We wish to test H0 : µ = 16.25 v.s H1 : µ > 16.25. Our null distribution is

T =X − 16.25

S/√

4∼ t3.

Set the α level to 5%. Our rejection region is

R =

{x − 16.25

s/√

4> t3,0.95

}


Earlier Example

t3,0.95 = qt(0.95, df=3) = 2.35336 (can also find this in given table) so

R = {observed value > 2.35336} .

Our observed value was 0.27133, which doesn’t lie in R . Therefore under a5% level we fail to reject H0.

Same conclusion reached when comparing p-values. As such the chosenmethod doesn’t matter that much, but selecting the correct alternatehypothesis does.


Earlier Example

t3,0.95 = qt(0.95, df=3) = 2.35336 (can also find this in given table) so

R = {observed value > 2.35336} .

Our observed value was 0.27133, which doesn’t lie in R . Therefore under a5% level we fail to reject H0.

Same conclusion reached when comparing p-values. As such the chosenmethod doesn’t matter that much, but selecting the correct alternatehypothesis does.


Error

Definition (Type I Error)

Type I error is when H0 is true, but was rejected.

Definition (Type II Error)

Type II error is when H0 is false, but was accepted.

Lemma (The whole point of α)

The α-level is the significance level. The smaller α is, the more type I erroris controlled.


Asymptotic Test

Assume that θ is an asymptotically normal estimator of θ. We wish to testH0 : θ = θ0 v.s. H1 : θ 6= θ0.

Definition (Wald Test Statistic)

The Wald test statistic is

W =θ − θ0

Se(θ)

with null distribution N (0, 1).

The p-value is

P(|Z | > |observed value|) = 2P(Z > |observed value|)


Quick Example

Example

Suppose in the earlier example we computed σMLE = 0.72706 under asample size n = 20. Assume that σMLE is asymptotically normal and thatwe can estimate Se(σMLE ) ≈ σMLE√

2n. Test the hypotheses

H0 : σ = 0.5 v.s. H1 : σ 6= 0.5

at a significance level of 5%.

We use the Wald statistic

W =σ − 0.5

Se(σ)=

√2× 20(σMLE − 0.5)

σMLE∼ N (0, 1)

under the null hypothesis.

The observed value is 1.97515.


Quick Example

Example

Suppose in the earlier example we computed σMLE = 0.72706 under asample size n = 20. Assume that σMLE is asymptotically normal and thatwe can estimate Se(σMLE ) ≈ σMLE√

2n. Test the hypotheses

H0 : σ = 0.5 v.s. H1 : σ 6= 0.5

at a significance level of 5%.

We use the Wald statistic

W =σ − 0.5

Se(σ)=

√2× 20(σMLE − 0.5)

σMLE∼ N (0, 1)

under the null hypothesis.

The observed value is 1.97515.


Quick Example

Under the null, we know that this is asymptotically standard normal and sowe look at the critical values for a standard normal r.v. Z ∼ N (0, 1).

R =

{W =

∣∣∣∣ σ − 0.5

Se(σ)

∣∣∣∣ > |Z0.975| = 1.96

}As the observed value is in the rejection region 1.97515 > 1.96, we canreject the null hypothesis at a significance level of 5%. Thus we concludethe σ 6= 0.5.


Likelihood Ratio Test (Asymptotic Version)

Consider the following hypothesis test: H0 : θ = θ0 vs H1 : θ 6= θ0.

Asymptotic Likelihood Ratio Test

Λ(θn(X )) = −2 ln

[L(θ0;X )

L(θn;X )

]D→ χ2

1

under certain regularity conditions (this is pretty much assumed to be truewithin this course).

Note: θn is the MLE of θ based on n observations.


Likelihood Ratio Test (Asymptotic Version)

Consider the following hypothesis test: H0 : θ = θ0 vs H1 : θ 6= θ0.

Asymptotic Likelihood Ratio Test

Λ(θn(X )) = −2 ln

[L(θ0;X )

L(θn;X )

]D→ χ2

1

Intuitively speaking, if the null hypothesis assumption is very unlikely to betrue, then L(θ0) would be much smaller than L(θn), i.e. we would reject thenull if Λ(θn) is large enough.

One should note that Λ(θn) ≥ 0, and so we end up having a one-sided testin this case.

P-value isP(Λ(θn(X ) > observed value).


Asymptotic LRT Example

Example

Consider the same hypothesis test as before, but this time we’ll be using theasymptotic likelihood ratio test instead. Hypotheses are:

H0 : σ = 0.5 v.s. H1 : σ 6= 0.5

with σMLE = 0.72706.



Example

H0 : σ = 0.5 v.s. H1 : σ 6= 0.5


We need to sub in the observed values into the relevant distributions to findL(0.5; x) and L(0.72706; x).Noting that Xi ∼ LogNormal(0, σ2) for i = 1, 2, . . . , 20 we have for σ = σ1

L(σ1; x) =1

σ201 × (2π)10

20∏i=1

x−1i × exp

[−

1

2σ21

20∑i=1

(ln xi )2

]

∝ σ−201 × exp

[−

1

2σ21

20∑i=1

(ln xi )2

].



Example

H0 : σ = 0.5 v.s. H1 : σ 6= 0.5


We need to sub in the observed values into the relevant distributions to findL(0.5; x) and L(0.72706; x).Noting that Xi ∼ LogNormal(0, σ2) for i = 1, 2, . . . , 20 we have for σ = σ1

L(σ1; x) =1

σ201 × (2π)10

20∏i=1

x−1i × exp

[−

1

2σ21

20∑i=1

(ln xi )2

]

∝ σ−201 × exp

[−

1

2σ21

20∑i=1

(ln xi )2

].



Subbing everything into the fraction yields:

−2 ln

{L(σ0; x)

L(σMLE ; x)

}= −2 ln

σ−20

0 × exp

[−

1

2σ20

∑20i=1(ln xi )

2

]

σ−20MLE × exp

[−

1

2σ2MLE

∑20i=1(ln xi )2

]

= −2 ln

(σMLE

σ0

)20

exp

[(1

2σ2MLE

−1

2σ20

)20∑i=1

(ln xi )2

]= −40 ln

(σMLE

σ0

)−

[(1

σ2MLE

−1

σ20

)20∑i=1

(ln xi )2

]

We sub things in now and deal with it rickety-split.




−2 ln

{L(σ0; x)

L(σMLE ; x)

}= −2 ln

σ−20

0 × exp

[−

1

2σ20

∑20i=1(ln xi )

2

]

σ−20MLE × exp

[−

1

2σ2MLE

∑20i=1(ln xi )2

]

= −2 ln

(σMLE

σ0

)20

exp

[(1

2σ2MLE

−1

2σ20

)20∑i=1

(ln xi )2

]

= −40 ln

(σMLE

σ0

)−

[(1

σ2MLE

−1

σ20

)20∑i=1

(ln xi )2

]





−2 ln

{L(σ0; x)

L(σMLE ; x)

}= −2 ln

σ−20

0 × exp

[−

1

2σ20

∑20i=1(ln xi )

2

]

σ−20MLE × exp

[−

1

2σ2MLE

∑20i=1(ln xi )2

]

= −2 ln

(σMLE

σ0

)20

exp

[(1

2σ2MLE

−1

2σ20

)20∑i=1

(ln xi )2

]= −40 ln

(σMLE

σ0

)−

[(1

σ2MLE

−1

σ20

)20∑i=1

(ln xi )2

]




Example

H0 : σ = 0.5 v.s. H1 : σ 6= 0.5


Recall from way earlier:

σMLE =

√√√√1

n

n∑i=1

(ln xi )2

n∑i=1

(ln xi )2 = nσ2

MLE

and so we just replace n with 20 and we can get rid of the log-sum thing.



Example

H0 : σ = 0.5 v.s. H1 : σ 6= 0.5


Thus, our test statistic is:

Λ(σMLE (x)) = −2 ln

{L(σ0; x)

L(σMLE ; x)

}

= −40 ln

(σMLE

σ0

)−

[(1

σ2MLE

−1

σ20

)20σ2

MLE

]

= −40 ln

(0.72706

0.5

)− 20

[(1−

0.727062

0.52

)]= 7.31326.



Example

H0 : σ = 0.5 v.s. H1 : σ 6= 0.5


Thus, our test statistic is:

Λ(σMLE (x)) = −2 ln

{L(σ0; x)

L(σMLE ; x)

}

= −40 ln

(σMLE

σ0

)−

[(1

σ2MLE

−1

σ20

)20σ2

MLE

]

= −40 ln

(0.72706

0.5

)− 20

[(1−

0.727062

0.52

)]= 7.31326.



Example

H0 : σ = 0.5 v.s. H1 : σ 6= 0.5


Reject or fail to reject the null?

p-value:

p-value = P(Λ(σMLE (X )) > Λ(σMLE (x)))

= P(χ21 > 7.31326)

= pchisq(7.31326, df = 1, lower.tail = FALSE)

= 0.0068448 < 0.05.

Thus at a significance level of 5%, we can reject the null hypothesis.



Example

H0 : σ = 0.5 v.s. H1 : σ 6= 0.5


Reject or fail to reject the null?

p-value:

p-value = P(Λ(σMLE (X )) > Λ(σMLE (x)))

= P(χ21 > 7.31326)

= pchisq(7.31326, df = 1, lower.tail = FALSE)

= 0.0068448 < 0.05.

Thus at a significance level of 5%, we can reject the null hypothesis.


Table of Contents





Working with S = SX


3 Coolio Doolio


Good Luck on your finals!


Documents

MATH2901 Final Revision · 2020. 8. 13. · MATH2901 Final Revision Part II: Statistical Inference Presenter: Henry Lam Based o Rui Tong’s 2018 Slides UNSW Society of Statistics/UNSW