Hypothesis Testing

Distribution of Estimator

• To see the impact of the sample on estimates, try different samples

• Plot histogram of answers– Is it “normal”

• Key point: even if we have the correct model we could get an answer that is way off just because we are unlucky in the sample.– Sampling error

• How do we know if we have been unlucky? How can we minimise the chances of bad luck?

Hypothesis Testing

• Non-trivial because of sampling distribution– See examples

• Is the evidence consistent with the hypothesis being true?

• General approach– what would the distribution look like if the (null)

hypothesis was true?– Where on the distribution is our estimate?– How likely is our estimate to occur when the

hypothesis is true?– How likely is the hypothesis to be true?

A Criminal Trial

• Metaphor of a criminal trial

• We have the hypothesis that the accused is innocent

• We ask if the evidence is consistent with the hypothesis being true

• If not we can reject the hypothesis

• If yes we fail to reject – Note: don’t “accept” the hypothesis

The Distribution of

• Typically we will not observe the distribution, we will have only one estimate

• We could create the distribution– Lots of small samples: but loose consistency

• Classical approach is to assume that the distribution is “normal” – i.e. use stylized distribution in place of the

histograms

Normality• The assumption of normality is made for

convenience. Is it reasonable?• Two ways in which it could be true

– Data could be normal– Central limit theorem applies

• Data is normal if residual is normal– Bell shape is reasonable– Normal is more restrictive: approximation?– Since OLS estimator is a linear function of data it will

also have a normal distribution– This is a small sample property i.e. applies for all

sizes

CLT

• Central limit theorem states that even if the residuals are not normal OLS will be

• Process of applying the OLS formula creates a new random variable (the estimator) that has a normal distribution.– See example

• This is an asymptotic property i.e. for large samples

• Implications for Classical hypothesis testing– Data must be normal– Or you must have lots of it (100+ obs)

Distribution of OLS

• OLS is unbiased so the distribution is centered on the true value

• Its variance depends on the variance of the residuals– Estimate produced by stata

• Using either the CLT or the normality of u, OLS is normally distributed

N

ii

OLS

N

ii

OLS

OLS

N

ii

N

iii

OLS

i

iii

xxN

xxVar

E

yy

yyxx

uVar

uxy

1

2

2

1

2

2

1

2

1

2

)(,~

)()(

)(

)(

))((

)(

TRUE-1.96*se TRUE TRUE+1.96*se

95% of distribution

Mechanism for Hypothesis Test

1. State the Hypothesis we want to testH0: MPC = 0.7 H1: MPC ≠ 0.7

2. Calculate the distribution of OLS assuming that H0 is true.

3. Find our estimate on the distribution4. What is the probability that our estimate

would have come from this distribution?5. Does this lead us to believe the null

hypothesis?

0.72

95% of distribution

• Any estimate is possibly consistent with any hypothesis– Always the possibility of a real fluke even with no

mistakes– But some are clearly more likely than others

• We can measure the probability of an estimate occurring if we know the distribution of the estimator

• Clearly we are unlikely to get an estimate of 0.72 if the true value is 0.7– But it is possible!

– P(bOLS≥0.72| MPC=0.7)=0.000001

– Usually use 5%,10% or 1% as the threshold value

Another Example

Use the individual consumption data1. State the Hypothesis we want to test

H0: MPC = 0.81 H1: MPC ≠ 0.81

1. Calculate the distribution of OLS assuming that H0 is true.

2. Find our estimate on the distribution3. What is the probability that our estimate

would have come from this distribution?4. Does this lead us to believe the null

hypothesis?

0.80

95% of distribution

• Again any estimate could be consistent with this hypothesis

• What is the probability of a fluke now?

• Clearly fairly likely to get an estimate of 0.801 if the true value is 0.81

• Probability is much greater than 5%– So we “cannot reject” the hypothesis

– P(bOLS≤0.801| MPC=0.81)=0.11

• How did I get this “p-value”?

Some Comments• Criminal Trial metaphor: The null

hypothesis (innocence) will only be overturned if there is overwhelming evidence

• What constitutes “overwhelming”– Not with regard to the size of the difference

between values for(compare the two examples).

– it is the difference in probability – i.e. the distance on distribution

Test Statistics

• Clearly some duplication between our two tests even though they were on different data

• So we can systematize things a little better

• We make use of property of normal distributions

)1,0(~)(

)(,~

Nse

Z

VarN

OLS

OLS

OLSOLS

• So now we only ever have to deal with one distribution, the “standard” normal

• Note also how the construction of Z explicitly removes the issue of scale

• Test procedure– State Hypothesis

– Calculate Z assuming H0 is true

• Now we can compare the calculated values of Z with the standard normal distribution

Our Two Examples

• Aggregate Consumption:1. State the Hypothesis we want to test

H0: MPC = 0.7 H1: MPC ≠ 0.7

1. Calculate the test statistic assuming that H0 is true.– Z=(0.7216591 -0.7)/(0 .0022551)=9.6

2. Find our estimate on the distribution– Either find the test statistic on the standard normal distribution

– Or compare with one of the traditional threshold (“critical”) values: 2.58(1%), 1.96 (5%), 1.64(10%)

3. |Z|>all the critical values and Prob (Z>9.6)=0

4. So we reject the null hypothesis

9.6

95% of distribution

• Comment– We will reject the idea that MPC = 0.7 if there is

overwhelming evidence that MPC is bigger or smaller

– The evidence is our estimate (0.72)– Is this big enough?– Remove the scale from the problem by calculating the

test statistic: Z=9.6 – Is this big enough?– Traditionally 1.96 would be the “critical value”

because of 5% probability of Z>1.96 as fluke– “beyond reasonable doubt”– Free to decide for ourselves (p-value)

• Individual Consumption1.State the Hypothesis we want to test

H0: MPC = 0.81 H1: MPC ≠ 0.81

1.Calculate Z assuming that H0 is true– Z=(0.8012171-0.81)/0 .0074599=-1.17

2. Compare with critical values– |z|<1.96– Prob (Z<-1.17)=0.12

3.Cannot Reject the Null Hypothesis

95% of distribution

• Comment– We will reject the idea that MPC = 0.81 if there

is overwhelming evidence that MPC is bigger or smaller

– The evidence is our estimate (0.80)– Is this big enough?– Remove the scale from the problem by

calculating the test statistic: Z=-1.17 – Is this big enough?– Traditionally 1.96 would be the “critical value”

because of 5% of Z<-1.96 as fluke– p-value of 0.12

Issues in Hypothesis Testing

• Test of significance

• “t-test”

• Rule of thumb

• Significance level

Test of Significance

• A test of H0: = 0 is given the special name of “test of significance”

• Test statistic is simpleZ=(OLS – se(OLS)= OLS/se(OLS)Which is calculated by most statistical software

• Simple eyeball test of significance• Variable is or is not “statistically

significant”• Not the same as economically significant

t-test• Strictly speaking the Z test is only valid when

, the variance of u is known as it is used to calculate se()

• will almost never be known and will have to be estimated

• When it is estimated the distribution of the estimator (and therefore the test statistic) is no longer normal

• Has a t-distribution with N-K degrees of freedom

• Fortunately t≈Z when N-K is large

Rule of Thumb• Easy to “learn off” test procedure

– Form the test statistic– Reject hypothesis if test statistic>2 in absolute

terms– Useful for “eyeball” tests of significance

• Stata command “test” does the whole procedure automatically– But does “F-test”– This is the square of t-test

Significance Level

• Probability of a fluke• When we choose a critical value we choose a

significance level also:– 2.58(1%), 1.96 (5%), 1.64(10%)

• If we reject the null because |Z|>1.96, we say we reject the null at the 5% significance level.

• We acknowledge that there is a 5% chance that Z>1.96 even though the null is true

• This is type 1 error: Rejecting a true Null– Criminal trial: Convicting the innocent

• The test is set up make this as low as possible– i.e. reject only if overwhelming evidence

• Why not make it zero? Cant because would never reject any null– Criminal: always acquit

• This matters because setting up a hypothesis is setting up a procedure that is deliberately biased against rejecting

• Make sure that is what you want for your null