No criminal on the run

The concept of test of significance

FETP India

Competency to be gained from this lecture

Formulate and test null hypotheses

Key issues

• Null and alternate hypotheses• Type I and Type II errors• Statistical testing

What is the question at hand?

• Estimating a quantity?• Test a hypothesis?

Hypotheses

Taking into account the sampling variation in decision-making

• Studies are on sample of subjects and not on an entire population

• There is sampling variation• Allowance should be given for sampling

variation while a decision taking

Hypotheses

Rationalizing decision-making

• Research studies test hypotheses Experiment and data collection

• Hypotheses are tested on the basis of inference from available data

• Considering a difference as significant may be subjective

• The concept of statistical significance is a decision-making tool to make a subjective decision objective

Hypotheses

A man is brought to court accused of a crime

• The judge needs to start from the hypothesis that the person is innocent

• The evidence is brought in: Fingerprints Pictures

Hypotheses

Assessing whether the evidence is caused by chance or not

• The judge assesses whether the evidence could be due to chance

• If the probability that the evidence is caused by chance is high: The judge accepts the hypothesis of

innocence

• If the probability that the evidence is caused by chance is low: The judge rejects the hypothesis of innocence

Hypotheses

Hypotheses formulated by epidemiologists

• Ho: Null hypothesis (=“innocence”) The difference observed is caused by

chance, or sampling variation

• H1: Alternate hypothesis The probability that the difference observed

is caused by chance alone is low

Hypotheses

From sampling distribution to hypothesis testing

• Epidemiologists decide a critical / rejection region That decision is arbitrary

• If the value falls under an extreme, rejection region, the null hypothesis is rejected

Hypotheses

Type I and type II errors

• Type I Rejection error, also called alpha error Rejecting the null hypothesis when it is true Punishing an innocent Particularly unacceptable to society Must be minimized

• Type II Acceptance error, also called beta error Accepting the null hypothesis when it is false Releasing a guilty person charged

Errors

Balancing the risk of errors

• If the judge wants to always avoid type I error, he can release everyone He will always commit the type II error

• If the judge wants to always avoid type II error, he can charge everyone He will always commit the type I error

• To balance the risk of errors, we will fix one error and try to minimize the other

Errors

Which error is more important?

Hypertension HIV

Effective drugs already available?

Many Few

Concluding that new treatment is better when it is not

UnfortunateNot so

unfortunate

Concluding that new treatment is no better when it is better

Not so unfortunate

Very unfortunate

Which error is more important? Type I Type II

Errors

Examples of errors

• An example where type I error is important If a new drug becomes available for HIV, we

must minimize the risk to reject a drug that would work

• An example where type II error is important If a new drug becomes available for

hypertension, since lots of anti-hypertensive are already available, we cannot take a risk and can only accept a drug that is completely safe

Errors

Behind errors are the right decisions

• 1-alpha Probability of accepting the null hypothesis

when it is the right decision

• 1-beta Probability of rejecting the null hypothesis

when it is the right decision Also called statistical power

Errors

Alpha and beta error

Decision

Accept Ho Reject Ho

Truth

Ho is trueGood

decision1-alpha

Alpha error

Ho is false Beta errorGood

decision1 - beta

Errors

Populationof 10,000

Mean height = 65”S.d. = 10”

66”

63” 65” 64”67”

= 1

Sampling fluctuation in samples of 100 subjects for height

measurement• Even when statistically

sound sampling techniques are employed The mean in samples of

100 will not necessarily be 65”

Variation from sample to sample

• This must be taken into account when interpreting differences

• This method is called a significance test

Sampling error of mean

Testing

Magnitude of allowance

• Consider an expected difference of 0% 1%, 2%, 3%

• Not large

20%, 30%• Large, not willing to consider the difference as 0%

• WHY? If the true difference is 0%, the chance

(probability) of getting a difference exceeding 20% is very small

Testing

Decision rule

• Formulate a decision rule based on the probability of getting the observed difference Null hypothesis (Ho)

• Assuming Ho is true, compute the probability of obtaining the observed difference

• If the probability is low: Reject Ho

• Else, accept HoTesting

Choosing a rejection level

• The definition of low probability is subjective• Conventionally:

Low probability = 5% (P=0.05) If P < 0.05, the observed difference is ‘significant

(Statistically) P< 0.01, sometimes termed as ‘Highly significant’

• Computation of P-values: Statistical exercise Depends on the nature of data and design of the study

• Necessary condition: Probability sample No test of significance on convenience or quota

samplesTesting

Population of 10, 000

A random sampleof size 100 is drawn

Mean height = 68”

Concept of test of significance

• Question: Could the population mean

be 65” ?• Hypothesis:

Population mean = 65”• Question:

What is the probability of obtaining a sample mean of 68” from this population when sample size = 100 ?

• If this probability is small (e.g. < 5%) Reject the Hypothesis

• If not, accept the Hypothesis

Testing

Testing

Test of significance: Computation of probability

• Observed mean = 68” Postulated mean = 65”• Standard deviation = 10” Sample size = 100• Sampling error (s.e.) of mean = 10 / 100 = 1• Compute: Observed mean - Postulated mean 68-65 ----------------------------------------- = -------- = 3 s.e. of mean 1• Critical value for significance at 5% level = 1.96• Since 3 > 1.96, the difference is statistically significant• Exact probability = 0.0027 , i.e., 0.27%

What if the distribution is not normal?

• Transform the data (e.g., drug concentration, cell counts) to some other scale to obtain a normal distribution e.g., logarithm, square root

• If not feasible, and provided sample size exceeds 30, make use of the result that mean is approximately normally distributed

Testing

Estimating the sample size

• The epidemiologist examines the willingness to commit: Alpha error Beta error

• Sample size calculation is the step at which decisions will be made in this respect

Testing

Interpretation of significance

• “Significant” does not necessarily mean that the observed difference is REAL or IMPORTANT

• “Significant” only means that it is unlikely (<5%) that the difference is due to chance

• Trivial differences can be statistically significant if they are based on large numbers

Testing

Interpretation of non-significance

• “Non - significant” does not necessarily mean that there is no real difference

• “Non - significant” means only that the observed difference could easily be due to chance Probability of at least 5%

• There could be a real or important difference but due to inadequate sample size we might have obtained a non-significant result

Testing

Significance does not systematically mean causation: Potential

explanations for a significant association

x Chance: Addressed by the significance test

? Bias? Confounding factor? Causation

Consider after the first three have been ruled out

Test for causality criteria

Testing

The choice of a one-sided test depends

upon the alternate hypothesis• One-sided test• When the alternate hypothesis is in one

direction• The actual P-values need to be quoted

instead of stating just p < 0.05 or p < 0.01

Testing

Testing

Quick checklist for statistical testing

A statistical test is indeed neededThe test used is adapted The test is calculated correctlyThe interpretation of the test is

appropriate

Key messages

• Under the null hypotheses, differences observed are caused by chance alone

• Type I error consists in rejecting the null hypothesis when it is true while type II error consists in accepting the null hypothesis when it is false

• Statistical tests estimate the probability that a difference observed may be caused by chance alone