Concept of Power

http://www.youtube.com/watch?feature=player_detailpage&v=7yeA7a0uS3A

Power is the probability of rejecting the null hypothesis

• The power of a hypothesis test is the probability that it will lead to a rejection of the null hypothesis.

• When is false, power = 1 -

What affects power?

1. The significance level of the test. If all other things are held constant, then as increases, so does the power of the test. This is because a larger means a larger rejection region for the test and thus a greater probability of rejecting the null hypothesis.

2. The sample size n. As n increases, so does the power of the significance test. This is because a larger sample size narrows the distribution of the test statistic. The hypothesized distribution of the test statistic and the true distribution of the test statistic (should the null hypothesis in fact be false) become more distinct from one another as they become narrower, so it becomes easier to tell whether the observed statistic comes from one distribution or the other. (Remember that this takes more time and will cost more!)

3. The inherent variability in the measured response variable. As the variability increases, the power of the test of significance decreases. One way to think of this is that a test of significance is like trying to detect the presence of a “signal,” such as the effect of a treatment, and the inherent variability in the response variable is “noise” that will drown out the signal if it is too great. Researchers can’t completely control the variability in the response variable, but they can sometimes reduce it through especially careful data collecting and conscientiously uniform handling of experimental units or subjects. The design of the study may also reduce unexplained variability, and one primary reason for choosing such a design is that it allows for increased power without necessarily having exorbitantly costly sample sizes.

4. The difference between the hypothesized value of a parameter and its true value. This is sometimes called the “magnitude of the effect” in the case when the parameter of interest is the difference between parameter values (say, means) for two treatment groups. The larger the effect, the more powerful the test is. This is because when the effect is large, the true distribution of the test statistic is far from its hypothesized distribution so the two distributions are distinct, and it’s easy to tell which one an observation came from. The intuitive ideas is simply that it’s easier to detect a large effect than a small one.

Example: You send a friend into your bedroom to look for a book that you really need. He comes back and says “it isn’t there”. What do you

conclude? Is the book there or not? There is no way to be sure.

So let’s express the answer as a probability. The question you really want to answer is: “If the book really is in the bedroom, what is the chance your friend would have found it”?The answer depends on the answers to these questions:• How long did he spend looking? If he looked for a long time, he is more likely to have

found the book.• How big is the book? It is easier to find a text book than a tiny paper back book.• How messy is the bedroom? If the bedroom is a real mess, he was less likely to find

the book than if it is super organized. • How large is the bedroom? It would be easier to find the book in a smaller room

than a larger one. So if he spent a long time looking for a large book in an organized bedroom, there is a high chance that he would have found the book it were there. So you can be quite confident of his conclusion that the book isn’t there. If he spent a short time looking for a small book in a messy bedroom, his conclusion that “the book isn’t there” doesn’t really mean very much.

Analogy with sample size and powerSo how is this related to computing the power of a completed experiment? The question about finding the book, is similar to asking about the power of a completed experiment. Power is the answer to this question: If an effect (of a specified size) really occurs, what is the chance that an experiment of a certain size will find a “statistically significant” result?• The time searching the bedroom is analogous to sample size. If you collect more

data you have a higher power to find an effect.• The size of the book is analogous to the effect size you are looking for. You always

have more power to find a big effect than a small one.• The messiness of the bedroom is analogous to the standard deviation of your data.

You have less power to find an effect if the data are very scattered.If you use a large sample size looking for a large effect using a system with a small standard deviation, there is a high chance that you would have obtained a “statistically significant effect” if it existed. So you can be quite confident of a conclusion of “no statistically significant effect”. But if you use a small sample size looking for a small effect using a system with a large standard deviation, then the find of “no statistically significant effect” really isn’t very helpful.

Conclusion

1. The larger the sample size, the higher the power of the test. 2. The larger the significance level, , the higher the power of the

test.3. The smaller the standard deviation, the larger the power.4. The larger the size of the discrepancy between the hypothesized

value and the true value of the population characteristic, the higher the power.

𝛼