GGGC8024 KAEDAH PENYELIDIKAN KUANTITATIF TUTORIAL 4

EFFECT SIZE AND CONFIDENCE INTERVAL REPORTING PRACTICES

Poor conduct or design of research must always be a concern. The less ethical researcher might

alter how they report findings, including only information that supports his or her hypothesis, or

present results in such a way as to misinform or mislead the reader. Today’s educational

atmosphere is highly laden with assessment and accountability issues. Researchers need to be

attuned to the need for effectively communicating the practical impact of research results in

addition to merely reporting findings that are statistically significant. The use of effect sizes and

confidence intervals can be key elements in aiding in this communication. Effect sizes provide a

means of measuring practical significance and confidence intervals convey the precision of

results. ‘Effect size’ is simply a way of quantifying the size of the difference between two

groups. It is easy to calculate, readily understood and can be applied to any measured outcome

in Education or Social Science. It is particularly valuable for quantifying the effectiveness of a

particular intervention, relative to some comparison. It allows us to move beyond the simplistic,

‘Does it work or not?’ to the far more sophisticated, ‘How well does it work in a range of

contexts?’ Moreover, by placing the emphasis on the most important aspect of an intervention –

the size of the effect – rather than its statistical significance (which conflates effect size and

sample size), it promotes a more scientific approach to the accumulation of knowledge. For

these reasons, effect size is an important tool in reporting and interpreting effectiveness.

Appropriate, effective and meaningful reporting practices are critical for communicating

research results correctly. Thoughtful interpretation of research and the ability of readers to sift

through good and bad research have gone beyond being merely a part of courses in research

methodology. Effect size has become increasingly recognized as an important statistic that needs

to be reported. Numerous field experts have stressed the need for effect size reporting throughout

the social sciences, including education. (Nix & Barnette, 1998). Both the fourth and fifth

editions of the American Psychological Association (1994 and 2001) highly recommend that

researchers report effect sizes. Often termed practical significance or, sometimes substantive

significance (Robinson & Levin, 1997), effect sizes provide a different, albeit related, piece of

information about how a treatment or other variable is impacting the issue of interest.

There are various effect size indices available as well as different terms used when referencing

effect sizes. Some of the various descriptors for effect size estimates include percent of variance

accounted for, strength of association, and magnitude of effect, among others (Plucker, 1997).

Additionally, correlation coefficients such as Spearman rho and the Pearson Product Moment

Correlation Coefficient are sometimes considered a type of effect size (Plucker 1997). Hedge’s

g, Glass’s ∆ , and Cohen’s d are all variations of effect sizes for differences in means between

two groups (Rosenthal, 1994 and Cohen, 1988). Effect sizes for studies using statistical methods

examining correlational relationships or variance relationships have measures such as eta-

squared (η2), R-squared (R2), and omega squared (ω 2) available for use (Snyder & Lawson,

1993).

Confidence intervals have been accepted for quite some time as a useful method for describing

statistical parameter estimates such as sample means. Although there are issues associated with

the lack of universal use of confidence intervals in research reporting, there have been recent

advances in using confidence intervals for statistics other than the mean and standard deviation.

The use of confidence intervals for other statistical estimates is quickly growing as an improved

way of reporting more informative measures of estimates than point estimates. Cumming and

Finch (2001) provide four reasons for researchers to give confidence interval estimates when

reporting research findings: (1) confidence intervals provide both point and interval information

that improves understanding and interpretation, (2) the use of intervals enhances the practice of

traditional null hypothesis reporting, it does not negate it. That is, if a specific null value is being

tested and is found to fall outside of the computed interval, it is rejecting the null hypothesis, but

with more precision, (3) the use of CIs may serve meta-analytical methods which focus on

estimation using many sources of study data, and (4) information about the precision of the study

and subsequent.

To sum up, an effect size is simply an amount of something of interest. It can be as simple as a

mean, a percentage increase, or a correlation; or it may be a standardized measure of a

difference, a regression weight, or the percentage of variance accounted for. Most research

questions in the social sciences are best answered by finding estimated effect sizes, meaning

point estimates of the true effect sizes in the population.

A confidence interval is an interval estimate of a population effect size. It is an interval that

extends above and below the point of effect size estimate; it indicates the precision of the point

estimate. In the social sciences, statistical analysis is still dominated by null hypothesis

significance testing. However, there is extensive evidence that null hypothesis significance

testing is poorly understood, frequently misused, and often leads to incorrect conclusions. That

is why reporting appropriate, effective and meaningful report are critical for communicating

research results correctly.

References

American Psychological Association. (2009). Publication manual of the American Psychological

Association(6th ed.). Washington, DC.

Belia, S., Fidler, F., Williams, J., & Cumming, G. (2005). Researchers misunderstand confidence

intervals and standard error bars. Psychological Methods, 10, 389-396.

Cumming, G., & Finch, S. (2001). A primer on the understanding, use and calculation of

confidence intervals based on central and noncentral distributions. Educational and

Psychological Measurement, 61, 530-572.

Cumming, G., & Finch, S. (2005). Inference by eye: Confidence intervals, and how to read

pictures of data.American Psychologist, 60, 170–180.

Cumming, G., Williams, J., & Fidler, F. (2004). Replication and researchers’ understanding of

confidence intervals and standard error bars. Understanding Statistics, 3, 299–311.

Fidler, F., & Thompson, B. (2001). Computing correct confidence intervals for ANOVA fixed

and random effect sizes. Educational and Psychological Measurement, 61, 575-604.