Education 795 Class Notes

Data Analyst PitfallsDifference Scores

Effects Sizes

Note set 12

Today’s Agenda

Announcements (ours and yours)

Data Analyst Pitfalls

Difference Scores

Effect Sizes

Multiple Comparisons

Data Analyst

We meet the qualitative paradigm and position ourselves in the research, quantitative discourse now includes how we the researcher affects the following:

What is to be investigated

How it is to be done

What are the facts? Hypotheses?

What are the findings? Intepretations?

Pitfalls

Improper analyses usedWe leave out a lot of the details that enable others to properly evaluate the researchUse rules that have no consensus in the fieldSome try to replicate with the same data to refute results… (we’ve seen this before)We often ignore Validity and it is actually the crux of what we concludeWe given limited attention to Reliability

Difference Scores

The most common difference scoreCalculate Pretest-Postest for each subjectAttain mean difference scores for treatment and comparison groupsTest the difference of the difference scores for statistical signficance

LimitationThis requires the pre and post test to have the same factor structure.

Problems with Difference Scores

Often referred to as ‘gain scores’Improvement ranges are often not equal for individuals along the continuum.

Example: Room for improvement is greater for those that start lower on the pre-test scale.

If there is a ceiling effect (too easy) or a floor effect (too hard) difference scores are meaningless because there will likely be no changeShort time intervals often don’t allow change

Alternatives to Raw Gain Scores

Standardized gain scores

Residualized gain scores

ANCOVA design predicting the post-test controlling for the pre-test

Here we lose some valuable information about individual change

Effect Sizes

Effect sizes are used to refer to magnitude, importance and meaningfulness. Cohen (1988) defined ES as “the degree to which the phenomenon is present in the population”. For Cohen’s d (specifically designed to asses the difference between groups), the rule is .2 small, .5 medium and .8 as large effects. For correlation coefficients, .1 small, .3 medium, .5 large effects

Connecting ES to Power

Cohen (1962) showed that the median power to detect small, medium and large effects was .17, .46 and .89.

In other words, for a small effect, holding sample sizes constant, a test will only correctly reject the null hypothesis 17% of the times

For large effects present in the population, tests will correctly reject the null hypothesis 89% of the time.

Note that large effects are rarely found in sociobehavioral research

Effect Sizes

There are two major classes of effect sizes (not counting a third "miscellaneous" category described by Kirk (1996)):

(a) variance-accounted-for effect sizes analogous to a squared correlation coefficient

(b) standardized mean differences

    

Types of Effect Sizes

Squared Multiple Correlation Coefficient We’ve seen this before, it is R2

R2=Sum of Squares Regression (Effect) /Sum of Squares TotalPercent of variance in the outcome explained

Note there is no effect of sample size on this statistic. This statistic is similar to 2 or Eta squared for ANOVA.Note we can calculate 2 for each separate main effect in an ANOVA table.

Effect Sizes for Means1. Cohen's d = M1 - M2 / spooled

    where spooled = [((s1)²+ (s2)²) / 2]

Example: Assume equal n’s. M1 = 50, M2 = 60, s1=10, s2=15

spooled = sqrt((102 + 152)/2) = sqrt(325/2) = sqrt(162.5) = 12.7

Cohen’s d=60-50/12.7 = .78 ---LARGE EFFECT

Effect Size for Means

We can also calculate an effect size by the t-statistic.

Cohen's d = 2t(df)

In 1999, the APA Task Force on Statistical Inference published its report:http://www.apa.org/journals/amp/amp548594.html

Laptop Project

In groups of 4Run a Regression using at least one group membership variable. Use the descriptive statistics to calculate an effect size for the group, compare that to the statistical significance.

Present your results to the class.