Alpha to Omega and Beyond! Presented by Michael Toland Educational Psychology & Dominique Zephyr...

Alpha to Omega and Beyond!

Presented by Michael Toland

Educational Psychology &

Dominique Zephyr Applied Statistics Lab

Hypothetical ExperimentSuppose you measured a person’s perceived

self-efficacy with the general self-efficacy scale (GSES) 1,000 times

Suppose the measurements we observe vary between 13 and 17 points

The person’s perceived self-efficacy score has seemingly remained constant, yet the measurements fluctuate

The problem is that it is difficult to get at the true score because of random errors of measurement

True Score vs. Observed Score An observed score is made up of 2 components

Observed Score = True score + Random Error

True score is a person’s average score over repeated (necessarily hypothetical) administrations

The 1,000 test scores are observed scores

Reliability Estimation:Classical Test Theory (CTT)

Across people we define the variability among scores in a similar way

SX2 = ST

2 + SE2

Reliability is the ratio of true-score variance relative to its observed-score variance

According to CTT xx’= σ2

T/σ2x

Unfortunately, the true score variability can’t be measured directly.

So, we have figured out other ways of estimating how much of a score is due to true score and measurement error

Concept of Reliability Reliability is not an all-or-nothing concept, but there

are degrees of reliability

High reliability tells us if people were retested they would probably get similar scores on different versions of a test

It is a property of a set of test scores – not a test

But we are not interested in just performance on a set of items

Why is reliability important?

Reliability affects not only observed score interpretations, but poor reliability estimates can lead to deflated effect size estimates nonsignificant results

When something is more reliable it is closer to the true score The more we can differentiate among individuals of

different levels

When something is less reliable it is further away from the true score The less it differentiates among individuals

Measurement Models Under CTT

Assumptions ParallelTau

EquivalentEssentially

Tau-Equivalent Congeneric

Unidimensional Y Y Y Y

Equal item covariances Y Y Y

Equal item-construct relations

Y Y Y

Equal item variance Y Y

Equal item error variance

Y

Alpha ()Cronbach

(1951)

Omega () McDonald

(1970, 1999)

Coefficient Alpha

covavg = average covariance among items

k = the number of items

ScaleV = scale score variance

VScale

kavg2)(cov

ˆ

Why have we been using alpha for so long?

Simple equation

Easily calculated in standard software (default)

Tradition

Easy to understand

Researchers are not aware of other approaches

Problems with alpha ()

Unrealistic to assume all items have same equal item-construct relation and item covariances are the same

Underestimates population reliability coefficient when congeneric model assumed

What do we gain with Omega?

Does not assume all items have the same item-construct relations and equal item covariances (assumptions relaxed)

More consistent (precise) estimator of reliability

Not as difficult to estimate as folks come to believe

One-Factor CFA Model

Item3

3

1

Item2

2

1

Item1

1

1

Item6

6

Item5

5

Item4

4

111

1

Perceived General Self-Efficacy

1 2 3 45 6

Coefficient Omega

k

iii

k

ii

k

ii

1

2

1

2

1

i = factor pattern loading for item i k = the number of items ii = unique variance of item I Assumes latent variance is fixed at 1 within

CFA framework

Include CI along Reliability Point Estimate

Measures a range that estimates the true population value for reliability, while acknowledging the uncertainty in our reliability estimate

Recommended by APA and most peer reviewed journals

Mplus input for alpha ()

Mplus output for alpha ()

Mplus input for omega

Mplus output for omega

APA style write-up for coefficients with CIs

= .61, Bootstrap corrected [BC] 95% CI [.56, .66]

= .62, Bootstrap corrected [BC] 95% CI [.56, .66]

Limitations of CTT Reliability Coefficients and Future QIPSR Talks

Although a better estimate of reliability than alpha, CTT still assumes a constant amount of reliability exists across the score continuum

However, it is well known in the measurement community that reliability/precision is conditional on a person’s location along the continuum

Modern measurement techniques such as Item Response Theory (IRT) do not make this assumption and focus on items instead of the total scale itself

References Revelle, W., Zinbarg, R. E. (2008). Coefficients Alpha, Beta, Omega, and the glb: Comments on Sijtsma.

Psychometrika, 74, 145-154. doi:10.1007/s11336-008-9102-z Gadermann, A. M., Guhn, M., & Zumbo, B. D. (2012). Estimating ordinal reliability for Likert-type and ordinal item

response data: A conceptual, empirical, and practical guide. Practical Assessment, Research and Evaluation, 17. Retrieved from: http://pareonline.net/getvn.asp?v=17&n=3

Zumbo, B. D., Gadermann, A. M., & Zeisser, C. (2007). Ordinal versions of coefficients alpha and theta for Likert rating scales. Journal of Modern Applied Statistical Methods, 6. Retrieved from: http://digitalcommons.wayne.edu/jmasm/vol6/iss1/4

Starkweather, J. (2012). Step out of the past: Stop using coefficient alpha; there are better ways to calculate reliability. Benchmarks RSS Matters Retrieved from: http://web3.unt.edu/benchmarks/issues/2012/06/rss-matters

Sijtsma, K. (2009). On the use, the misuse, and the very limited usefulness of Cronbach's alpha. Psychometrika, 74, 107-120.doi:10.1007/s11336-008-9101-0

Peters., G-J. Y. (2014). The alpha and the omega of scale reliability and validity: Why and how to abandon Cronbach's alpha and the route towards more comprehensive assessment of scale quality. The European Health Psychologist, 16, 56–69. Retrieved from: http://www.ehps.net/ehp/issues/2014/v16iss2April2014/4%20%20Peters%2016_2_EHP_April%202014.pdf

Crutzen, R. (2014). Time is a jailer: What do alpha and its alternatives tell us about reliability?. The European Health Psychologist, 16, 70-74. Retrieved from: http://www.ehps.net/ehp/issues/2014/v16iss2April2014/5%20Crutzen%2016_2_EHP_April%202014.pdf

Dunn, T., Baguley, T., & Brunsden, V. (2014). From alpha to omega: A practical solution to the pervasive problem of internal consistency estimation. British Journal of Psychology, 105, 399-412. doi:10.1111/bjop.12046

Geldhof, G., Preacher, K. J., & Zyphur, M. J. (2014). Reliability estimation in a multilevel confirmatory factor analysis framework. Psychological Methods, 19, 72-91. doi:10.1037/a0032138

Acknowledgements

APS Lab members Angela Tobmbari Zijia Li Caihong Li Abbey Love Mikah Pritchard