Upload
darlene-black
View
216
Download
2
Embed Size (px)
Citation preview
Alpha to Omega and Beyond!
Presented by Michael Toland
Educational Psychology &
Dominique Zephyr Applied Statistics Lab
Hypothetical ExperimentSuppose you measured a person’s perceived
self-efficacy with the general self-efficacy scale (GSES) 1,000 times
Suppose the measurements we observe vary between 13 and 17 points
The person’s perceived self-efficacy score has seemingly remained constant, yet the measurements fluctuate
The problem is that it is difficult to get at the true score because of random errors of measurement
True Score vs. Observed Score An observed score is made up of 2 components
Observed Score = True score + Random Error
True score is a person’s average score over repeated (necessarily hypothetical) administrations
The 1,000 test scores are observed scores
Reliability Estimation:Classical Test Theory (CTT)
Across people we define the variability among scores in a similar way
SX2 = ST
2 + SE2
Reliability is the ratio of true-score variance relative to its observed-score variance
According to CTT xx’= σ2
T/σ2x
Unfortunately, the true score variability can’t be measured directly.
So, we have figured out other ways of estimating how much of a score is due to true score and measurement error
Concept of Reliability Reliability is not an all-or-nothing concept, but there
are degrees of reliability
High reliability tells us if people were retested they would probably get similar scores on different versions of a test
It is a property of a set of test scores – not a test
But we are not interested in just performance on a set of items
Why is reliability important?
Reliability affects not only observed score interpretations, but poor reliability estimates can lead to deflated effect size estimates nonsignificant results
When something is more reliable it is closer to the true score The more we can differentiate among individuals of
different levels
When something is less reliable it is further away from the true score The less it differentiates among individuals
Measurement Models Under CTT
Assumptions ParallelTau
EquivalentEssentially
Tau-Equivalent Congeneric
Unidimensional Y Y Y Y
Equal item covariances Y Y Y
Equal item-construct relations
Y Y Y
Equal item variance Y Y
Equal item error variance
Y
Alpha ()Cronbach
(1951)
Omega () McDonald
(1970, 1999)
Coefficient Alpha
covavg = average covariance among items
k = the number of items
ScaleV = scale score variance
VScale
kavg2)(cov
ˆ
Why have we been using alpha for so long?
Simple equation
Easily calculated in standard software (default)
Tradition
Easy to understand
Researchers are not aware of other approaches
Problems with alpha ()
Unrealistic to assume all items have same equal item-construct relation and item covariances are the same
Underestimates population reliability coefficient when congeneric model assumed
What do we gain with Omega?
Does not assume all items have the same item-construct relations and equal item covariances (assumptions relaxed)
More consistent (precise) estimator of reliability
Not as difficult to estimate as folks come to believe
One-Factor CFA Model
Item3
3
1
Item2
2
1
Item1
1
1
Item6
6
Item5
5
Item4
4
111
1
Perceived General Self-Efficacy
1 2 3 45 6
Coefficient Omega
k
iii
k
ii
k
ii
1
2
1
2
1
i = factor pattern loading for item i k = the number of items ii = unique variance of item I Assumes latent variance is fixed at 1 within
CFA framework
Include CI along Reliability Point Estimate
Measures a range that estimates the true population value for reliability, while acknowledging the uncertainty in our reliability estimate
Recommended by APA and most peer reviewed journals
Mplus input for alpha ()
Mplus output for alpha ()
Mplus input for omega
Mplus output for omega
APA style write-up for coefficients with CIs
= .61, Bootstrap corrected [BC] 95% CI [.56, .66]
= .62, Bootstrap corrected [BC] 95% CI [.56, .66]
Limitations of CTT Reliability Coefficients and Future QIPSR Talks
Although a better estimate of reliability than alpha, CTT still assumes a constant amount of reliability exists across the score continuum
However, it is well known in the measurement community that reliability/precision is conditional on a person’s location along the continuum
Modern measurement techniques such as Item Response Theory (IRT) do not make this assumption and focus on items instead of the total scale itself
References Revelle, W., Zinbarg, R. E. (2008). Coefficients Alpha, Beta, Omega, and the glb: Comments on Sijtsma.
Psychometrika, 74, 145-154. doi:10.1007/s11336-008-9102-z Gadermann, A. M., Guhn, M., & Zumbo, B. D. (2012). Estimating ordinal reliability for Likert-type and ordinal item
response data: A conceptual, empirical, and practical guide. Practical Assessment, Research and Evaluation, 17. Retrieved from: http://pareonline.net/getvn.asp?v=17&n=3
Zumbo, B. D., Gadermann, A. M., & Zeisser, C. (2007). Ordinal versions of coefficients alpha and theta for Likert rating scales. Journal of Modern Applied Statistical Methods, 6. Retrieved from: http://digitalcommons.wayne.edu/jmasm/vol6/iss1/4
Starkweather, J. (2012). Step out of the past: Stop using coefficient alpha; there are better ways to calculate reliability. Benchmarks RSS Matters Retrieved from: http://web3.unt.edu/benchmarks/issues/2012/06/rss-matters
Sijtsma, K. (2009). On the use, the misuse, and the very limited usefulness of Cronbach's alpha. Psychometrika, 74, 107-120.doi:10.1007/s11336-008-9101-0
Peters., G-J. Y. (2014). The alpha and the omega of scale reliability and validity: Why and how to abandon Cronbach's alpha and the route towards more comprehensive assessment of scale quality. The European Health Psychologist, 16, 56–69. Retrieved from: http://www.ehps.net/ehp/issues/2014/v16iss2April2014/4%20%20Peters%2016_2_EHP_April%202014.pdf
Crutzen, R. (2014). Time is a jailer: What do alpha and its alternatives tell us about reliability?. The European Health Psychologist, 16, 70-74. Retrieved from: http://www.ehps.net/ehp/issues/2014/v16iss2April2014/5%20Crutzen%2016_2_EHP_April%202014.pdf
Dunn, T., Baguley, T., & Brunsden, V. (2014). From alpha to omega: A practical solution to the pervasive problem of internal consistency estimation. British Journal of Psychology, 105, 399-412. doi:10.1111/bjop.12046
Geldhof, G., Preacher, K. J., & Zyphur, M. J. (2014). Reliability estimation in a multilevel confirmatory factor analysis framework. Psychological Methods, 19, 72-91. doi:10.1037/a0032138
Acknowledgements
APS Lab members Angela Tobmbari Zijia Li Caihong Li Abbey Love Mikah Pritchard