S052/III.1(a): Applied Data Analysis Roadmap of the Course – What Is Today’s Topic Area?

© Willett, Harvard University Graduate School of Education, 04/21/23 S052/III.1(a) – Slide 1

S052/III.1(a): Applied Data Analysis Roadmap of the Course – What Is Today’s Topic Area?

S052/III.1(a): Applied Data Analysis Roadmap of the Course – What Is Today’s Topic Area?

More details can be found in the “Course Objectives and Content” handout on the course webpage.More details can be found in the “Course Objectives and Content” handout on the course webpage.

Multiple Regression

Analysis (MRA)

Multiple Regression

Analysis (MRA)••

iiii XXY 22110

Do your residuals meet the required assumptions?

Test for residual normality

Use influence statistics to detect atypical datapoints

If your residuals are not independent, replace OLS by GLS regression analysis

Use Individual growth modeling

Specify a Multi-level Model

If your sole predictor is continuous, MRA is identical to correlational analysis

If your sole predictor is dichotomous, MRA is identical to a t-test

If your several predictors are categorical, MRA is identical to ANOVA

If time is a predictor, you need discrete-time survival analysis…

If your outcome is categorical, you need to use…

Binomial logistic regression analysis (dichotomous outcome)

Multinomial logistic regression analysis (polytomous outcome)

If you have more predictors than you can deal with,

Create taxonomies of fitted models and compare them.Form composites

of the indicators of any common construct.

Conduct a Principal Components Analysis

Use Cluster Analysis

Use non-linear regression analysis.

Transform the outcome or predictor

•If your outcome vs. predictor relationship is non-linear,

How do you deal with missing data?

Today’s Topic Area


S052/III.1(a): Classical Methods For Compositing Multiple Indicators Of A Construct Printed Syllabus – What Is Today’s Topic?

S052/III.1(a): Classical Methods For Compositing Multiple Indicators Of A Construct Printed Syllabus – What Is Today’s Topic?

Please check inter-connections among the Roadmap, the Daily Topic Area, the Printed Syllabus, and the content of today’s class when you pre-read the day’s materials.

Please check inter-connections among the Roadmap, the Daily Topic Area, the Printed Syllabus, and the content of today’s class when you pre-read the day’s materials.

Today, in Syllabus Section III.1(a), on Classical Methods For Compositing Multiple Indicators Of A Construct, I will: Explore issues in forming composites from multiple

indicators (#3 - #5). Comment on the role of indicator variability & indicator-

indicator correlation in composite formation (#6 - #10). Review fundamental tenets of classical test theory and

define the reliability parameter (#11 - #12). Show how reliability depends on the number of

indicators included in the composite (#13). Estimate and interpret the internal consistency reliability

of a group of indicators, using Cronbach’s (#14 - #16). Demonstrate how estimates of Cronbach’s can be used

to conduct simple item-analyses of the indicators within a composite (#17 - #18).

Appendix 1: Listwise vs. pairwise deletion (#19).


A dataset in which the investigators measured multiple indicators of what they thought was a single underlying construct that represented Teacher Job Satisfaction: The data described in TSUCCESS_info.pdf.

A dataset in which the investigators measured multiple indicators of what they thought was a single underlying construct that represented Teacher Job Satisfaction: The data described in TSUCCESS_info.pdf.

Dataset TSUCCESS.txt

Overview Responses of national sample of teachers to six questions about job satisfaction.

SourceAdministrator and Teacher Survey of the High School and Beyond (HS&B) dataset, 1984 administration, National Center for Education Statistics (NCES).

Sample Size 5269 teachers (4955 with complete data).

More Info

HS&B was established to study the educational, vocational, and personal development of young people beginning in their elementary or high school years and following them over time as they began to take on adult responsibilities. The HS&B survey included two cohorts: (a) the 1980 senior class, and (b) the 1980 sophomore class. Both cohorts were surveyed every two years through 1986, and the 1980 sophomore class was also surveyed again in 1992.

S052/III.1(a): Classical Methods For Compositing Multiple Indicators Of A Construct Introducing the TSUCCESS Dataset

S052/III.1(a): Classical Methods For Compositing Multiple Indicators Of A Construct Introducing the TSUCCESS Dataset

http://nces.ed.gov/surveys/hsb/

http://nces.ed.gov/


Col Var Variable Description Labels

1 X1You have high standards of teacher performance.

1 = strongly disagree 2 = disagree3 = slightly disagree 4 = slightly agree5 = agree 6 = strongly agree

2 X2You are continually learning on the job.


3 X3You are successful in educating your students.

1 = not successful 2 = a little successful3 = successful 4 = very successful

4 X4It’s a waste of time to do your best as a teacher.

1 = strongly agree 2 = agree,3 = slightly agree 4 = slightly disagree,5 = disagree 6 = strongly disagree

5 X5You look forward to working at your school.


6 X6How much of the time are you satisfied with your job?

1 = never 2 = almost never3 = sometimes 4 = always

S052/III.1(a): Classical Methods For Compositing Multiple Indicators Of A Construct Multiple Indicators of the Construct of Teacher Job Satisfaction Are Present

S052/III.1(a): Classical Methods For Compositing Multiple Indicators Of A Construct Multiple Indicators of the Construct of Teacher Job Satisfaction Are Present

As is typical of many datasets, TSUCCESS contains: Several variables – or “indicators” – that record

teacher’s responses to survey items. These multiple items were included in the survey

instrument tp provide teachers with replicate opportunities to report their job satisfaction (“teacher job satisfaction” being the central “construct” in the research).

To incorporate the multiple indicators successfully into subsequent analysis – whether as outcome or predictor – you must deal with several issues:

1. Should each of the indicators be treated as a separate variable in subsequent analyses, or should the indicators be combined to form a “composite” measure of the underlying construct of teacher job satisfaction?

2. If you form a composite, how do you confirm that the multiple indicators actually “belong together” in a single composite?

3. If the multiple indicators do indeed belong together in a single composite, what’s the best way to form that composite?


VarVariable

Description Labels

X1

You have high standards of teacher performance.


X2 You are continually learning on the job.


X3You are successful in educating your students.

1 = not successful 2 = a little successful3 = successful 4 = very successful

X4It’s a waste of time to do your best as a teacher.

1 = strongly agree 2 = agree,3 = slightly agree 4 = slightly disagree,5 = disagree 6 = strongly disagree

X5You look forward to working at your school.


X6

How much of the time are you satisfied with your job?

1 = never 2 = almost never3 = sometimes 4 = always

S052/III.1(a): Classical Methods For Compositing Multiple Indicators Of A Construct Some Potentially Serious Problems With The Indicators Are Immediately Obvious!!!

S052/III.1(a): Classical Methods For Compositing Multiple Indicators Of A Construct Some Potentially Serious Problems With The Indicators Are Immediately Obvious!!!

1. Different indicators have scales of different “length”:

i. Indicators X1, X2, X4, & X5 are measured on 6-point scales.

ii. Indicators X3 & X6 are measured on 4-point scales.

iii. Does this matter, and how do we deal with it in the compositing process?

iv. Is there a “preferred” scale length?

2. Some indicators “point” in a “positive” direction and some in a “negative” direction:

i. Notice the coding direction of X4, compared to the directions of the rest of the indicators.

ii. When we composite the indicators, what should we do about this?

3. Simply coding indicators on the “same” scale does not necessarily mean that they have the same “value” at the same scale points:

i. Compare scale point “3” for indicators X3 and X6, for instance.

ii. How do we deal with this, in compositing?

Of course, not all indicators are always created equally …Of course, not all indicators are always created equally …


*-----------------------------------------------------------------------* Input the dataset, name & label six indicators of teacher satisfaction*-----------------------------------------------------------------------*; DATA TSUCCESS; INFILE 'C:\DATA\S052\TSUCCESS.txt'; INPUT X1-X6; LABEL X1 = 'Have high standards of teaching' X2 = 'Continually learning on job' X3 = 'Successful in educating students' X4 = 'Waste of time to do best as teacher' X5 = 'Look forward to working at school' X6 = 'Time satisfied with job';

PROC FORMAT; VALUE AFMT 1='Strongly disagree' 2='Disagree' 3='Slightly disagree' 4='Slightly agree' 5='Agree' 6='Strongly agree'; VALUE BFMT 1='Strongly agree' 2='Agree' 3='Slightly agree' 4='Slightly disagree' 5='Disagree' 6='Strongly disagree'; VALUE CFMT 1='Not successful' 2='Somewhat successful' 3='Successful' 4='Very Successful'; VALUE DFMT 1=‘Never' 2=‘Almost never' 3=‘Sometimes' 4='Always';

*-----------------------------------------------------------------------* Input the dataset, name & label six indicators of teacher satisfaction*-----------------------------------------------------------------------*; DATA TSUCCESS; INFILE 'C:\DATA\S052\TSUCCESS.txt'; INPUT X1-X6; LABEL X1 = 'Have high standards of teaching' X2 = 'Continually learning on job' X3 = 'Successful in educating students' X4 = 'Waste of time to do best as teacher' X5 = 'Look forward to working at school' X6 = 'Time satisfied with job';

PROC FORMAT; VALUE AFMT 1='Strongly disagree' 2='Disagree' 3='Slightly disagree' 4='Slightly agree' 5='Agree' 6='Strongly agree'; VALUE BFMT 1='Strongly agree' 2='Agree' 3='Slightly agree' 4='Slightly disagree' 5='Disagree' 6='Strongly disagree'; VALUE CFMT 1='Not successful' 2='Somewhat successful' 3='Successful' 4='Very Successful'; VALUE DFMT 1=‘Never' 2=‘Almost never' 3=‘Sometimes' 4='Always';

Here’s some preliminary analyses of the multiple indicators of teacher satisfaction. in Data-Analytic Handout III,1(a).1…Here’s some preliminary analyses of the multiple indicators of teacher satisfaction. in Data-Analytic Handout III,1(a).1…

S052/III.1(a): Classical Methods For Compositing Multiple Indicators Of A Construct Other Issues Are More Subtle, And Require Some EDA For Their Detection


Standard data input statements

Standard formatting of

indicators


*------------------------------------------------------------------* Print out 40 cases from the dataset, for inspection.*------------------------------------------------------------------*; PROC PRINT DATA=TSUCCESS(OBS=35); VAR X1-X6;*------------------------------------------------------------------* Estimate selected univariate summary statistics for each indicator*------------------------------------------------------------------*; PROC TABULATE DATA=TSUCCESS; VAR X1-X6; TABLE (X1 X2 X3 X4 X5 X6),(N NMISS MEAN VAR);*------------------------------------------------------------------* Summarize bivariate relationships among indicators of satisfaction*------------------------------------------------------------------*;* With pairwise deletion of cases with missing values; PROC CORR NOPROB NOSIMPLE DATA=TSUCCESS; VAR X1-X6;* With listwise deletion of cases with missing values; PROC CORR NOPROB NOSIMPLE NOMISS DATA=TSUCCESS; VAR X1-X6;

*------------------------------------------------------------------* Print out 40 cases from the dataset, for inspection.*------------------------------------------------------------------*; PROC PRINT DATA=TSUCCESS(OBS=35); VAR X1-X6;*------------------------------------------------------------------* Estimate selected univariate summary statistics for each indicator*------------------------------------------------------------------*; PROC TABULATE DATA=TSUCCESS; VAR X1-X6; TABLE (X1 X2 X3 X4 X5 X6),(N NMISS MEAN VAR);*------------------------------------------------------------------* Summarize bivariate relationships among indicators of satisfaction*------------------------------------------------------------------*;* With pairwise deletion of cases with missing values; PROC CORR NOPROB NOSIMPLE DATA=TSUCCESS; VAR X1-X6;* With listwise deletion of cases with missing values; PROC CORR NOPROB NOSIMPLE NOMISS DATA=TSUCCESS; VAR X1-X6;

Obtain some univariate and bivariate descriptive statistics on the indicators …Obtain some univariate and bivariate descriptive statistics on the indicators …



Print out a few cases for inspection

Examine univariate descriptive statistics on

each indicator

Examine bivariate inter-relationships among the

multiple indicators

Missing values in the indicators are always a problem, when you are forming composites.

There are many ways to deal with them: Pairwise & listwise deletion, Mean substitution Regression imputation, Hotdecking, Multiple imputation …


„ƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒ…ƒƒƒƒƒƒƒƒƒƒƒƒ…ƒƒƒƒƒƒƒƒƒƒƒƒ…ƒƒƒƒƒƒƒƒƒƒƒƒ…ƒƒƒƒƒƒƒƒƒƒƒƒ† ‚ ‚ N ‚ NMiss ‚ Mean ‚ Var ‚ ‡ƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒˆƒƒƒƒƒƒƒƒƒƒƒƒˆƒƒƒƒƒƒƒƒƒƒƒƒˆƒƒƒƒƒƒƒƒƒƒƒƒˆƒƒƒƒƒƒƒƒƒƒƒƒ‰ ‚Have high ‚ ‚ ‚ ‚ ‚ ‚standards of ‚ ‚ ‚ ‚ ‚ ‚teaching ‚ 5097.00 ‚ 173.00 ‚ 4.33 ‚ 1.19 ‚ ‡ƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒˆƒƒƒƒƒƒƒƒƒƒƒƒˆƒƒƒƒƒƒƒƒƒƒƒƒˆƒƒƒƒƒƒƒƒƒƒƒƒˆƒƒƒƒƒƒƒƒƒƒƒƒ‰ ‚Continually ‚ ‚ ‚ ‚ ‚ ‚learning on job ‚ 5109.00 ‚ 161.00 ‚ 3.87 ‚ 1.56 ‚ ‡ƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒˆƒƒƒƒƒƒƒƒƒƒƒƒˆƒƒƒƒƒƒƒƒƒƒƒƒˆƒƒƒƒƒƒƒƒƒƒƒƒˆƒƒƒƒƒƒƒƒƒƒƒƒ‰ ‚Successful in ‚ ‚ ‚ ‚ ‚ ‚educating ‚ ‚ ‚ ‚ ‚ ‚students ‚ 5144.00 ‚ 126.00 ‚ 3.15 ‚ 0.45 ‚ ‡ƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒˆƒƒƒƒƒƒƒƒƒƒƒƒˆƒƒƒƒƒƒƒƒƒƒƒƒˆƒƒƒƒƒƒƒƒƒƒƒƒˆƒƒƒƒƒƒƒƒƒƒƒƒ‰ ‚Waste of time to ‚ ‚ ‚ ‚ ‚ ‚do best as ‚ ‚ ‚ ‚ ‚ ‚teacher ‚ 5121.00 ‚ 149.00 ‚ 4.22 ‚ 2.79 ‚ ‡ƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒˆƒƒƒƒƒƒƒƒƒƒƒƒˆƒƒƒƒƒƒƒƒƒƒƒƒˆƒƒƒƒƒƒƒƒƒƒƒƒˆƒƒƒƒƒƒƒƒƒƒƒƒ‰ ‚Look forward to ‚ ‚ ‚ ‚ ‚ ‚working at school‚ 5116.00 ‚ 154.00 ‚ 4.42 ‚ 1.78 ‚ ‡ƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒˆƒƒƒƒƒƒƒƒƒƒƒƒˆƒƒƒƒƒƒƒƒƒƒƒƒˆƒƒƒƒƒƒƒƒƒƒƒƒˆƒƒƒƒƒƒƒƒƒƒƒƒ‰ ‚Time satisfied ‚ ‚ ‚ ‚ ‚ ‚with job ‚ 5125.00 ‚ 145.00 ‚ 2.84 ‚ 0.33 ‚ Šƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒ‹ƒƒƒƒƒƒƒƒƒƒƒƒ‹ƒƒƒƒƒƒƒƒƒƒƒƒ‹ƒƒƒƒƒƒƒƒƒƒƒƒ‹ƒƒƒƒƒƒƒƒƒƒƒƒŒ

„ƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒ…ƒƒƒƒƒƒƒƒƒƒƒƒ…ƒƒƒƒƒƒƒƒƒƒƒƒ…ƒƒƒƒƒƒƒƒƒƒƒƒ…ƒƒƒƒƒƒƒƒƒƒƒƒ† ‚ ‚ N ‚ NMiss ‚ Mean ‚ Var ‚ ‡ƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒˆƒƒƒƒƒƒƒƒƒƒƒƒˆƒƒƒƒƒƒƒƒƒƒƒƒˆƒƒƒƒƒƒƒƒƒƒƒƒˆƒƒƒƒƒƒƒƒƒƒƒƒ‰ ‚Have high ‚ ‚ ‚ ‚ ‚ ‚standards of ‚ ‚ ‚ ‚ ‚ ‚teaching ‚ 5097.00 ‚ 173.00 ‚ 4.33 ‚ 1.19 ‚ ‡ƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒˆƒƒƒƒƒƒƒƒƒƒƒƒˆƒƒƒƒƒƒƒƒƒƒƒƒˆƒƒƒƒƒƒƒƒƒƒƒƒˆƒƒƒƒƒƒƒƒƒƒƒƒ‰ ‚Continually ‚ ‚ ‚ ‚ ‚ ‚learning on job ‚ 5109.00 ‚ 161.00 ‚ 3.87 ‚ 1.56 ‚ ‡ƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒˆƒƒƒƒƒƒƒƒƒƒƒƒˆƒƒƒƒƒƒƒƒƒƒƒƒˆƒƒƒƒƒƒƒƒƒƒƒƒˆƒƒƒƒƒƒƒƒƒƒƒƒ‰ ‚Successful in ‚ ‚ ‚ ‚ ‚ ‚educating ‚ ‚ ‚ ‚ ‚ ‚students ‚ 5144.00 ‚ 126.00 ‚ 3.15 ‚ 0.45 ‚ ‡ƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒˆƒƒƒƒƒƒƒƒƒƒƒƒˆƒƒƒƒƒƒƒƒƒƒƒƒˆƒƒƒƒƒƒƒƒƒƒƒƒˆƒƒƒƒƒƒƒƒƒƒƒƒ‰ ‚Waste of time to ‚ ‚ ‚ ‚ ‚ ‚do best as ‚ ‚ ‚ ‚ ‚ ‚teacher ‚ 5121.00 ‚ 149.00 ‚ 4.22 ‚ 2.79 ‚ ‡ƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒˆƒƒƒƒƒƒƒƒƒƒƒƒˆƒƒƒƒƒƒƒƒƒƒƒƒˆƒƒƒƒƒƒƒƒƒƒƒƒˆƒƒƒƒƒƒƒƒƒƒƒƒ‰ ‚Look forward to ‚ ‚ ‚ ‚ ‚ ‚working at school‚ 5116.00 ‚ 154.00 ‚ 4.42 ‚ 1.78 ‚ ‡ƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒˆƒƒƒƒƒƒƒƒƒƒƒƒˆƒƒƒƒƒƒƒƒƒƒƒƒˆƒƒƒƒƒƒƒƒƒƒƒƒˆƒƒƒƒƒƒƒƒƒƒƒƒ‰ ‚Time satisfied ‚ ‚ ‚ ‚ ‚ ‚with job ‚ 5125.00 ‚ 145.00 ‚ 2.84 ‚ 0.33 ‚ Šƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒ‹ƒƒƒƒƒƒƒƒƒƒƒƒ‹ƒƒƒƒƒƒƒƒƒƒƒƒ‹ƒƒƒƒƒƒƒƒƒƒƒƒ‹ƒƒƒƒƒƒƒƒƒƒƒƒŒ

S052/III.1(a): Classical Methods For Compositing Multiple Indicators Of A Construct Other Issues Are More Subtle – Each Indicator Has A Unique Variance, For Instance

S052/III.1(a): Classical Methods For Compositing Multiple Indicators Of A Construct Other Issues Are More Subtle – Each Indicator Has A Unique Variance, For Instance

Notice the impact of the missing values: Sample size differs from

indicator to indicator.

Not unexpectedly, different indicators have different sample means

More importantly, different different indicators have markedly indicators have markedly different variancesdifferent variances: This is a big problem when

forming composites. Because indicators with

larger variancelarger variance play a larger rolelarger role in the composite composite scorescore.


Indicator X1 X2 X3 X4 X5 X6

X1: Have high standards of teaching

0.555 0.161 0.213 0.253 0.192

X2: Continually learning on the job

0.552(5058)

0.166 0.231 0.270 0.222

X3: Successful in educating students

0.161(5069)

0.164(5082)

0.299 0.356 0.433

X4: Waste of time to do best as teacher

0.211(5071)

0.232(5079)

0.296(5094)

0.448 0.399

X5: Look forward to working at school

0.253(5069)

0.271(5070)

0.356(5088)

0.446(5091)

0.553

X6: Time satisfied with job

0.193(5060)

0.224(5069)

0.437(5094)

0.395(5082)

0.550(5081)

S052/III.1(a): Classical Methods For Compositing Multiple Indicators Of A Construct Other Issues Are More Subtle – Each Indicator Has A Different Correlation With Other Indicators

S052/III.1(a): Classical Methods For Compositing Multiple Indicators Of A Construct Other Issues Are More Subtle – Each Indicator Has A Different Correlation With Other Indicators

Bivariate correlations estimated under pairwise deletion

Bivariate correlations estimated under listwise deletion

To justify forming a single compositesingle composite, you must be able to argue successfully that all indicators measure the same constructsame construct: Here, generally positive inter-correlations support a “uni-dimensional” view. But, the small & heterogeneous values of indicator inter-correlations also

suggest: Either there is considerable measurement error in each indicator, Or that some, or all, of indicators may also measure other unrelated

constructs. This is bad news for the overall quality (reliability) of the ultimate composite.

Sample inter-correlations among the indicators: Are all positivepositive

(thankfully!), Are of smallsmall to

moderatemoderate magnitudemagnitude but differ widelydiffer widely (unfortunately!).


IndicatorSt.Dev.

Angle of Bisection/Bivariate Correlation

X1 X2 X3 X4 X5 X6

X1: Have high standards of teaching 1.09 0.555 0.161 0.213 0.253 0.192X2: Continually learning on the job 1.25 56 0.166 0.231 0.270 0.222X3: Successful in educating students 0.67 81 80 0.299 0.356 0.433X4: Waste of time to do best as teacher 1.67 78 77 73 0.448 0.399X5: Look forward to working at school 1.33 75 74 69 63 0.553X6: Time satisfied with job 0.57 79 77 64 67 56

S052/III.1(a): Classical Methods For Compositing Multiple Indicators Of A Construct Other Issues Are More Subtle – An Interesting Geometric Presentation Of The Problem …S052/III.1(a): Classical Methods For Compositing Multiple Indicators Of A Construct

Other Issues Are More Subtle – An Interesting Geometric Presentation Of The Problem …

Regard the correlation between two indicators as the cosine of the angle between them:

… etc.

X1

1.09

X2

1.25

56

Regard the standard deviation of the indicator as its “lengthlength”:

… etc.

X11.09

X21.25

X30.67

Inter-correlated indicators are like “forces” diverging from a point. In compositing the indicators, you seek their “resultant” … recall Newton’s Newton’s Parallelogram of ForcesParallelogram of Forces?

Inter-correlated indicators are like “forces” diverging from a point. In compositing the indicators, you seek their “resultant” … recall Newton’s Newton’s Parallelogram of ForcesParallelogram of Forces?

X1

X2

X3

56

81 80•

1.09

1.25

0.67… etc.

Putting it all together …


Classical test theory hypothesizes that, when we measure the value of any construct, the act of measurement introduces random errors into the measurement:

1. At the individual level, the theory assumes that:

1. And so, at the group level, the theory requires that:

Classical test theory hypothesizes that, when we measure the value of any construct, the act of measurement introduces random errors into the measurement:

1. At the individual level, the theory assumes that:

1. And so, at the group level, the theory requires that:

ErrortMeasuremen

RandomScoreTrue ScoreObserved

construct underlying theon scoresperson' The

Person Eachof

ErrortMeasuremen

RandomScoreTrue ScoreObserved

construct underlying theon scoresperson' The

Person Eachof

VarianceError

ScoresTrue the ofVariance

Scores Observedthe ofVariance

VarianceError

ScoresTrue the ofVariance

Scores Observedthe ofVariance

ObservedVarianceObservedVariance

When composites are created to measure an underlying construct, we need a statistical criterion for judging the quality of the composite thus formed … this requires a side trip into classical test theory …When composites are created to measure an underlying construct, we need a statistical criterion for judging the quality of the composite thus formed … this requires a side trip into classical test theory …

TrueVariance

ErrorVariance

S052/III.1(a): Classical Methods For Compositing Multiple Indicators Of A Construct Classical Test Theory – Introducing the Notion of Observed, True and Error Scores

S052/III.1(a): Classical Methods For Compositing Multiple Indicators Of A Construct Classical Test Theory – Introducing the Notion of Observed, True and Error Scores


How Is Reliability Estimated?How Is Reliability Estimated?

Many methods, but all involve replicate measurement:• Parallel forms estimation,• Test/Retest estimation,• Split-halves estimation,• Internal consistency estimation, …

All estimation methods are based on notion that the only reason for two indicators to be correlated is their mutual interest in revealing the same underlying true variance.

S052/III.1(a): Classical Methods For Compositing Multiple Indicators Of A Construct Classical Test Theory – What Is Reliability?

S052/III.1(a): Classical Methods For Compositing Multiple Indicators Of A Construct Classical Test Theory – What Is Reliability?

What Values Can Reliability Take?What Values Can Reliability Take?Because it is a proportion, reliability can take on numerical values between 0 and 1: When all the observed variance is error

variance, reliability is zero. When all the observed variance is true

variance, reliability is one.

Typical ranges for reliability values include:` .6 - .8, self-reported attitude measurement. .8 - .9, self-penned skills measures. >.9 – standardized achievement measures.

Under a “classical” approach, compositing of multiple indicators is guided by the composite’s reliability:Under a “classical” approach, compositing of multiple indicators is guided by the composite’s reliability:

Reliability is a population parameter that describes how much of the observed varianceobserved variance in a measure (or composite) is actually true variancetrue variance:Reliability is a population parameter that describes how much of the observed varianceobserved variance in a measure (or composite) is actually true variancetrue variance:

Scores of Variance Population


Observed

True



Observed

TrueT

TE

(X)= =


Providing that each indicator to be included in a composite is a measure of the same single underlying construct:

The more indicators you include in a composite, the higher the reliability of the composite will be.

Why? Because the measurement errors in

each indicator are random and will tend to cancel each other out in the composite.

This leaves any true variation in each indicator to combine and be revealed in the composite measure.

Number of Items in

Composite

CompositeReliability(r = .2)




1 0.2000 0.4000 0.6000 0.80002 0.3333 0.5714 0.7500 0.88893 0.4286 0.6667 0.8182 0.92314 0.5000 0.7273 0.8571 0.94125 0.5556 0.7692 0.8824 0.95246 0.6000 0.8000 0.9000 0.96007 0.6364 0.8235 0.9130 0.96558 0.6667 0.8421 0.9231 0.96979 0.6923 0.8571 0.9310 0.9730

10 0.7143 0.8696 0.9375 0.9756

0.00

0.25

0.50

0.75

1.00

1 2 3 4 5 6 7 8 9 10

Number of Indicators, I

Rel

iabi

lity

of

Com

posi

te

Number of Items in

Composite





1 0.2000 0.4000 0.6000 0.80002 0.3333 0.5714 0.7500 0.88893 0.4286 0.6667 0.8182 0.92314 0.5000 0.7273 0.8571 0.94125 0.5556 0.7692 0.8824 0.95246 0.6000 0.8000 0.9000 0.96007 0.6364 0.8235 0.9130 0.96558 0.6667 0.8421 0.9231 0.96979 0.6923 0.8571 0.9310 0.9730

10 0.7143 0.8696 0.9375 0.9756

0.00

0.25

0.50

0.75

1.00

1 2 3 4 5 6 7 8 9 10

Number of Indicators, I

Rel

iabi

lity

of

Com

posi

te

= .8

= .6

= .4

= .2

As a consequence, the reliability of a composite of I indicators, each of separate reliability , can be estimated using the Spearman-Brown Prophesy Formula:

)1(1

I

Itotal

S052/III.1(a): Classical Methods For Compositing Multiple Indicators Of A Construct How Does the Reliability of a Composite Depend on the Number of Indicators Combined?S052/III.1(a): Classical Methods For Compositing Multiple Indicators Of A Construct How Does the Reliability of a Composite Depend on the Number of Indicators Combined?


*------------------------------------------------------------------------* Estimate Cronbach's Alpha (internal consistency) reliability for a composite of all the indicators of the construct of teacher satisfaction*------------------------------------------------------------------------*;* Listwise deletion of missing values ensures correct computation of alpha; PROC CORR ALPHA NOMISS NOCORR NOSIMPLE DATA=TSUCCESS; VAR X1-X6;

*------------------------------------------------------------------------* Estimate Cronbach's Alpha (internal consistency) reliability for a composite of all the indicators of the construct of teacher satisfaction*------------------------------------------------------------------------*;* Listwise deletion of missing values ensures correct computation of alpha; PROC CORR ALPHA NOMISS NOCORR NOSIMPLE DATA=TSUCCESS; VAR X1-X6;

S052/III.1(a): Classical Methods For Compositing Multiple Indicators Of A Construct Estimating Cronbach’s Alpha – An Internal Consistency Estimate of the Reliability of A Composite

S052/III.1(a): Classical Methods For Compositing Multiple Indicators Of A Construct Estimating Cronbach’s Alpha – An Internal Consistency Estimate of the Reliability of A Composite

Requests a Cronbach’s alpha estimate of reliability for a composite of indicators X1

through X6 Cronbach’s is an internal-consistency estimate of reliability:

Estimated by treating each item in the composite as a replicate measure of the underlying construct.

It’s a weighted average of the indicator-indicator correlations (actually, covariances).

Assesses the extent to which sampled teachers responded consistently across all six indicators:

If all the indicator-indicator correlations are 1, then the estimated value of is 1.

If all the indicator-indicator correlations are 0, then the estimated value of is 0.

Computation of is correct only in complete data, or under list-wise deletion of missing values


Cronbach Coefficient Alpha

Variables AlphaƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒRaw 0.696594Standardized 0.735530



The output is pretty incontrovertible, but provides two estimates of the alpha coefficient …The output is pretty incontrovertible, but provides two estimates of the alpha coefficient …

S052/III.1(a): Classical Methods For Compositing Multiple Indicators Of A Construct Two Estimates of Cronbach’s Coefficient Alpha -- Unstandardized Version

S052/III.1(a): Classical Methods For Compositing Multiple Indicators Of A Construct Two Estimates of Cronbach’s Coefficient Alpha -- Unstandardized Version

Here, the estimated reliability suggests that 69.7% of the observed variance in the unstandardized composite score is true variance in teacher satisfaction:

• Use this estimate of reliability if you have simply summed raw indicator scores to form a composite.

• Is this a bad idea? Yes, it is when the indicators have heterogeneous metrics and variabilities …

Here’s the estimated -reliability for a “raw” composite:

• In a “raw” composite, each indicator remains in its original metric.

• The composite score is formed by adding the raw scores on each indicator together:

where i represents the ith teacher, X1i is the raw score of the ith teacher on the 1st indicator, and so on …

iiiiiii XXXXXXC 654321 iCiC

iX 2iX 2

iX1iX1

iX 5iX 5

iX 3iX 3

iX 4iX 4

iX 6iX 6






When the indicators have heterogeneous metrics and variances, it’s better to use a standardized composite …When the indicators have heterogeneous metrics and variances, it’s better to use a standardized composite …

S052/III.1(a): Classical Methods For Compositing Multiple Indicators Of A Construct Two Estimates of Cronbach’s Coefficient Alpha -- Standardized Version

S052/III.1(a): Classical Methods For Compositing Multiple Indicators Of A Construct Two Estimates of Cronbach’s Coefficient Alpha -- Standardized Version

Here, the estimated reliability suggests that 73.6% of observed variance in the standardized composite is true variance in teacher satisfaction:

• Use this estimate if you have formed a composite by summing standardized indicator scores.

• In this composite, each indicator has an identical metric and variance, and so contributes equally to the composite

Here’s the estimated -reliability for a “standardized” composite:

• Each indicator is first standardized to a mean of 0 and a standard deviation of 1:

• The composite is then formed by summing the standardized indicator scores: *

6*5

*4

*3

*2

*1

*iiiiiii XXXXXXC

6

66*6

5

55*5

4

44*4

3

33*3

2

22*2

1

11*1

s

XXX

s

XXX

s

XXX

s

XXX

s

XXX

s

XXX

ii

ii

ii

ii

ii

ii

*iC*iC

iX 2

iX 2

iX1

iX1

iX 5

iX 5

iX 4

iX 4

iX 3

iX 3

iX 6

iX 6



Variables Alpha ƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒ Raw 0.696594 Standardized 0.735530 Cronbach Coefficient Alpha with Deleted Variable Raw Variables Standardized Variables Deleted Correlation Correlation Variable with Total Alpha with Total Alpha ƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒ X1 0.419404 0.659181 0.400722 0.718260 X2 0.428629 0.656491 0.423916 0.711781 X3 0.392822 0.677513 0.414038 0.714550 X4 0.459576 0.665171 0.472577 0.697946 X5 0.544084 0.613596 0.573332 0.668234 X6 0.537640 0.660448 0.544912 0.676762



S052/III.1(a): Classical Methods For Compositing Multiple Indicators Of A Construct Using Cronbach’s Alpha To Conduct “Item Analysis” of the Indicators


Here is the item-analysis for the composite of the

six raw indicators.

Here is the item-analysis for the

composite of the six standardized indicators

Let’s focus on this item analysis, because the

original teacher satisfaction indicators

had heterogeneous metrics and variances.


Recommended data-analytic strategy:Recommended data-analytic strategy:Compare the estimated reliability in the deleted condition with the overall estimated reliability. If the estimated reliability is smaller after the removal of

the indicator, then the indicator was needed in the composite.

If estimated reliability is greater after the removal of the indicator, then the indicator was not needed in the composite..







Here’s the bivariate correlation of the score on each indicator with the total score on the other indicators …

It’s often referred to as the item-total point-biserial correlation.

Here is the estimated reliability of the full composite, 0.7355

Here are the estimated reliabilities for

additional composites, each with the one listed

indicators deleted.


Obs X1 X2 X3 X4 X5 X6

1 5 5 3 3 4 2 2 4 3 2 1 1 2 3 4 4 2 2 2 2 4 . 6 3 5 3 3 5 4 4 3 2 4 3 6 . 5 2 4 3 3 7 4 4 4 4 5 3 8 6 4 4 1 1 2 9 6 6 3 6 5 3 10 3 5 3 6 3 3 11 4 2 1 3 2 2 12 5 6 2 6 6 4 13 4 3 3 2 5 3 14 3 3 3 3 4 3 15 4 4 3 6 3 2 16 6 4 3 6 6 3 17 2 1 2 4 5 3 18 3 4 3 2 3 3 19 4 4 4 6 5 3 20 2 2 3 3 3 3

Obs X1 X2 X3 X4 X5 X6

1 5 5 3 3 4 2 2 4 3 2 1 1 2 3 4 4 2 2 2 2 4 . 6 3 5 3 3 5 4 4 3 2 4 3 6 . 5 2 4 3 3 7 4 4 4 4 5 3 8 6 4 4 1 1 2 9 6 6 3 6 5 3 10 3 5 3 6 3 3 11 4 2 1 3 2 2 12 5 6 2 6 6 4 13 4 3 3 2 5 3 14 3 3 3 3 4 3 15 4 4 3 6 3 2 16 6 4 3 6 6 3 17 2 1 2 4 5 3 18 3 4 3 2 3 3 19 4 4 4 6 5 3 20 2 2 3 3 3 3

S052/III.1(a): Classical Methods For Compositing Multiple Indicators Of A Construct Appendix I: Listwise and Pairwise Deletion of Missing Values

S052/III.1(a): Classical Methods For Compositing Multiple Indicators Of A Construct Appendix I: Listwise and Pairwise Deletion of Missing Values

Listwise deletion removes the entire case from an analysis …Default approach in PC-SAS.Conservative, eliminates the most cases.Ensures that the important positive definiteness

property required of covariance and correlation matrices will continue to hold.

Can be disastrous for: Sample size. Sample representativeness.

Pairwise deletion only removes a case when the variable on which it is missing is involved in the analysis: Preserves the case in the analysis whenever it can contribute

information. Can lead to the violation of the positive definiteness property of

covariance and correlation matrices. Different parts of the analysis end up with different sample sizes. Can be disastrous for sample representativeness.

Documents

S052/III.1(a): Applied Data Analysis Roadmap of the Course – What Is Today’s Topic Area?