MeasurementMeasurement

Class 8Class 8

Purposes of MeasurementPurposes of Measurement Make connections between concepts and data Make connections between concepts and data

A measure is a representation of a variable, or a constructA measure is a representation of a variable, or a construct A measure also has to capture the true meaning of the A measure also has to capture the true meaning of the

construct. construct. The accuracy of this capture is called The accuracy of this capture is called validityvalidity But there are many ways to operationalize and measure a But there are many ways to operationalize and measure a

construct such as “damages,” “job satisfaction,” construct such as “damages,” “job satisfaction,” “productivity,” “social class,” “segregation” or “attitudes “productivity,” “social class,” “segregation” or “attitudes toward the law."toward the law."

What is critical to understand is that decisions What is critical to understand is that decisions that we make in measuring any of these that we make in measuring any of these constructs can bear on the results of the constructs can bear on the results of the research and the decision to reject the null research and the decision to reject the null hypothesis. This is called measurement error.hypothesis. This is called measurement error. There is no bigger cancer on social research than There is no bigger cancer on social research than

measurement errormeasurement error

Variables and MeasuresVariables and Measures The critical idea of a variable is that it has The critical idea of a variable is that it has

to vary. That is, it must represent the to vary. That is, it must represent the range of values implicit in the construct. range of values implicit in the construct.

Variables are operationally defined by how Variables are operationally defined by how they are measured. Precision in they are measured. Precision in measurement, and close relationship measurement, and close relationship between measure and construct, allows between measure and construct, allows other researchers to replicate findings other researchers to replicate findings from other studiesfrom other studies No replication No replication → No validity in the causal claim→ No validity in the causal claim

By operationally measuring a variable, you By operationally measuring a variable, you allow others to reach an independent allow others to reach an independent judgment about the meaning of your judgment about the meaning of your results, and you allow others to try to results, and you allow others to try to reach those conclusions on their own.reach those conclusions on their own.

Illustrations of Theoretical Illustrations of Theoretical Meanings in MeasurementMeanings in Measurement

Some variables can be operationalized only in Some variables can be operationalized only in one way: gender, for example, or jury size. one way: gender, for example, or jury size.

But most variables can be operationalized and But most variables can be operationalized and measured in many different ways.measured in many different ways. Age Age can conceptualized as a number, as a range, or as a can conceptualized as a number, as a range, or as a

descriptive—qualitative assessmentdescriptive—qualitative assessment Race and ethnicity – two constructs or one ?Race and ethnicity – two constructs or one ?

Problematic – census categories – see Brent Staples in Problematic – census categories – see Brent Staples in today’s NYT Editorial today’s NYT Editorial

But gBut genderender too can be operationalized in more than one too can be operationalized in more than one wayway

Consider Consider dangerousnessdangerousness in a civil proceeding, or in a civil proceeding, or exposureexposure in a tort case in a tort case

Violence – acts or consequences? threats or just Violence – acts or consequences? threats or just physical acts? what about robbery?physical acts? what about robbery?

Types or Levels of Types or Levels of MeasurementMeasurement

Examples of Examples of nominal nominal measures (measures measures (measures that indicate distinct categories or types)that indicate distinct categories or types) GenderGender Religious preferenceReligious preference RegionRegion Type of defense counselType of defense counsel

Examples of Examples of ordinal ordinal measures (measures measures (measures that enable the ranking of categories but that enable the ranking of categories but offer no information about the meanings of offer no information about the meanings of the intervals between ranks)the intervals between ranks) Birth orderBirth order School gradeSchool grade Types of criminal sentencesTypes of criminal sentences

Examples of Examples of interval interval scales (scales where the scales (scales where the distance between the points are equal and signify distance between the points are equal and signify actual differences in the construct)-- - the "zero" actual differences in the construct)-- - the "zero" point remains arbitrary.point remains arbitrary. Attitude scale scoresAttitude scale scores Crime rates (do we actually know what the "zero" point Crime rates (do we actually know what the "zero" point

is?)is?) TemperatureTemperature

Examples of Examples of ratio ratio scales (scales where the scales (scales where the distances between units are based on a meaningful distances between units are based on a meaningful "zero" point, and "zero" point, and where differences signify relative where differences signify relative distances as well as actual distances)distances as well as actual distances) Age, incomeAge, income Exposure to toxinsExposure to toxins Prison sentences lengthsPrison sentences lengths Punitive damage awardsPunitive damage awards SegregationSegregation Consumer confusionConsumer confusion

ScalesScales Scales are composites of separate Scales are composites of separate

items to form composite and complex items to form composite and complex representations of constructs representations of constructs

Scales avoids reliance on individual Scales avoids reliance on individual items for representation of a construct items for representation of a construct or phenomenonor phenomenon

Most scales demand that the face Most scales demand that the face validity of the items comprising the validity of the items comprising the scale is highscale is high

Never trust a single item scale !!!Never trust a single item scale !!!

Types of ScalesTypes of Scales Likert ScalesLikert Scales

Scores are obtained directly from respondents Scores are obtained directly from respondents and there is no discarding of items with and there is no discarding of items with disagreement. Assignment of arbitrary #’s to disagreement. Assignment of arbitrary #’s to low or high values. Use of reversals within low or high values. Use of reversals within related items to avoid “response sets).”related items to avoid “response sets).”

The scale is developed in stages where we The scale is developed in stages where we begin with a large number of items and begin with a large number of items and reduce them through item analysis. reduce them through item analysis. Redundant items (where high and low scorers Redundant items (where high and low scorers answered specific questions in the same way) answered specific questions in the same way) are eliminated from the analysis. Scale scores are eliminated from the analysis. Scale scores represent the total of responses to items in represent the total of responses to items in the scalethe scale

Example:Example:Collective Efficacy ScaleCollective Efficacy Scale

For each of these statements, please tell me whether you For each of these statements, please tell me whether you (1) strongly agree, (2) agree, (3) neither agree nor disagree,(1) strongly agree, (2) agree, (3) neither agree nor disagree,

(4) disagree, or (5) strongly disagree(4) disagree, or (5) strongly disagree

If there is a problem around here the neighbors get together to If there is a problem around here the neighbors get together to deal with it. deal with it.

This is a close-knit neighborhood. This is a close-knit neighborhood. When you get right down to it, no one in this neighborhood cares When you get right down to it, no one in this neighborhood cares

much about what happens to me much about what happens to me There are adults in this neighborhood that children can look up There are adults in this neighborhood that children can look up

to to People around this neighborhood are willing to help their People around this neighborhood are willing to help their

neighbors neighbors People in this neighborhood generally don’t get along with each People in this neighborhood generally don’t get along with each

otherother If I had to borrow $30 in an emergency, I could borrow it from a If I had to borrow $30 in an emergency, I could borrow it from a

neighbor neighbor People in this neighborhood do not share the same values People in this neighborhood do not share the same values People in the neighborhood can be trusted People in the neighborhood can be trusted Parents in this neighborhood know their children’s friends Parents in this neighborhood know their children’s friends

Guttman Scales Guttman Scales A scale where the responses indicate the precise order A scale where the responses indicate the precise order

of response to the constituent items. That is, responses of response to the constituent items. That is, responses to one item predict responses to other items. It to one item predict responses to other items. It constructs a scale based on the logic that if one scores constructs a scale based on the logic that if one scores positively or high at the upper boundaries of the scale, positively or high at the upper boundaries of the scale, they also (at least 90% of the time) score high or they also (at least 90% of the time) score high or positively on the lower ranges of the scale. It is a positively on the lower ranges of the scale. It is a convenient way of organizing ordinal or even nominal convenient way of organizing ordinal or even nominal data into an interval or ratio scale.data into an interval or ratio scale.

Statistical tests are available to assess whether items fit Statistical tests are available to assess whether items fit into a Guttman scale and what the scale properties are into a Guttman scale and what the scale properties are (the coefficient of reproducibility).(the coefficient of reproducibility).

Raises questions of the temporal dimension of scaling: Raises questions of the temporal dimension of scaling: items may scale well when the time span is long but the items may scale well when the time span is long but the scale may be plagued by errors if the time span is short.scale may be plagued by errors if the time span is short.

Example:Example: Spouse assault scale -- What happens Spouse assault scale -- What happens if lengthen or shorten the time frame? Validity if lengthen or shorten the time frame? Validity threats from this concern?threats from this concern?

Example: Conflict Tactics Example: Conflict Tactics ScaleScale

We also are interested in whether your partner has done any of these things have happened We also are interested in whether your partner has done any of these things have happened to you during any relationship you’ve been in over the past year. Please tell me if this has to you during any relationship you’ve been in over the past year. Please tell me if this has happened in the past [YEAR], and, if so, how many times you think it has happened. happened in the past [YEAR], and, if so, how many times you think it has happened.

In the last [YEAR]…..In the last [YEAR]….. NoNo YesYes # Times# Times

Has your [PARTNER] pushed, grabbed, shoved, slapped, or shaken you? Has your [PARTNER] pushed, grabbed, shoved, slapped, or shaken you?

Has your [PARTNER] punched, choked, strangled, kicked, bitten, or you? Has your [PARTNER] punched, choked, strangled, kicked, bitten, or you?

Has your [PARTNER] thrown an object at you or tried to hit you with an object? Has your [PARTNER] thrown an object at you or tried to hit you with an object? Has your [PARTNER] threatened you with a knife or gun? Has your [PARTNER] threatened you with a knife or gun? Has your [PARTNER] ever shot at or stabbed you? Has your [PARTNER] ever shot at or stabbed you? Has your [PARTNER] tried to stop you from working or studying?Has your [PARTNER] tried to stop you from working or studying? Has your [PARTNER] tried to stop you from having contact with family, friends or co-workers?Has your [PARTNER] tried to stop you from having contact with family, friends or co-workers? Has your [PARTNER] become angry (e.g., yelled, gotten real upset) when you disagreed with Has your [PARTNER] become angry (e.g., yelled, gotten real upset) when you disagreed with

his or her point of view?his or her point of view? Has your [PARTNER] damaged, destroyed, hid or thrown out any of your clothes or Has your [PARTNER] damaged, destroyed, hid or thrown out any of your clothes or

possessions?possessions? Has your [PARTNER] damaged or destroyed any other property when angry with you? Has your [PARTNER] damaged or destroyed any other property when angry with you? Has your [PARTNER] locked you out of the house? Has your [PARTNER] locked you out of the house? Has your [PARTNER] insulted or shamed you in front of others?Has your [PARTNER] insulted or shamed you in front of others? Has your [PARTNER] threatened to leave you? Has your [PARTNER] threatened to leave you? Has your [PARTNER] called you stupid, fat or ugly?Has your [PARTNER] called you stupid, fat or ugly? Has your [PARTNER] used physical force or threats of force to make you have sex when you Has your [PARTNER] used physical force or threats of force to make you have sex when you

didn’t want to?didn’t want to?

Simple additive scales or arbitrary scalesSimple additive scales or arbitrary scales Example: social class – a combination of income, Example: social class – a combination of income,

job category, and location (plus subjective or job category, and location (plus subjective or perceptual factors)perceptual factors)

Factor analysisFactor analysis Rather than use an additive or other Rather than use an additive or other

computational technique to arrange items in a computational technique to arrange items in a scale, factor analysis offers the possibility of scale, factor analysis offers the possibility of identifying and quantifying underlying patterns or identifying and quantifying underlying patterns or dimensions among the items in a scale. It avoids dimensions among the items in a scale. It avoids the assumption that all items tap the same the assumption that all items tap the same dimension of the phenomenon or construct.dimension of the phenomenon or construct.

Examples:Examples: NeighborhoodsNeighborhoods Childhood exposure to violenceChildhood exposure to violence Organizational climateOrganizational climate

I am not paid enough for what I do. The demands and nature of police work and the personal problems it could cause (like alcoholism, emotional depression, and anxiety) are about the same as in most other jobs. (reverse scored) The public criticizes the members of my profession unfairly. The administration/management causes pressure and job stress.

The criminal sentences given in court are about the right length and severity. (reverse scored) The administration/management seems to understand and care about my problems. (reverse scored) When people are wrongly criticized, the administration/management does not give them much support. Working a night shift disrupts my eating and sleeping. Working a night shift does not cause problems for my family or social life. (reverse scored) Work rules unnecessarily restrict me from earning extra money on my own time. I find myself in conflicts with coworkers that escalate into angry exchanges.

Source: Liberman et al., Work Environment Inventory, 2002

Procedures: The researcher constructs a correlation Procedures: The researcher constructs a correlation matrix, and factors that share a high correlation matrix, and factors that share a high correlation among themselves are organized computationally among themselves are organized computationally as a "factor." Each items is given a factor "loading" as a "factor." Each items is given a factor "loading" that shows its relative correlation within the factor.that shows its relative correlation within the factor.

The researcher has the choice of either selecting The researcher has the choice of either selecting those items within each factor that best represent those items within each factor that best represent the factor (those with the highest factor loads), or the factor (those with the highest factor loads), or using the using the factor score, factor score, a composite index of the a composite index of the items based on weights assigned that reflect their items based on weights assigned that reflect their factor loads.factor loads.

Criticisms:Criticisms: It is sample-specific. The results will vary with the It is sample-specific. The results will vary with the

response patterns of the sample. Addition of a few cases response patterns of the sample. Addition of a few cases can alter the factor scores.can alter the factor scores.

It often is abused and used in a context void of theory. In It often is abused and used in a context void of theory. In this regard, it is simply an exercise in "barefoot this regard, it is simply an exercise in "barefoot empiricism:" aimlessly tiptoeing through the data.empiricism:" aimlessly tiptoeing through the data.

When used to analyze data that may itself have When used to analyze data that may itself have measurement errors (e.g., arrest records or bad scales, the measurement errors (e.g., arrest records or bad scales, the errors approach a level that cannot be tolerated.errors approach a level that cannot be tolerated.

Evaluating Scales or Evaluating Scales or Measures: Validity and Measures: Validity and

ReliabilityReliability Validity and reliability are Validity and reliability are

assessment tools that are, like a assessment tools that are, like a Greek chorus, a constant presence in Greek chorus, a constant presence in the background, standing off-center the background, standing off-center or off-stage, commenting on the or off-stage, commenting on the state of affairs and suggesting plots state of affairs and suggesting plots and weaknesses that undermine the and weaknesses that undermine the affairs of state at the center stage.affairs of state at the center stage.

ValidityValidity Validity asks, quite simply, did we measure Validity asks, quite simply, did we measure

what we thought we measured? what we thought we measured? A variable is validly measured if it accurately A variable is validly measured if it accurately

measures what you want it to measure. measures what you want it to measure. Thus, we ask whether what we observed a Thus, we ask whether what we observed a function of the actual phenomena and function of the actual phenomena and relationships we have hypothesized, or is it relationships we have hypothesized, or is it an artifact of the research design (especially an artifact of the research design (especially measurement) that we used to generate measurement) that we used to generate these data?these data?

In fact, some of the trends that we think In fact, some of the trends that we think have held up consistently regarding some have held up consistently regarding some simple relationships in criminality are simple relationships in criminality are explained better by the artifacts of the study explained better by the artifacts of the study design than by the behaviors themselves.design than by the behaviors themselves.

Face validityFace validity Am I measuring what I think I am measuring?Am I measuring what I think I am measuring? Example: Family supervision -- LIKE PARENTSExample: Family supervision -- LIKE PARENTS

Content validityContent validity Does the item measure the concept it addresses? This is Does the item measure the concept it addresses? This is

dimension of validity similar to face validity. Yet it differs dimension of validity similar to face validity. Yet it differs in one important way: it refers to the ability of the item in one important way: it refers to the ability of the item to distinguish among people within a population.to distinguish among people within a population.

Examples: Test scores -- all students score the same Examples: Test scores -- all students score the same SRD scales -- kids with black eyes report NONE on the SRD scales -- kids with black eyes report NONE on the

item relating to item relating to fightingfighting

Types of ValidityTypes of Validity

Construct validityConstruct validity Am I measuring what the theory states and what the Am I measuring what the theory states and what the

construct implies? The match between the theoretical and construct implies? The match between the theoretical and the operational definitions of the concept. Error could lie in the operational definitions of the concept. Error could lie in the measurement, or it could be in the formulation of the the measurement, or it could be in the formulation of the construct, but something is mismatched in this critical construct, but something is mismatched in this critical relationship.relationship.

Examples: Fighting -> deviance --- maybe some behaviors Examples: Fighting -> deviance --- maybe some behaviors are normative!are normative!

Concurrent validityConcurrent validity Does the item have the ability to accurately state the present Does the item have the ability to accurately state the present

state of another variable?state of another variable? Example: Measures of spouse assault from one member of Example: Measures of spouse assault from one member of

the couplethe couple Predictive validityPredictive validity

Does the item accurately forecast behaviors or outcomes in Does the item accurately forecast behaviors or outcomes in the future?the future?

Examples: "Dangerousness" and future behavior after releaseExamples: "Dangerousness" and future behavior after release "Rehabilitation" and parole outcome"Rehabilitation" and parole outcome

Convergent validityConvergent validity How consistent or distinctive are multiple measures of the How consistent or distinctive are multiple measures of the

same or multiple constructs? Three measures of the same same or multiple constructs? Three measures of the same construct should point in the same direction. One measure of construct should point in the same direction. One measure of different constructs should yield distinctive results.different constructs should yield distinctive results.

Examples: Using husband and wife self-reports of both Examples: Using husband and wife self-reports of both victimization and offending to measure spouse assaultvictimization and offending to measure spouse assault

Using MMPI (standardized)and Rorschach (projective, Using MMPI (standardized)and Rorschach (projective, subjective) tests to measure psychopathologysubjective) tests to measure psychopathology

Another element of this strategy is the use of Another element of this strategy is the use of multiple multiple methods. methods. For example, using participant observation to For example, using participant observation to determine whether drinking precedes gang fights determine whether drinking precedes gang fights

Techniques for assessing validity in Techniques for assessing validity in surveyssurveys Social desirability scalesSocial desirability scales Lie detector testsLie detector tests Known-group testsKnown-group tests Secondary sources of dataSecondary sources of data

ReliabilityReliability A reliability coefficient is a measure of the A reliability coefficient is a measure of the

consistency and stability of measurement consistency and stability of measurement across subjects and populations.across subjects and populations. ConsistencyConsistency: a reliable variable is one where you : a reliable variable is one where you

keep getting the same value every time you keep getting the same value every time you measure it.measure it.

Stability Stability refers to the consistency of measurement refers to the consistency of measurement across periods. Simplest example is the test-retest across periods. Simplest example is the test-retest score -- -scores should be consistent across time score -- -scores should be consistent across time (after controlling for rival causal factors such as (after controlling for rival causal factors such as history or maturation) Internal history or maturation) Internal consistency consistency refers refers to the associations among items within the to the associations among items within the measurement of a complex phenomenon.measurement of a complex phenomenon.

Examples:Examples: Weight – how consistent is a digital v mechanical Weight – how consistent is a digital v mechanical

scale?scale? Parental supervisionParental supervision

Types of ReliabilityTypes of Reliability Inter-rater reliabilityInter-rater reliability refers to the degree of agreement refers to the degree of agreement

between two independent people on a measure. The between two independent people on a measure. The judges of Olympic skating provide a good test each year of judges of Olympic skating provide a good test each year of the reliability of the judging technique. When the East the reliability of the judging technique. When the East German judge holds up a number very dissimilar from the German judge holds up a number very dissimilar from the other scores, we question whether the measure of scoring other scores, we question whether the measure of scoring has good inter-rater reliability. has good inter-rater reliability.

Test-retest reliabilityTest-retest reliability refers to the relationship between the refers to the relationship between the score a person gets on one occasion on one variable and score a person gets on one occasion on one variable and the score he or she receives on a subsequent occasion. The the score he or she receives on a subsequent occasion. The LSAT has good test-retest reliability: in general, scores on LSAT has good test-retest reliability: in general, scores on one administration are similar to the score he or she one administration are similar to the score he or she receives on a second administration (compared to others receives on a second administration (compared to others who also are taking a second administration) who also are taking a second administration)

Multiple instrumentationMultiple instrumentation -- e.g., randomization of items -- e.g., randomization of items within multiple administrations of a test (used widely by within multiple administrations of a test (used widely by the Educational Testing Service and the major polling the Educational Testing Service and the major polling organizations).organizations).

Split-half methodsSplit-half methods -- random sampling within a test -- random sampling within a test population to determine internal consistency.population to determine internal consistency.

Measuring ReliabilityMeasuring Reliability Cronbach's alpha is used most widely, and is Cronbach's alpha is used most widely, and is

available on most statistical packagesavailable on most statistical packages

where N  is equal to the number of items and r-bar is the where N  is equal to the number of items and r-bar is the average inter-item correlation among the items.  average inter-item correlation among the items. 

If you increase the number of items, you increase If you increase the number of items, you increase Cronbach's alpha.  Additionally, if the average Cronbach's alpha.  Additionally, if the average inter-item correlation is low, alpha will be low.  As inter-item correlation is low, alpha will be low.  As the average inter-item correlation increases, the average inter-item correlation increases, Cronbach's alpha increases as well. Cronbach's alpha increases as well.

CFI, RMSEA, etc. – “latent construct” measures – CFI, RMSEA, etc. – “latent construct” measures – shows whether all items form a single construct shows whether all items form a single construct when measured concurrently.when measured concurrently.

The Relationship between The Relationship between Validity and ReliabilityValidity and Reliability

You must have reliability in order to have (measurement) validity. You must have reliability in order to have (measurement) validity. If everyone rating a variable came up with a different score every If everyone rating a variable came up with a different score every time the variable was measured, you can't draw an inference time the variable was measured, you can't draw an inference about the measure of the variable. You don't know if the measure about the measure of the variable. You don't know if the measure is accurate, or if it is not tapping into the dimension you really is accurate, or if it is not tapping into the dimension you really want to measure.want to measure.

But you can have reliability without validity. For example, you But you can have reliability without validity. For example, you could use hat size as a measure of IQ. Reliable yes, but valid? could use hat size as a measure of IQ. Reliable yes, but valid? Hardly.Hardly. Nothing wrong with the tape measure, differences from one Nothing wrong with the tape measure, differences from one

measurement to the next aren't likely to vary much. But hat size has measurement to the next aren't likely to vary much. But hat size has little to do with IQ, so why measure?little to do with IQ, so why measure?

Reasonable people will argue about validity. It is rarely an all or Reasonable people will argue about validity. It is rarely an all or nothing assessment, as in the case of age. Validity means not just nothing assessment, as in the case of age. Validity means not just that the measure was accurate and clear, but that it was true.that the measure was accurate and clear, but that it was true.

Consider the different measures of faculty productivity: article Consider the different measures of faculty productivity: article credits, pages per faculty member, prestige of law reviews or credits, pages per faculty member, prestige of law reviews or journals, footnotes per faculty member, etc. (What about teaching journals, footnotes per faculty member, etc. (What about teaching productivity???) All measures are quite reliable, but how valid are productivity???) All measures are quite reliable, but how valid are they re: "productivity"?they re: "productivity"?