Psych 818Psych 818Measurement Types Measurement Types

& Construction& Construction

DeShon - 2005DeShon - 2005

Readings!Readings!

Dawis (1987)Dawis (1987) Test ConstructionTest Construction

Reckase (1996)Reckase (1996) Test constructionTest construction

Roberts et al (1990)Roberts et al (1990) ScalingScaling

Friedman & Amoo (1999)Friedman & Amoo (1999) Rating the rating scalesRating the rating scales

Do you know your level?Do you know your level?

Levels of analysis and measurementLevels of analysis and measurement A fundamental decision of any A fundamental decision of any

measurement is to select a level of measurement is to select a level of analysisanalysis

This decision is often made This decision is often made implicitly based on existing research implicitly based on existing research normsnorms

Consideration of levels of analysis can Consideration of levels of analysis can greatly improve your conceptualization greatly improve your conceptualization and measurementand measurement

Levels of AnalysisLevels of Analysis

Chemical & Physical

Biological

Neural

Cognitive

Behavioral

Social Adapted from: Newell (1990) - Unified Theories of Cognition

Levels PrinciplesLevels Principles

Mesarovic, Macko, & Takahara (1970)Mesarovic, Macko, & Takahara (1970)1.1. Level selection depends upon the Level selection depends upon the

investigator’s knowledge and interest in investigator’s knowledge and interest in the operation of the systemthe operation of the system

2.2. Contexts in which the system operates Contexts in which the system operates on different levels are not, in general, on different levels are not, in general, mutually relatedmutually related

3.3. The principles or laws used to The principles or laws used to characterize the system at one level characterize the system at one level cannot be derived from the principles cannot be derived from the principles used at other levelsused at other levels

Levels PrinciplesLevels Principles

Mesarovic, Macko, & Takahara (1970)Mesarovic, Macko, & Takahara (1970)4.4. Asymmetrical interdependence between Asymmetrical interdependence between

the functioning of a system at different the functioning of a system at different levelslevels

Higher levels constrain the functioning of Higher levels constrain the functioning of lower levelslower levels

5.5. Each level has its own set of terms, Each level has its own set of terms, concepts, and principlesconcepts, and principles

6.6. Understanding is increased by crossing Understanding is increased by crossing stratastrata

Lower = howLower = how Higher = whyHigher = why

Levels PrinciplesLevels Principles

Systems differ in decomposabilitySystems differ in decomposability Near decomposibility - SimonNear decomposibility - Simon

the degree to which the behavior of a system at the degree to which the behavior of a system at any one level is free of the interactions on a any one level is free of the interactions on a lower level and the degree which its interactions lower level and the degree which its interactions are irrelevant to the higher levels of the systemare irrelevant to the higher levels of the system

““Scientific knowledge is organized in levels, not Scientific knowledge is organized in levels, not because reduction in principle is impossible, but because reduction in principle is impossible, but because nature is organized in levels, and the because nature is organized in levels, and the pattern at each level is most clearly discerned by pattern at each level is most clearly discerned by abstracting from the details of the levels far abstracting from the details of the levels far below” below”

Levels PrinciplesLevels Principles

Example: Example: ConsciousnessConsciousness

Psychological Measurement Psychological Measurement Crosses the levelsCrosses the levels

Psychology is a very broad field of Psychology is a very broad field of investigationinvestigation Hormones, chemicals, genes to …Hormones, chemicals, genes to … Politics of nationsPolitics of nations

PhysiologicalPhysiological

Hormones (estrogen) , Neurohormones Hormones (estrogen) , Neurohormones (dopamine) & Neurotransmitters (Seritonin)(dopamine) & Neurotransmitters (Seritonin)

GenesGenes Circadian cyclesCircadian cycles PET ScansPET Scans CAT scansCAT scans EEG (electroencephalograph)EEG (electroencephalograph) fMRIfMRI PupilometerPupilometer

fMRIfMRI

Based substantially on Donder’s subtraction method

Interviews and JudgmentInterviews and Judgment

Job InterviewsJob Interviews Structured vs. UnstructuredStructured vs. Unstructured

Clinical InterviewsClinical Interviews Observer AttributionsObserver Attributions Jury deliberationJury deliberation

Often categorical Often categorical Dawes – Mechanical vs. judgmental Dawes – Mechanical vs. judgmental

accuracyaccuracy

Ability & AchievementAbility & Achievement

Multiple ChoiceMultiple Choice Constructed ResponseConstructed Response Physical AbilitiesPhysical Abilities

A tale of Firefighters and Ice skaters…A tale of Firefighters and Ice skaters…

Self-ReportSelf-Report

PersonalityPersonality Experiences & BiodataExperiences & Biodata AttitudesAttitudes BeliefsBeliefs Affect?Affect?

Cognitive process? Cognitive process? Nisbett & Wilson (1977). Telling more than we Nisbett & Wilson (1977). Telling more than we

can know. can know.

Other ReportOther Report PeersPeers Significant othersSignificant others FamilyFamily TeachersTeachers SupervisorsSupervisors

Opportunity to observe slices of the Opportunity to observe slices of the personperson

Attribution problemsAttribution problems The “other” also has a personality and The “other” also has a personality and

fallible memoryfallible memory

ObservationalObservational NarrativesNarratives

attempts to record as much as possible of what happens attempts to record as much as possible of what happens within the focus of the observation. within the focus of the observation.

open-ended, flexible & Richopen-ended, flexible & Rich time consuming to both record and interprettime consuming to both record and interpret Diary descriptionDiary description

a chronological record of behavior made after the behavior a chronological record of behavior made after the behavior occursoccurs

Anecdotal recordAnecdotal record a descriptive narrative, recorded after the behavior occursa descriptive narrative, recorded after the behavior occurs

Running recordRunning record a sequential record over a given time recorded while the a sequential record over a given time recorded while the

behavior is occurringbehavior is occurring Specimen descriptionSpecimen description

detailed notes on an identified situation, recorded while the detailed notes on an identified situation, recorded while the behavior is occurringbehavior is occurring

ObservationalObservational

Time SamplingTime Sampling an observation of what happens within a given an observation of what happens within a given

period of time, coded with tallies or symbols period of time, coded with tallies or symbols while the behavior is occurring; used to while the behavior is occurring; used to document the frequency of specific behaviors. document the frequency of specific behaviors.

more objective than narrative records, less time more objective than narrative records, less time consuming, and it offers a way to observe and record consuming, and it offers a way to observe and record two or more people simultaneously.two or more people simultaneously.

closed-ended, limited to what happens in the closed-ended, limited to what happens in the specified time interval, and lacking in behavioral and specified time interval, and lacking in behavioral and contextual detail. contextual detail.

ObservationalObservational

Event SamplingEvent Sampling an observation of an event that has an observation of an event that has

been defined in advance and what been defined in advance and what happens before and after, recorded happens before and after, recorded briefly while it is taking placebriefly while it is taking place

objective objective event sampling is closed-ended and limited event sampling is closed-ended and limited

Experience SamplingExperience Sampling

AdvantagesAdvantages quantitatively study in-depth, quantitatively study in-depth,

everyday experiences, moods, and everyday experiences, moods, and behaviors in their natural context behaviors in their natural context

measures administered at random measures administered at random or scheduled intervals during the or scheduled intervals during the activity of interest to researchers activity of interest to researchers

less subject to biases in recall of less subject to biases in recall of events, mood, behaviors, or other events, mood, behaviors, or other psychologically important variables psychologically important variables

Experience SamplingExperience Sampling

DisadvantagesDisadvantages Intrusiveness Intrusiveness Missing data & lack of commitmentMissing data & lack of commitment InterruptionsInterruptions Hawthorne effects?Hawthorne effects? Huge amounts of dataHuge amounts of data

Internet AssessmentInternet Assessment

Equivalence?Equivalence? Time of dayTime of day ContextContext SecuritySecurity Breaks or pausesBreaks or pauses Technological problems – missing dataTechnological problems – missing data

Measurement MistakesMeasurement Mistakes

PhrenologyPhrenology

More MistakesMore Mistakes Craniometry …Craniometry …

Gould’s Mismeasure of manGould’s Mismeasure of man

Broca - demonstrated that Broca - demonstrated that women are  inferior to men women are  inferior to men because of their smaller because of their smaller crania. He argued against crania. He argued against higher education for women higher education for women because their small brains because their small brains couldn't handle the couldn't handle the demands demands

Effect of societal values on Effect of societal values on sciencescience

Measures are like handgunsMeasures are like handguns Nothing wrong with Nothing wrong with

measuring skulls…it’s the measuring skulls…it’s the inference that must made inference that must made cautiously and responsiblycautiously and responsibly

More MistakesMore Mistakes

Polygraph

More MistakesMore MistakesHandwriting Analysis

Probable MistakesProbable Mistakes

Penile PlethysmographologyPenile Plethysmographology http://www.smith-lawfirm.com/Scientific_Evidence_Brief.htmlhttp://www.smith-lawfirm.com/Scientific_Evidence_Brief.html

Started in Czechoslovakia to detect Started in Czechoslovakia to detect sexual preferencesexual preference

Now used in pedophilia casesNow used in pedophilia cases

Major issues with reliability and Major issues with reliability and fakability (validity)fakability (validity)

Constructing Multiple ChoiceConstructing Multiple Choice and Self-Report Measures and Self-Report Measures

1.1. Clear Construct DefinitionClear Construct Definition What is it…and what isn’t it!What is it…and what isn’t it! Identify & define sub-dimensionsIdentify & define sub-dimensions

goal orientation examplegoal orientation example

2.2. Specification ChartSpecification Chart Relative importance of sub-dimensionsRelative importance of sub-dimensions Number & type of items per dimensionNumber & type of items per dimension

3.3. Identify Population of interestIdentify Population of interest4.4. Review existing measures/interview sampleReview existing measures/interview sample5.5. Select scaling methodSelect scaling method6.6. Write ItemsWrite Items7.7. Pilot Test & Item AnalysisPilot Test & Item Analysis8.8. Item RevisionItem Revision

Item Writing PrinciplesItem Writing Principles See distributed fileSee distributed file General principlesGeneral principles

1.1. vocabulary level –should match the populationvocabulary level –should match the population Use reading level algorithms to checkUse reading level algorithms to check

2.2. avoid slang – differences in familiarityavoid slang – differences in familiarity3.3. avoid outdated/obsolete termsavoid outdated/obsolete terms4.4. Don’t use conjunctionsDon’t use conjunctions5.5. avoid absolute terms – difficult to defend due to avoid absolute terms – difficult to defend due to

exceptionsexceptions6.6. avoid unfamiliar abbreviations avoid unfamiliar abbreviations 7.7. use non-sexist language – pronouns and possessive use non-sexist language – pronouns and possessive

verbs for both sexesverbs for both sexes8.8. brevity – item stem and alternatives brief with all brevity – item stem and alternatives brief with all

necessary informationnecessary information9.9. one content topic per item - no compound itemsone content topic per item - no compound items10.10. Items should cover domain of constructItems should cover domain of construct

Writing the Item StemWriting the Item Stem1.1. Don’t include implicit assumptions in the stem (e.g., how Don’t include implicit assumptions in the stem (e.g., how

concerned are you about…)concerned are you about…)2.2. one clear, central problem completelyone clear, central problem completely3.3. meaningful without reference to alternativemeaningful without reference to alternative4.4. complete enough to serve as a short answer questioncomplete enough to serve as a short answer question5.5. Avoid irrelevant informationAvoid irrelevant information6.6. avoid awkward, weak sentence structureavoid awkward, weak sentence structure7.7. avoid vague or ambiguous words – more than one interpretation avoid vague or ambiguous words – more than one interpretation

possiblepossible8.8. avoid negative prefixes and double negatives – increases avoid negative prefixes and double negatives – increases

difficultydifficulty9.9. emphasize negative terms – prepares mental set (underline)emphasize negative terms – prepares mental set (underline)10.10. use left-branching for modifying clauses added to the main use left-branching for modifying clauses added to the main

clauseclause11.11. avoid dangling participlesavoid dangling participles12.12. avoid passive negative voiceavoid passive negative voice13.13. avoid central embedding – clause between subject and verbavoid central embedding – clause between subject and verb

Constructing AlternativesConstructing Alternatives

1.1. 4 or 5 alternatives per item – one correct and 3 4 or 5 alternatives per item – one correct and 3 or 4 distractorsor 4 distractors

2.2. all alternatives should be plausible, especially all alternatives should be plausible, especially the distractorsthe distractors

answered by those with knowledge correctly and answered by those with knowledge correctly and incorrectly by those without knowledgeincorrectly by those without knowledge

increases discriminability and reliability of scoresincreases discriminability and reliability of scores each distractor should be selected by a proportion of the each distractor should be selected by a proportion of the

examinees without knowledge examinees without knowledge distractors of equal difficulty and attractiveness to correct distractors of equal difficulty and attractiveness to correct

responseresponse

3.3. one best answerone best answer careful of evidence for incorrect alternativescareful of evidence for incorrect alternatives obtain opinion of expertsobtain opinion of experts document source of correct answerdocument source of correct answer

Constructing AlternativesConstructing Alternatives4.4. ““all of the above” alternative – lists of all of the above” alternative – lists of

characteristics/alternativescharacteristics/alternatives several solutions of equal merit, no obviously wrong distractorseveral solutions of equal merit, no obviously wrong distractor all alternatives of equal difficultyall alternatives of equal difficulty proportionate number of times used in items – 50%proportionate number of times used in items – 50% correct response a proportionate number of times correct response a proportionate number of times last alternativelast alternative

5.5. ““none of the above” alternative – for correct answer test none of the above” alternative – for correct answer test (math, spelling)(math, spelling)

notnot appropriate for “best answer” tests appropriate for “best answer” tests solutions of equal difficulty an meritsolutions of equal difficulty an merit used in terms of proportionate number of times – 50%used in terms of proportionate number of times – 50% correct response a proportional number of timescorrect response a proportional number of times last alternativelast alternative lengthens test timelengthens test time

6.6. do notdo not use “all” and “none” alternatives in the same test use “all” and “none” alternatives in the same test

Constructing AlternativesConstructing Alternatives

7.7. correct answer in same position and correct answer in same position and equal number of timesequal number of times

8.8. position of correct response should position of correct response should be randomizedbe randomized

9.9. numerical sequences should be in numerical sequences should be in nonnon-random order-random order

Constructing AlternativesConstructing Alternatives

Direct clue errorsDirect clue errorsdistractors correct answer

specific instances general rule

short statements long statements

negative statements positive statements

positive statements negative statements

unqualified statements qualified statements

means to end end product

professional jargon basic, simple, concise terms

laymen language/general technical definition

incomplete statement rule complete statement rule

all point in common direction point in opposite

direction or frame of reference frame of reference

Constructing AlternativesConstructing Alternatives More direct cluesMore direct clues

items asked for best plan, combination or sequenceitems asked for best plan, combination or sequence one alternative contains all stepsone alternative contains all steps one alternative omits a certain stepone alternative omits a certain step one alternative has a unique set of elementsone alternative has a unique set of elements

obviously wrong distractor – direct opposite of correct answerobviously wrong distractor – direct opposite of correct answer alternative association/clang association – similarity in sound of item stem alternative association/clang association – similarity in sound of item stem

and correct answerand correct answer duplicating language – similar phrases or synonyms in stem and answerduplicating language – similar phrases or synonyms in stem and answer items asking the meaning of self descriptive terms – infer answer from items asking the meaning of self descriptive terms – infer answer from

common sensecommon sense overlapping alternatives among items – alternatives or answer to one item overlapping alternatives among items – alternatives or answer to one item

gives away answer to another item (ex.: same alternative, different answer)gives away answer to another item (ex.: same alternative, different answer) item duplicity – inadvertently give away answer to other item with extraneous item duplicity – inadvertently give away answer to other item with extraneous

information in stem of alternativesinformation in stem of alternatives

consequences of direct clue errorsconsequences of direct clue errors all people answer the item correctly – no discriminationall people answer the item correctly – no discrimination clues used only by knowledgeable testee – less harmfulclues used only by knowledgeable testee – less harmful item easier for test-wise individual – able to infer correct answeritem easier for test-wise individual – able to infer correct answer

measures test wiseness rather then subject matter, lowers difficulty of test, testees can determine the measures test wiseness rather then subject matter, lowers difficulty of test, testees can determine the correct answer through reason rather then knowledgecorrect answer through reason rather then knowledge

Constructing AlternativesConstructing Alternatives Indirect Clue Errors - less obvious, require more common sense Indirect Clue Errors - less obvious, require more common sense

reasoning to detect and infer correct answerreasoning to detect and infer correct answer1.1. grammatical construction of stem and alternatives are inconsistent grammatical construction of stem and alternatives are inconsistent

except for the correct alternative, grammatical inconsistencies – except for the correct alternative, grammatical inconsistencies – source of confusionsource of confusion

tense of verbtense of verb use of articles “a”, and “an”use of articles “a”, and “an” use of pronoun without appropriate referent in stemuse of pronoun without appropriate referent in stem use of plural and singular nounsuse of plural and singular nouns

2.2. overlapping distractors – say same thing in different waysoverlapping distractors – say same thing in different ways neither is “one best” answerneither is “one best” answer

3.3. overlapping/logically impossible numerical seriesoverlapping/logically impossible numerical series cancel overlapping series outcancel overlapping series out

4.4. stray alternatives – not parallel with others alternatives in form or stray alternatives – not parallel with others alternatives in form or conceptual levelconceptual level

ambiguous and multiple right answersambiguous and multiple right answers5.5. concept split among alternatives – stem requires selection of concept split among alternatives – stem requires selection of

appropriate consequence and alternatives from two separate groupsappropriate consequence and alternatives from two separate groups do do not not follow from the central problem in the stemfollow from the central problem in the stem represent further qualifications of stem that are reasononablerepresent further qualifications of stem that are reasononable

6.6. tag along phrases – embellish alternative with implausible phrases to tag along phrases – embellish alternative with implausible phrases to make the distractor more attractivemake the distractor more attractive

discredits the distractordiscredits the distractor

Item Response FormatsItem Response Formats Know what your anchors mean to Know what your anchors mean to

respondents (see Freidman & Amoo (1989)respondents (see Freidman & Amoo (1989) Don’t force a response (e.g., 4 point scales)Don’t force a response (e.g., 4 point scales)

Respondents will express an attitude or belief Respondents will express an attitude or belief that they don’t actually holdthat they don’t actually hold

Scale should be balanced (equal number of Scale should be balanced (equal number of good and bad scale points)good and bad scale points)

Understand that the asymmetry of Understand that the asymmetry of comparisons (do you prefer x to y is not the comparisons (do you prefer x to y is not the same as y to x)same as y to x)

# of scale points = 7+ or -2 (5-9)# of scale points = 7+ or -2 (5-9) Much more on this when we get to scalingMuch more on this when we get to scaling