Psych 818Psych 818Measurement Types Measurement Types
& Construction& Construction
DeShon - 2005DeShon - 2005
Readings!Readings!
Dawis (1987)Dawis (1987) Test ConstructionTest Construction
Reckase (1996)Reckase (1996) Test constructionTest construction
Roberts et al (1990)Roberts et al (1990) ScalingScaling
Friedman & Amoo (1999)Friedman & Amoo (1999) Rating the rating scalesRating the rating scales
Do you know your level?Do you know your level?
Levels of analysis and measurementLevels of analysis and measurement A fundamental decision of any A fundamental decision of any
measurement is to select a level of measurement is to select a level of analysisanalysis
This decision is often made This decision is often made implicitly based on existing research implicitly based on existing research normsnorms
Consideration of levels of analysis can Consideration of levels of analysis can greatly improve your conceptualization greatly improve your conceptualization and measurementand measurement
Levels of AnalysisLevels of Analysis
Chemical & Physical
Biological
Neural
Cognitive
Behavioral
Social Adapted from: Newell (1990) - Unified Theories of Cognition
Levels PrinciplesLevels Principles
Mesarovic, Macko, & Takahara (1970)Mesarovic, Macko, & Takahara (1970)1.1. Level selection depends upon the Level selection depends upon the
investigator’s knowledge and interest in investigator’s knowledge and interest in the operation of the systemthe operation of the system
2.2. Contexts in which the system operates Contexts in which the system operates on different levels are not, in general, on different levels are not, in general, mutually relatedmutually related
3.3. The principles or laws used to The principles or laws used to characterize the system at one level characterize the system at one level cannot be derived from the principles cannot be derived from the principles used at other levelsused at other levels
Levels PrinciplesLevels Principles
Mesarovic, Macko, & Takahara (1970)Mesarovic, Macko, & Takahara (1970)4.4. Asymmetrical interdependence between Asymmetrical interdependence between
the functioning of a system at different the functioning of a system at different levelslevels
Higher levels constrain the functioning of Higher levels constrain the functioning of lower levelslower levels
5.5. Each level has its own set of terms, Each level has its own set of terms, concepts, and principlesconcepts, and principles
6.6. Understanding is increased by crossing Understanding is increased by crossing stratastrata
Lower = howLower = how Higher = whyHigher = why
Levels PrinciplesLevels Principles
Systems differ in decomposabilitySystems differ in decomposability Near decomposibility - SimonNear decomposibility - Simon
the degree to which the behavior of a system at the degree to which the behavior of a system at any one level is free of the interactions on a any one level is free of the interactions on a lower level and the degree which its interactions lower level and the degree which its interactions are irrelevant to the higher levels of the systemare irrelevant to the higher levels of the system
““Scientific knowledge is organized in levels, not Scientific knowledge is organized in levels, not because reduction in principle is impossible, but because reduction in principle is impossible, but because nature is organized in levels, and the because nature is organized in levels, and the pattern at each level is most clearly discerned by pattern at each level is most clearly discerned by abstracting from the details of the levels far abstracting from the details of the levels far below” below”
Levels PrinciplesLevels Principles
Example: Example: ConsciousnessConsciousness
Psychological Measurement Psychological Measurement Crosses the levelsCrosses the levels
Psychology is a very broad field of Psychology is a very broad field of investigationinvestigation Hormones, chemicals, genes to …Hormones, chemicals, genes to … Politics of nationsPolitics of nations
PhysiologicalPhysiological
Hormones (estrogen) , Neurohormones Hormones (estrogen) , Neurohormones (dopamine) & Neurotransmitters (Seritonin)(dopamine) & Neurotransmitters (Seritonin)
GenesGenes Circadian cyclesCircadian cycles PET ScansPET Scans CAT scansCAT scans EEG (electroencephalograph)EEG (electroencephalograph) fMRIfMRI PupilometerPupilometer
fMRIfMRI
Based substantially on Donder’s subtraction method
Interviews and JudgmentInterviews and Judgment
Job InterviewsJob Interviews Structured vs. UnstructuredStructured vs. Unstructured
Clinical InterviewsClinical Interviews Observer AttributionsObserver Attributions Jury deliberationJury deliberation
Often categorical Often categorical Dawes – Mechanical vs. judgmental Dawes – Mechanical vs. judgmental
accuracyaccuracy
Ability & AchievementAbility & Achievement
Multiple ChoiceMultiple Choice Constructed ResponseConstructed Response Physical AbilitiesPhysical Abilities
A tale of Firefighters and Ice skaters…A tale of Firefighters and Ice skaters…
Self-ReportSelf-Report
PersonalityPersonality Experiences & BiodataExperiences & Biodata AttitudesAttitudes BeliefsBeliefs Affect?Affect?
Cognitive process? Cognitive process? Nisbett & Wilson (1977). Telling more than we Nisbett & Wilson (1977). Telling more than we
can know. can know.
Other ReportOther Report PeersPeers Significant othersSignificant others FamilyFamily TeachersTeachers SupervisorsSupervisors
Opportunity to observe slices of the Opportunity to observe slices of the personperson
Attribution problemsAttribution problems The “other” also has a personality and The “other” also has a personality and
fallible memoryfallible memory
ObservationalObservational NarrativesNarratives
attempts to record as much as possible of what happens attempts to record as much as possible of what happens within the focus of the observation. within the focus of the observation.
open-ended, flexible & Richopen-ended, flexible & Rich time consuming to both record and interprettime consuming to both record and interpret Diary descriptionDiary description
a chronological record of behavior made after the behavior a chronological record of behavior made after the behavior occursoccurs
Anecdotal recordAnecdotal record a descriptive narrative, recorded after the behavior occursa descriptive narrative, recorded after the behavior occurs
Running recordRunning record a sequential record over a given time recorded while the a sequential record over a given time recorded while the
behavior is occurringbehavior is occurring Specimen descriptionSpecimen description
detailed notes on an identified situation, recorded while the detailed notes on an identified situation, recorded while the behavior is occurringbehavior is occurring
ObservationalObservational
Time SamplingTime Sampling an observation of what happens within a given an observation of what happens within a given
period of time, coded with tallies or symbols period of time, coded with tallies or symbols while the behavior is occurring; used to while the behavior is occurring; used to document the frequency of specific behaviors. document the frequency of specific behaviors.
more objective than narrative records, less time more objective than narrative records, less time consuming, and it offers a way to observe and record consuming, and it offers a way to observe and record two or more people simultaneously.two or more people simultaneously.
closed-ended, limited to what happens in the closed-ended, limited to what happens in the specified time interval, and lacking in behavioral and specified time interval, and lacking in behavioral and contextual detail. contextual detail.
ObservationalObservational
Event SamplingEvent Sampling an observation of an event that has an observation of an event that has
been defined in advance and what been defined in advance and what happens before and after, recorded happens before and after, recorded briefly while it is taking placebriefly while it is taking place
objective objective event sampling is closed-ended and limited event sampling is closed-ended and limited
Experience SamplingExperience Sampling
AdvantagesAdvantages quantitatively study in-depth, quantitatively study in-depth,
everyday experiences, moods, and everyday experiences, moods, and behaviors in their natural context behaviors in their natural context
measures administered at random measures administered at random or scheduled intervals during the or scheduled intervals during the activity of interest to researchers activity of interest to researchers
less subject to biases in recall of less subject to biases in recall of events, mood, behaviors, or other events, mood, behaviors, or other psychologically important variables psychologically important variables
Experience SamplingExperience Sampling
DisadvantagesDisadvantages Intrusiveness Intrusiveness Missing data & lack of commitmentMissing data & lack of commitment InterruptionsInterruptions Hawthorne effects?Hawthorne effects? Huge amounts of dataHuge amounts of data
Internet AssessmentInternet Assessment
Equivalence?Equivalence? Time of dayTime of day ContextContext SecuritySecurity Breaks or pausesBreaks or pauses Technological problems – missing dataTechnological problems – missing data
Measurement MistakesMeasurement Mistakes
PhrenologyPhrenology
More MistakesMore Mistakes Craniometry …Craniometry …
Gould’s Mismeasure of manGould’s Mismeasure of man
Broca - demonstrated that Broca - demonstrated that women are inferior to men women are inferior to men because of their smaller because of their smaller crania. He argued against crania. He argued against higher education for women higher education for women because their small brains because their small brains couldn't handle the couldn't handle the demands demands
Effect of societal values on Effect of societal values on sciencescience
Measures are like handgunsMeasures are like handguns Nothing wrong with Nothing wrong with
measuring skulls…it’s the measuring skulls…it’s the inference that must made inference that must made cautiously and responsiblycautiously and responsibly
More MistakesMore Mistakes
Polygraph
More MistakesMore MistakesHandwriting Analysis
Probable MistakesProbable Mistakes
Penile PlethysmographologyPenile Plethysmographology http://www.smith-lawfirm.com/Scientific_Evidence_Brief.htmlhttp://www.smith-lawfirm.com/Scientific_Evidence_Brief.html
Started in Czechoslovakia to detect Started in Czechoslovakia to detect sexual preferencesexual preference
Now used in pedophilia casesNow used in pedophilia cases
Major issues with reliability and Major issues with reliability and fakability (validity)fakability (validity)
Constructing Multiple ChoiceConstructing Multiple Choice and Self-Report Measures and Self-Report Measures
1.1. Clear Construct DefinitionClear Construct Definition What is it…and what isn’t it!What is it…and what isn’t it! Identify & define sub-dimensionsIdentify & define sub-dimensions
goal orientation examplegoal orientation example
2.2. Specification ChartSpecification Chart Relative importance of sub-dimensionsRelative importance of sub-dimensions Number & type of items per dimensionNumber & type of items per dimension
3.3. Identify Population of interestIdentify Population of interest4.4. Review existing measures/interview sampleReview existing measures/interview sample5.5. Select scaling methodSelect scaling method6.6. Write ItemsWrite Items7.7. Pilot Test & Item AnalysisPilot Test & Item Analysis8.8. Item RevisionItem Revision
Item Writing PrinciplesItem Writing Principles See distributed fileSee distributed file General principlesGeneral principles
1.1. vocabulary level –should match the populationvocabulary level –should match the population Use reading level algorithms to checkUse reading level algorithms to check
2.2. avoid slang – differences in familiarityavoid slang – differences in familiarity3.3. avoid outdated/obsolete termsavoid outdated/obsolete terms4.4. Don’t use conjunctionsDon’t use conjunctions5.5. avoid absolute terms – difficult to defend due to avoid absolute terms – difficult to defend due to
exceptionsexceptions6.6. avoid unfamiliar abbreviations avoid unfamiliar abbreviations 7.7. use non-sexist language – pronouns and possessive use non-sexist language – pronouns and possessive
verbs for both sexesverbs for both sexes8.8. brevity – item stem and alternatives brief with all brevity – item stem and alternatives brief with all
necessary informationnecessary information9.9. one content topic per item - no compound itemsone content topic per item - no compound items10.10. Items should cover domain of constructItems should cover domain of construct
Writing the Item StemWriting the Item Stem1.1. Don’t include implicit assumptions in the stem (e.g., how Don’t include implicit assumptions in the stem (e.g., how
concerned are you about…)concerned are you about…)2.2. one clear, central problem completelyone clear, central problem completely3.3. meaningful without reference to alternativemeaningful without reference to alternative4.4. complete enough to serve as a short answer questioncomplete enough to serve as a short answer question5.5. Avoid irrelevant informationAvoid irrelevant information6.6. avoid awkward, weak sentence structureavoid awkward, weak sentence structure7.7. avoid vague or ambiguous words – more than one interpretation avoid vague or ambiguous words – more than one interpretation
possiblepossible8.8. avoid negative prefixes and double negatives – increases avoid negative prefixes and double negatives – increases
difficultydifficulty9.9. emphasize negative terms – prepares mental set (underline)emphasize negative terms – prepares mental set (underline)10.10. use left-branching for modifying clauses added to the main use left-branching for modifying clauses added to the main
clauseclause11.11. avoid dangling participlesavoid dangling participles12.12. avoid passive negative voiceavoid passive negative voice13.13. avoid central embedding – clause between subject and verbavoid central embedding – clause between subject and verb
Constructing AlternativesConstructing Alternatives
1.1. 4 or 5 alternatives per item – one correct and 3 4 or 5 alternatives per item – one correct and 3 or 4 distractorsor 4 distractors
2.2. all alternatives should be plausible, especially all alternatives should be plausible, especially the distractorsthe distractors
answered by those with knowledge correctly and answered by those with knowledge correctly and incorrectly by those without knowledgeincorrectly by those without knowledge
increases discriminability and reliability of scoresincreases discriminability and reliability of scores each distractor should be selected by a proportion of the each distractor should be selected by a proportion of the
examinees without knowledge examinees without knowledge distractors of equal difficulty and attractiveness to correct distractors of equal difficulty and attractiveness to correct
responseresponse
3.3. one best answerone best answer careful of evidence for incorrect alternativescareful of evidence for incorrect alternatives obtain opinion of expertsobtain opinion of experts document source of correct answerdocument source of correct answer
Constructing AlternativesConstructing Alternatives4.4. ““all of the above” alternative – lists of all of the above” alternative – lists of
characteristics/alternativescharacteristics/alternatives several solutions of equal merit, no obviously wrong distractorseveral solutions of equal merit, no obviously wrong distractor all alternatives of equal difficultyall alternatives of equal difficulty proportionate number of times used in items – 50%proportionate number of times used in items – 50% correct response a proportionate number of times correct response a proportionate number of times last alternativelast alternative
5.5. ““none of the above” alternative – for correct answer test none of the above” alternative – for correct answer test (math, spelling)(math, spelling)
notnot appropriate for “best answer” tests appropriate for “best answer” tests solutions of equal difficulty an meritsolutions of equal difficulty an merit used in terms of proportionate number of times – 50%used in terms of proportionate number of times – 50% correct response a proportional number of timescorrect response a proportional number of times last alternativelast alternative lengthens test timelengthens test time
6.6. do notdo not use “all” and “none” alternatives in the same test use “all” and “none” alternatives in the same test
Constructing AlternativesConstructing Alternatives
7.7. correct answer in same position and correct answer in same position and equal number of timesequal number of times
8.8. position of correct response should position of correct response should be randomizedbe randomized
9.9. numerical sequences should be in numerical sequences should be in nonnon-random order-random order
Constructing AlternativesConstructing Alternatives
Direct clue errorsDirect clue errorsdistractors correct answer
specific instances general rule
short statements long statements
negative statements positive statements
positive statements negative statements
unqualified statements qualified statements
means to end end product
professional jargon basic, simple, concise terms
laymen language/general technical definition
incomplete statement rule complete statement rule
all point in common direction point in opposite
direction or frame of reference frame of reference
Constructing AlternativesConstructing Alternatives More direct cluesMore direct clues
items asked for best plan, combination or sequenceitems asked for best plan, combination or sequence one alternative contains all stepsone alternative contains all steps one alternative omits a certain stepone alternative omits a certain step one alternative has a unique set of elementsone alternative has a unique set of elements
obviously wrong distractor – direct opposite of correct answerobviously wrong distractor – direct opposite of correct answer alternative association/clang association – similarity in sound of item stem alternative association/clang association – similarity in sound of item stem
and correct answerand correct answer duplicating language – similar phrases or synonyms in stem and answerduplicating language – similar phrases or synonyms in stem and answer items asking the meaning of self descriptive terms – infer answer from items asking the meaning of self descriptive terms – infer answer from
common sensecommon sense overlapping alternatives among items – alternatives or answer to one item overlapping alternatives among items – alternatives or answer to one item
gives away answer to another item (ex.: same alternative, different answer)gives away answer to another item (ex.: same alternative, different answer) item duplicity – inadvertently give away answer to other item with extraneous item duplicity – inadvertently give away answer to other item with extraneous
information in stem of alternativesinformation in stem of alternatives
consequences of direct clue errorsconsequences of direct clue errors all people answer the item correctly – no discriminationall people answer the item correctly – no discrimination clues used only by knowledgeable testee – less harmfulclues used only by knowledgeable testee – less harmful item easier for test-wise individual – able to infer correct answeritem easier for test-wise individual – able to infer correct answer
measures test wiseness rather then subject matter, lowers difficulty of test, testees can determine the measures test wiseness rather then subject matter, lowers difficulty of test, testees can determine the correct answer through reason rather then knowledgecorrect answer through reason rather then knowledge
Constructing AlternativesConstructing Alternatives Indirect Clue Errors - less obvious, require more common sense Indirect Clue Errors - less obvious, require more common sense
reasoning to detect and infer correct answerreasoning to detect and infer correct answer1.1. grammatical construction of stem and alternatives are inconsistent grammatical construction of stem and alternatives are inconsistent
except for the correct alternative, grammatical inconsistencies – except for the correct alternative, grammatical inconsistencies – source of confusionsource of confusion
tense of verbtense of verb use of articles “a”, and “an”use of articles “a”, and “an” use of pronoun without appropriate referent in stemuse of pronoun without appropriate referent in stem use of plural and singular nounsuse of plural and singular nouns
2.2. overlapping distractors – say same thing in different waysoverlapping distractors – say same thing in different ways neither is “one best” answerneither is “one best” answer
3.3. overlapping/logically impossible numerical seriesoverlapping/logically impossible numerical series cancel overlapping series outcancel overlapping series out
4.4. stray alternatives – not parallel with others alternatives in form or stray alternatives – not parallel with others alternatives in form or conceptual levelconceptual level
ambiguous and multiple right answersambiguous and multiple right answers5.5. concept split among alternatives – stem requires selection of concept split among alternatives – stem requires selection of
appropriate consequence and alternatives from two separate groupsappropriate consequence and alternatives from two separate groups do do not not follow from the central problem in the stemfollow from the central problem in the stem represent further qualifications of stem that are reasononablerepresent further qualifications of stem that are reasononable
6.6. tag along phrases – embellish alternative with implausible phrases to tag along phrases – embellish alternative with implausible phrases to make the distractor more attractivemake the distractor more attractive
discredits the distractordiscredits the distractor
Item Response FormatsItem Response Formats Know what your anchors mean to Know what your anchors mean to
respondents (see Freidman & Amoo (1989)respondents (see Freidman & Amoo (1989) Don’t force a response (e.g., 4 point scales)Don’t force a response (e.g., 4 point scales)
Respondents will express an attitude or belief Respondents will express an attitude or belief that they don’t actually holdthat they don’t actually hold
Scale should be balanced (equal number of Scale should be balanced (equal number of good and bad scale points)good and bad scale points)
Understand that the asymmetry of Understand that the asymmetry of comparisons (do you prefer x to y is not the comparisons (do you prefer x to y is not the same as y to x)same as y to x)
# of scale points = 7+ or -2 (5-9)# of scale points = 7+ or -2 (5-9) Much more on this when we get to scalingMuch more on this when we get to scaling