Chapter 7: Types of Chapter 7: Types of Educational MeasuresEducational Measures

EDUC 502EDUC 502November 7, 2005November 7, 2005

Opening taskOpening taskForm six groups of five students.Form six groups of five students.Have the first student hand copies of Have the first student hand copies of

his/her abstract out to everyone in the his/her abstract out to everyone in the group to read.group to read.

As you read through the abstract, make As you read through the abstract, make notes on any points that seem unclear or notes on any points that seem unclear or incorrect (BE BRUTAL!!)incorrect (BE BRUTAL!!)

Repeat until everyone has had their Repeat until everyone has had their abstract evaluated.abstract evaluated.

Classifying educational measures Classifying educational measures

Cognitive or non-cognitive Cognitive or non-cognitive Cognitive measures focus on what a person Cognitive measures focus on what a person

knows or is able to do mentally knows or is able to do mentally Non-cognitive measures focus on affective Non-cognitive measures focus on affective

traits or characteristics (e.g, personality traits, traits or characteristics (e.g, personality traits, attitudes, values, interests, preferences, etc.) attitudes, values, interests, preferences, etc.)

Classifying educational measuresClassifying educational measures

Commercially prepared or locally Commercially prepared or locally developed developed Commercially prepared measures are Commercially prepared measures are

developed for widespread use with a focus on developed for widespread use with a focus on technical merit technical merit

Locally prepared measures are developed by Locally prepared measures are developed by a researcher for specific situations with some, a researcher for specific situations with some, but not extensive, concern for technical but not extensive, concern for technical characteristics characteristics

Classifying educational measures Classifying educational measures

Commercially prepared or locally Commercially prepared or locally developed developed Commercially prepared measures are Commercially prepared measures are

developed for widespread use with a focus on developed for widespread use with a focus on technical merit technical merit

Locally prepared measures are developed by Locally prepared measures are developed by a researcher for specific situations with some, a researcher for specific situations with some, but not extensive, concern for technical but not extensive, concern for technical characteristics characteristics

Classifying educational measuresClassifying educational measures

Self-report or observations by others Self-report or observations by others Self-report measures require the subjects to Self-report measures require the subjects to

supply the response (e.g., tests, supply the response (e.g., tests, questionnaires, interviews, etc.) questionnaires, interviews, etc.)

Observations by others require subjects to be Observations by others require subjects to be observed by others who record the data (e.g., observed by others who record the data (e.g., observations, unobtrusive measures, etc.) observations, unobtrusive measures, etc.)

Four types of educational Four types of educational measures used in quantitative measures used in quantitative

research research

TestsTestsQuestionnairesQuestionnairesObservationsObservations InterviewsInterviews

TestsTests Norm-referenced - individual scores are Norm-referenced - individual scores are

interpreted relative to the scores of others in a interpreted relative to the scores of others in a well defined norming group (e.g., John's scores well defined norming group (e.g., John's scores places him in the 95th percentile; Sally's score is places him in the 95th percentile; Sally's score is in the bottom quartile) in the bottom quartile)

Criterion-referenced - individual scores are Criterion-referenced - individual scores are interpreted in terms of the student's performance interpreted in terms of the student's performance relative to some standard or criterion (e.g., relative to some standard or criterion (e.g., Jeanne passed the Louisiana High School Jeanne passed the Louisiana High School Graduate Exit Exam; Sammy did not make the Graduate Exit Exam; Sammy did not make the cutoff for being promoted to the 7th grade) cutoff for being promoted to the 7th grade)

TestsTestsStandardized tests have uniform Standardized tests have uniform

procedures for administration, scoring, and procedures for administration, scoring, and interpreting test scores interpreting test scores

Types of standardized tests Types of standardized tests Achievement - tests of content knowledge or Achievement - tests of content knowledge or

skills skills Aptitude - tests which are used to predict Aptitude - tests which are used to predict

future cognitive performance future cognitive performance

Questionnaires Questionnaires A questionnaire is an instrument containing A questionnaire is an instrument containing

statements designed to obtain a subject's statements designed to obtain a subject's perceptions, attitudes, beliefs, values, opinions, perceptions, attitudes, beliefs, values, opinions, or other non-cognitive traits or other non-cognitive traits

Personality inventories Personality inventories Psychological orientation (i.e., general psychological Psychological orientation (i.e., general psychological

adjustment) adjustment) Educational orientation (i.e., traits such as self-Educational orientation (i.e., traits such as self-

concept or self-esteem that are related to learning concept or self-esteem that are related to learning and motivation) and motivation)

QuestionnairesQuestionnaires Likert scales Likert scales Response options require the subject to Response options require the subject to

determine the extent to which they agree with determine the extent to which they agree with a statement a statement

An odd number of options provides for a middle or An odd number of options provides for a middle or neutral response (e.g., strongly agree, agree, neutral response (e.g., strongly agree, agree, neutral, disagree, or strongly disagree) neutral, disagree, or strongly disagree)

An even number of options eliminates a response An even number of options eliminates a response of neutral (e.g., strongly agree, agree, disagree, or of neutral (e.g., strongly agree, agree, disagree, or strongly disagree) strongly disagree)

QuestionnairesQuestionnairesChecklists - responses require subjects to Checklists - responses require subjects to

identify specific options from which they identify specific options from which they choose those options that appeal to them choose those options that appeal to them (e.g., which of the following dispositions (e.g., which of the following dispositions are important to a first year teacher)are important to a first year teacher)

Ranked items - responses require Ranked items - responses require students to place a limited number of students to place a limited number of items into sequential order (e.g., rank items into sequential order (e.g., rank these dispositions in order of importance)these dispositions in order of importance)

Problems with measuring non-Problems with measuring non-cognitive traits cognitive traits

Difficulty clearly defining what is being measured Difficulty clearly defining what is being measured (e.g., self-concept or self-esteem) (e.g., self-concept or self-esteem)

Response set - a tendency to respond the same Response set - a tendency to respond the same way to all items (e.g., strongly agreeing with way to all items (e.g., strongly agreeing with each statement) each statement)

Faking - a tendency to respond inaccurately Faking - a tendency to respond inaccurately (e.g., agreeing with statements because of the (e.g., agreeing with statements because of the negative consequences associated with negative consequences associated with disagreeing) disagreeing)

Observations Observations Direct observation of behaviors Direct observation of behaviors

Natural or controlled settings Natural or controlled settings Structured or unstructured observations Structured or unstructured observations Detached or involved observers Detached or involved observers

ObservationsObservations Inference Inference

Low inference - involves little if any inference Low inference - involves little if any inference on the observers part on the observers part Children are in their seats Children are in their seats Teacher uses math manipulatives Teacher uses math manipulatives

High inference - involves high levels of High inference - involves high levels of inference on the observers part inference on the observers part Children are happy Children are happy Teacher lectures effectively Teacher lectures effectively

ObservationsObservationsLaboratory observation Laboratory observation

Specified environment Specified environment Use of structured forms and procedures Use of structured forms and procedures Concern with demand characteristics Concern with demand characteristics

Structured field observation Structured field observation Natural setting Natural setting Use of structured forms and procedures Use of structured forms and procedures

Frequency counts Frequency counts Duration Duration

ObservationsObservationsAdvantages of observations Advantages of observations

Yields firsthand data without the Yields firsthand data without the contamination that can arise from tests, contamination that can arise from tests, inventories, or other self-report instruments inventories, or other self-report instruments

Allows for the description of behavior as it Allows for the description of behavior as it occurs naturally occurs naturally

Allow for the consideration of contextual Allow for the consideration of contextual factors that can influence the interpretation factors that can influence the interpretation and use of the results and use of the results

InterviewsInterviews Interviews involve orally questioning subjects Interviews involve orally questioning subjects

and recording their responses and recording their responses Advantages Advantages

Establish rapport Establish rapport Enhance motivation Enhance motivation Clarify responses through additional questioning Clarify responses through additional questioning Capture the depth and richness of responses Capture the depth and richness of responses Allow for flexibility Allow for flexibility Reduce "no response" and/or "neutral" responses Reduce "no response" and/or "neutral" responses

InterviewsInterviews Disadvantages Disadvantages

Time consuming Time consuming Expensive Expensive Small samples Small samples Subjective Subjective

Types of questions Types of questions Structured Structured Semi-structured Semi-structured Unstructured Unstructured Leading Leading

Locating and evaluating Locating and evaluating educational measures educational measures

Reviews Reviews Test authors and publishers Test authors and publishers Computerized ERIC search for relevant Computerized ERIC search for relevant

articles articles Reference sections of articles already Reference sections of articles already

read read

Criteria for evaluating instruments Criteria for evaluating instruments

Validity evidence Validity evidence Reliability evidence Reliability evidence Administration procedures Administration procedures Norming information for norm-referenced tests Norming information for norm-referenced tests

(NRTs) (NRTs) Standards for criterion-referenced tests (CRTs) Standards for criterion-referenced tests (CRTs) Avoidance of response problems in non-Avoidance of response problems in non-

cognitive measures cognitive measures Training observers and interviewers Training observers and interviewers High standards for observers using high High standards for observers using high

inference observations inference observations

Homework AssignmentHomework AssignmentP. 174 (3, 5, 9, 10, 13, 14)P. 174 (3, 5, 9, 10, 13, 14)