CHAPTER OVERVIEW Deciding on a Method Tests and Their Development Types of Tests Observational...

CHAPTER OVERVIEW

• Deciding on a Method• Tests and Their Development• Types of Tests• Observational Techniques• Questionnaires

SOME IMPORTANT FIRST CONSIDERATIONS

• Is the tool you propose to use reliable and valid?

• Base your choice of research tools on how you have asked the research question

TESTS AND THEIR DEVELOPMENT

• A test should measure the nature and extent of individual differences

• A good test differentiates people based on true scores

WHY USE TESTS?

• Help determine outcomes of experiments

• Can be used to diagnose strengths and weaknesses

• Assist in placing individuals in appropriate programs

• Assist in selecting applicants• Used to evaluate a program’s

effectiveness

TYPES OF TESTSOverview

What It Does

Achievement Test Assesses an individual’s knowledge of a specific area

Attitude Test Assesses an individual’s feelings about an issue

Personality Test Assesses stable individual behavior patterns

ACHIEVEMENT TEST SOURCES

• Standardized– Commercially prepared for wide use– Scoring instructions included

• Researcher-made– Designed by user for specific purpose– Scoring instructions specific to test

ACHIEVEMENT TESTS REFERENCING

What Comparison Do Tests Make?

• Norm-referenced—Individual’s scores are compared to results from a larger group

• Criterion-referenced—Individual’s scores are compared to defined performance standards

MULTIPLE-CHOICE ACHIEVEMENT ITEMS

Anatomy of a Multiple-Choice Item

12. Intelligence tests that are given to preschool children a. favor middle-class children b. have questionable construct validity c. are based on motor skills d. are no fun at all

STEMClearly written

CORRECT ANSWER

DISTRACTERSShould be plausible (b & c), not easily ruled out

(d)

TO USE OR NOT TO USE?• Advantages

– Versatile– Easy to score– Simple to take– Poor writers not

penalized– Good items used

again– Good distracters are

diagnostic– Hard to fake correct

answer

• Disadvantages– Limit student’s

options– No opportunity to

practice writing– Some people don’t do

well on them– Limits content to be

assessed– Items must be well

written

ITEM ANALYSIS: HOW TO TELL IF YOUR ITEMS WORK• Questions should discriminate

those who know the material from those who don’t

• Item analysis provides two measures of a question’s ability to discriminate– Difficulty index– Discrimination index

COMPUTING INDICESFirst Steps

1. Rank scores from highest to lowest2. Choose top 27% of scores for “high” group3. Choose bottom 27% of scores for “low”

group

COMPUTING INDICES12. Intelligence tests that are given to preschool children

a. favor middle-class children b. have questionable construct validity c. are based on motor skills

d. are no fun at all

Item Alternative A B C D Total

High Group (n = 41)

23 12 4 2 41

Low Group (n = 41)

11 9 15 6 41

Total 34 21 19 8 82

COMPUTING INDICESDifficulty index• Proportion who

answered item correctly

• D = NCh + NCl

T

Discrimination index• Proportion in high

group who answered correctly – proportion in low group who answered correctly

• D = NCh - NCl

(.5)T

NCh

= number of people in high group answering correctly

NCl= number of people in low group answering correctly

T = total number of people in high and low groups

RELATIONSHIP BETWEEN ITEM DISCRIMINATION AND DIFFICULTY

0%

+1.00

50% 100%

-1.00

0

Difficulty Level

Dis

crim

inat

ion

Lev

el

Perfect Discrimination When: ½ gets item right, ½ gets item wrong

& ½ right is in upper half, ½ wrong is in lower half

ATTITUDE TESTSAssess an Individual’s Feelings About

a Topic

Item Agree No Strong Feeling

Disagree

The day before Thanksgiving should be a holiday.

 ____

 ____

 ____

Final exams should be elective. ____ ____ ____

The dining room should serve gourmet food.

 ____

 ____

 ____

My parents don’t appreciate how smart I am.

 ____

 ____

 ____

My professors don’t appreciate how smart I am, either.

 ____

 ____

 ____

THURSTONE SCALESMethod of Equal Appearing

Intervals

• Statements indicating an attitude toward a topic are written

• Judges rank the statements from least favorable to most favorable

• Statements receiving consistent ratings are given the average score

• A set of statements covering the entire range of attitudes is selected

THURSTONE SCALESAdministration

• Respondents check items with which they agree– Well-formed attitudes indicated by

consistently checking either high or low items

– Poorly-formed or inconsistent attitudes indicated by inconsistent pattern or by checking off many neutral items

LIKERT SCALES

• Statements indicating an attitude toward a topic are written

• Items with clearly positive or negative attitudes are selected

• Statements are listed with a space for respondent to indicate degree of agreement

Item Rating

Government has no business funding child care programs.

SD D U A SA

Child care should be supported by federal, state, and local tax dollars.

SD D U A SA

A LIKERT SCALE

Directions: Indicate to what extent you agree or disagree with the statements listed below by circling one of the following:

SA means that you strongly agree with the statementA means that you agree with the statementU means that you are undecided about the statementD means that you disagree with the statementSD means that you strongly disagree with the statement

SCORING LIKERT RESPONSESMethod of Summated Ratings

• Items are weighted• Weights of unfavorable items are reversed• Average score is computed

Item Rating

Government has no business funding child care programs.

SD D U A SA

Child care should be supported by federal, state, and local tax dollars.

SD D U A SA

PERSONALITY TESTS

• Projective tests– Present respondent with ambiguous

stimulus

• Structured tests– Questions are objective

OBSERVATIONAL TECHNIQUES

• Researcher observes and records• Does not interfere with behavior

TECHNIQUES FOR RECORDING BEHAVIOR

• Duration—how long the behavior lasts• Frequency—how frequently the

behavior occurs • Interval (time sampling)—target

behaviors of each subject during set period of time

• Continuous—all behaviors of the subject

TECHNIQUES FOR RECORDING BEHAVIOR

Technique How it Works Example

Duration recording The researcher records the length of time that a behavior occurs.

How much time is spent in verbal interaction between two children?

Frequency recording The researcher records the number of times a behavior occurs.

How often are questions asked?

Interval recording The researcher observes a subject for a fixed amount of time.

Within a 60-second period, how many times do members of the group talk to another person?

Continuous recording The researcher records everything that happens.

During a 1-hour period, all the behavior of a 6-year-old boy is recorded.

OBSERVATIONAL TECHNIQUES? BE CAREFUL!

Pitfalls to Avoid

• Observer effects• Observer bias• Fatigue• Changing definitions

QUESTIONNAIRES

• What they are– Paper and pencil tests with structured

questions– Self-administered

QUESTIONNAIRES

• Advantages– Can be mailed out– Survey broad geographic area– Cheaper than one-on-one interview– Respondents may be more honest– Data easy to share with other

researchers• Disadvantage

– Low return rate

BASIC ASSUMPTIONS OF A QUESTIONNAIRE

• Does not make unreasonable demands upon the respondent

• Does not have a hidden purpose• Requests information that

respondents presumably have• Contains interesting questions• Does not request information that

could be obtained by other means

THE QUESTIONS

• The questionnaire contains questions that can be answered

• The questionnaire contains questions that are straightforward

THE FORMAT• The presentation is attractive,

professional, and easy-to-understand• Questions and pages are clearly

numbered• Directions are clear and explicit• Questions are objective• Questions are ordered sensibly• Transitions are used from one topic to

the next

THE IMPORTANCE OF USING A COVER LETTER

• Informs the recipient about the research

• Establishes the importance of the research

• Makes the recipient a part of the research