Upload
yvonne-yin
View
223
Download
0
Embed Size (px)
Citation preview
7/27/2019 Portage Collage Item Objektif
1/47
Multiple Choice Item
ConstructionD. Michele Jacobsen
Assistant Professor
7/27/2019 Portage Collage Item Objektif
2/47
Test Development Process
1. Statement of Goals
2. Content Outline
3. Table of Specifications
4. Item Selection 5. Item Construction
6. Composition of Instructions
7. Development of Answer Sheets
8. Construction of Answer Keys
9. Test Administration
10. Test Revision
7/27/2019 Portage Collage Item Objektif
3/47
Todays Focus
3. Table of Specifications
4. Item Selection5. Item Construction
7/27/2019 Portage Collage Item Objektif
4/47
Two Important Characteristics of Tests
Reliability
consistency
free from extraneous sources of error
Validity
how well a test measures what it is
supposed to measure
7/27/2019 Portage Collage Item Objektif
5/47
Formative vs. Summative Tests
Formative
monitor progress toward goals within a
course of study
Summative
assess overall achievement of course
goals
7/27/2019 Portage Collage Item Objektif
6/47
A. Table of Specifications
Blue print for test
Purpose
ensure proper emphasis given to all
elements of a course of study
content validity
Guide for writing items
7/27/2019 Portage Collage Item Objektif
7/47
Table of Specifications (2)
Start with Instructional Objectives
Biology 30: Circulation Unit Test
1. Identifies parts of circulation system.
2. Explains function of parts in relationto whole system.
3. Distinguishes between circulationsystem and other bodily systems.
7/27/2019 Portage Collage Item Objektif
8/47
Biology 30: Circulation
Levels Of Understanding
Content Knowledge Comprehension Application
Parts
Function
Relationto othersystems
7/27/2019 Portage Collage Item Objektif
9/47
Need to Consider and Decide:
Length of Test
Weight to be given to each objective
Weight to be given to each level oftaxonomy
Estimate number of items in each
cell
7/27/2019 Portage Collage Item Objektif
10/47
Biology 30: Circulation
Levels Of Understanding
Content Knowledge Comprehension Application
Parts 10 5 0 15
Function
5 5 5 15
Relationto other
systems
0 5 10 15
15 15 15 45
7/27/2019 Portage Collage Item Objektif
11/47
B. Item Selection
Types of Items
Objectively Scored (Selection)
true/false completion
matching
multiple choice
Subjectively Scored (Supply) interpretive exercises
essay
7/27/2019 Portage Collage Item Objektif
12/47
C. Item Construction
Selected Response Test Items
Item construction skills for valid and
reliable measures of student
achievement.
Guidelines which apply to all types
Specific Suggestions for writing each type
Advantages and Disadvantages of Each
7/27/2019 Portage Collage Item Objektif
13/47
Guidelines for Writing Objective Items
1. Construct at appropriate level ofdifficulty for examinees
2. Include Items at appropriate level ofdifficulty for purpose of test.
3. Test significant elements of acourse.
4. Write independent items.
7/27/2019 Portage Collage Item Objektif
14/47
5. Construct questions free fromextraneous reasons for problems.
6. Communicate the question in clear,
concise language.
7. In the correct alternatives,paraphrase statements from the text.
8. Exclude clues to correct answer.
II. Guidelines for Writing Objective Items
7/27/2019 Portage Collage Item Objektif
15/47
III. Guidelines for Writing Objective Items
9. Provide one correct answer.
10. Edit the Items.
7/27/2019 Portage Collage Item Objektif
16/47
Alternate Response Items
Involves the selection of one of twoalternatives
true / false
yes / no
right / wrong
fact / opinion
Mainly for Knowledge & Comprehension
Can be written at higher levels
7/27/2019 Portage Collage Item Objektif
17/47
True / False
Word statements clearly. Vague or ambiguous wording
will confuse students.
Avoid overgeneralizing.
Poor: Heavy smokin g causes lung cancer. T F
Better: Heavy smokin g often causes lung cancer. T F
Avoid Trick questions.
i .e., General Wolseley led Canadian tro op s to Mani tob a in
1870.
3 tr icks: Colon el, Br i t ish , Not yet a pro vinc e
Do not use trivial statements to pad out the number of
questions and marks to arrive at a predetermined level.
7/27/2019 Portage Collage Item Objektif
18/47
True / False
Statements should be entirely true, or
entirely false:
Unacceptable:
In King Lear, Regan ordered Gloucesters eyes to beplucked out and Gloucester died when he jumped off
the cliff of Dover.
Acceptable:
In King Lear, Regan ordered Gloucesters eyes to beplucked out. (T)
In King Lear, Gloucester died when he jumped off
the cliff at Dover. (F)
7/27/2019 Portage Collage Item Objektif
19/47
True / False
Avoid using universal descriptors such as
never, none, always, and all.
Testw ise stud ents w i l l recog nize that there are few
absolutes.
Avoid negative words, as they are often
overlooked by students. Poor: It was no t un heard of for Henry VIII to close
monaster ies in England. T F Better: Henry VIII clos ed som e monaster ies in England.
(T)
7/27/2019 Portage Collage Item Objektif
20/47
True / False
Do not include two ideas in one statement unless you areevaluating students understanding of cause and effect
relationships.
Poor: Porpo ises are able to commun icate because
they are mammals. T F
Better: Porpo ises are mammals. T F
Porpo ises are able to commun icate. T F
Provide a T and F beside each statement and ask
students to circle correct answer.
Avoids problem of stud ents wr i t ing i l leg ib le let ters .
7/27/2019 Portage Collage Item Objektif
21/47
True / False & Variations
Include more false than true statments in any given testand vary the number of false statements from test to test.
tendency to mark more statements true than false.
discrimination between those who know the content
and those who do not is greater for false expressions.
Avoid using negative statements.
Under the demands of the testing situation, students
may fail to see the negative qualifier.
7/27/2019 Portage Collage Item Objektif
22/47
Matching Items
Consist of
a column of premises
a column of responses
directions for matching the two.
Similar to multiple choice, but easier and more
efficient to construct
Can be written to assess Knowledge,Comprehension, Application, Analysis level
behaviors
7/27/2019 Portage Collage Item Objektif
23/47
Guidelines for Writing Matching Items
Provide clear instructions on how to indicate thecorrect answers.
Indicate whether the same response can be used
more than once.
Maintain grammatical consistency within and betweencolumns.
within a column: either sentence or point form
between columns: one or the other
Ensure that any matching question appears entirely
on one page.
7/27/2019 Portage Collage Item Objektif
24/47
2. Guidelines for Writing Matching Items
Provide an unequal number of premises and
responses
reduces guessing and elimination
increases measure of comprehension
Avoid designing questions which require
students to draw lines between premise and
response.
confusing for student and marker provide space for letter or number answers
7/27/2019 Portage Collage Item Objektif
25/47
3. Guidelines for Writing Matching Items
Make sure lists are homogeneous.
i.e., do not include items testing names, dates, and
events.
Instead, make every response plausible
Make the wording of the premises longer than
the wording of the responses.
Identify the items in one list with numbers andthose in the second list with letters.
7/27/2019 Portage Collage Item Objektif
26/47
7/27/2019 Portage Collage Item Objektif
27/47
Short Answer Test Items
Typically, the student is asked to reply with aword, phrase, name, or sentence, rather than a
more extended response.
Direct Questions / Short Answer
Who is the current Prime Minister of Canada?
Incomplete Sentences / Fill In the Blanks
The current Prime Minister of Canada is _____?
Items are fairly easy to construct and mark
Assess mainly knowledge, comprehension, and
some application.
7/27/2019 Portage Collage Item Objektif
28/47
7/27/2019 Portage Collage Item Objektif
29/47
7/27/2019 Portage Collage Item Objektif
30/47
III. Guidelines for Writing Short Answer Items
When an answer is to be expressed in numerical units, theunit should be stated.
Poor: If a room measures 7 metres by 4 metres, theperimeter is _____ .
Better: If a room measures 7 metres by 4 metres, theperimeter is _____ metres (or m).
Do not use too many blanks in completion items. Theemphasis should be on knowledge and comprehension,not mind reading!
Consider:In the year _____ , Prime Minister _________ signed the
__________ , which led to a ___________ which was__________ .
7/27/2019 Portage Collage Item Objektif
31/47
Multiple Choice Items
Interpretive Exercise
Guidelines for Writing
Advantages & Disadvantages
7/27/2019 Portage Collage Item Objektif
32/47
1. The capital city of Canada isa. Vancouverb. Montrealc. Toronto
*d. Ottawa
Terminology: Multiple Choice
1. Stem:presents the problem
2. Keyed Response:correct or best answer
3. Distracters: appear to be reasonable answers to the
examinee who does not know the content
4. Options:include the distracters and the keyed
response.
4
1
2
3
7/27/2019 Portage Collage Item Objektif
33/47
Ambiguity
Extrinsic
Desirable quality in
multiple choiceitems
Outside the item
Allows
discrimination
between those who
know material and
those who do not.
Intr insic
Undesirable
Shou ld be avo ided Ins ide the item
poor word ing
more than on e answer
Even those who know
content have dif f icu l ty
choo sing correct answer
7/27/2019 Portage Collage Item Objektif
34/47
Types of Multiple Choice Items
Correct Answer*
Only one correct response
Best Answer
requires examinee to select alternative
closest to being correct
fine distinctions
Multiple Answer
More than one correct or best answer
7/27/2019 Portage Collage Item Objektif
35/47
Interpretive Exercise
Usually begins with verbal, tabular or graphic
information which is the basis for 1 or more
multiple choice questions.
map, passage from a story, a poem, a cartoon
Can challenge students at various levels of
understanding
appl icat ion, analysis , syn thesis, evaluat ion
Exercise contains all information needed toanswer questions
Readily adaptive to the more important
outcomes of disciplines.
7/27/2019 Portage Collage Item Objektif
36/47
Interpretive Exercises (cont)
Examples
If student answers incorrectly it is because
they have not mastered the thinking or
reasoning required by the question, NOT
because they failed to memorize background
information.
Math questions: give students the formulas,
test ability to apply concepts, rather than ability
to memorize formulas.
7/27/2019 Portage Collage Item Objektif
37/47
Guidelines for Writing:
Multiple Choice Items
State stem in the form of a question. Weak
Canada is
a) a country
b) where you live
c) between the Pacific and Atlantic oceans
d) all of the above
Better
Between which two oceans is Canada located?
a) Atlantic and Indian
b) Atlantic and Pacific
c) Pacific and Indian
7/27/2019 Portage Collage Item Objektif
38/47
Guidelines for Writing: Stems
Place most of the subject matter in the Stem ensures full statement of problem
Eliminate extraneous material from the Stem
goal is to measure student achievement, not to present
new material maximize use of time for demonstrating understanding,
not reading ability
Avoid Negatively phrased Stems
students may miss the qualifier
use only when learning outcome requires this type of
differentiation
7/27/2019 Portage Collage Item Objektif
39/47
7/27/2019 Portage Collage Item Objektif
40/47
Example: Length of Alternatives
Neurotics are more likely than psychotics to
a. be dangerous to society
b. have delusional symptoms
c. be dangerous to themselves
*d. have insight into their own inappropriatebehavior but nevertheless feel rather
helpless in terms of dealing with theirdifficulties
7/27/2019 Portage Collage Item Objektif
41/47
II. Guidelines for Writing: MC
Make one of the alternatives the most clearly
correct or best answer
exception: multiple answer form
reduces intrinsic ambiguity reduces f rustrat ionduring test
Make distracters plausible
desire to attract students who really do NOT know the
answer to the question create distracters from elements of the correct response
improves reliability of item
7/27/2019 Portage Collage Item Objektif
42/47
7/27/2019 Portage Collage Item Objektif
43/47
III. Guidelines for Writing: MC
Avoid parallel language between the Stem andthe Correct Response
gives clues to keyed response
emphasizes testwiseness, not knowledge
Randomly distribute answers across the
alternative positions
inexperienced test writers emphasize b and c
alternatives (hide the answer!!)
do NOT use an interpretable order of keyed responses
7/27/2019 Portage Collage Item Objektif
44/47
IV. Guidelines for Writing: MC
Use qualifiers such as all of the
above and none of the above
sparingly
testwise students w i l l use proc ess ofel im inat ion to select answer
do NOT use to pad out the distracters
because you cannot th ink of another
one.
7/27/2019 Portage Collage Item Objektif
45/47
Advantages of Multiple Choice Items
allow more adequate sampling of content.
tend to more effectively structure the problem
to be addressed
items can be more efficiently and reliably
scored than supply items
different response alternatives can provide
diagnostic feedback (item analysis) items can be constructed to address various
levels of cognitive complexity
7/27/2019 Portage Collage Item Objektif
46/47
Disadvantages of Mult.Choice Items
difficult & time consuming to construct gooditems
leads to emphasis on other selected response item types
can lead the instructor to favour simple recall
of facts high degree of dependence on students
reading and instructors writing ability
can be difficult to achieve clarity of expression
measuring synthesis and evaluation can bedifficult
inappropriate for measuring outcomes that
require skilled performance
7/27/2019 Portage Collage Item Objektif
47/47
Wrap-Up