Upload
julie-sievers
View
352
Download
1
Tags:
Embed Size (px)
Citation preview
Designing (More)Effective Tests & Quizzes
@kristobolsloan@StEds_CTE
Thursday, February 12, 3:30-4:45Fleck 314
Assumptions
Effective assessments can drive, propel even student learning.
Effective assessment and evaluation leads to more clear course outcomes.
Effective assessment and evaluation can better help a professor determine their own teaching effectiveness
Assumptions
Most importantly:
To better ensure greater access to academic success of a diverse group of learners, I must offer my students an array of assessments and evaluations to better understand the fuller range of what my students know and can do vis-à-vis my course outcomes
Time Management
Electronic management of scoring/grading
Provides immediate feedback/correct misconceptions
Can offer students opportunities for input on test creation
Socrative
Pre-assessments example
Using student questions as a source of student-to-student talk and input into test creation.
(Educational) LiteracyThe questions that will be graded on the following
three criteria:
Explicit connection to course content—not reflective of outside/internet sources
Specificity (question could only be answered by the term below)
Relevance to (contemporary educational context).
Provide the correct answer (with citation of where correct answer can be found)
Limit of 20 words!
What’s “wrong” with these ?s
1. A ______ is used in a house to help prevent the ________of the electrical wiring.
2. _____is the largest export product in Brazil?
3. When did the English first arrive in North America?
4. Discuss what you learned about the British parliament and the U.S. congress.
Instructor-created assessments
What sorts of assessments items do you offer on a typical instructor-created tests & quizzes?
Which types of items are most common?
Which are easiest/most difficult to create?
What are some possibilities/limitations of each type of item?
Instructor-Created Quizzes/Tests
Validity/Reliability concerns:
What is the difference between reliability and validity?
Instructor-Created Quizzes/Tests
Validity: Does the quiz/test item actually measure what it is intended/supposed to measure relative to the content objectives?
Reliability: Does the quiz/test item consistently estimate what the learner knows and can do relative to the content objectives?
Two general categories of items
Supply (“Completion”) itemssentence completionfill-insshort answeressay-response items
“Objective items”true/false multiple choicematching
Supply items
Tip #1: Use only one blank per item
Poor: A ______ is used in a house to help prevent the ________of the electrical wiring.
Supply ItemsTip #1: Use only one blank per item
The overheating of electrical wiring in a house may be prevented if _____ are installed.
Supply Items
Tip #2: Blanks should appear either at the end or near the end of an item.
Poor: _____is the largest export product in Brazil?
Supply Items
Tip #2: Blanks should appear either at the end or near the end of an item.
The largest export product in Brazil is ______?
Supply Items
Tip #3: Use precise language, or language/terms that cannot be construed in multiple and correct ways
The largest export product in Brazil is ______?
Supply Items
Tip #3: Use precise language
The largest export product in Brazil is ______?
By “largest” do you mean is size (e.g., jackfruit, palm trees), biggest revenue producer (e.g. $?), or by “units” (e.g., number of bananas, pounds of coffee)
Supply Items
The highest revenue-producing export of Brazil is ______?
Instead of…
The largest export product in Brazil is ______?
Supply Items
Tip #4: Include a word or phrase that indicates the category to which the answer must belong.
Poor: When did the English first arrive in North America?
Supply Items
Tip #4: Include a word or phrase that indicates the category to
which the answer must belong.
What year did the English first arrive in North America?
True/False ItemsTip #1: Write more false than
true
items.
Researchers have concluded that teachers tend to do the reverse and students tend to mark items they are unsure about as “true”
True/False Items
Tip #2: Avoid words like “most” (which usually indicates a “true” statement) and “never” (which usually suggests a false statement).
True/False Items
Tip #3: Keep wording clear and specific.
Avoid broad, general statements such as always, never, all, may seldom, usually, etc
Poor: __ Birds eat more than mammals.
True/False Items
Tip #3: Keep wording clear and specific.
____Relative to their body weights, birds eat more than mammals.
True/False Items
Tip #4: Avoid simply taking statements directly from text (even if modified slightly) because it may encourage rote memorization with little understanding.
True/False
Tip #5: Generate binary-choice items in pairs not singly.
Because True/False questions need to be mutually exclusive when reversed, check it.
Multiple Choice
Which of these is the most valid question?
How old is Bob?How many years ago was Bob born?In what year was Bob born?
Multiple ChoiceWhich of these is the most valid question?
How old is Bob?How many years ago was Bob born?
In what year was Bob born?
Time sensitive questions are tricky because the answers come in many different forms and can change
Multiple Choice ItemsTip #1: Stem should present enough
information so that answer choices can be relatively short.
Tip #2: Avoid negatives (e.g., not) as well as words like all, some, often, usually, etc
If you do use word not in stem, italicize or underline it.
Multiple Choice Items: Stems and Choices
Tip #3: All choices should be feasible
Tip #4: Don’t expect students to make narrow distinctions (e.g., “Freezing”: 30’F, 31’F, 32’F, 33’F)
Multiple Choice Items
Tip #5: All choices should be grammatically consistent
The largest tree-borne fruit is a:
A. Banana
B. Apple
C. Jackfruit
D. Strawberries
Multiple Choice Items
Poor:
An example of:
A. An animal that is a mollusk is a whale.
B. An animal that is a mollusk is a clam.
C. An animal that is a mollusk is a crab.
D. An animal that is a mollusk is a lobster.
Multiple Choice Items
Tip #6: Do not overuse “none of the above”/ “All of the above”
&
variations of “A and B” or “C and D, but not A.”
Multiple Choice Items Strategies to find the elusive“higher-level”* multiple choice question:
Analogy based items:
Bandura is to social cognitive theory as _____________ is to social constructivist theory:
a. Piagetb. Sieglerc. Vygotskyd. Skinner
*Bloom’s Taxonomy
Knowledge-Level Items
Identify meaning of term:
Reliability is the same as:
A. consistency.
B. relevancy.
C. representativeness.
D. usefulness.
Comprehension-Level Item
Identifies an example of a concept or principle .
Which of the following is an example of a criterion-referenced interpretation?
A. Derik earned the highest score in science.
B. Erik completed his experiment faster than his classmates.
C. Edna’s test score was higher than 50 percent of the class.
D. Tricia set up her laboratory equipment in five minutes.
Application-Level ItemsDistinguish between properly and improperly stated outcomes
Which one of the following learning outcomes is properly stated in terms of student performance?
A. Develops an appreciation of the importance of testing.
B. Writes valid test questions.
C. Realizes the importance of validity.
Analysis-level ItemsAnalyze whether a word fits with the accepted definition of pragmatism.
Directions: Read carefully through the paragraph below, and decide which of the options A-D is correct.
“The basic premise of pragmatism is that questions posed by speculative metaphysical propositions can often be answered by determining what the practical consequences of the acceptance of a particular metaphysical proposition are in this life. Practical consequences are taken as the criterion for assessing the relevance of all statements or ideas about truth, norm and hope.”
A. The word “acceptance” should be replaced by “rejection.”
B. The word “often” should be replaced by “only.”
C. The word “speculative” should be replaced by hypothetical.”
D. The word “criterion” should be replaced by “measure.”
Synthesis-level ItemsDirections: Read the following comments a teacher made about testing. Then answer the questions that follow by circling the letter of the best answer.
“Students go to school to learn, not to take tests. In addition, tests cannot be used to indicate a student’s absolute level of learning. All tests can do is rank students in order of achievement, and this relative ranking is influenced by guessing, bluffing, and the subjective opinions of the teacher doing the scoring. The teacher-learning process would benefit if we did away with tests and depended on student self-evaluation.”
Identifies relationships:
Which one of the following propositions is most essential to the final conclusion?
A. Effective self-evaluation does not require the use of tests.
B. Tests place students in rank order only.
C. Test scores are influenced by factors other than achievement.
D. Students do not go to school to take tests.
Multiple Choice Items
Advantages:Efficient way of testing students’ ability to recall facts
Demonstrates understanding
DisadvantagesHard to create, time consuming
Inference Questions
Ask students draw conclusions
Ask you to find assumptions made in the passage.
Assesses your ability to go beyond what is on the page.
Inference Question Stems….
The author suggests which of the following…
It can be inferred from the passage that…
The author assumes that…
Inference Question Stems….
The main point of the passage is that...
Which of the following statements about... is best supported by the statements above?
Which of the following best states the author's conclusion in the passage above?
Inference Question Stems….
Which of the following conclusions can be most properly drawn from the data above?
Which of the following is [implied, must be true, implicit, most reasonably drawn] in the passage above?
Which of the following conclusions can most properly be drawn if the statements above are true?
Making Inferences/Drawing Conclusions
Which conclusion is best supported by the sentence: “The Golden Gate Bridge spans the San Francisco Bay, connecting northern California to the peninsula of San Francisco.”
a. The bridge enables many people to go from SF to northern California
b. The bridge has the longest span of any other suspension bridge
c. The cables are best
d. The bridge is the most popular bridge in the bay area
Making Inferences/Drawing Conclusions
Joe woke up and could hear the roosters crowing.
We can infer the time of the day is__________.
a. At noon
b. Early in the morning
c. In the afternoon
d. At night
Making Inferences/Drawing Conclusions
Which of the following statements would make the most accurate conclusion to the passage
a. Bacteria are often an indication that a person’s health has been seriously compromised.
b. Both measures would help protect the health of countless people throughout the country.
c. Researchers are always looking at ways to provide people with better dental health care.
d. Sensors are so incredibly small that they can easily be placed on teeth or even tinier spots.
Matching
1. ___The Seine river divides this city into two famous banks.
2. ___The capital city of the Island of Taiwan.
3. ___Home city to the Statue of Liberty.
4. ___The Kremlin is located in this city.
…….
A. Paris
B. Moscow
C. Taipei
D. New York City
E. Hawaii
F. Newark
……………….
MatchingAdvantages:
especially suitable for who, what, when, and where questions.Can ask students to discriminate among, and to apply concepts.efficient use of space, easy to score
Disadvantages:Poor for diagnosing strengths and weaknesses
Matching
Tip: Include more responses than premises!
This increases the difficulty and decreases the likelihood that correct answers are guesses.
Essay Items/Prompts
Tip #1:Clearly indicate a task. Be precise about the information you expect, and how much information you expect
Poor: Discuss what you learned about the British parliament and the U.S. congress.
Essay Items/Prompts
Tip #1:Clearly indicate a task. Be precise about the information you expect, and how much information you expect
Identify and describe two major differences and one similarity between the BP and U.S. congress.
Compare and contrast….??
Essay Items/PromptsIdentify and describe at least three
assumptions about B.F. Skinner’s version of behaviorism as it relates to educational psychology.
Define Vygotsky’s notion of the Zone of Proximal Development AND provide a clear and convincing example that one might see in a typical classroom. Your example should include specific content or specific examples of a teaching strategy
Essay Items/Prompts
Tip #2: Try to avoid using “why” questions*
Poor: If you were a Catholic priest in Texas in the early 1800’s, why would you be Texas?
*unless your purpose is to have your students produce a teleological response.
Essay Items/Prompts
Tip #2: Try to avoid using “Why” questions
Identify and describe at least three activities a typical Catholic priest would have been engaged in during the early 1800’s in the Spanish Texas.
Culturally Response Assessment Literacy
More equitable, culturally responsive assessment practices requires teachers to……
Culturally Response Assessment Literacy
The use of an array of assessments/evaluations
Evaluation of students against their own records (vs. comparisons with other students)
Inclusion of socioemotional measures as well as measures of academic content (look for both verbal/nonverbal information)
Culturally Response Assessment Literacy
Items should measure only the content of interest, not one’s knowledge of the item context.
Write items to measure what students know, not what they do not know.
Never couch correct answers around the notion of “intelligence” (See Claude Steele’s notion of “Stereotype Threat”)