10 Things to Ask Your Local Psychometrician
about the Keystone Exams
Dr. Stefan BiancanielloOnHand Schools, Inc.Quality Classroom ConsortiumOctober 3, 2012
First Some Background First Some Background InformationInformation
OnHand Schools Works OnHand Schools Works with Common with Common AssessmentsAssessments
• Started in Florida—Hard to Measure Subjects
• Next came the Keystone Exams and inquiries from Penn Hills and Elizabeth Forward
• Others we are/expect to be working with this year :– Albert Gallatin
– Carlynton
– Highlands
– Clairton
– Sto-Rox
– Keystone Consortium
• 2012-13 - Algebra I, Literature, Biology
• 2015-16 – English Composition
• 2016-17 – Civics & Government, Geometry
• 2017-18 – U.S. History
• 2018-19 – Algebra II
• 2019-20 – Chemistry
• 2020-21 – World History
The Timeline of the The Timeline of the Keystone ExamsKeystone Exams
• …to develop Common Benchmark Assessments – similar in content/rigor to the Keystones in Algebra I, Biology, & Literature
• ….to develop a pool of items to be used for valid and reliable final exams
• ….to develop a pool of items for valid and reliable baseline and mid-course exams
• The results will be used ……
The purpose of the The purpose of the Albert Gallatin Albert Gallatin
assessments is…assessments is…
Now For the 10 Now For the 10 Questions for Your Questions for Your Psychometrician!Psychometrician!
#1#1What the Heck is a Psychometrician?What the Heck is a Psychometrician?
A psychometrician is someone who practices the science of educational and psychological measurement, i.e., testing. Psychometricians measure the validity, reliability, and fairness of an exam program, and are an integral part in the process of creating valid and reliable tests.
Psychometricians are often skilled in statistics, and psychology.
Psychometrician
#2 #2 What is the Difference between What is the Difference between Multiple Choice and Constructed Multiple Choice and Constructed
Response Items?Response Items?
Why are we writing items?
A. to populate a state item bank
B. to build common assessments
C. to work with other teachers
D. to gain experience writing
The Stem
Distractor
Distractor
CORRECT ANSWER
Distractor
Constructed Response ItemsConstructed Response Items•Constructed response items require students to provide a written response. These questions typically ask students to describe, explain, critique, or evaluate the scenario provided in the stimulus
•Constructed response items have multiple correct responses. However, there may be specific content that is required in the response to receive full credit.•Constructed response items can assess one or more benchmarks and range from low to high complexity.
•Constructed response items are scored using a rubric
Module 1 Module 2
Assessment Anchors Covered
Operations & Linear Equations & Inequalities
Linear Functions & Data Organizations
# of Eligible Content 18 15
# of Multiple Choice 18 18
# of Constructed Response 3 3
Multiple Choice Points 18 18
Constructed Response Points 12 12
Algebra Keystone – Algebra Keystone – from other PDE from other PDE
documents documents
60%
40%
Module 1 Module 2
Assessment Anchors Covered
Cells and Cell Processes
Continuity and Unity of Life
# of Eligible Content 16 22
# of Multiple Choice 24 24
# of Constructed Response 3 3
Multiple Choice Points 24 24
Constructed Response Points 9 9
Biology Keystone – Biology Keystone – from other PDE from other PDE
documents documents
73%
27%
Module 1 Module 2
Assessment Anchors Covered Fiction Literature Nonfiction Literature
# of Eligible Content 25 31
# of Multiple Choice 17 17
# of Constructed Response 3 3
Multiple Choice Points 17 17
Constructed Response Points 9 9
Literature Keystone – Literature Keystone – from other PDE from other PDE
documents documents
65%
35%
#3#3How Do You Train Teachers for This?How Do You Train Teachers for This?
We Teach them A New We Teach them A New Language! Try This Language! Try This
ExerciseExercise• Each of you received a stapled
piece of paper when you registered this morning.
• Please open the paper now. It contains a Multiple Choice Question written in “Franzipanic” —take just two minutes and figure out the correct answer!
The Moral of this TestThe Moral of this Test• With poorly written items, you can
get them correct without knowing the intended content.
• Reliability issue?
• Validity issue?
#4#4How Hard are the Keystone Exams?How Hard are the Keystone Exams?
Field Test Results—Field Test Results—Half or More Failed!Half or More Failed!
• Of the 47,000 students who took the Biology exam, 64.1% were NOT proficient: Basic (32.7%) Below Basic (31.4%)
• Of the 95,000 students tested in Algebra I, 61.0% were NOT proficient: Basic (41.2%) Below Basic (19.8%)
• Of the 42,600 tested in Literature, 49.7% were NOT proficient: Basic (31.4%) Below Basic (18.3%)
A recently released A recently released problemproblem
Tim’s scores for the first 5 times he played a video game are listed below:
4526 4599 4672 4745 4818
Tim’s scores follow a pattern. Which expression can be used to determine his score after he played the video game n times?
a.73n+4453
b.73 (n+4453)
c.4453n +73
d.4526n
Keystone Exam Keystone Exam DifficultyDifficulty• The Keystone Exam Test Questions
will almost all be at the Level 2 or 3 from Webb’s Depth of Knowledge
• Which leads us to the next question…
#5#5What Is Webb’s Depth of KnowledgeWhat Is Webb’s Depth of Knowledge
________________________________ ________________________________
Webb’s Depth of Webb’s Depth of KnowledgeKnowledge
• Norman Webb developed a process and criteria for systematically analyzing the alignment between standards and assessments.
• Webb’s Depth of Knowledge Model can be used to analyze the cognitive complexity required for the student to master a standard or complete an assessment task.
Webb’s Depth of Webb’s Depth of KnowledgeKnowledge
• Cognitive complexity refers to the cognitive demand associated with a test item.
• The Depth of Knowledge level of the item is determined by the complexity of the mental processing that the student must use to answer the item.
Level 1 - Recall - Recall of a fact, information, or procedure
Level 2 - Basic Application - of Skill/Concept - Use of information, conceptual knowledge, procedures, two or more steps, etc.
Level 3 - Strategic Thinking - Requires reasoning, developing a plan or sequence of steps; has some complexity; more than one possible answer; generally takes less than 10 minutes to do.
Level 4 - Extended Thinking - Requires an investigation; time to think and process multiple conditions of the problem or task; and more than 10 minutes to do non-routine manipulations.
Webb’s four levels of Webb’s four levels of complexitycomplexity
Cognitive Complexity/ Cognitive Complexity/ VerbsVerbsLow Cognitive Complexity—Level 1• Remember • Recall• Memorize
• Recognize• Translate• Rephrase
• Describe• Explain• Repeat
Moderate Cognitive Complexity—Level 2• Apply• Execute• Solve
• Connect• Classify• Break
Down
• Distinguish• Compare• Contrast
High Cognitive Complexity—Levels 3 & 4• Integrate• Extend• Combine
• Design• Create• Judge
• Perform• Value• Assess
Same Verb—Three Same Verb—Three Different DOK LevelsDifferent DOK Levels
DOK 1- Describe three characteristics of metamorphic rocks. (Requires simple recall)
DOK 2- Describe the difference between metamorphic and igneous rocks. (Requires cognitive processing to determine the differences in the two rock types)
DOK 3- Describe a model that you might use to represent the relationships that exist within the rock cycle. (Requires deep understanding of rock cycle and a determination of how best to represent it)
Describe
#6 #6 Does Depth of Knowledge Does Depth of Knowledge
Mean Difficulty?Mean Difficulty?______________________________________________________________
DOK is NOT about DOK is NOT about difficultydifficulty
DifficultyDifficulty is a reference to how many students answer an item correctly•How many of you know the definition of exaggerate?
DOK Low = Recall
If all or most of you know the definition , this item is an easy one.•How many of you know the definition of illeist?
DOK Low = Recall
If most of you do not know the definition, this item is a difficult one.
Complexity vs. Complexity vs. DifficultyDifficulty• The Depth of Knowledge levels are
based on the complexity of the mental processes the student must use to find the correct answer, not the difficulty of the item!
#7 How Does Webb’s DOK Relate to How Does Webb’s DOK Relate to
Bloom’s TaxonomyBloom’s Taxonomy__________________________________________________________________
COGNITIVE LEVEL COMPARISON MATRIX: BLOOM AND WEBB
Action WordsBLOOM WEBB
Rememberdefine, identify, name, select, state, order (involves a one-step process)
1.0define, identify, name, select, state, order (involves a one-step process)
Understand
convert, estimate, explain, express, factor, generalize, give example, identify, indicate, locate, picture graphically (involves a 2-step process)
2.0
apply, choose, compute, employ, interpret, graph, modify, operate, plot, practice, solve, use (involves a two-step process)
Apply
apply, choose, compute, employ, interpret, graph, modify, operate, plot, practice, solve, use, (involves a three-or-more step process)
Analyze
compare, contrast, correlate, differentiate, discriminate, examine, infer, maximize, minimize, prioritize, subdivide, test
3.0
compare, contrast, correlate, differentiate, discriminate, examine, infer, maximize, minimize, prioritize, subdivide, test
Evaluate
arrange, collect, construct, design, develop, formulate, organize, set up, prepare, plan, propose, create, experiment and record data 4.0
arrange, collect, construct, design, develop, formulate, organize, set up, prepare, plan, propose, create, experiment and record data
Createappraise, assess, defend, estimate, evaluate, judge, predict, rate, validate, verify
#8#8Do You Have Item Writing Rules?Do You Have Item Writing Rules?
Item Writing RulesItem Writing Rules• Yes, we offer training in constructing
quality Multiple Choice and Constructed Response Items
• A one-page summary explains the rules that we used last week while training in Florida
• Shula Nedley and I will be available to discuss this in more detail at one of the breakout sessions today
#9#9Can’t I Just Buy Assessment Questions?Can’t I Just Buy Assessment Questions?
Yes You Can Buy or Yes You Can Buy or Find Assessment Find Assessment Questions But… Questions But…
• Concern over Alignment
• Concern over Depth of Knowledge
• Penn Hills Will Discuss Why They Are Creating Their Own
#10#10Is it Worth All This Effort?Is it Worth All This Effort?
Yes, It’s Worth the Yes, It’s Worth the EffortEffort• The Bar is being raised
• The Common Core changes the game
• The Keystone Exams change the game
• The world our students will enter is FAR MORE COMPLEX
• We need to raise the Level of our game
A Tool To Ease the A Tool To Ease the PainPain
• Assessment BuilderAssessment Builder
A demonstration of this tool will take A demonstration of this tool will take place during the breakout sessions place during the breakout sessions later today.later today.
You are trapped in a room with 2 doors, one of which leads to freedom and the other to certain death. You do not know which is which.
Each door is guarded by a man, one of whom always lies, while the other always tells the truth. You do not know which is which.
You can ask but one question of either man to gain your freedom. Which question should you ask?
A.How do I get out of this room and stay alive?
B.Which one of the two door is the correct one?
C.Who is the man who never tells the truth & always lies?
D.Which door would the other man say leads to death?
Test Your Depth of Test Your Depth of KnowledgeKnowledge