Embed Size (px)
Citation preview
Sterling Practices in Design & Scoring of Performance-Based Exams
#156F. Jay Breyer
Presented at the 2005 CLEAR Annual ConferenceSeptember 15-17 Phoenix, Arizona
Presented at the 2005 CLEAR Annual ConferenceSeptember 15-17 Phoenix, Arizona
The Players• F. Jay Breyer, PhD Thomson Prometric• Ron Bridwell, PE National Council of Examiners for Engineering & Surveying• Beth Sabin, DVM, PhD
American Veterinary Medical Association• Ron Rodgers, PhD CTS/Employment Research & Development• Elizabeth Witt, PhD American Board of Emergency Medicine
Scoring Procedures for STRUCTURAL II
September 2005
Ron Bridwell, P.E.
Presented at the 2005 CLEAR Annual ConferenceSeptember 15-17 Phoenix, Arizona
Introduction
The Exam
Before the Scoring Session
The Scoring Process
The Cut Score (Passing Point) Process
Presented at the 2005 CLEAR Annual ConferenceSeptember 15-17 Phoenix, Arizona
Introduction
The Exam
Before the Scoring Session
The Scoring Process
The Cut Score (Passing Point) Process
Presented at the 2005 CLEAR Annual ConferenceSeptember 15-17 Phoenix, Arizona
Introduction
The Exam
Before the Scoring Session
The Scoring Process
The Cut Score (Passing Point) Process
Presented at the 2005 CLEAR Annual ConferenceSeptember 15-17 Phoenix, Arizona
Introduction
The Exam
Before the Scoring Session
The Scoring Process
The Cut Score (Passing Point) Process
Presented at the 2005 CLEAR Annual ConferenceSeptember 15-17 Phoenix, Arizona
The Exam
Scoring Protocol DevelopmentNeed to Standardize the STR II Scoring
guidelines using a benchmark holistic method.
Scoring can drift due to fatigue or anger.
Scoring Criteria DevelopmentDeveloped by the exam committee as the
problems are developed.Candidates may respond differently.
Presented at the 2005 CLEAR Annual ConferenceSeptember 15-17 Phoenix, Arizona
Before the Scoring Session
Tasks
Identifying Scoring Committee MembersMost familiar with problemsCoordinators work with staffEmpowered to modify criteria as needed.
Presented at the 2005 CLEAR Annual ConferenceSeptember 15-17 Phoenix, Arizona
Before the Scoring Session
Tasks
Identify Sample Papers5 benchmarks for trainingRange finders for training5 benchmarks for certification
Presented at the 2005 CLEAR Annual ConferenceSeptember 15-17 Phoenix, Arizona
The Scoring Process
Tasks
Certifying Scorers
5 benchmark papers are given to scorers as test
Pass or FailScorers have two chances to be certified
Training the ScorersScorers should be skilled at assigning scores to
specific problemsScorers are trained with benchmark papers
Presented at the 2005 CLEAR Annual ConferenceSeptember 15-17 Phoenix, Arizona
The Scoring Process
TasksScoring
Care is taken to insure the scorers do not know the names or jurisdictions of the examinees
Papers are scored blind as if by machine Each paper is scored by two scorersIf the scores agree or are off by no more than 1
the score is assigned (averaged)If off by more than 1, the coordinator
adjudicatesAny scorer can be replaced by any other and
the same score would resultDatabase provides feedback
Presented at the 2005 CLEAR Annual ConferenceSeptember 15-17 Phoenix, Arizona
Kinds of Information
Discrepancy
Agreement
Summary
Shows how many papers scored
Shows consistency
Shows consistency of each scorer paired with all partners
Useful for Scoring Reliability
Aggregate &
Separate
Number, Mean, SD across entire test for each scorer and coordinator
Book keeping
Records
RoadTo Fair &
Quality Scores
Number of Papers to be adjudicated
Total Required Adjudications by Scorer
Re-Training may be necessary if too many
Number, Mean, SD read by each scorer & coordinator by problem
Monitoring Solution for Fair Scoring: Report Components
Adjudication Resolution Training
Presented at the 2005 CLEAR Annual ConferenceSeptember 15-17 Phoenix, Arizona
Overview ofStandard Setting Process
Report Results
Definition of Competence
Practice Session
Real Rating
3 Uniform Solution Samples Selected
Training undertaken
Assign candidates to PASS/FAIL status based on comparison of total performance to Standard
Presented at the 2005 CLEAR Annual ConferenceSeptember 15-17 Phoenix, Arizona
Questions?
