View
214
Download
0
Category
Tags:
Preview:
Citation preview
Standard Setting: Standard Setting: Grade 3 MathematicsGrade 3 Mathematics
Sheraton Four Points HotelNorwood, MA
August 15-16, 2007
Massachusetts Massachusetts Comprehensive Comprehensive
Assessment System (MCAS) Assessment System (MCAS)
Wednesday, August 15Wednesday, August 15Overview of Plenary Overview of Plenary
SessionSession Welcome/Introductions Overview of MCAS Program Purpose of 2007 Standard Setting Body of Work Method and
Procedures Ground Rules for Standard Setting Agenda (Wednesday-Thursday)
Department of Department of Education Education Bob Bickerton, Associate Commissioner Wayne Fernald, MCAS Mathematics Lead
Developer Haley Freeman, MCAS Mathematics Development
Specialist Mark Johnson, Director of MCAS Test Development Bob Lee, MCAS Chief Analyst Matt O’Connor, Administrator for Administration,
Analysis and Reporting Kit Viator, Director of Student Assessment
Measured Measured Progress Progress
Sally Blake, MCAS Lead Developer, Mathematics Lee Butler, Administrative Assistant Lisa Ehrlich, Assistant Vice President Kevin Haley, Manager of Data Analysis Renee Jordan, Service Center Representative Mark Peters, Program Assistant Miechelle Poulin, Program Assistant Michael J. Richards, Program Manager Kevin Sweeney, Assistant Vice President, Research &
Analysis David Tong, Assistant Director, MCAS Program
Management Eric Wigode, Director of MCAS Test Development
Standard Setting Standard Setting FacilitatorFacilitator
Sally Blake
Welcome Grade 3 Mathematics Welcome Grade 3 Mathematics PanelistsPanelists
Karen Anderson Associate Professor & Chair, Education Dept. Stonehill CollegeNancy Buell Elementary Mathematics Specialist William H. Lincoln SchoolBruce Carter Case Manager Urban League of Eastern Mass.Robert Cote 3rd Grade Classroom Teacher Jordan/Jackson ElementaryLinda Gauthier ** Curriculum Coordinator Saugus Public SchoolsCheryl Goguen ** Grade 4 General Educator Miriam F. McCarthy SchoolRebecca Gutierrez 4th Grade Teacher Newton Elementary SchoolSteven Kaczmarczyk Special Education Teacher Ellen Bigelow SchoolKristine Klumpp ** Grade 3 Teacher Alden Elementary SchoolCarol LaPolice ** Math Instructional Leadership Specialist-Elementary Daniel B. Brunton SchoolMarlena McCoy Grade 4 Teacher Mittineague Elementary SchoolElaine McNamara Title I Director and Teacher Parker Avenue SchoolLyudmila Moiseyeva ** ELL Teacher Baker Elementary SchoolJudy Moore ** Grade 3 Teacher Harvard Elementary SchoolStephanie Morris ** Grade 4 Teacher Craneville SchoolJudith Richards Mathematics Teacher Graham & Parks SchoolJennifer Rubera ** Grade 4 Teacher Pentucket Lake Elementary Michael Stanton ** Principal Boyden Elementary SchoolDeborah Stewart Community Representative Urban LeagueElizabeth Sweeney ** Assistant Program Director Boston Public SchoolDenise Young ** Grade 3 Teacher Brown School
**Served on 2006 panel
Historical Background of the Historical Background of the MCAS TestsMCAS Tests
First MCAS operational tests introduced (ELA, Math, and Science & Technology, grades 4, 8, and 10)
1998
NCLB requires states to annually test reading & math in grades 3-8
Grade 3 Math test administered
2006
Grade 3 Reading, grade 6 Math, and grade 7 ELA tests introduced2001
Massachusetts Education Reform Law passed
1993
Class of 2003 first graduating class required to earn a CD (ELA and Math)
2003
Grade 3 Math standard setting
2006
Grade 3 Math standard setting revisited
2007
Purpose of MCAS Purpose of MCAS ProgramProgram
Inform/improve curriculum and instruction
Evaluate student, school, and districtperformance according to Curriculum Framework content standards and MCAS performance standards
Certify eligibility for high school Competency Determination (CD)
Selected Features of Selected Features of MCASMCAS
Custom developed based on Massachusetts Curriculum Framework content standards and MCAS performance standards
100% of questions used to determine student scores released annually
Measures performance of ALL students educated with public funds
Results reported according to raw scores and performance levels
Overview of 2006 Overview of 2006 Standards Setting Event Standards Setting Event
and Outcomesand Outcomes• Cut scores successfully established at
Warning/Needs Improvement and at Needs Improvement/Proficient
• Some panelists expressed concern about whether any test questions existed at the Above Proficient level; cut score at Proficient/Above Proficient set at 40 (out of 40)
• 2007 test designed to have sufficient questions at Above Proficient level
Purpose:Purpose:2007 Grade 3 2007 Grade 3 MathematicsMathematics
Standard SettingStandard SettingPrimary purpose:• Establish a cut score at Proficient/Above
Proficient
Secondary purpose• Validate cut scores at Warning/Needs
Improvement and Needs Improvement/Proficient
Standard Setting vs.Standard Setting vs.Standards ValidationStandards Validation
Standard setting (top cut point)– Process of establishing original cut
scores– Panelists are not provided initial cut points
Standards validation (bottom two cut points) – Process of validating cut scores– Panelists are provided initial cut points
2007 Standard 2007 Standard Setting/ValidationSetting/Validation
Warning
Needs Improvement
Proficient
Cut score to
be validate
d
Cut score
needed
Cut score to
be validate
d
Above Proficient
Development of Content Development of Content StandardsStandards
Supplement to the CF was created, pulling out specific content standards for grades 3, 5, and 7; no “brand-new” standards were written
2000
Mathematics Curriculum Framework content standards written for grade spans (e.g., grades 5-6 and grades 7-8)
2004
Content Standards Content Standards vsvs. . Performance StandardsPerformance Standards
Content standards = “What”Describe the knowledge and skills students should acquire in a particular content and grade
Performance Standards = “How well”
Describe student work on MCAS tests at the Needs Improvement, Proficient, and Above Proficient levels
General MCAS General MCAS Performance Level Performance Level
DescriptorsDescriptorsNeeds Improvement Students at this level demonstrate partial understanding of subject matter and solve simple problems
Proficient Students at this level demonstrate a solid understanding of challenging subject matter and solve a wide variety of problems
Above ProficientStudents at this level demonstrate a comprehensive and in-depth understanding of rigorous subject matter, and provide sophisticated solutions to complex problems
Linking Performance Linking Performance Standards with Student Standards with Student
WorkWork What is standard setting?
Establishment of cut scores to distinguish between performance levels
What is your job?Use the PLDs to evaluate student work and make recommendation for Proficient/Above Proficient cut score
Purpose of Standard Purpose of Standard SettingSetting
Determine cut scores for reporting assessment results
Answer the question:– How much is enough?
General Phases of Standard General Phases of Standard Setting/Setting/
Standards Validation Standards Validation Data-collection phase
Policy-making/decision-making phase
Standard-Setting MethodsStandard-Setting Methods
Angoff Bookmark Body of Work
Choosing a Standard-Choosing a Standard-Setting MethodSetting Method Prior usage/history Recommendation/
requirement by policy-making authority
Type of assessment
Body of Work method chosen for MCAS test inGrade 3 Mathematics
What is the Body of Work What is the Body of Work Procedure?Procedure?
Panelists examine student work (actual responses to test questions) and make a judgment regarding the performance level to which the student work most closely corresponds.
Top cut Top cut Standard Setting:Standard Setting:Panelists examine student work that has not been previously classified and determine how that work should be classified.
Lower cutsLower cutsStandards Validation:Standards Validation:Panelists examine student work that has been initially classified into a performance level based on starting cut points and determine if they agree with these classifications or recommend changes to them.
Initial Classification of Initial Classification of Student WorkStudent Work
Initial classification of student work in grade 3 mathematics based on 2006 test results.
Step 1:Step 1: Equate the 2007 grade 3 mathematics test to the 2006 test.
Step 2:Step 2: Find the raw score cuts on the 2007 form that are equivalent to the cut points established in August 2006.
Step 3:Step 3: Select student work with scores ranging from very low to very high; classify them into performance levels based on preliminary cut points found in Step 2.
Selected Student WorkSelected Student Work
Example Distribution of Selected Student Work: Grade 3 Math
Warning Needs Improvement ProficientAbove
Proficient
X X XXX
XXX
XXX
XX X X X X X X X X X X
XX
XXX
XXX
XX
XX
XX
XX
XX
XX
XX
XX
XX
XX
XX
XX
XX
XX
XX
How to Classify Student How to Classify Student WorkWork
Performance Level Definitions• General• Grade and content specific
Materials you will need:
Bodies of Student Work• Responses to constructed-response
questions• Multiple-choice summary sheet
Rating Forms
How to Classify Student How to Classify Student WorkWork
Examine the student’s responses to multiple-choice questions
Examine the student’s responses to open-response questions
Judge the student’s knowledge and skills demonstrated relative to the PLDs
Panelists do not need to reach consensus on the classifications
How to Classify Student How to Classify Student WorkWork
Grade 3 mathematics test General MCAS and grade 3 math
Performance Level Descriptors Bodies of student work
• Responses to multiple-choice items AND constructed-response items
To help prepare you to do these ratings, you will spend time becoming familiar with the following:
How to Classify Student How to Classify Student WorkWork
You will have the opportunity to discuss your classifications and change them if desired.
Don’t worry! We have procedures, materials, and staff to assist you in this process.
What Next?What Next?
Take the assessment Complete the Item Map Discuss the Performance Level Definitions Complete training round Complete individual ratings Receive feedback from first round of ratings Discuss feedback and provide final ratings Complete an evaluation form
Top 8 Most Top 8 Most Misunderstood Misunderstood Things about Things about Standard SettingStandard Setting
5.We should use this time to rework Math performance level definitions.
8.Standard setting is a great opportunity to rewrite Curriculum Framework standards.7.The process is rigged.
6.This is a good time to vent about all the things you hate about MCAS.
Top 8 Most Top 8 Most Misunderstood Misunderstood Things about Things about Standard SettingStandard Setting
1.Disagreement is bad.
4.Standard setting is scoring.
3.Only Mathematics scholars should be doing this work.
2.Only teachers should be doing this work.
Ground RulesGround Rules Role of facilitator is to “facilitate” and keep
process on track Process solely focused on recommending
performance standards (cut scores) for MCAS MCAS performance level definitions are
integral to process but are not up for debate Panelists’ recommendations are vital;
however, final cut scores determined by the MDOE
Each panelist must be in attendance for the duration of the process for his/her judgments to be considered
Each panelist must complete evaluation form at the end of the event
Cell phones off, please!
Agenda Agenda Wednesday, August 15Breakfast 8:00 am – 9:00 amWork session 9:00 am – 12:00 pmLunch 12:00 pm – 1:00 pmWork session 1:00 pm – 4:00 pm
Thursday, August 16Breakfast 8:00 am – 9:00 am Work session 9:00 am –12:00 pmLunch 12:00 pm – 12:45 pmWork session 12:45 pm – Until completion
Room Room AssignmeAssignmentntGrade 3 Math –
105/106
Questions?Questions?
Recommended