Standard Setting: Standard Setting: Grade 3 MathematicsGrade 3 Mathematics

Sheraton Four Points HotelNorwood, MA

August 15-16, 2007

Massachusetts Massachusetts Comprehensive Comprehensive

Assessment System (MCAS) Assessment System (MCAS)

Wednesday, August 15Wednesday, August 15Overview of Plenary Overview of Plenary

SessionSession Welcome/Introductions Overview of MCAS Program Purpose of 2007 Standard Setting Body of Work Method and

Procedures Ground Rules for Standard Setting Agenda (Wednesday-Thursday)

Department of Department of Education Education Bob Bickerton, Associate Commissioner Wayne Fernald, MCAS Mathematics Lead

Developer Haley Freeman, MCAS Mathematics Development

Specialist Mark Johnson, Director of MCAS Test Development Bob Lee, MCAS Chief Analyst Matt O’Connor, Administrator for Administration,

Analysis and Reporting Kit Viator, Director of Student Assessment

Measured Measured Progress Progress

Sally Blake, MCAS Lead Developer, Mathematics Lee Butler, Administrative Assistant Lisa Ehrlich, Assistant Vice President Kevin Haley, Manager of Data Analysis Renee Jordan, Service Center Representative Mark Peters, Program Assistant Miechelle Poulin, Program Assistant Michael J. Richards, Program Manager Kevin Sweeney, Assistant Vice President, Research &

Analysis David Tong, Assistant Director, MCAS Program

Management Eric Wigode, Director of MCAS Test Development

Standard Setting Standard Setting FacilitatorFacilitator

Sally Blake

Welcome Grade 3 Mathematics Welcome Grade 3 Mathematics PanelistsPanelists

Karen Anderson Associate Professor & Chair, Education Dept. Stonehill CollegeNancy Buell Elementary Mathematics Specialist William H. Lincoln SchoolBruce Carter Case Manager Urban League of Eastern Mass.Robert Cote 3rd Grade Classroom Teacher Jordan/Jackson ElementaryLinda Gauthier ** Curriculum Coordinator Saugus Public SchoolsCheryl Goguen ** Grade 4 General Educator Miriam F. McCarthy SchoolRebecca Gutierrez 4th Grade Teacher Newton Elementary SchoolSteven Kaczmarczyk Special Education Teacher Ellen Bigelow SchoolKristine Klumpp ** Grade 3 Teacher Alden Elementary SchoolCarol LaPolice ** Math Instructional Leadership Specialist-Elementary Daniel B. Brunton SchoolMarlena McCoy Grade 4 Teacher Mittineague Elementary SchoolElaine McNamara Title I Director and Teacher Parker Avenue SchoolLyudmila Moiseyeva ** ELL Teacher Baker Elementary SchoolJudy Moore ** Grade 3 Teacher Harvard Elementary SchoolStephanie Morris ** Grade 4 Teacher Craneville SchoolJudith Richards Mathematics Teacher Graham & Parks SchoolJennifer Rubera ** Grade 4 Teacher Pentucket Lake Elementary Michael Stanton ** Principal Boyden Elementary SchoolDeborah Stewart Community Representative Urban LeagueElizabeth Sweeney ** Assistant Program Director Boston Public SchoolDenise Young ** Grade 3 Teacher Brown School

**Served on 2006 panel

Historical Background of the Historical Background of the MCAS TestsMCAS Tests

First MCAS operational tests introduced (ELA, Math, and Science & Technology, grades 4, 8, and 10)

1998

NCLB requires states to annually test reading & math in grades 3-8

Grade 3 Math test administered

2006

Grade 3 Reading, grade 6 Math, and grade 7 ELA tests introduced2001

Massachusetts Education Reform Law passed

1993

Class of 2003 first graduating class required to earn a CD (ELA and Math)

2003

Grade 3 Math standard setting

2006

Grade 3 Math standard setting revisited

2007

Purpose of MCAS Purpose of MCAS ProgramProgram

Inform/improve curriculum and instruction

Evaluate student, school, and districtperformance according to Curriculum Framework content standards and MCAS performance standards

Certify eligibility for high school Competency Determination (CD)

Selected Features of Selected Features of MCASMCAS

Custom developed based on Massachusetts Curriculum Framework content standards and MCAS performance standards

100% of questions used to determine student scores released annually

Measures performance of ALL students educated with public funds

Results reported according to raw scores and performance levels

Overview of 2006 Overview of 2006 Standards Setting Event Standards Setting Event

and Outcomesand Outcomes• Cut scores successfully established at

Warning/Needs Improvement and at Needs Improvement/Proficient

• Some panelists expressed concern about whether any test questions existed at the Above Proficient level; cut score at Proficient/Above Proficient set at 40 (out of 40)

• 2007 test designed to have sufficient questions at Above Proficient level

Purpose:Purpose:2007 Grade 3 2007 Grade 3 MathematicsMathematics

Standard SettingStandard SettingPrimary purpose:• Establish a cut score at Proficient/Above

Proficient

Secondary purpose• Validate cut scores at Warning/Needs

Improvement and Needs Improvement/Proficient

Standard Setting vs.Standard Setting vs.Standards ValidationStandards Validation

Standard setting (top cut point)– Process of establishing original cut

scores– Panelists are not provided initial cut points

Standards validation (bottom two cut points) – Process of validating cut scores– Panelists are provided initial cut points

2007 Standard 2007 Standard Setting/ValidationSetting/Validation

Warning

Needs Improvement

Proficient

Cut score to

be validate

d

Cut score

needed

Cut score to

be validate

d

Above Proficient

Development of Content Development of Content StandardsStandards

Supplement to the CF was created, pulling out specific content standards for grades 3, 5, and 7; no “brand-new” standards were written

2000

Mathematics Curriculum Framework content standards written for grade spans (e.g., grades 5-6 and grades 7-8)

2004

Content Standards Content Standards vsvs. . Performance StandardsPerformance Standards

Content standards = “What”Describe the knowledge and skills students should acquire in a particular content and grade

Performance Standards = “How well”

Describe student work on MCAS tests at the Needs Improvement, Proficient, and Above Proficient levels

General MCAS General MCAS Performance Level Performance Level

DescriptorsDescriptorsNeeds Improvement Students at this level demonstrate partial understanding of subject matter and solve simple problems

Proficient Students at this level demonstrate a solid understanding of challenging subject matter and solve a wide variety of problems

Above ProficientStudents at this level demonstrate a comprehensive and in-depth understanding of rigorous subject matter, and provide sophisticated solutions to complex problems

Linking Performance Linking Performance Standards with Student Standards with Student

WorkWork What is standard setting?

Establishment of cut scores to distinguish between performance levels

What is your job?Use the PLDs to evaluate student work and make recommendation for Proficient/Above Proficient cut score

Purpose of Standard Purpose of Standard SettingSetting

Determine cut scores for reporting assessment results

Answer the question:– How much is enough?

General Phases of Standard General Phases of Standard Setting/Setting/

Standards Validation Standards Validation Data-collection phase

Policy-making/decision-making phase

Standard-Setting MethodsStandard-Setting Methods

Angoff Bookmark Body of Work

Choosing a Standard-Choosing a Standard-Setting MethodSetting Method Prior usage/history Recommendation/

requirement by policy-making authority

Type of assessment

Body of Work method chosen for MCAS test inGrade 3 Mathematics

What is the Body of Work What is the Body of Work Procedure?Procedure?

Panelists examine student work (actual responses to test questions) and make a judgment regarding the performance level to which the student work most closely corresponds.

Top cut Top cut Standard Setting:Standard Setting:Panelists examine student work that has not been previously classified and determine how that work should be classified.

Lower cutsLower cutsStandards Validation:Standards Validation:Panelists examine student work that has been initially classified into a performance level based on starting cut points and determine if they agree with these classifications or recommend changes to them.

Initial Classification of Initial Classification of Student WorkStudent Work

Initial classification of student work in grade 3 mathematics based on 2006 test results.

Step 1:Step 1: Equate the 2007 grade 3 mathematics test to the 2006 test.

Step 2:Step 2: Find the raw score cuts on the 2007 form that are equivalent to the cut points established in August 2006.

Step 3:Step 3: Select student work with scores ranging from very low to very high; classify them into performance levels based on preliminary cut points found in Step 2.

Selected Student WorkSelected Student Work

Example Distribution of Selected Student Work: Grade 3 Math

Warning Needs Improvement ProficientAbove

Proficient

X X XXX

XXX

XXX

XX X X X X X X X X X X

XX

XXX

XXX

XX

XX

XX

XX

XX

XX

XX

XX

XX

XX

XX

XX

XX

XX

XX

How to Classify Student How to Classify Student WorkWork

Performance Level Definitions• General• Grade and content specific

Materials you will need:

Bodies of Student Work• Responses to constructed-response

questions• Multiple-choice summary sheet

Rating Forms

How to Classify Student How to Classify Student WorkWork

Examine the student’s responses to multiple-choice questions

Examine the student’s responses to open-response questions

Judge the student’s knowledge and skills demonstrated relative to the PLDs

Panelists do not need to reach consensus on the classifications

How to Classify Student How to Classify Student WorkWork

Grade 3 mathematics test General MCAS and grade 3 math

Performance Level Descriptors Bodies of student work

• Responses to multiple-choice items AND constructed-response items

To help prepare you to do these ratings, you will spend time becoming familiar with the following:

How to Classify Student How to Classify Student WorkWork

You will have the opportunity to discuss your classifications and change them if desired.

Don’t worry! We have procedures, materials, and staff to assist you in this process.

What Next?What Next?

Take the assessment Complete the Item Map Discuss the Performance Level Definitions Complete training round Complete individual ratings Receive feedback from first round of ratings Discuss feedback and provide final ratings Complete an evaluation form

Top 8 Most Top 8 Most Misunderstood Misunderstood Things about Things about Standard SettingStandard Setting

5.We should use this time to rework Math performance level definitions.

8.Standard setting is a great opportunity to rewrite Curriculum Framework standards.7.The process is rigged.

6.This is a good time to vent about all the things you hate about MCAS.

Top 8 Most Top 8 Most Misunderstood Misunderstood Things about Things about Standard SettingStandard Setting

1.Disagreement is bad.

4.Standard setting is scoring.

3.Only Mathematics scholars should be doing this work.

2.Only teachers should be doing this work.

Ground RulesGround Rules Role of facilitator is to “facilitate” and keep

process on track Process solely focused on recommending

performance standards (cut scores) for MCAS MCAS performance level definitions are

integral to process but are not up for debate Panelists’ recommendations are vital;

however, final cut scores determined by the MDOE

Each panelist must be in attendance for the duration of the process for his/her judgments to be considered

Each panelist must complete evaluation form at the end of the event

Cell phones off, please!

Agenda Agenda Wednesday, August 15Breakfast 8:00 am – 9:00 amWork session 9:00 am – 12:00 pmLunch 12:00 pm – 1:00 pmWork session 1:00 pm – 4:00 pm

Thursday, August 16Breakfast 8:00 am – 9:00 am Work session 9:00 am –12:00 pmLunch 12:00 pm – 12:45 pmWork session 12:45 pm – Until completion

Room Room AssignmeAssignmentntGrade 3 Math –

105/106

Questions?Questions?