35
1 New England Common Assessment Program (NECAP) Setting Performance Standards

1 New England Common Assessment Program (NECAP) Setting Performance Standards

Embed Size (px)

Citation preview

Page 1: 1 New England Common Assessment Program (NECAP) Setting Performance Standards

1

New EnglandCommon Assessment Program

(NECAP)Setting Performance Standards

Page 2: 1 New England Common Assessment Program (NECAP) Setting Performance Standards

2

Purpose

• Provide data to establish the following three cut scores for Reading, Grades 3-8 and Writing, Grades 5 and 8:– Proficient/Proficient with Distinction– Partially Proficient/Proficient– Substantially Below Proficient/Partially

Proficient

Page 3: 1 New England Common Assessment Program (NECAP) Setting Performance Standards

3

What is Standard Setting?

• Set of activities that result in the determination of threshold or cut scores on an assessment

• We are trying to answer the question:– How much is enough?

Page 4: 1 New England Common Assessment Program (NECAP) Setting Performance Standards

4

What is Standard Setting

• Data collection phase

• Policy/Decision making phase

Page 5: 1 New England Common Assessment Program (NECAP) Setting Performance Standards

5

Many Standard Setting Methods

• Angoff

• Body of Work

• Bookmark

Page 6: 1 New England Common Assessment Program (NECAP) Setting Performance Standards

6

Choice of Method is Based on Many Factors

• Prior usage/history

• Recommendation/requirement by some policy making authority

• Type of assessment

Page 7: 1 New England Common Assessment Program (NECAP) Setting Performance Standards

7

Choice of Method is Based on Many Factors

• Weighing all these factors, it was determined that the methods to be used are:– Reading: Bookmark– Writing: Body of Work

• They are both well-established procedures that have been successfully used on many assessments

• They have produced defensible results

Page 8: 1 New England Common Assessment Program (NECAP) Setting Performance Standards

8

Choice of Method is Based on Many Factors

• Bookmark is appropriate for assessments that consist primarily of multiple-choice items but also include some constructed-response items

• Body of Work method works well for assessments that consist primarily of constructed-response items

Page 9: 1 New England Common Assessment Program (NECAP) Setting Performance Standards

9

Standard Setting vs. Standards Validation

• Standard setting– Generally three rounds:

• Round 1: individual ratings

• Round 2: table group discussion of Round 1 results, followed by second round ratings

• Round 3: whole room discussion of Round 2 results, followed by final ratings

Page 10: 1 New England Common Assessment Program (NECAP) Setting Performance Standards

10

Standard Setting vs. Standards Validation

• Standards validation– Starting cut points are provided to panelists at the

beginning of the process

– Two rounds:• Round 1: table group discussion of starting cut points,

followed by first round ratings

• Round 2: whole group discussion of Round 1 results, followed by second round ratings

• The process you will be following will be closer to a standards validation process

Page 11: 1 New England Common Assessment Program (NECAP) Setting Performance Standards

11

Details for Standards Validation using the Bookmark and Body of Work Procedures

Page 12: 1 New England Common Assessment Program (NECAP) Setting Performance Standards

12

Starting Cut Points

• Starting cuts for Reading and Writing were determined using teacher judgment data:– At the time of testing, teachers were asked to categorize

each student into an achievement level category, based on classroom performance (not test performance)

– Using these categorizations as well as the students’ test scores, Measured Progress staff calculated starting cut points

Page 13: 1 New England Common Assessment Program (NECAP) Setting Performance Standards

13

Starting Cut Points

• Analyses of teacher judgment data indicate:– Good participation rates– Teachers appear to have been conscientious in

completing the task

• In other words, available evidence supports the validity of the starting cuts

Page 14: 1 New England Common Assessment Program (NECAP) Setting Performance Standards

14

Starting Cut Points

• However, standard setting is also a critical part of the process because:– the teachers based their judgments on classroom

performance, not performance on NECAP

– different teachers have different conceptions of what it means to be, for example, Partially Proficient

– they did not have the benefit of discussion with their colleagues from across the three states

– etc.

Page 15: 1 New England Common Assessment Program (NECAP) Setting Performance Standards

15

Starting Cut Points

• Both sources of information are important. Therefore:– You are free to recommend changing the

starting cuts, but we don’t expect to see major changes

– The changes you recommend should be based on the test content and the Achievement Level Descriptions

Page 16: 1 New England Common Assessment Program (NECAP) Setting Performance Standards

16

The Bookmark Procedure(Reading)

• The Bookmark procedure is a standard setting method that uses a book of items (ordered from easiest to hardest)

• Panelists place bookmarks in that book of items

Page 17: 1 New England Common Assessment Program (NECAP) Setting Performance Standards

17

What is the bookmark procedure?

Page 18: 1 New England Common Assessment Program (NECAP) Setting Performance Standards

18

What is the bookmark procedure?

• For purposes of NECAP standard setting, you will be provided with starting cut points and will either validate those cuts, or recommend modifications to them

Page 19: 1 New England Common Assessment Program (NECAP) Setting Performance Standards

19

A Technical Detail regarding the Bookmark Method

• Note that the ordered item cut point for a given cut does not equal the raw score a student must obtain to be categorized into the higher achievement level

• For example, the Substantially Below Proficient/ Partially Proficient starting cut for grade 3 reading is between ordered items 6 and 7; however, a student must obtain at least 20 points on the test in order to be classified into the Partially Proficient level

Page 20: 1 New England Common Assessment Program (NECAP) Setting Performance Standards

20

How to Place a Bookmark

• A few concepts you will need to know:– The starting cut points– The achievement level descriptions– ‘Borderline’ students– What knowledge, skills, and abilities (KSAs)

are needed to answer each question

Page 21: 1 New England Common Assessment Program (NECAP) Setting Performance Standards

21

How to Place a Bookmark

• Start at the beginning of the ordered item book• Evaluate whether at least 2 out of 3 students demonstrating

skills at the ‘borderline’ of Partially Proficient would correctly answer item 1

• Moving through the book, make this evaluation of each item• The bookmark should go where you no longer think 2 out of

3 Partially Proficient ‘borderline’ students would correctly answer the question.

• You may decide that the starting cuts are in the right place, or you might recommend moving them.

Page 22: 1 New England Common Assessment Program (NECAP) Setting Performance Standards

22

How to Place a BookmarkItem Number

Would at least 2 out of 3 students who demonstrate skills at the Partially Proficient 'borderline' correctly answer this question?

1 Yes

2 Yes

3 Yes

4 Yes

5 Yes

6 Yes

7 Yes

8 Yes

9 No

10 No

11 No

12 No

13 No

14 No

15 No

… No

Page 23: 1 New England Common Assessment Program (NECAP) Setting Performance Standards

23

How to Place a Bookmark

• In the example, the bookmark would go between items 8 and 9

• However, it won’t be that easy; there will be gray areas

• Place one bookmark for each cut score

• You will have the opportunity to discuss your bookmark placements and change them if desired

Page 24: 1 New England Common Assessment Program (NECAP) Setting Performance Standards

24

How to Place a Bookmark

• To place your bookmarks you will need to be familiar with the achievement level descriptions and the assessment items

Page 25: 1 New England Common Assessment Program (NECAP) Setting Performance Standards

25

How to Place a Bookmark

• Don’t worry, we have procedures, materials and staff to assist you in this process.

Page 26: 1 New England Common Assessment Program (NECAP) Setting Performance Standards

26

Any questions about the Bookmark Procedure?

Page 27: 1 New England Common Assessment Program (NECAP) Setting Performance Standards

27

The Body of Work Procedure (Writing)

• A standard method that uses approximately 30 complete sets of student work, including their answers to open-ended items and a display of their responses to the multiple-choice items

• Bodies of work cover the range of possible scores and are presented in order from lowest to highest total score

Page 28: 1 New England Common Assessment Program (NECAP) Setting Performance Standards

28

What is the Body of Work procedure?

• You will classify each BOW into the achievement level in which you feel it belongs:– Proficient with Distinction– Proficient– Partially Proficient – Substantially Below Proficient

Page 29: 1 New England Common Assessment Program (NECAP) Setting Performance Standards

29

The Body of Work Procedure

• Prior to beginning the process of rating the BOWs, you will:– take the assessment– become very familiar with the Achievement

Level Descriptions– define, with your grade-level group,

‘borderline’ students– individually review the full set of BOWs

Page 30: 1 New England Common Assessment Program (NECAP) Setting Performance Standards

30

Making Your Ratings

• Beginning with the first BOW, and referring to the Achievement Level Descriptions and the definition of ‘borderline’ students, determine into which achievement level the BOW should be categorized

• Continue through each BOW in turn, assigning each to an achievement level

• You may agree with the categorizations based on the starting cuts, or you may recommend changing some of them

Page 31: 1 New England Common Assessment Program (NECAP) Setting Performance Standards

31

Any questions about the Body of Work Procedure?

Page 32: 1 New England Common Assessment Program (NECAP) Setting Performance Standards

32

What Next?

• After this session, you will break into grade-level groups, where you will:– take the assessment to familiarize yourself with the test

items

– Reading groups: complete the item map, which is a document that will help you with the bookmark placement process

– discuss the Achievement Level Descriptions and develop definitions of “borderline” Partially Proficient, Proficient, and Proficient with Distinction students

Page 33: 1 New England Common Assessment Program (NECAP) Setting Performance Standards

33

What Next?

• You will:– discuss the starting cut points in table groups

and do the first round of ratings– discuss the first round ratings as a whole room

then do the second round of ratings

Page 34: 1 New England Common Assessment Program (NECAP) Setting Performance Standards

34

What Next?

• After the second round of ratings, you will have an opportunity to provide feedback about the Achievement Level Descriptions

• As the final step, we will ask you to complete an evaluation of the standard setting process

Page 35: 1 New England Common Assessment Program (NECAP) Setting Performance Standards

35

Good Luck!