1

New EnglandCommon Assessment Program

(NECAP)Setting Performance Standards

2

Purpose

• Provide data to establish the following three cut scores for Reading, Grades 3-8 and Writing, Grades 5 and 8:– Proficient/Proficient with Distinction– Partially Proficient/Proficient– Substantially Below Proficient/Partially

Proficient

3

What is Standard Setting?

• Set of activities that result in the determination of threshold or cut scores on an assessment

• We are trying to answer the question:– How much is enough?

4

What is Standard Setting

• Data collection phase

• Policy/Decision making phase

5

Many Standard Setting Methods

• Angoff

• Body of Work

• Bookmark

6

Choice of Method is Based on Many Factors

• Prior usage/history

• Recommendation/requirement by some policy making authority

• Type of assessment

7

Choice of Method is Based on Many Factors

• Weighing all these factors, it was determined that the methods to be used are:– Reading: Bookmark– Writing: Body of Work

• They are both well-established procedures that have been successfully used on many assessments

• They have produced defensible results

8

Choice of Method is Based on Many Factors

• Bookmark is appropriate for assessments that consist primarily of multiple-choice items but also include some constructed-response items

• Body of Work method works well for assessments that consist primarily of constructed-response items

9

Standard Setting vs. Standards Validation

• Standard setting– Generally three rounds:

• Round 1: individual ratings

• Round 2: table group discussion of Round 1 results, followed by second round ratings

• Round 3: whole room discussion of Round 2 results, followed by final ratings

10

Standard Setting vs. Standards Validation

• Standards validation– Starting cut points are provided to panelists at the

beginning of the process

– Two rounds:• Round 1: table group discussion of starting cut points,

followed by first round ratings

• Round 2: whole group discussion of Round 1 results, followed by second round ratings

• The process you will be following will be closer to a standards validation process

11

Details for Standards Validation using the Bookmark and Body of Work Procedures

12

Starting Cut Points

• Starting cuts for Reading and Writing were determined using teacher judgment data:– At the time of testing, teachers were asked to categorize

each student into an achievement level category, based on classroom performance (not test performance)

– Using these categorizations as well as the students’ test scores, Measured Progress staff calculated starting cut points

13

Starting Cut Points

• Analyses of teacher judgment data indicate:– Good participation rates– Teachers appear to have been conscientious in

completing the task

• In other words, available evidence supports the validity of the starting cuts

14

Starting Cut Points

• However, standard setting is also a critical part of the process because:– the teachers based their judgments on classroom

performance, not performance on NECAP

– different teachers have different conceptions of what it means to be, for example, Partially Proficient

– they did not have the benefit of discussion with their colleagues from across the three states

– etc.

15

Starting Cut Points

• Both sources of information are important. Therefore:– You are free to recommend changing the

starting cuts, but we don’t expect to see major changes

– The changes you recommend should be based on the test content and the Achievement Level Descriptions

16

The Bookmark Procedure(Reading)

• The Bookmark procedure is a standard setting method that uses a book of items (ordered from easiest to hardest)

• Panelists place bookmarks in that book of items

17

What is the bookmark procedure?

18

What is the bookmark procedure?

• For purposes of NECAP standard setting, you will be provided with starting cut points and will either validate those cuts, or recommend modifications to them

19

A Technical Detail regarding the Bookmark Method

• Note that the ordered item cut point for a given cut does not equal the raw score a student must obtain to be categorized into the higher achievement level

• For example, the Substantially Below Proficient/ Partially Proficient starting cut for grade 3 reading is between ordered items 6 and 7; however, a student must obtain at least 20 points on the test in order to be classified into the Partially Proficient level

20

How to Place a Bookmark

• A few concepts you will need to know:– The starting cut points– The achievement level descriptions– ‘Borderline’ students– What knowledge, skills, and abilities (KSAs)

are needed to answer each question

21

How to Place a Bookmark

• Start at the beginning of the ordered item book• Evaluate whether at least 2 out of 3 students demonstrating

skills at the ‘borderline’ of Partially Proficient would correctly answer item 1

• Moving through the book, make this evaluation of each item• The bookmark should go where you no longer think 2 out of

3 Partially Proficient ‘borderline’ students would correctly answer the question.

• You may decide that the starting cuts are in the right place, or you might recommend moving them.

22

How to Place a BookmarkItem Number

Would at least 2 out of 3 students who demonstrate skills at the Partially Proficient 'borderline' correctly answer this question?

1 Yes

2 Yes

3 Yes

4 Yes

5 Yes

6 Yes

7 Yes

8 Yes

9 No

10 No

11 No

12 No

13 No

14 No

15 No

… No

23

How to Place a Bookmark

• In the example, the bookmark would go between items 8 and 9

• However, it won’t be that easy; there will be gray areas

• Place one bookmark for each cut score

• You will have the opportunity to discuss your bookmark placements and change them if desired

24

How to Place a Bookmark

• To place your bookmarks you will need to be familiar with the achievement level descriptions and the assessment items

25

How to Place a Bookmark

• Don’t worry, we have procedures, materials and staff to assist you in this process.

26

Any questions about the Bookmark Procedure?

27

The Body of Work Procedure (Writing)

• A standard method that uses approximately 30 complete sets of student work, including their answers to open-ended items and a display of their responses to the multiple-choice items

• Bodies of work cover the range of possible scores and are presented in order from lowest to highest total score

28

What is the Body of Work procedure?

• You will classify each BOW into the achievement level in which you feel it belongs:– Proficient with Distinction– Proficient– Partially Proficient – Substantially Below Proficient

29

The Body of Work Procedure

• Prior to beginning the process of rating the BOWs, you will:– take the assessment– become very familiar with the Achievement

Level Descriptions– define, with your grade-level group,

‘borderline’ students– individually review the full set of BOWs

30

Making Your Ratings

• Beginning with the first BOW, and referring to the Achievement Level Descriptions and the definition of ‘borderline’ students, determine into which achievement level the BOW should be categorized

• Continue through each BOW in turn, assigning each to an achievement level

• You may agree with the categorizations based on the starting cuts, or you may recommend changing some of them

31

Any questions about the Body of Work Procedure?

32

What Next?

• After this session, you will break into grade-level groups, where you will:– take the assessment to familiarize yourself with the test

items

– Reading groups: complete the item map, which is a document that will help you with the bookmark placement process

– discuss the Achievement Level Descriptions and develop definitions of “borderline” Partially Proficient, Proficient, and Proficient with Distinction students

33

What Next?

• You will:– discuss the starting cut points in table groups

and do the first round of ratings– discuss the first round ratings as a whole room

then do the second round of ratings

34

What Next?

• After the second round of ratings, you will have an opportunity to provide feedback about the Achievement Level Descriptions

• As the final step, we will ask you to complete an evaluation of the standard setting process

35

Good Luck!