Pamela K. Kaliski, Stefanie A. Wind, George Engelhard Jr., Deanna L.Morgan,

Using the Many-FacetedRasch Model to Evaluate

Standard Setting Judgments:An IllustrationWith the Advanced Placement

Environmental Science Exam

Pamela K. Kaliski, Stefanie A. Wind,

George Engelhard Jr., Deanna L.Morgan,

Barbara S. Plake, and Rosemary. Reshetar

psychology and applied statistics

http://ncaase.com/about/bio?id=4

http://des.emory.edu/home/people/faculty/Engelhard.html

Contents

2. Multiple Yes-No (MYN) method

1. Many-Faceted Rasch Model

4. Results and Conclusion

3. Instrument

22

introduction

Standard setting ‘‘. . . standard setting refers to the process of establishing one or

more cut scores on examinations. The cut scores divide the distribution of examinees’ test performances in two or more categories’’ Cizek and Bunch (2007)

criteria The criteria for evaluating panelist judgments Procedural validity ： implementation issues and

documentation Internal validity ： interpanelist and intrapanelist consistency External validity ： comparisons with other methods

33

Many-Faceted Rasch Model

n: panelist k: a standard setting modified Angoff rating i :item ； j : round n is the judged severity for panelist n, i is the average judged item difficulty for item i, j is the judged average performance level for round j jk is the cut score, or threshold coefficient, from round j for

standard setting ratings of k. （ rating k relative to k − 1）

44

Rating quality indices

Rating quality indices(a) panelist severity/leniency measures:

separation statistics/chi-square statistic(b) model–data fit ： Outfit MSE

(c) the creation of a visual display for comparing panelist judgments on the latent variable

55

Multiple Yes-No (MYN) method

MYN requires panelists to consider the borderline examinee at each cut score and to identify at which level the borderline examinee would be able to answer each item correctly.

panelists considered each item and decided whether or not the borderline examinee in

each category would be able to identify the correct answer

66

PLDs

77

Would a borderline-1/2 student be able to answer this item correctly? If yes, then the panelist would circle the 1/2 cut score on the rating form and move on to the next item.

If no, then the panelists would consider the next question about the same item: Would a borderline-2/3 student be able to answer this item correctly?

If yes, the 2/3 cut score would be circled for that item and the panelist would move on to the next item. If no, the panelist would consider the next question about the same item: Would a borderline-3/4 student be able to answer this item correctly?

If yes, the 3/4 cut score would be circled for that item and the panelist would move on to the next item.

If no, then the panelists would consider the next question about the same item: Would a borderline-4/5 student be able to answer this item correctly? If yes, the 4/5 cut score would be circled for that item and the panelist would move on to the next item. If no, then the panelist would consider the final question about the same item: Would the above borderline-5 student be able to answer this item correctly?

If yes (which is likely given that all other possible borderline students have

been considered), then the Above 5 score would be circled for that item.

88

99

instuments

The Advanced Placement (AP) program (Advanced Placement Environmental Science (APES) examination) is composed of 34 courses and corresponding examinations in 22 subject areas.

Data used in this study come from the 2011 administration of the APES exam and the standard setting for this examination.

100 MC items and four CR itemsData used in this study are the ratings that resulted

from two rounds of item-level judgments provided by the 15 APES panelists

1010

Research Purpose

the MFR model is used to evaluate the quality of judgments on MC items provided by panelists who participated in a modified Angoff standard setting that used the MYN method for MC items, the 2011 APES exam.

panelist characteristics (gender and level of teaching) are incorporated into the MFR model to determine whether or not these are explanatory variables that account for differences in panelist ratings

1111

Results and conclusion

1212

Results and conclusion P397

1313

P398

1414

MSE[0.6-1.5]

P400

1515

P401

1616

1717

P402

1818

P404

P405

1919

P405

2020

Future study

additional explanatory variablesadditional statistical modelsMFR model+other modified Angoff procedures,

or Bookmark proceduresoverall contribution of each facetCR questions

2121

Thus the interaction between theta and omega should be considered in Equation 1

different rating scale structure and a random effect approach

Through the PC power, let panelists use computer to do standard setting. We can record the time spent

transform cut score of each category to the expected scores

2222