Item Analysis

ITEM ANALYSIS Ann Meredith Garcia, MD

What is item analysis? • Judging the quality of test items by examining the students’ responses

• Competent vs. less competent students?

• Difficulty of items?

How is item analysis done?

Administer the test

Check the students’

responses to separate items

Check the total scores

Tasks of item analysis

1st task: Item discrimination

• Sort the students who know the topic well from those who do not

• Correlate performance on a single test item with total test performance

•  (+) Correlation à better discrimination

2nd task: Item difficulty

Electronic item analysis

Student Item 1 Score

Total Score / 30

A 1 25 B 1 19 C 1 18 D 0 16 E 1 12 F 0 10

• Average total score (correct Item 1) = 18.5

• Average total score (incorrect Item 1) = 13

• Computed correlation coefficient = 0.53 •  Item is to some extent

related to the total score Correct = 1

Incorrect = 0


r = correlation of an option with the total score p = percentage of students who chose that option (n = 65)

•  Correct options should show positive correlations; distractors should show negative correlations

Item 1: r = 0.25 à low correlation Item 2: r = 0.49 à fairly good correlation

Item 3: r = 0.34 à modest correlation •  r ≤ 0.15 à course content is not being assessed well à

eliminate the item (OR revise)

Item A B C 1 r = -0.27

p = 13.89 r = 0.25

p = 50.00 r = -0.06 p = 36.11

2 r = -0.46 p = 5.56

r = 0.49 p = 88.86

r = -0.22 p = 5.56

3 r = -0.30 p = 16.67

r = -0.13 p = 27.78

r = 0.34 p = 55.56


r = correlation of an option with the total score p = percentage of students who chose that option (n = 65)

• Standard error (SE) = 1 / √ (number of students – 1) = 0.12

• Any r > 2(SE) will be accepted as other than a chance relationship between the item and the total score

Item 1: r = 0.25 > 0.24 [2(SE)] à very marginal but acceptable

Item A B C 1 r = -0.27

p = 13.89 r = 0.25

p = 50.00 r = -0.06 p = 36.11

2 r = -0.46 p = 5.56

r = 0.49 p = 88.86

r = -0.22 p = 5.56

3 r = -0.30 p = 16.67

r = -0.13 p = 27.78

r = 0.34 p = 55.56

Item analysis by hand • Step 1: Arrange the students’ papers according to their

test scores (highest to lowest). • Step 2: Divide these into “high scorers” vs. “low scorers”. • Step 3: Tabulate the number of students who chose each

option in both groups.

•  Step 4: Compute for the discrimination index.

Item 1 A B C D Total High scorers 2 4 0 16 22 Low scorers 12 7 0 4 23

Item analysis by hand

•  Step 4: Compute for the discrimination index (DI).

DI = (NumHigh – NumLow) x Number of students in larger group = (16 – 4) / 23 = 0.52

* Ranges from 0 – 1.00 * Can also be negative (for distractors)


Item analysis by hand

Alternative: Straight difference method • Steps 1 – 3: Same • Step 4: Compute for NumHigh – NumLow.

•  If ≥ 0.10(n) à adequate (used across all items) 16 – 4 = 12

0.10(45) = 4.5


Analysis of distractors

• Distractor C was not chosen by any student à 3-option item (0.33 instead of 0.25 chance level of guessing the item correctly)

• Good item – each distractor will be chosen more often by the low scorers

Item 1 A B C D High scorers 4 13 0 3 Low scorers 7 9 0 4 p 27.5 55.0 0.0 18.5



• Difficulty index/facility index = proportion of students who get an item correctly

•  Step 1: Award a score to each student.

•  Step 2: Arrange the scored tests from highest to lowest.

•  Step 3: Identify the upper and lower 27%.

•  Step 4: Count the response counts in each group.

Item 1 A B C D Total

High scorers

2 4 0 16 22

Low scorers

11 7 0 4 22




• Step 5: Calculate the difficulty index.

= H + L N

H = no. of students in the high group with a correct answer

L = no. of students in the low group with a correct answer

N = total no. of students


High scorers

2 4 0 16 22

Low scorers

11 7 0 4 22




• Ranges from 0 – 1.00 • Best ≈ 0.50 (0.30 – 0.70) •  Larger index à easier

item; smaller index à more difficult item

DI = (16 + 4) / 80

= 0.25 • Criteria:

•  ≥0.35 = excellent question •  0.25 – 0.34 = good question •  0.15 – 0.24 = marginal

question à revise •  <0.15 = poor question à

discard


High scorers

2 4 0 16 22

Low scorers

11 7 0 4 22

Item analysis for essay tests

•  Step 1: Identify the upper and lower 25% of the students. •  Step 2: Compute for the following:

Disc. = (Sum of scores for highs – sum of score for lows) N x (max. possible score on item)

Diff. = (Sum of scores for highs + sum of score for lows) 2N x (max. possible score on item) N = 25% of the number tested

Item Score

High Group Low Group No. of Students No. of Students

x Score No. of Students No. of Students

x Score

10 9 90 1 10

8 6 48 0 0

6 2 12 4 24

4 3 12 7 28

2 0 0 8 16

Total 20 162 20 78


•  Step 2: Compute for the following: Disc. = (Sum of scores for highs – sum of score for lows)

N x (max. possible score on item) = (162 – 78) / [(0.25 x 80) x 10] = 0.42

Satisfactory discrimination

Item Score



x Score

10 9 90 1 10

8 6 48 0 0

6 2 12 4 24

4 3 12 7 28

2 0 0 8 16

Total 20 162 20 78


•  Step 2: Compute for the following: Diff. = (Sum of scores for highs + sum of score for lows)

2N x (max. possible score on item) = (162 + 78) / [(2 x 0.25 x 80) x 10] = 0.60

Satisfactory difficulty

Item Score



x Score

10 9 90 1 10

8 6 48 0 0

6 2 12 4 24

4 3 12 7 28

2 0 0 8 16

Total 20 162 20 78

Item response theory and item analysis • Calculates the odds of getting an item right à converts

this number to a natural logarithm • Allows faculty to equate the item difficulty scale on one

test to the scale of another test and across different student groups

• Useful system for building item banks

Thank you! J

Education

Item Analysis