Diesta Irt Presentation

8/9/2019 Diesta Irt Presentation

http://slidepdf.com/reader/full/diesta-irt-presentation 1/39

The Case of De La Salle-Santiago

Zobel School (DLSZ)



Shift from Classical Test Theoryapproach to Item Response Theory

(IRT)

IRT was introduced to DLSZ in School

Year 2008-2009 by

Dr. Carlo Magno.



Curriculum-based Achievement tests

Grade/Year Levels

Senior Prep

Grades 1 to Grade 7

First year to Fourth year high school

Subject Areas

English Math

Filipino Science

Christian Living Social Studies



Two (2) forms of the achievementtests were created

Experts in the subject areas content

validated and externally reviewed

the test items and the curriculum



Subject Area Teachers¶ workshop wereconducted

Teachers /Subject Area Coordinators

revised some items

Consultants reviewed revisions via email



Test item pool and TOS to IPA

Orientation of Teachers for the pilot testing

Pilot testing for all levels

Gathering of test data from the pilot tests

Statistical analysis of test items



` Descriptive statistics

` Mean and standard deviations were obtained to

determine the central tendency and variability of

the scores for each form of the tests.



Two ways:

` Internal consistency ± Cronbach¶s alpha value

of .70

` Parallel forms of the test

-Pearson r was used to correlate the

scores of each examinee for both forms.

-All correlation coefficients obtainedwere significant (p<.05)



` Content validity

The items were reviewed by experts in the

field based on its intended skills and content.

` Concurrent validity

Each form of the test was correlated with

each other. Majority of the correlation

coefficients were significant.



` The one-parameter Rasch model was used todetermine the difficulty and fit of the items.

Item difficulty

` Positive logit values indicate that the

item is difficult and negative logit values

indicate that the item is easy.



SY µ08-¶09 SY ¶09-¶10



Item Fit

Means Square Values (MNSQ) within 0.8 to 1.2 are

said to have good fit.

Bad items are determined if the ability of the persons

do not match difficulty of the items.



Legend: - -

- - 1

ITEM FIT RESULTS

ENIOR PREP

FormNo. of

Items

No. of Good Fit

Item/s

No. of Bad Fit

Item/s

N MEAN TDEV PROPORTION

CORRECT

P 3 1 33 1 .16 1 .4 4.67 4. .64 .65

FORM SY2008-2009 taken by the Upper Level (PILOT TESTING)

SY2008-2009 SY2009-2010

Seni r r ep 7, , , 7, , 19, 18, 18,

FORM No. of Items

SY2008-2009 SY2009-2010

No. of Difficult

Items

No. of Easy

Items

No. of Difficult

Items

No. of Easy

Items

SP 30 16 14 16 14

FORM DIFFICULT to EASY items EASY to DIFFICULT items

SP 23, 28 2, 5



Legend: - SY 2008-2009

- SY 2009-2010

ENGLISH

Form

No.

of

Items

No. of

Good Fit

Item/s

No. of

Bad Fit

Item/s

N MEAN STDEVPROPORTION

CORRECT

Engl 1 45 42 45 3 - 215 241 31.51 26.60 6.52 7.45 0.70 0.59

Engl 2 44 43 43 1 1 210 291 29.07 26.53 6.66 6.66 0.66 0.60

Engl 3 60 56 58 4 2 215 328 32.53 31.44 9.55 8.91 0.54 0.52

Engl 4 70 65 69 5 1 265 313 37.48 40.23 11.18 8.02 0.54 0.57

Engl 5 70 62 68 8 2 198 290 39.73 37.77 11.22 10.14 0.57 0.54

Engl 6 75 75 74 - 1 269 285 47.19 42.73 9.34 10.07 0.63 0.57

Engl 7 80 79 80 1 - 292 316 51.42 45.82 10.72 10.66 0.64 0.57



FORM

SY2008-2009 taken by the

Upper Level

(PILOT TESTING)

SY2009-2010

Engl 1 19, 27, 31 NO REVISIONS NEEDED

Engl 2 46 23

Engl 3 9, 10, 21, 40 39, 50

Engl 4 20, 22, 34, 35, 53 34

Engl 5 28, 32, 33, 37, 44, 46, 50, 53 12, 44

Engl 6 NO REVISIONS NEEDED 31

Engl 7 78 NO REVISIONS NEEDED


Engl 1 19 40, 41

Engl 2 NONE NONE

Engl 3 9, 10, 18, 33, 42, 55 5, 16, 40, 43, 47, 52Engl 4 20, 25,50, 61, 70 27, 52, 53, 60

Engl 5 24, 32, 37, 39, 40, 48, 59, 65, 68 28, 46, 53, 54

Engl 6 29, 42, 53 21, 34, 35, 57, 64

Engl 7 3, 8, 11, 20 37, 55, 74



Form

No. of

Items

SY2008-2009 SY2009-2010

No. of Difficult Items

No. of EasyItems

No. of

Difficult Items

No. of Easy Items

Engl 1 45 21 24 22 23

Engl 2 44 21 23 23 21

Engl 3 60 32 28 32 28

Engl 4 70 36 34 35 35

Engl 5 70 36 34 31 39

Engl 6 75 37 38 39 36

Engl 7 80 39 41 39 41



Legend: - SY 2008-2009

- SY 2009-2010

r

.

f Ite

. f

Good itIte /

. f

d itIte /

S

I

L 1 30 28 29 2 1 198 241 25.69 25.00 2.82 3.17 0.86 0.83

L 2 30 27 29 3 1 179 289 21.16 20.73 4.12 3.84 0.71 0.69

L 3 40 39 39 1 1 255 323 25.58 24.20 5.28 5.24 0.64 0.61

L 4 50 50 49 - 1 283 318 27.25 27.66 6.57 6.31 0.55 0.55

L 5 50 50 48 - 2 282 301 36.15 35.04 5.86 7.29 0.72 0.70

L 6 60 60 59 - 1 285 310 39.56 37.76 6.68 7.76 0.66 0.63

L 7 60 60 58 - 2 233 320 42.14 43.06 6.29 8.05 0.70 0.72

H IS I LI I G



FORMSY2008-2009 taken by the

Upper Level (PILOT TESTING)SY2009-2010

CL 1 22, 23 10

CL 2 2, 13, 25 30

CL 3 12 34

CL 4 NO REVISIONS NEEDED 40

CL 5 NO REVISIONS NEEDED 33, 44

CL 6 NO REVISIONS NEEDED 7

CL 7 NO REVISIONS NEEDED 2, 60


CL 1 5 28, 13, 14,

CL 2 14, 24 13, 26

CL 3 8, 21, 35, 36 1, 13, 22, 23, 33, 40CL 4 6, 15, 17, 27, 38, 43, 47, 50 9, 12, 14, 44, 46

CL 5 5, 9, 21, 23, 35 14, 17, 24, 33

CL 6 30, 37 4, 25, 40, 58

CL 7 3, 6, 9, 29, 35, 51 4, 5, 14, 16, 47, 57



Form No. of Items

SY2008-2009 SY2009-2010

No. of

Difficult Items

No. of Easy

Items

No. of Difficult

Items

No. of Easy

Items

CL 1 30 14 16 16 14

CL 2 30 16 14 16 14

CL 3 40 18 22 21 19CL 4 50 26 24 22 28

CL 5 50 26 24 27 23

CL 6 60 28 32 31 29

CL 7 60 28 32 28 32



Legend: - SY 2008-2009

- SY 2009-2010

FORMNo. of

Items

No. of

Good Fit

Item/s

No. of

Bad Fit

Item/s

N MEAN STDEV

PROPORTIO

N

CORRECT

Math 1 30 27 28 3 2 190 239 24.08 19.81 4.42 4.81 0.80 0.66

Math 2 40 35 36 5 4 212 292 26.68 21.69 6.62 5.35 0.67 0.54

Math 3 29 23 27 6 2 244 318 21.81 19.43 4.79 4.39 0.75 0.65

Math 4 49 47 49 3 - 216 312 30.25 32.24 7.72 8.49 0.62 0.66

Math 5 50 48 48 2 2 273 287 30.05 23.83 7.80 7.48 0.60 0.48

Math 6 58 59 55 1 3 260 281 31.92 28.59 8.16 9.01 0.55 0.48

Math 7 60 59 59 1 1 251 332 30.68 29.41 9.00 8.10 0.51 0.49

MATH



FORM

SY2008-2009 taken by the

Upper Level (PILOT

TESTING)

SY2009-2010

Math 1 9, 16, 20 16, 25

Math 2 5, 19, 20, 24, 32 23, 26, 31, 32

Math 3 2, 3, 7, 18, 21, 25 18, 25

Math 4 31, 36, 39 NO REVISIONS NEEDED

Math 5 25, 41 25, 41

Math 6 14 8, 14, 50

Math 7 6 50


Math 1 5 12, 26, 20

Math 2 38, 25, 5 20, 32, 23, 18, 31, 19

Math 3 21, 27, 24, 14 7

Math 4 42, 36, 38 6, 27, 1

Math 5 39, 46, 44 8, 28, 4

Math 6 44, 37, 20, 42, 15, 34, 26, 18, 31, 28 45, 6, 25

Math 7 30, 31, 49 57, 52, 47, 46, 17, 36, 18, 11



Form No. of Items

SY2008-2009 SY2009-2010

No. of Difficult

Items

No. of Easy

Items

No. of Difficult

Items

No. of Easy

Items

Math 1 30 16 14 18 12

Math 2 40 16 24 19 21

Math 3 29 15 14 12 17

Math 4 49 25 24 25 24

Math 5 50 27 23 27 23

Math 6 58 32 26 26 32

Math 7 60 26 34 32 28






Upper Level (PILOT ESTING)SY2009-2010

Sci 1 10, 27 12

Sci 2 10, 22, 24, 25, 27 NO REVISIONS NEEDED

Sci 3 17 15, 32

Sci 4 30 NO REVISIONS NEEDED

Sci 5 6, 7, 8, 10, 11, 22, 44 10

Sci 6 NO REVISIONS NEEDED NO REVISIONS NEEDED

Sci 7 27 NO REVISIONS NEEDED


Sci 1 6, 7, 26 4, 1, 10

Sci 2 11, 26, 28 22, 25

Sci 3 24, 27 2, 17

Sci 4 11, 12, 23, 26,45 2, 10, 15, 20, 25, 27, 33, 37, 46

Sci 5 2, 13, 15, 16, 33, 36, 43, 47, 49 8, 11, 22, 24

Sci 6 11, 15, 18, 21, 23, 26, 33, 37, 48, 53, 54 8, 31, 32, 35, 41, 42, 56

Sci 7 31, 32, 36, 39, 51, 52, 54, 58 1, 3, 7, 12, 13, 20, 26, 28, 30, 41



Form No. of

Items

SY2008-2009 SY2009-2010

No. of Difficult

Items

No. of Easy

Items

No. of Difficult

Items

No. of Easy

Items

Sci 1 30 13 17 13 17

Sci 2 30 14 16 13 17

Sci 3 40 22 18 22 18

Sci 4 50 22 28 26 24

Sci 5 50 26 24 21 29

Sci 6 60 31 29 27 33

Sci 7 60 29 31 31 29



Legend: - SY 2008-2009

- SY 2009-2010

FORM

No. of

Items

No. of

Good FitItem/s

No. of

Bad FitItem/s N MEAN STDEV

PROPORTION

CORRECT

Fil 1 30 27 27 3 3 229 239 21.40 16.85 5.22 5.25 0.71 0.56

Fil 2 35 32 34 3 1 174 292 23.92 21.40 6.15 5.75 0.68 0.61

Fil 3 40 37 40 3 - 204 328 24.53 22.42 8.64 7.40 0.61 0.56

Fil 4 50 45 49 5 1 258 318 27.55 26.42 9.03 8.38 0.55 0.53

Fil 5 50 54 47 6 3 266 287 27.81 26.52 8.90 7.95 0.56 0.53

Fil 6 60 55 55 5 5 276 305 39.24 32.42 9.43 10.25 0.65 0.54

Fil 7 60 56 59 4 1 240 319 37.46 31.83 9.58 8.80 0.62 0.53

FILIPINO




Upper Level (PILOT TESTING)SY2009-2010

Fil 1 4, 9, 26 9, 20, 26

Fil 2 24, 28, 29 8

Fil 3 2, 5, 24, 27 NO REVISIONS NEEDED

Fil 4 15, 19, 21, 39, 43 38

Fil 5 5, 16, 22, 26, 36, 38 5, 25, 26

Fil 6 5, 15, 20, 37, 54 5, 10, 11, 19, 20

Fil 7 4, 30, 39, 48 9


Fil 1

Fil 2 33 21

Fil 3 4, 9, 14, 29, 38 10, 16, 30

Fil 4 2, 11, 40 16, 35, 49

Fil 5 31, 40, 42 3, 5, 9, 30, 34, 36

Fil 6 1, 31, 40, 44, 49, 56

Fil 7 6, 8, 12, 22, 23, 42, 43, 45, 57 2, 13, 27, 32, 34



Form No. of Items

SY2008-2009 SY2009-2010

No. of Difficult

Items

No. of Easy

Items

No. of Difficult

Items

No. of Easy

Items

Fil 1 30 13 17 14 16

Fil 2 35 17 18 17 18

Fil 3 40 22 18 20 20 Fil 4 50 27 23 27 23

Fil 5 50 22 28 25 25

Fil 6 60 28 32 33 27

Fil 7 60 29 31 25 35



Which of the tests obtained the sameresults when administered to 2 different

groups for«

Item Difficulty?English 2 and Filipino 1 only

Item Fit?

Science 6 only (no bad fit items)



Batch 1 ± IPA Personnel and Coordinators

Batch 2 ± Math and Science Teachers

What did you like in the activity?

- workshop is very useful and

very relevant.

- Everything about the latest

technology that we can use for

Item Analysis

My suggestions to improve the activity are:

- more workshops, it was an interesting

topic but then we need more time so we

can absorb all the important details



Batch 3 ± Social Studies & CL teachers

Batch 4 ± English & Filipino teachers

What i li i the activit

- It is helpful in determining items difficulty and

good fitting items in tests.It will further help usteachers in creating reliable and valid

assessments.

- The topic is interesting because it's ''new''

- New insights.

- Actual/hands on.

- It was just too fast.

What did you dislike in the activity?


- Study further its advantage before bringing

this down to us, teachers.



Batch 4 ± English & Filipino teachers

What did you like in the activity?

What did you dislike in the activity?


- the IRT is ok but would need more time

to familiarize with it.

- The topic is very applicable in test construction and testing.

- it's a different way to analyze items (basically multiple choice) although theconcern about why do it tediously needs to be addressed (aside from the fact

that the teacher gets to use the improvements needed in the test)

- I really like the lecture part. the speaker explained it very well in very

interesting manner. congratulations!

- Longer time for practice/hands on.

- use of varied data so we can practice using the program more.



SUMMARY

Best Practices in the conduct of IRT:

1. The school sought the assistance of consultants from

De La Salle University , Manila, Dr. Alicia Bustos-Orosa,

to take the lead in the test development project and Dr.

Carlo Magno, for the statistical treatment and analysis of test data.

2. The Instructional and Performance Assessment Office

(IP A) in collaboration with the Principals and subject

area coordinators, worked closely with the consultants inrelation to planning the in-service programs for the

faculty.



3. Two of the IPA personnel were trained to

prepare the test data needed by Dr. Magno, which

was followed by consultative meetings on the

technical analysis of test data.

4. The administration provided the financial

support needed for the conduct of the IRT by

allotting a budget for the purchase of a scanner ,

software for making answer sheets, software for

IRT, training the faculty and hiring additional

personnel to perform the tasks.



5. The IPA technical support staff did the IRT for the

achievement tests administered in February and

March 2009 under the supervision of Dr. Magno.

6. The training for the IPA personnel, principals, andsubject area coordinators was held on November 7,

2009. This was followed by training for Science and

Math teachers on December 5, 2009. The CL and

Social Studies teachers had their turn on June 26, 2010.The last group, the English and Filipino teachers, had

their training on July 24, 2010.



7. The items in the first end termexams (September 15-17) will be

analyzed using the IRT approach and

this will be done by the teachers withthe assistance of the IPA Office.



1.As a whole, the academic community is receptive

and open to trying out new approaches to improve

their craft. It is therefore easier for DLSZ to

introduce innovations.

2.The IRT results can be further validated by looking

into other factors that have influenced the results.

IMPLICATIONS



3. Since there are still revisions required to improve

the test items, seminar- workshops specific for the

subject areas identified are recommended.

4. To further practice using the IRT, opportunities

should be provided to the teachers, not just

limited to end-term examinations.

IMPLICATIONS



³Item writing is an art. It requires an uncommoncombination of special abilities. It is mastered only

through extensive and critically supervised practice. It

demands and tends to develop high standards of quality

and a sense of pride in craftsmanship.´

Robert Edel

Thank you. God bless.

Documents

Diesta Irt Presentation