1 An Investigation of The Response Time for Maths Items in A Computer Adaptive Test C. Wheadon & Q. He, CEM CENTRE, DURHAM UNIVERSITY, UK Chris Wheadon

1

An Investigation of The Response An Investigation of The Response Time for Maths Items in A Computer Time for Maths Items in A Computer

Adaptive Test Adaptive Test

C. Wheadon & Q. He, CEM CENTRE, DURHAM UNIVERSITY, UK

Chris Wheadon and Qingping HeChris Wheadon and Qingping He

((CEM Centre, Durham University, UKCEM Centre, Durham University, UK))

2

Background

• The CEM Centre’s Baseline Assessments on Primary and Secondary School Students

• The CEM Baseline Tests for Secondary Schools (for students aged 11-18: three separate projects) Paper based, Efficiency, Cost

• Use of CTT and IRT in Computerised Testing and to Interpret Test Data

• To Develop A Computerised Test to Replace the Existing Paper-based Baseline Tests for the Three Secondary Projects


3

• CABT comprises a calibrated item bank and a question

display system

• The adaptive test contains an adaptive maths test and

an adaptive English vocabulary test

• Testing is delivered through the Web or from the

school’s local area network

CABT-The CEM Computer Adaptive Baseline Test


4

Establishing A Calibrated Item Bank


• Testing of items by school students from different year groups through the administration of a series of tests containing common items

• Calibration of items in each test using the Rasch model (items meeting the Rasch requirements are used)

• Setting of the reference test and equating of different tests using the Rasch model and common items

• Further new item calibration through the embedding of new items in the adaptive test

5

Developing The Question Display System - Conducting Adaptive Testing


• Realisation of adaptivity is achieved through the implementation in the question display system of the Rasch model for ability estimation (MLE method is used) and question selection

• Rules for stopping test: minimum number of question; maximum number of questions; and convergence value

• Variable starting difficulty for the first question for different year groups.

6


Preliminary Results from theAdaptive Maths Test

• There are over 500 items in the maths item bank

• Effort has been made to make the items as curriculum-free as possible

• Items are content-independent to each other

• Item types include MCQ, short free text entry questions and interactive questions

• Items cover a wide range of difficulties in order for the three secondary projects (for three different year groups) to use the same adaptive test as their baseline test

7

70

80

90

100

110

120

130

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20

Item Sequence Number

Sc

ale

d A

bili

ty

0

20

40

60

80

100

120

140

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20


Tim

e T

aken

(s)


Individual Student: The Distribution of Ability and the Distribution of Item Response Time

8

80

85

90

95

100

105

110

115

120

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20


Sc

ale

d A

bili

ty

0

20

40

60

80

100

120

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20


Tim

e T

aken

(s)


Individual Student: The Distribution of Ability and the Distribution of Item Response Time

9

-8.00 -6.00 -4.00 -2.00 0.00 2.00 4.00 6.00

Item Difficulty (logits)

0

50

100

150

Re

sp

on

se

Tim

e (

s)


All Items: The Effect of Item Difficulty on Response Time

10

40

60

80

100

120

140

160

180

200

2202

39

9

67

2

22

99

58

22

8

64

1

61

2

47

5

23

60

22

65

62

6

63

9

64

9

22

66

26

9

62

4

65

4

57

3

21

24

22

81

49

3

23

34

21

22

23

52

Item Id

Re

sp

on

se

Tim

e (

s)

Year 7Year 10Year 12


A Selection of Items: The Effect of Age of Test Takers on Item Response Time

11

40

60

80

100

120

140

160

180

2359 67

0

2300 33

4

478

490 58 32

648

2360 64

9

2265 51

0

493

2270 64

1

733

228

504

2266 53

626 71

658 39

Item Id

Res

po

nse

Tim

e (s

)

Correct

Incorrect


A Selection of Items: The Relationship between Response Time for Correct Answers and Response Time for Incorrect Answers

12

Q.631 Understanding a simple algebraic relationship. Difficulty: -0.6 logits.

The table represents a relationship between x and y. What is the missing number in the table?

a) 9 b) 10 c) 11 d) 12 e) 13

X Y 2 5 3 7 4 ? 7 15


Individual Item: The Effect of Ability and Age on Response Time (Difficulty: -0.6 logits) – Easy Item

13

Correct Answers

3.1

3.2

3.3

3.4

3.5

3.6

3.7

3.8

3.9

<= -.50

-.49 --.20

-.19 -.10

.11 -.50

.51 -1.27

1.28 -2.65

2.66+

Ability Band (logits)

Sca

led

Res

po

nse

Tim

e


Incorrect Answers

3

3.2

3.4

3.6

3.8

4

4.2

<= -1.05

-1.04- -.70

-.69 --.45

-.44 --.21

-.20 -.08

.09 -.79

.80+


Sca

led

Res

po

nse

Tim

e



14

<= -.56 -.55 - -.28 -.27 - .05 .06 - .45 .46+

Ability Measure (Banded, logits)

2

3

4

5

6S

ca

led

Re

sp

on

se

Tim

e


Ability Bands within a Year Group for the Specific Item (Year 7)

15


Q .490 C om paring tw o fractions. D ifficu lty : 1 .1 log its .

C om pare the tw o exp ress ions:

E xpress ion A : 5

1817161514

E xpress ion B : 4

20191817

E xpress ion A is g rea te r than express ion B E xpress ion B is g rea te r than express ion A The express ions a re equa l

Individual Item: The Effect of Ability and Age on Response Time (Difficulty: 1.1 logits) – Medium-difficulty Item

16

Correct Answers

3.9

4

4.1

4.2

4.3

4.4

4.5

4.6

4.7

4.8

4.9

<=1.73

1.74 -1.98

1.99 -2.19

2.20 -2.41

2.42 -2.80

2.81 -3.64

3.65+


Sca

led

Res

po

nse

Tim

e


Incorrect Answers

3

3.2

3.4

3.6

3.8

4

4.2

4.4

4.6

4.8

<=1.15

1.16 -1.45

1.46 -1.65

1.66 -1.82

1.83 -2.06

2.07 -2.44

2.45+


Sca

led

Res

po

nse

Tim

e



17

<= 1.82 1.83 - 2.09 2.10 - 2.38 2.39 - 3.05 3.06+


2

3

4

5

6

7

Sca

led

Re

sp

on

se

Tim

e



18

Q.2359 Reading a pie chart, working with fractions. Difficulty: 2.9 logits.

The pie chart represents the different colours of cars in Albert Street. If there are 144 cars in total, how many are blue (segment z)?

Free response answer.

1/6

1/9

2/3

z


Individual Item: The Effect of Ability and Age on Response Time (Difficulty: 2.9 logits) – Difficult Item

19

Correct Answers

4.2

4.4

4.6

4.8

5

5.2

5.4

5.6

<=2.14

2.15 -2.92

2.93 -3.49

3.50 -4.05

4.06 -4.80

4.81 -5.78

5.79+


Scale

d R

esp

on

se T

ime


Incorrect Answers

3

3.5

4

4.5

5

5.5

<=.24

.25 -.99

1.00 -1.57

1.58 -2.05

2.06 -2.51

2.52 -3.29

3.30+


Scale

d R

esp

on

se T

ime



20

<= 1.82 1.83 - 2.09 2.10 - 2.38 2.39 - 3.05 3.06+


2

3

4

5

6

7

Sc

ale

d R

es

po

ns

e T

ime



21

Conclusions and Further Work

• Response time for all the maths items in the test generally increases with item difficulty but shows great variability

• Item difficulty levels and the age and ability of test takers have significant influence on item response time

• The information obtained in this study can be used for constructing more efficient tests

• Further work will involve investigating the relationship between CABT results and students’ subsequent academic performance


Documents

1 An Investigation of The Response Time for Maths Items in A Computer Adaptive Test C. Wheadon & Q. He, CEM CENTRE, DURHAM UNIVERSITY, UK Chris Wheadon