Computerized Adaptive Testing for the

Singapore Employability Skills System

Patricia Rickard – CASAS

Linda Taylor - CASAS

Richard Ackermann – Team Code

Jim Olsen – Alpine Testing Solutions

Background

• The Singapore Workforce Development

Agency plays a key role in implementation

of Employability Skills System (ESS)

“to enhance the employability and

competitiveness of employees and job

seekers, building a workforce to meet

changing needs of Singapore’s economy.”

The Singapore Employability Skills System

(ESS) aims to …• equip the workforce with generic and portable skills that

will enable workers to better adapt to new job demands,

work challenges and changing work environment

• help lower skilled and low waged workers (LWWs) gain

and improve their jobs through higher skills

• enhance recognition of ESS as an alternate entry level

qualification for the 504,800 Singaporean workers

without “N” and “O” level qualifications

Objectives

Understand how the ESS system

– meets the time constraints of workers

– uses CAT and CBT together with human scoring

for a battery of tests

– serves walk-ins, businesses, remote testing

locations and prisons

– offers assessment in multiple deployments

Objectives

• Understand challenges in customizing

computer delivered assessments for

adults with low to high literacy skills

CASAS developed and

customized for Singapore • CAT tests for

– math

– reading

– listening

• Computer-based tests for

– writing

– speaking

7

Level 1 Pre-Beginning

Level 2 Beginning

Level 3 High Beginning

Level 4 Low Intermediate

Level 5 High Intermediate

Level 6 Advanced

Level 7 High Advanced

Level 8 Proficient

Nil

SOA 1

SOA 2

SOA 3

SOA 4

SOA 5

SOA 6

SOA 7

SOA 8Test Score

250

245

235

220

210

200

190

180

150

WPLN

Assessment

Statement of

Attainment

WPLN Training

N

R L

WS

WPLN training modules are designed to raise

a learner’s proficiency gradually

Test Administration Environment

• Secure proctored locations

– Assessment Center

– Business Sites

– Prisons

• Test Security

• Multiple proctors

Lab

Lab Designed with Headsets for Listening & Speaking

Off to business sites

Demonstration

• Sample test items

– Multiple choice

– Performance-based

• Test delivery and administration system

• Scoring

– Computer

– Human

Sample Low Level Reading Item

A to teach young people

how to stay healthy

C to provide nutritious meals for

young people

B to identify schools with

unhealthy conditions

D to interest teenagers in health

care careers

2. What is the purpose of the new project?

Sample Math Item

Sample Speaking Prompt

CAT Psychometric Procedures

• Selection of initial item

• Rasch model calibration and proficiency

estimation

• Stopping rule based on multiple criteria

• Review of test score results and

psychometric analyses

CAT Mechanism

00.511.522.5

2121.52222.52323.524

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18

SE

Ab

ilit

y

Items

18 Items

Item correct

Item incorrect

SE

Ability

Score Distribution-Reading N=12,625

0

50

100

150

200

250

300

13

6

15

9

16

7

17

0

17

4

17

7

18

0

18

3

18

6

18

9

19

2

19

5

19

8

20

1

20

4

20

7

21

0

21

3

21

6

21

9

22

2

22

5

22

8

23

1

23

4

23

7

24

0

24

3

24

6

24

9

25

2

25

5

25

8

26

1

26

4

27

1

Fre

qu

en

cy

Score

SOA = 2 SOA = 3 SOA = 4 SOA = 5 SOA = 6 SOA = 7 SOA = 8SOA = 1SOA = 0

Score Distribution-Math N=10,095

SOA = 0 SOA = 2 SOA = 3 SOA = 4 SOA = 5 SOA = 6 SOA = 7 SOA = 8SOA = 1

Descriptive Statistics

Descriptive Statistics N Minimum Maximum Mean Std. Error Std. Deviation

Scaled Score 12,625 139 270 220.0 0.15 17.03

Starting Ability 12,625 195 205 199.48 0.01 19.91

Final Ability (at exit) 12,625 139.39 269.72 219.97 0.02 17.03

Starting SE 12,625 2.00 2.66 2.08 0.00 0.08

Final SE 12,625 0.48 3.57 0.51 0.00 0.07

Descriptive Statistics N Minimum Maximum Mean Std. Error Std. Deviation

Scaled Score 10,095 136 274 218.9 0.19 19.52

Starting Ability 10,095 195 205 198.03 0.01 6.82

Final Ability 10,095 135.69 273.74 218.88 0.02 19.51

Starting SE 10,095 2.00 3.03 2.08 0.00 0.08

Final SE 10,095 0.48 3.25 0.52 0.00 0.08

Reading

Math

Mean Reading Ability Estimate N=12,625

185

190

195

200

205

210

215

220

225

230

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19

Ab

ility

Sco

re

Items

Mean

Reading Standard Error (SE) N=12,625

0

0.5

1

1.5

2

2.5

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20

SE

Items

SE

Mean Math Ability Estimate N=10,095

180

190

200

210

220

230

240

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20

Ab

ilit

y Sc

ore

Items

Mean

Math Standard Error (SE) N=10,095

0

0.5

1

1.5

2

2.5

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20

SE

Items

SE

Reading RIT Value and Measure

Correlation is .943 between Singapore Reading Item Rasch

Calibration RITS and the Rasch Item Measure computed from the

concurrent calibration of the CAT item vectors.

-6

-4

-2

0

2

4

6

180 190 200 210 220 230 240 250 260

Measu

re

RIT Value

Math RIT Value and Measure

-8.00

-6.00

-4.00

-2.00

0.00

2.00

4.00

6.00

8.00

10.00

170 180 190 200 210 220 230 240 250 260 270

Mea

sure

RIT Value

Correlation is .968 between Singapore Math Item Rasch Calibration RITS

and the Rasch Item Measure computed from the concurrent calibration of

the CAT item vectors.

Stopping Rule – Reading N=12,625

0.0%

20.0%

40.0%

60.0%

80.0%

100.0%

120.0%

139

162

171

175

179

183

187

191

195

199

203

207

211

215

219

223

227

231

235

239

243

247

251

255

259

263

Score

20 Items max.

SE <0.5

Stopping Rule – Math N=10,095

0.0%

20.0%

40.0%

60.0%

80.0%

100.0%

120.0%136

165

169

174

178

182

186

190

194

198

202

206

210

214

218

222

226

230

234

238

242

246

250

254

258

262

267

Score

20 Items max.

SE <0.5

Reporting Results

• Reporting Test results

• Plans for future enhancements and

extensions