Applying Open Source to Adaptive Testing

7/24/2019 Applying Open Source to Adaptive Testing

1/25

Applying Open Source to

Adaptive TestingMichal [email protected]

TAO DAYS Berlin, 10/09/2012

www.psychometrics.cam.ac.uk

Michal Kosinski

[email protected]


2/25

PART I: Open-source

PART II: Item Response Theory and Computerized

Adaptive Testing

PART III: What do you need to build your CAT (or anyother test or e-learning platform)

and where to get a right open-source software

PLAN


3/25

Part I:

Open-source

Michal Kosinski

[email protected]


4/25

Who is not convinced to open-source software?

Problems? Concerns?

Using open-source


5/25

High reliability / free updates / constant debugging

Open source gives you independence: Own your software copy and access all the updates

Enhance your software as (and for how much) you wish

See what is under the hood

Low switching costs = innovation and progress

Usually relatively low costs

Why open-source instead ofcommercial one?


6/25

all the aforementioned advantages, plus:

Avoids rediscovering the wheel

Minimize dependence on your suppliers / employees

Decrease maintenance costs (e.g. debugging /updating)

Open-source elements as a part ofyour own software dev. project


7/25

It doesnt mean that you have to stop offering commercials0lutions (e.g. mySQL model)

Low-cost marketing Build trust

Get free input from the community

Ideas / Improvements / Enhancements / Debugging

Increased opportunities for funding

Improve the world! (and get competitive edge!)

Publish your stuff as open-source


8/25

How do I sell products based on open-source?

In the same way as commercial ones!

Most open-source can be used for commercial purposes

Open-source based products based can be sold / licensed / etc.

Who do I blame/sue when something goes wrong?

Read your agreement with a commercial software developercan you really blame them?

Choose your software wisely Pay someone to give you a warranty

Problems and myths


9/25

Part II:Item Response Theory

and ComputerizedAdaptive Testing


Michal Kosinski

[email protected]


10/25

Modelling on the item level (and not on the test level) Robust item parameters

Can be used for test development and scoring

Score is item independent

Advantages:

Increased accuracy of the tests

Cheaper test development

On-going test development Test security / IP protection

Enables Adaptive Testing (CAT)

IRT


11/25

Select items fit for a person and purpose

Advantages:

Much shorter tests

Less frustration / more attention

Detect react to inattention, cheating, etc.

Tests fit for purpose (e.g. screening)

Further increased test security and protection

CAT


12/25

Adaptive learning

Innovative question formats

Incorporate non-question information

IRT and CAT beyond testing!


13/25

Part III:

What do you need to buildyour CAT (or any other testor learning process)


Michal Kosinski

[email protected]


14/25

1. Build a test = develop an Item bank Item writing / translations

2. Establish the psychometric properties of the test

CAT / IRT analysis

3. Choose the most appropriate CAT approach

CAT simulation

4. Deliver your test on the testing platform

5. Analyse the results

What you need:


15/25

Consider open source items: Avoid rediscovering the wheel!

IPIP.orgi.org

myPersonality.org/wiki

Crowd-source your items / translations e.g. Amazon Mechanical Turk

Wiki-source your items?

Generate your items automatically:

Prof. Heinz Hollings

Cambridge IQ items generator (coming soon!)

Build an item bank


16/25

R language for statistical analysis

Classical and IRT analyses (+ anything else you may

want) ltm, irttoys, psych, and loads of other psychometric

libraries

Powerful graphing capabilities

Flexible programming interface

Great wealth of analyses available within oneenvironment

Establish the psychometricproperties of the test


17/25

Issues:

Where to start?

How to select next item?

When to stop?

What is the expected accuracy and test length?

CAT simulation :

catR package

catIrt

MAT (Multidimensional Adaptive Testing, M3PL model)

Choose the most appropriate CATapproach


18/25

Use web-based technology:

Compatibility

Can be used locally

Plan ahead:

Choose the platform on the piloting stage

Use CAT/IRT enabled platform even if do you need it yet

Testing platform choice


19/25

It is open source

It relies on open-source software (e.g. R, MySQL)

Ease of use develop advanced tests without a softwareengineer

Flexible:

Robust R engine

Works with any CAT / IRT algorithm

Nicely integrates with other software e.g. Moodle, TAO, etc.

Concerto:an open source adaptive testing

platform


20/25

Design of Concerto

Concerto Platform

Database

MySQL

Statistical

Engine

R server

Item Editor

CKEditorQTI

Editor

Open-source elements Flexible R engine

Standard language inpsychometrics

Reliable / up-to-date methods Powerful MySQL database Modular design

Add / Replace modules ifnecessary


21/25

Internet

Secured LAN e.g. testing centre

Locally e.g. Tablet with local copy of the platform

Administration channels


22/25

Concerto: brief tour


23/25

Ror

GNU Octave

Analyse your results:


24/25

Avoid rediscovering the wheel Do your homework BEFORE you decide to buy/develop

software

Stay independent: Insist on using open-source software EVEN if you decide

to develop your own platform

Use IRT and CAT it is cheaper, more accurate, andgives you a competitive edge!

Insist on using R in statistical analyses

Help us to fully integrate TAO with Concerto Anyone has spare programmer or few euros?

TAKE-HOME MESSAGES


25/25

More about Concerto:www.psychometrics.cam.ac.uk

Michal Kosinski, [email protected]

Prof. John Rust, [email protected]


Michal Kosinski

[email protected]

Documents

Applying Open Source to Adaptive Testing