Advancements in Large-scale Assessment Heiko Rölke & Krisztina Toth DIPF - Deutsches Institut für Internationale Pädagogische Forschung Frankfurt am Main

Advancements in Large-scale Assessment

Heiko Rölke & Krisztina TothDIPF - Deutsches Institut für Internationale

Pädagogische ForschungFrankfurt am Main

TAO Days | 10.09.2012Seite

1

Overview

• Trends in LSA• Closer Look at Trends• Example: Making Use of Data• Challenges in LSA

TAO Days| 10.09.2012 Seite 2

Trends

Main trend: CBA

Sub-trends/effects of CBA:•CAT• towards formative assessment

•Complex items• Simulation, interconnection

•Interweaving• different tests, questionnaires, etc.

•Big data

TAO Days | 10.09.2012 Seite 3

Trend: CBA

Computer-based Assessment has to serve a purpose•Costs?•Time!•Validity!


CAT

• Also not practiced for its own sake• Not as important in pure summative assessment• But: • Strict time limits (e.g. PIAAC)• Growing demand for formative aspects in summative

assessment


Complex Items

• Closer at reality• SimulationMost important:• Ability to assess new domains


Examples of Complex Items Simulation of an “Email / Web Scenario”

The test person receives an email and should bookcinema tickets online.

Dynamic Model (MicroDYN)The test person should explore and master a dynamicsystem with input (exogenous) variables influencingoutput (endogenous) variables.

Automaton with Finite State Machine (MicroFIN)The test person should interact with a mobile phone and set time tosummertime.

Interweaving of Tests and Questionnaires• Framing• test only if preconditions are fulfilled

• Double-check • findings of questionnaire


Big Data

• Log data, not only results• Find out what is going on• Make use of data• E.g. partial scoring

• -> Examples from PIAAC PS-TRE


Simulation-based assessment

• Modern assessments:• real life situations: e.g. web environments• complexity of instrument – various ways

• Monitoring students‘ actions - log files• Analysis of test-taking paths is a relatively new field in education

(reason: printed vs. computer-based data collection)• Process data requires to consider and evaluate methods

Berlin, 10.9.2012 | H. Rölke & K. Tóth | TAO Days 2012 | Advancements in Large-Scale Assessments

10

Aim

• to investigate how log data can be integrated into the process of educational assessment and evaluation to support researchers and practitioners in making use of the data assembled in computer-based test delivery


11

Aim

• to investigate how log data can be integrated into the process of educational assessment and evaluation to support researchers and practitioners in making use of the data assembled in computer-based test delivery


12

Hypertext Item

13

14

15

829709 15 {"sender":"15","type":"loading","data":""}833463 15 {"sender":"15","type":"loaded","data":""}848153 15 {"sender":"15","type":"user_interaction","":"<?xml version=\"1.0\">

\u000d\u000a<cbaloggingmodel:EmbeddedLinkLogEntryxmlns:cbaloggingmodel=\"http://www.softcon.de/cba/cbaloggingmodel\" id=\„-b576759f1:-7fed\" sourcePageId=\"Item15_linklist\" targetPageId=\"Item15_website1\" textFieldId=\"cbaTextField_71"/>\u000d\u000a"}849094 15 {"sender":"15","type":"variable_change","data":{"name":

"snapshot_url„,"value":"http://localhost:8101/cba-runtime/itemjsessionid=LB1?custom_servicehandler=downloadService&file=C:\\...snapshot5537437125227002523.xml"}}

855852 15 {"sender":"15","type":"user_interaction","data":"<?xmlversion=\"1.0\“encoding=\"UTF8\"?>\u000d\u000a<cbaloggingmodel:ButtonLogEntryxmlns:cbaloggingmodel=\"http://www.soft

con.de/cba/cbaloggingmodel\id=\"cbaBackButton_14_13454937565947\"/>\u000d\u000a„}":"15","type":"unloaded","data":""}

Example log

Promising methods

• Statistics and visualisation• Clustering• Classification


16

Process Measures


17

1. Number of page visits2. Number of different page visits3. Visit of relevant page4. Time spent on the relevant page5. Ratio of time spent on the relevant page6. Ratio of time spent on the opening screen7. Completion time

Statistics

Process measures MeanStandard Deviation

Number of page visits 5.18 3.13Number of different page visits 2.38 1.38Time spent on the relevant page 10.87 9.77Ratio of time spent on the relevant page .14 .11Ratio of time spent on the opening screen .64 .18Completion time 67.74 27.49


18

Distribution of different page visits


19

Distribution of different page visits


20

Visualisation

21Berlin, 10.9.2012 | H. Rölke & K. Tóth | TAO Days 2012 | Advancements in Large-Scale Assessments

Visualisation

22Berlin, 10.9.2012 | H. Rölke & K. Tóth | TAO Days 2012 | Advancements in Large-Scale Assessments

Cluster Analysis

Features Cluster 1 Cluster 2 Cluster 3 Cluster 4

Nr. of page visits

Nr. of different page visits

Completion time

Relevant page visited (Y/N)

Ratio of time on relevant page

Ratio of time on starting page

Distribution of sequences (%) 23% 16% 34% 27%

Ratio of correct responses

23


Cluster Analysis


Nr. of page visits

Nr. of different page visits

Completion time

Relevant page visited (Y/N)

Ratio of time on relevant page

Ratio of time on starting page


Ratio of correct responses 25.0% 1.7% 92.6% 84.4%

24


Cluster Analysis


Nr. of page visits .72 5.41

Nr. of different page visits .36 2.54

Completion time 28.38 62.08

Relevant page visited (Y/N) No No

Ratio of time on relevant page 0 0

Ratio of time on starting page .93 .60

Distribution of sequences (%) 23% 16%

Ratio of correct responses 25.0% 1.7%

25


Cluster Analysis


Nr. of page visits 4.73 9.57

Nr. of different page visits 2.29 4.53


Relevant page visited (Y/N) Yes Yes

Ratio of time on relevant page .20 .17




26


Cluster Analysis


Nr. of page visits 5.41 4.73

Nr. of different page visits 2.54 2.29


Relevant page visited (Y/N) No Yes

Ratio of time on relevant page 0 .20




27


Cluster Analysis


Nr. of page visits .72 5.41 4.73 9.57

Nr. of different page visits .36 2.54 2.29 4.53

Completion time 28.38 62.08 59.60 91.60

Relevant page visited (Y/N) No No Yes Yes

Ratio of time on relevant page 0 0 .20 .17

Ratio of time on starting page .93 .60 .62 .45



28


Summary and Conclusions


29

Summary and Conclusions


Nr. of page visits .72 5.41 4.73 9.57

Nr. of different page visits .36 2.54 2.29 4.53

Completion time 28.38 62.08 59.60 91.60

Relevant page visited (Y/N) No No Yes Yes

Ratio of time on relevant page 0 0 .20 .17

Ratio of time on starting page .93 .60 .62 .45



30


Future work

• Pilot study – sample size• Validation

• Other types of items require new process measures


31

32

Future work

• Pilot study – sample size• Validation

• Other types of items require new process measures• We have a lot of work to do:• Software developer• Test developer• Psychometricans


33

(Selected) Challenges

• Authoring and Management• Delivery• Re-use and exchange











Documents

Advancements in Large-scale Assessment Heiko Rölke & Krisztina Toth DIPF - Deutsches Institut für Internationale Pädagogische Forschung Frankfurt am Main