Assessment outcomes from the TENCompetence project

Assessment in TENCompetence:Sharing Assessment Process Information

Yongwu Miao

Educational Technology Expertise CentreOpen University of The Netherlands

CETIS Assessment SIG meetingGlasgow, Sep. 26, 2007

Overview

• TENCompetence Project

• A Standard-based and Service-oriented Approach to Support Assessment Process

• Assessment Process Specification (APS)

• Assessment Process Modelling Tool

• Conclusions and Future Work

TENCompetence Project

TENCompetence is a 4-year (Dec. 2005 - Nov. 2009) EU-funded Integrated IST-TEL project and fifteen partners are involved.

Project Aims:

to build the European Network for Lifelong Competence Development

• How? By providing a technical and organizational Infrastructure

• Who for? For use by any individual, group or organization

• What for? To develop their competences

TENCompetence Project

Assessment in TENCompetence (WP6.B)

- Supporting assessment that is integrated with learning and instruction (which steps are to be carried out and by whom), e.g., formative & summative assessment

- Supporting new forms of assessment, which are process-based and are normally involved with multi-role/users, e.g., peer assessment, 360 degree feedback, and portfolio assessment

Objectives of WP6.B

Develop a formal assessment specification and supporting tools that combines new assessment types and the ones included in the IMS QTI.

A Standard-based and Service-oriented Approach to Support Assessment Process

A combined use of IMS LD, IMS QTI, and assessment-specific services to model assessment processes – Using IMS LD to model multi-role/user-involved

assessment processes – Using IMS QTI to model classical assessment

tasks and using assessment-specific services to model special assessment tasks

– Using IMS LD to model a learn-flow with complicated process controls and decision rules

– Using IMS QTI, IMS LD, and assessment-specific services to model data-flow

learning design (or assessment design)

unit of learning (or unit of assessment)

role activity environment property

component

method condition

expression action

play

act

role-part

learning support servicelearning objectstaff learner

internal change-property

notification

show/hide

assesseeassessor

assessment-specificservice

item

activity-description

resourceQTI_type

external

assessment

An extended IMS LD model

assessment_item outcome

item_body

assessment_test

interaction response

response_processing

A simplified IMS QTI model

associate

choice

. . . . . .

Architecture of Run-time Environment

Player Player Player

internet

Dispatcher/Service broker

IMS LD engine IMS QTI engine ePortfolio editor

Concept mappingLatent Semantic Analysis (LSA)

. .

. .A A AA

A

generic assessment service

special assessment services

UoL managementand enactment service

client client


An example of run-time environment

SLeD SLeD SLeD

internet

CopperCore Service Integration (CCSI)

CopperCore APIS

A A

generic assessment service

UoL managementand enactment service

client client


A peer assessment example

design assignment

writing article1

reviewing article2

improving article1

writing article2

reviewing article1

improving article2

final assessment2final assessment1

learner1 learner2tutor

article2

feedback1

improved a2

paper1

article1

feedback2

improved a1

paper2

tutor’s feedback1 tutor’s feedback2

assessment form1 assessment form2


A screenshot of execution of the peer assessment example


Assessment Process SpecificationProblem:

It is too difficult for practitioners to specify an assessment process model by using IMS LD, IMS QTI, and assessment-specific services.

Solution:

Developing an additional layer, assessment process specification (APS) used to describe assessment processes in an interoperable and abstract way. By using APS, every assessment practice can be represented in a design description (an abstract assessment plan), which captures underlying design ideas and principles. Thus, practitioners can share and exchange their assessment process models (assessment designs).

role activity

assessment design

artefact rule

certifier

designer

assessee

assessor

stage

rating

responding

instructing

editing portfolio

constructing item

performing

input

tool

output

assignment

portfolio

assessment test

assessment item

evidence

assessment result

expression action

portfolio editor

concept mapping

QTI editor

feedback

simulator

LSA

QTI player

giving feedback

react on feedback

preparation

evaluation

evi. collection

processing

teaching-learning

add/remove role member

trigger activity

alloca. activity to role

send email

change property

……

……

evi. provider

consultant

decision maker

…… ……

distribute artifacts

present information

……

Conceptual Model

Assessment process model

Programmed instruction

model

Peer assessment

model<…>

360 degree feedback

model

Domain-specific models

Assessment process

specification

Unit of assessment

mapping

IMS LD +

IMS QTI +

Assessment-specific services

Assessment Process Specification

Assessment Process Modeling Tool

Aims:• Enabling skilled users to define their own

assessment process models by providing basic modeling facilities

• Enabling practitioners to understand, reuse, and customize pre-defined assessment process models by providing design guidance

• Transforming assessment process models into executable models represented in IMS LD + IMS QTI + assessment-specific services

The tool is still in design and will be implemented in the next year. The design and use of the tool will be explained by using a mock-up of the UI


• Four kinds of stages in a peer assessment process• In each stage, people with a certain role perform activities• In each activity, there may be input and output artefacts

Design assessment Give feedbackDo assignment React to feedback

constructing assi./designing instruction

/designing criteria/setting time

viewing/rating feedback/requiring more

/improving

responding/performing

0,11 1 10,1

0,1

1

commenting/rating

/grading/elaborating

1preceding precedingpreceding

1

1, *

11

1, *

1

1, * 1, *

designer candidate reviewer candidate

assign-ment

1, *

1, *

1, *

1, *

1, *

1, *

1, *

1, *

0, *

1, * 1, *

1, *

assign-ment

outcome1, *

1, * 1, *

1, * feedback1, *

1, * 1, *

1, *

improvedassignment

outcome

1, *

0, *1, *

1, *

assessmentform

1, *

0, * 0, *

1, *

performing performing performing performing

producing used_in


• Guide to make design decisions by showing guidance and consequence• Make design decisions by selecting default or pre-defined options• Customize a process model (for skilled users)

define stages? define activities?define roles?

include design stage?

default default

additional feedbackcycles? how many?

for each stage,define sub-roles?

how many instancesfor each roles?

what are relations between candidates

and reviewers? will tutor be involved?

for each stage, define activities?

completion?

assign roles to activities?

define artifact-flow?

default

choose distribution patterns (1-1, 1-m, m-1,

m-m, m-n, rotate, mutual,reciprocal, random, ...)

define assignment, assessment form,calculation rules

Anonymous/confidential/

public

customize

default

completion?synchronizeactivities?

define services/tools?

File Edit View Tool Template Help

artifactrole service lanestage

Peer assessment

Here shows the guidance to make design decisions

State bar

user’s decisions

Specify role structure, artifacts, and services

Show and edit peer assessment model



Peer assessment

How many stages in the peer assessment process?

Please make a decision

user’s decisionsDefault (three stages)With design assessment stageWith additional feedback stageMore complicated setting



Peer assessment

How many roles will be involved?

Five stages have been defined

design assessment do assignment give feedback react to feedback give feedback 2

Default (two persons)With the involvement of tutorMore complicated setting


artifactrole service lanestagedesign assessment do assignment give feedback react to feedback

Peer assessment

give feedback 2

Would you like to define role structure?

reviewerR

designerR

candidateR

Default role setting and with the involvement of a tutor

learner1R

tutorR

learner2R

learner1R

learner2R

tutorR



Peer assessment

give feedback 2

Would you like to define a typical activity for each stage?

reviewerR

designerR

candidateR


learner1R

tutorR

learner2R

learner1R

learner2R

tutorR

Default settingMore complicated setting



Peer assessment

give feedback 2

Would you like to define a typical activity for each stage?

reviewerR

designerR

candidateR

Nine activities are defined and roles are assigned

learner1R

tutorR

learner2R

learner1R

learner2R

tutorR

learner1 learner1 learner1

learner 2 learner 2 learner 2

learner1

learner2

tutor



Peer assessment

give feedback 2

Which stages will the tutor be involved?

reviewerR

designerR

candidateR


learner1R

tutorR

learner2R

learner1R

learner2R

tutorR



learner1

learner2

Give feedback 1Give feedback 2More complicated setting

tutor



Peer assessment

give feedback 2

Which stages will the tutor be involved?

reviewerR

designerR

candidateR

The tutor is assigned to work in the stage of “give feedback 2”

learner1R

tutorR

learner2R

learner1R

learner2R

tutorR



tutor

tutor

tutor



Peer assessment

give feedback 2

Would you like to create artifacts ?

reviewerR

designerR

candidateR


learner1R

tutorR

learner2R

learner1R

learner2R

tutorR

Default settingMore complicated setting



tutor

tutor

tutor



Peer assessment

give feedback 2

Would you like to create typical artifacts ?

artifacts are created



tutor

tutor

tutor

feedback1.1A

feedback1.2A

outcome1A

outcome2A

assignment1A

assignment2A

improved outcome1A

improved outcome2A

feedback2.1A

feedback2.2A



Peer assessment

give feedback 2

Would you like to change the titles of artifacts?

artifacts are created



tutor

tutor

tutor

feedback1.1A

feedback1.2A

outcome1A

outcome2A

assignment1A

assignment2A

improved outcome1A

improved outcome2A

feedback2.1A

feedback2.2A

finished



Peer assessment

give feedback 2

Would you like to change the titles of artifacts?

The title of an artifact has been changed



tutor

tutor

tutor

feedback1A

feedback2A

article1A

article2A

paper1A

paper2A

improved article1A

improved article2A

tutor’s feedback1A


finished



Peer assessment

give feedback 2

Would you like to define artifacts?

An artifact has been created



tutor

tutor

tutor

feedback1A

feedback2A

article1A

article2A

paper1A

paper2A

improved article1A

improved article2A



finished

learner 2 learner 2

Review Form

How do you think about this article?

accept without modification

accept with minor/major modification

reject

Comments:

assessment form

QTI assessment editor

assessment form (QTI item)

A



Peer assessment

give feedback 2

Would you like to define artifacts?

An artifact has been defined



tutor

tutor

tutor

feedback1A

feedback2A

article1A

article2A

paper1A

paper2A

improved article1A

improved article2A



finished

assessment formA



Peer assessment

give feedback 2

How will the artifacts be distributed?




tutor

tutor

tutor

feedback1A

feedback2A

article1A

article2A

paper1A

paper2A

improved article1A

improved article2A



assessment formA

Reciprocally MutuallyRandomlyDefault setting



Peer assessment

give feedback 2


An artifacts flow has been defined



tutor

tutor

tutor

feedback1A

feedback2A

article1A

article2A

paper1A

paper2A

improved article1A

improved article2A



assessment formA



Peer assessment

give feedback 2


An artifact flow has been defined



tutor

tutor

tutor

feedback1A

feedback2A

article1A

article2A

paper1A

paper2A

improved article1A

improved article2A



assessment formA

Conclusions and Future Work

Conclusions:• IMS LD, IMS QTI, and assessment-specific

services can be used complementarily to support various assessment processes

• A higher level assessment process specification and corresponding supporting tools are expected to enable practitioners to share and exchange assessment process information

Future work:• Developing the assessment process modeling tool• Integrating QTI editor and other assessment services• Developing mapping algorithm to transform APS to

IMS LD + IMS QTI + assessment-specific services.

Thanks for your attention!

Questions?

Contact information:• Project website: www.tencompetence.org• For details of associate partnerships:

http://www.tencompetence.org/node/16 • Contact [email protected] for all other enquiries

http://www.tencompetence.org/

http://www.tencompetence.org/node/16