cerg.csse.monash.au/pedant

http://cerg.csse.monash.edu.au/pedant

Applying agent technology to evaluation tasks in e-learning

environments

Applying agent technology to evaluation tasks in e-learning

environments

Selby Markham, Jason Ceddia & Judy SheardCSSE, Monash

Colin Burvill & John WeirMechanical and Manufacturing Engineering, U M

Bruce Field Mechanical Engineering, Monash

Linda Stern & Leon SterlingCSSE, UM

The PEDANT project grew out of a desire/need to understand how current students deal with electronic materials.

Aims to investigate

The relationship between the way students use

on-line and interactive educational tools and the

quality of their learning experience.

Using automated, agent-oriented software tools

The educational tools

SiMLED

A simulator for exploring engineering principles.

Engineering

Algorithms in Action : AIA

An interactive, visual tool for exploring programming

algorithms

CSSE


MOMUS Tutor

A graphics-based tutorial system on mechanical

engineering principles

Engineering


Web Industrial Experience Resource : WIER

A Web-based resource to support students in the

Industrial Experience project

CSSE


What is a software agent?What is a software agent?

A bit like TRON

What is a software agent?What is a software agent?

Your computer Virus Checker

What is a software agent?What is a software agent?It has a purpose

It can be given intelligence

It is responsive

It can be designed to learn

It acts independently

What is a software agent?What is a software agent?It has a purpose

It is responsive

It can act independently

It has a purpose

To monitor software output

To collect appropriate output

To organise that output

It is responsive

The information can be analysed

Notifications can be given

It can act independently

When initiated it needs no support

It can be given intelligence

(Phase 2 of the project)

But does it have Ethics?

But why use agents?

Software tools create issues that are not easily addressed by self-report techniques or

observational techniques.

Have you tried to analyse the log data from a Web-based tool?

The extent of the data is mind-boggling

Have you tried to trace the informational path used by the learner?

How often have you been unable to trace your own path when working on the Web?

The development of agent technology is a productive direction for achieving both short-term and long-term goals

•Agent technology is technologically compatible with the teaching/learning tasks.

•Agent technology can provide data about what the learner is doing rather than what he/she remembers that he she was doing.

•Agent technology provides a tool with a highly pervasive ability to carry out functional formative assessment.

Conceptualising the basic processConceptualising the basic process

Educator

Learner

Software

Learning Objectives

Learner Motives

Evaluation

Learning Tasks

Learner Behaviour

Inferred Behaviour

Responses

Learner Outcomes

Measures

Educator

Learner

Software

Learning Objectives

Learner Motives

Evaluation

Learning Tasks

Learner Behaviour

Inferred Behaviour

Responses

Learner Outcomes

Measures

2003

Educator

Learner

Software

Learning Objectives

Learner Motives

Evaluation

Learning Tasks

Learner Behaviour

Inferred Behaviour

Responses

Learner Outcomes

Measures

2003

We will have achieved the primary aim of developing:

•A rationale for defining pedagogical elements•Prototypical agents that can monitor the software•Have analysis models that can apply to evaluation tasks

Educator

Learner

Software

Learning Objectives

Learner Motives

Evaluation

Learning Tasks

Learner Behaviour

Inferred Behaviour

Responses

Learner Outcomes

Measures

2003

The evaluation component:

•The evaluation of the functional usability will have begun

•The format for evaluating the processes being used by students - a broader view of formative assessment

Educator

Learner

Software

Learning Objectives

Learner Motives

Evaluation

Learning Tasks

Learner Behaviour

Inferred Behaviour

Responses

Learner Outcomes

Measures

2003

20042004

20042004

20042004

20042004

There are many more tasks to be done to fill in the underlying matrix

The agent in evaluation

Provides are means of monitoring process and relating process to educational goals

This creates what can be called a process evaluation – as defined in the broader evaluation literature.

This is similar to formative evaluation but more as originally conceptualised by Scriven.

Objectives Application Outcomes

Specified task/goal

Operationalised in software

Learner responses

Pedagogical Agent

Data Log Raw data generated

by Application

Information Log Conversion of Data Log to usable information by

AgentKnowledge Log

Intelligent conversion of Information Log by

Agent

Agent action definition

The current process model for the project



Specified task/goal


Learner responses


Specified task/goal


Learner responses

Pedagogical Agent


Specified task/goal


Learner responses

Pedagogical Agent


by Application


Specified task/goal


Learner responses

Pedagogical Agent


by Application



Specified task/goal


Learner responses

Pedagogical Agent


by Application


Agent



Specified task/goal


Learner responses

Pedagogical Agent


by Application


AgentKnowledge Log

Intelligent conversion of Information Log by

Agent


e.g.

Action (User, Type, Time, [parameter list] )


action (user code, page, time [URLdata])


action (user code, page, time [URLdata])

download (403, file_manager, 994867577, [doc_rep: 0 Requirements-Model.doc])

The practical meaning could be

Learning tasks that the software enables:

exploring a particular area accessing a particular resource

Learner behaviours:

passive lostwandering

directed interaction linear, orderly interaction exploration (at different levels of depth)

FIN

Documents

cerg.csse.monash.au/pedant