Quality estimation of adaptive tutoring systems

Volgograd State Technical UniversityCAD departmentPavel Vorobkalov

Faculty advisor Shabalina O.A.

Volgograd 2007

2

The aim and tasks

The aim:Quality management of learning process using adaptive tutoring systems

Tasks:1. To analyze problems of quality of adaptive tutoring systems2. To develop method of quality estimation of adaptive tutoring systems3. To design model of adaptive learning process4. To develop criteria of quality estimation of adaptive learning5. To implement computer-aided system of quality estimation of adaptive

tutoring systems

3

Quality management system of adaptive tutoring systems

Developer

Development Framework

Quality estimation system

Users

Adaptive tutoring systems

Users’Knowledge

User requirments

Users’ Knowledge

Criteria Values

4

Analysis of approaches and methods of quality estimation of adaptive tutoring systems

‘as a whole’ approach Questionnaires Return of investments as a result of

education Common quality criteria

(ex., according to ISO-9127) Standard Conformance

‘layered’ approach Estimation of interface layer Estimation of adaptation models Estimation of adaptation decisions Estimation of adaptation techniques Estimation of leaning content

To interpret results for further development

To generalize results

To use both approaches

5

Model of human-computer interaction during adaptive learning process in layered approach

1 layer 2 layer

Adaptation mechanism

Building adaptation

models

System adaptation

Adaptive applicationData

collection mechanism

Hi-level conclusions:· user is confused;· user couldn’t complete the task.

Low-level user information:· data from input devices;· information about task completion;· quiz answers.

Adaptation decisions:· popup help window;· changing hyperspace.

Adaptation process

Representation as a «white box» needed

?

?

6

Method of estimation and prediction of quality of adaptive tutoring system

1. Collecting data about learning process using adaptive tutoring system

2. Building model of learning process3. Analysis of built model

1. Computation of criteria of quality estimation of learning process2. Expert estimation of learning process (detecting issues)

4. Learning course modification1. Modification of concepts those are problematic2. Testing changed concepts by student groups3. Calculating new parameters of learning process model4. Computation of predicted results of changed learning process (quality

prediction)

7

- 1 student strategy

- 2 student strategy

Table of symbols:

Ci- unitConsists of learning content and a quiz

Cj- sectionSuperset of some units

C1

Domain area structure

C2 C3 C4 C5

C6

C7

C8

C9

C10

C11

C16

C12

C13 C14 C15 C17

C18 C19 C20

- learning dependency

- union

Adaptive learning process

Choosing learning strategy

Generation of page

Learning course development

Observing student

Student

Developer

8

Модели процесса обучения

Model Major advantages and disadvantages

Psychological models, models based on cognition theory

+Well known–No formal method of building model

Grammar-based models (BNF, EBNF) +Advanced approach–Difficult to estimate

Models based on algebraic and differential equations

+Well formalized–High level of abstraction

Machine models (Mur, Mille machines) +Wide range of described tasks–Hard to interpret

Net models (Markov chains, Petri nets) +Easy to interpret–Big, hard to build

1 2

21

2

1

Concept 1 Concept 2

Concept 3

Concept 4

1 2

1 2

2

Quiz 1 Quiz 2

Quiz 3

1

1 1

2

22

(Identity, Kind) = (1, ”Learned”)

(Identity, Kind) = (2, ”Learned”)

Concept 5

9

1 2Concept 1 Concept 2

Concept 3

Concept 4

1 2

1 2

Quiz 1 Quiz 2

Quiz 3

1

1 1

2

22

(Identity, Kind) = (1, ”Current”)

(Identity, Kind) = (2, ”Learned”)

Concept 5

Learning process model based on Petri nets

Set of colors:

kcccC ,,, 21

,

where

ic

– color of a token, an arc;

k

– count of Petri net colors.

KindIdentityci ,

,

where

Identity

– color component that identifies a

student’s role;

Kind

– color component that defines kind of token.

Kindlk

,

where

l

– count of roles of students that collaborate

during learning process using adaptive tutoring

system;

Kind

– count of different kinds of token.

.

.

10

Quantitative parameters of learning process

To random values correspond to each transition:

firing time of transition (learning duration in seconds)

0, DZD

;

delta of knowledge level

1,0, KLKLRKL

.

Probability distribution of 2D random value is:

,

where – probability distribution of 2D

lognormal random value , , and , – mean and standard

deviation values of

X

and

Y

, accordingly, and – correlation factor between

X

and

Y

.

Lognormal distribution of 1D random value

X with mean and standard

deviation

1.0X

.

XXxf ,,

x

Concept 1

Concept 2

Test 1

1

1 1

(D, KL)

1

1

As a quantitative parameters learning duration D and knowledge level KL have been chosen

Knowledge level KL1

Knowledge level KL2=KL1+KL

11

Building the learning process model

p2 p3

p4

Built learning process model

p1

t1 t2

t4

p2 p3

p4

p1

t1 t2

t4

p3

p4

p1

t1

t4

Studying strategy 1Use frequency 0.6

Studying strategy 2Use frequency 0.4

The algorithm

Collecting data about student’s actions

Preprocessing the data

Building Petri net model using join operation

User knowledge level for a unitLearning duration for a unit

Learning process model

0

00

Probability 0,6

Probability 0,4

12

Quantitative parameters of Petri net inferring learning process

Learning durationComputer experience

Correlation between net parameters and user characteristics

C fan

C density

C flex

p2 p3

p4

p1

t1 t2

t3

t4

Quantitative parameters of Petri net:

P – number of placesT – number of transitionsR – number of routes

Difficulty

Density

Flexibility

P

TC fan

1

PP

TCdensity

P

RC flex

17

,16

7fa

nC

23,0166

7

densityC

5,06

3flexC

p5

t5

t6

p6

t7

1

13

Criterion of adaptive decisions balance of tutoring system

Probability h11

Probability h21

Computing of balance criterion B

where i – number of adaptation strategy;

Hi – relative frequency of the strategy;

N – number of possible strategies.

where hij – probability of variant on the route corresponding to strategy i

,logi

iNi HHB

p2 p3

p4

p1

t1 t2

t4

0

ijj

i hH ,

Адаптацияне сбалансированаАдаптацияне сбалансирована

14

Expert estimation of learning process

C2

C3

C4

C1

t1 t2

t3

C5

t4

t5

C6

t6

KL(t1)D(t1)

KL(t2)D(t2)

KL(t3)D(t3)

1

KL(t6)D(t6)

KL(t5)D(t5)

KL(t4)D(t4)

Ra

tio

n i

s a

sc

en

din

g

KL(t3)D(t3)

=0,54311

KL(t1)D(t1)

= 0,25155

KL(t2)D(t2)

=0,71271

KL(t4)D(t4)

=0,80275

KL(t6)D(t6)

=0,82273

KL(t5)D(t5)

= 0,85255

Changes are required

Learning results - ?

KL(t1) D(t1),t1:

KL(t3)D(t3),t3:

15

Экран «Редактирование модели»

16

Экран «Запуск моделирования»

17

Экран «Результаты моделирования»

18

Conclusions

1. Problems of quality of adaptive tutoring systems and approaches to their solution have been analyzed

2. Method of quality estimation of adaptive tutoring systems has been developed

3. Model of adaptive learning process has been designed4. Criteria of quality estimation of adaptive learning have been created5. The prototype of computer-aided system of quality estimation of

adaptive learning has been implemented