Slide 1

Understanding Users Interaction Behavior with an Intelligent Educational Game: Prime Climb

Alireza Davoodi, Samad Kardan, Cristina Conati{Davoodi, Skardan, Conati}@cs.ubc.caIntelligent User Interface Lab

Department of Computer ScienceUniversity of British ColumbiaVancouver, BC, Canada

OELEs @ AIED 2013

Understanding Users Interaction Behavior with an Intelligent Educational Game: Prime Climb1

Objective: Understanding how different groups of users interact with an intelligent educational gameMotivation: Determining users prior knowledge on the target skills without taking a pre-testOnline classification of users into groups with different domain knowledgeImproving the user model and consequently the adaptive intervention mechanismMethod: Applying Clustering and Association Rule Mining techniques to discover user interaction patterns specific to groups of students with significantly different prior knowledge.

INTRODUCTION

PRIME CLIMB GAME

An intelligent educational game for students in grades 5 and 6 to practice number factorization skillsHintMagnifying GlassA mountain of numbers

Short version===========================A player and his partner pair up numbers not sharing a common factor to climb a series of mountains

Long version===========================Prime Climb: An intelligent educational game for students in grades 5 and 6 to practice number factorization skillsA player and her partner pair up the numbers on the mountains which have no factor in commonThe player falls down in case of making a wrong moveThe player can use a tool, Magnifying Glass (MG) to know about the factor tree of a number on the mountain. (top right corner of the figure)A hint is given once the student model believes does know have mastered a skill to answer a question.Prime Climb contains 11 mountains of numbers and upper level mountain contain more difficult numbers.Prime Climb: Is equipped with an intelligent pedagogical agent:Maintains a probabilistic model of the students knowledge on number factorization skills. Leverages the model to provide adaptive Hints.

3

Prime Climb: An intelligent educational game for students in grades 5 and 6 to practice number factorization skillsA player and her partner pair up the numbers on the mountains which have no factor in commonThe player falls down in case of making a wrong moveThe player can use a tool, Magnifying Glass (MG) to know about the factor tree of a number on the mountain. (top right corner of the figure)A hint is given once the student model believes does know have mastered a skill to answer a question.Prime Climb contains 11 mountains of numbers and upper level mountain contain more difficult numbers.Prime Climb: Is equipped with an intelligent pedagogical agent:Maintains a probabilistic model of the students knowledge on number factorization skills. Leverages the model to provide adaptive Hints.

PRIME CLIMB

PRIME CLIMB GAME

Interaction with Prime ClimbMaking a movement from one number on the mountain to another number on the mountainUsing Magnifying Glass (MG): Clicking MG on a number to see the factor tree of a numberReading hints

Short version===========================-A Click node is added for each movement. -The structure of the Click node is as shown here.-The CPT of the Click node is shown here. (Give then just as one exampleLong version===========================Explain a bit about Models Parameter.Play the video and say1- For each movement the model gets updated. 1-1 A click node is added. A Click node is added as a child of three nodes 1) Player number 2)partner number 3)common factor 1-2 Evidence is set (correct move or wrong move) 1-3 Affected nodess beliefs get updated (Prior nodes) 1-4 Click node is deleted3- Let the video goes and pause it in the second level when the partner is on N15 and player moves to N28 and explain the CPT table.4- A Causal structure5

BEHAVIOR DISCOVERY in PRIME CLIMB

Behavior Discovery Framework in Prime ClimbData CollectionUser RepresentationClusteringRule Mining

behavior6

45 students played Prime Climb2 students finished less than 9 levels43 students finished at least the first 9 mountains (levels) of Prime ClimbInteraction data from the first 9 levels collected for behavior discoveryMovement-related interaction dataMG-related interaction data

BEHAVIOR DISCOVERY in PRIME CLIMBData Collection

FEATURE EXTRACTIONMovement Features[Sum/Mean/STD] of number of [correct/wrong] movements made by a student across mountains[Sum/Mean/STD] of time on [correct/wrong] movements made by a student across mountains[Mean/STD] of length of sequences of [correct/wrong] moves made by a student[Mean/STD] of time spent per sequence of [correct/wrong] moves made by a student

Magnifying Glass Features[Sum/Mean/STD] of MG usageMean number of [correct/wrong] movements per each MG usageSTD of number of [correct/wrong] movements per each MG usage

Mountain-Generic Features [i-j]: Features calculated based on interaction data across mountains i to jMountain-Specific Features [i]: Features calculated based on interaction data across mountain iMountain-Generic+Specific Features [i-j]: Combination of the above two features

Example: Mountain-Generic+Specific-Movement+MG(19): Movement and MG-related features collected across mountains 1 to 9 and for each mountain individually.

BEHAVIOR DISCOVERY in PRIME CLIMBFeatures Calculation / User RepresentationUser Representation: Each user is represented by a vector of features

Feature Selection is done before clustering.The optimal number of cluster is found.The lowest number suggested by C-index, Calinski and Harabasz and Silhouette [13] measures of clustering validityGA-KMeans clustering algorithm (developed in IUI lab@UBC) is used for clustering.

BEHAVIOR DISCOVERY in PRIME CLIMBClustering / Feature SelectionRule MiningHotspot algorithm of Association Rule Mining is used for extracting frequent pattern of interaction.

BEHAVIOR DISCOVERY in PRIME CLIMB

Whether the discovered clusters are significantly different from other with respect to clusters average prior knowledge.Measure for comparing the clusters:Each subject took a pre-test before starting the gameMaximum possible pre-test score: 15

Behavior discovery has been done on two types of features set:Full-Feature Set: Data from all 9 mountains is included:Mountain-Generic-Movement(19)To understand how students in different groups have interacted with Prime Climb across all 9 mountainsTruncated-Feature Set: Data from some mountains is included:Mountains-Generic+Specific-MG+Movements(1-2)Mountains-Generic+Specific MG+Movements(1-4)To understand if students can be differentiated early in the interactionUseful for building online classifiers

BEHAVIOR DISCOVERY in PRIME CLIMB

Features Selected: 18 out of original 30 featuresOptimal Number of Cluster: 2

BEHAVIOR DISCOVERY in PRIME CLIMBBehavior Discovery on Mountain-Generic-Movement(19)Cluster1 (LPK)Cluster2 (HPK)StatisticsMeasuresMeanSDMeanSDp-valueCohen-dPrior Knowledge11.33.4513.02.0.030.53

BEHAVIOR DISCOVERY in PRIME CLIMBBehavior Discovery on Mountain-Generic-Movement(19)Rules for Cluster 1[HPK]: (Size: 10/43 = 23.26%) Mean-Time-on-Movements(1-9) = Higher, [6/6=100%] Mean-Time-Spent-On-Correct-Movements-On-Mountains(1-9) = Higher, ([5/5=100%])Rules for Cluster 2[LPK]: (Size: 33/43 = 76.74%) Mean-Time-On-Movements(1-9) = Lower, [33/37=89.19%]STD-Time-On-Wrong-Correct-Moves(1-9) = Lower, [33/35=94.29%] Mean-Time-On-Consecutive-Wrong-Movements(1-9) = Lower, [31/35=88.57%]STD-Time-On-Movements(1-9) = Lower, [31/33=93.94%]STD-Time-On-Correct-Movements(1-9) = Lower, [31/33=93.94]

When interaction data from all 9 mountains was included: HPK group were more engaged in the game and spent more time on making movements.LPK group spent less time on making movements, indicating that they were less involved in the game.BEHAVIOR DISCOVERY in PRIME CLIMBBehavior Discovery on Mountain-Generic-Movement(19)

Features Selected: 25 out of original 51 featuresOptimal Number of Cluster: 2

BEHAVIOR DISCOVERY in PRIME CLIMBBehavior Discovery on Mountain-Generic+Specific-MG+Movement(12)Cluster1 (LPK)Cluster2 (HPK)StatisticsMeasuresMeanSDMeanSDp-valueCohen-dPrior Knowledge9.223.9312.452.660.021.08

BEHAVIOR DISCOVERY in PRIME CLIMBBehavior Discovery on Mountain-Generic+Specific-MG+Movement(12)Rules for Cluster 1[HPK] (Size: 33/42=78.57%) Mean-Time-On-Movements(1)=Lower, [30/31 =96.77%] Mean-Time-On-Movements(1-2) = Lower, [29/30 = 96.67%]Rules for Cluster 2[LPK] (Size: 9/42=21.43%) Mean-Time-Spent-On-Mountain(1-2) = Higher, [7/7=100%] Total-Time-On-Mountain(1) = Higher, [5/5=100%]

HPK group were less engaged in the game and spent more time on making movements.LPK group spent more time on making movements, indicating that they were less involved in the game.

Inconsistency

BEHAVIOR DISCOVERY in PRIME CLIMBBehavior Discovery on Mountain-Generic+Specific-MG+Movement(14)Rules for Cluster 1[HPK] (Size: 7/43=16.28%) Mean-Time-On-Movements(4) = Higher, [5/5 = 100%] Mean-Time-On-Correct-Movements(3) = Higher, [3/3 = 100%]Rules for Cluster 2[LPK] (Size: 36/43=83.72%) Mean-Time-On-Correct-Movements(1-4) = Lower, [35/35 = 100%] Mean-Time-On-Movements(1-4)=Lower, [34/34 = 100%]

HPK group were more engaged in the gameLPK group were less involved in the game.Similar patterns to those extracted from (1-9) mountains

SUMMARY

Movement and MG-related interaction features can be used to differentiate groups of student with significantly different prior knowledge.Behavior discovery using interaction data from all 9 mountains showed students with higher prior knowledge showed more engagement in the game.Behavior discovery using interaction data from 4 mountains and more showed students with higher prior knowledge showed more engagement in the game.This result can be used to develop an online classifier which classifies students to different groups early in the interaction.

FUTURE WORKIncluding other types of features and doing rule mining:Hints-related features (time spent on reading hints, )Eye-gaze related features (fixation on hints, )

Online Classifier to classify students early in their interactionLearning Clustered Student ModelClustered Student Model = Model for each cluster as opposed to a similar model for all studentsAnalysis of impact of Clustered Student Model on Prime Climbs adaptive scaffolding mechanism

THANK YOU!QUESTIONS?