Upload
ronat
View
42
Download
0
Tags:
Embed Size (px)
DESCRIPTION
Generalizability of Goal Recognition Models in Narrative-Centered Learning Environments. Alok Baikadi Jonathan Rowe, Bradford Mott James Lester. North Carolina State University. Goal Recognition in Narrative-Centered Learning Environments. - PowerPoint PPT Presentation
Citation preview
1
Generalizability of Goal Recognition Models in
Narrative-Centered Learning Environments
Alok Baikadi Jonathan Rowe, Bradford Mott James Lester
North Carolina State University
2
Goal Recognition in Narrative-Centered Learning Environments
Task: Identify the specific objective that the player is attempting to achieve
Goal recognition models enable the following:• Preemptively augmenting
narrative experiences• Assessing problem solving in
narrative-centered learning environments
• Iteratively refining learning environment
3
Generalization of Goal Recognition
Goal Recognition is typically very domain dependent• Plan libraries• Many domain-independent techniques are only
evaluated on one domain Research Question: Can a domain-specific goal
recognition model be applied in a principled way to a new domain and achieve similar results?
4
Related Work
Goal recognition is a restricted form of plan recognition (Carberry 2001; Kautz & Allen, 1986; Singla & Mooney, 2011)
Investigated widely in numerous domains (Blaylock & Allen, 2003; Charniak & Goldman, 1993; Lesh, Rich & Sidner, 1999)
IO-HMM approach for recognizing high-level goals in simple action-adventure game (Gold, 2010)
PHATT-based approach for behavior recognition in real-time strategy game (Kabanza, Bellefeuille & Bisson, 2010)
N-gram and Bayesian network approaches for goal recognition to support dynamic narrative planning (Mott, Lee & Lester, 2006)
MLN-based approaches (Singla and Mooney, 2011 ; Ha et al., 2011 ; Sadilek and Kautz, 2012)
5
Outline
Goal Recognition Approach Goal Recognition Corpora Evaluation & Discussion Conclusions and Future Work
6
Markov Logic Networks (MLNs)
Statistical relational learning• Combines first-order relational reasoning with statistical learning• Input: A set of first-order predicate calculus formulae, along with
weights• Formulae can be expanded into a Markov Random Field for learning and
inference The joint probability distribution is defined as:
Toolkit: Markov TheBeast (Riedel, 2008)
7
Representation
Predicate Interpretationaction(t, a) Action a happens at time targument(t, a) Argument a observed at time tlocation(t, l) Player is at location l at time tstate(t, s) The narrative is in state s at time tgoal(t, g) Player is pursuing goal g at time t
8
Context in Goal Recognition
Actions are not always independent Traditional goal recognition formulation allows
for all previous observations• Can lead to sparsity issues
Solution: Look for key events in the history that provide insight to the player’s context
Use the structure of the narrative to provide the context
9
Discovery Events
Task progress is represented by a sequence of discovery events
Partial Answers to Central Questions are clues towards the solution
Provides a context for goal recognition: What has the user discovered?
10
Discovery Event Formulae
Milestone formulae recognize which discovery events have already occurred
Uses a cardinality constraint to capture existence
11
Outline
Goal Recognition Approach Goal Recognition Corpora Evaluation & Discussion Conclusions and Future Work
12
CRYSTAL ISLAND: OUTBREAK
8th grade microbiology
Valve Software’s Source engine
Science mystery Goal: Identify
source and treatment of outbreak
13
CRYSTAL ISLAND: Introduction
1. Student plays the role of a new visitor to the island.
2. Student discovers that several team members have fallen sick.
14
CRYSTAL ISLAND: Gathering Information
3. Student gathers clues from sick team members.
4. Student asks the camp’s pathogen experts about microbiology concepts.
15
CRYSTAL ISLAND: Gathering Information
5. Student views microbiology-themed posters.
6. Student reads books about different types of pathogens.
16
CRYSTAL ISLAND: Hypothesis Testing
7. Student conducts tests using laboratory equipment.
8. Student interacts with the lab technician to view microscopic images of pathogens.
17
CRYSTAL ISLAND: Reporting Findings
9. Student presents findings and recommended treatment to camp nurse.
18
Corpus Collection
Eighth grade class from public middle school
153 participants
No prior experience with CRYSTAL ISLAND
Played game for 1 hour, or until they were finished
7 goals available to students (Ha et al., 2011)
19
CRYSTAL ISLAND: UNCHARTED DISCOVERY Upper Elementary Science
Subject 5th grade science Standards aligned
Content Landforms Maps, models &
navigation
Story Adventurous adolescent Shipwrecked crew Complete quests to
explore island
20
CRYSTAL ISLAND: UNCHARTED DISCOVERY Video
Video
21
Corpus Collection
Onsite at 8 schools 831 fifth grade students 62% Caucasian, 14% African
American, 8% Asian, 16% Other
Teacher-driven implementation in classrooms
6 one hour sessions over 4 weeks
12 goals available during the first 2 weeks
22
Goal Extraction Procedure
Goal-achieving actions were identified Actions between previous goal and
current goal were labeled with current goal
Goal-achieving actions were removed
23
Outline
Goal Recognition Approach Goal Recognition Corpora Evaluation & Discussion Conclusions and Future Work
24
Empirical Evaluation
State of the Art Baseline:• Factored model (Ha et al., 2011)• Uses MLNs to relate the current time step to the previous time step
Each model was evaluated using 10-fold student-level cross-validation
Each model was evaluated according to three metrics:• Accuracy: Measured as F1 score• Convergence rate: Percent of sequences which eventually predicted the
correct goal• Convergence point: In sequences that converged, the percent of actions
that had to be observed before a consistent prediction was made
25
Baseline Model
26
CRYSTAL ISLAND: OUTBREAKDiscovery Events Model
27
CRYSTAL ISLAND: UNCHARTED DISCOVERYDiscovery Events
28
EXPERIMENTAL RESULTS
Model F1 Convergence Rate
Convergence Point
Baseline 0.488 30.906 50.865Discovery Events 0.546 50.056 35.862
Model F1 Convergence Rate
Convergence Point
Baseline 0.226 11.915 87.786
Discovery Events 0.244 29.973 79.350
Crystal Island: Outbreak
Crystal Island: Uncharted Discovery
29
Outline
Goal Recognition Approach Goal Recognition Corpora Evaluation & Discussion Conclusions and Future Work
30
Conclusions
Goal recognition models show considerable promise for enhancing the effectiveness of narrative-centered learning environments
Encoding narrative discovery events in Markov Logic is a natural approach for representing context for student actions in goal recognition
Experimental findings from two narrative-centered learning environments suggest that narrative discovery events enhance the accuracy and convergence of state-of-the-art MLN-based goal recognition models.
31
Future Work
Investigate combinations of discovery events• Some of the milestones may have provided more information
than others• Use automated feature selection
Integrate goal recognition into a runtime environment• Can establish intuition for how accurate a model is necessary
Elicit feedback from player• Assumes goals achieved are intended• May cause some bias
32
Collaborators
Research StaffEleni LobeneRob Taylor
PostdocEunyoung Ha
Digital Art StaffKirby CulbertsonSarah HeglerKaroon McDowell
Graduate StudentsJulius Goth Wookhee Min
Joe Grafsgaard Chris Mitchell
Eunyoung Ha Jennifer SabourinSeung Lee Andy Smith
Sam Leeman-Munk
Undergraduate StudentStephen Cossa
Affiliated FacultyCarol Brown (East Carolina University)Roger Conner (East Carolina University)Patrick FitzGerald (Art + Design)Elizabeth Hodge (East Carolina University)James Minogue (Elementary Education)John Nietfeld (Educational Psychology)Marc Russo (Art + Design)Hiller Spires (Curriculum & Instruction)Eric Wiebe (STEM Education)
Affiliated Post-Docs and Graduate Students (Art, Education, Psychology)Megan Hardy (Human Factors)Kristin Hoffman (Educational Psychology)Angela Meluso (Curriculum & Instruction)Lucy Shores (Educational Psychology)Sinky Zheng (Curriculum & Instruction)
33
Acknowledgments
Support provided by the National Science Foundation under grant DRL-0822200. Additional support was provided by the Bill and Melinda Gates Foundation, the William and Flora Hewlett Foundation, EDUCAUSE, and the Social Sciences and Humanities Research Council of Canada.
34
35
Goal RecognitionExample
36
Goal Recognition Example
37
Goal Recognition Example
38
Goal Recognition Example
39
Goal Recognition Example
40
Goal Recognition Example
41
Goal Recognition Example
?
42
Goal Recognition Example
43
Goal Recognition Example
44
Goal Recognition Example
45
Goal Recognition Example
?
46
Goal Recognition Example
47
Goal Recognition Example
48
Goal Recognition Example
49
Goal Recognition Example
What is the player’s current goal?
50
Goal Recognition Example