Center for Advanced Research in Technology for Education, USC / ISI The Tactical Language Training Project W. Lewis Johnson, Sunhee Choi, Stacy Marsella,

Center for Advanced Research in Technology for Education, USC / ISI

The Tactical Language Training The Tactical Language Training ProjectProject

W. Lewis Johnson, Sunhee Choi, Stacy W. Lewis Johnson, Sunhee Choi, Stacy Marsella, Nicolaus Mote, Shrikanth Narayanan, Marsella, Nicolaus Mote, Shrikanth Narayanan,

Hannes Hannes Vilhjálmsson, Shumin WuVilhjálmsson, Shumin Wu

University of Southern CaliforniaUniversity of Southern CaliforniaMicro Analysis & DesignMicro Analysis & Design

US Military AcademyUS Military AcademyUCLA CRESSTUCLA CRESST


Major Research ThemesMajor Research Themes

Study role of simulation games and intelligent Study role of simulation games and intelligent tutoring in promoting communicative skilltutoring in promoting communicative skill

Simulation motivates, builds fluencySimulation motivates, builds fluency Intelligent scaffolding built into simulationIntelligent scaffolding built into simulation

Tutoring provides form feedbackTutoring provides form feedback

Tailor speech recognition to learner languageTailor speech recognition to learner language Emulate free dialog within narrative structureEmulate free dialog within narrative structure Manage learner motivationManage learner motivation

Strong task orientationStrong task orientation Promote early communicative successPromote early communicative success Employ motivational tutorial tacticsEmploy motivational tutorial tactics Adjust level of challenge to maximize engagementAdjust level of challenge to maximize engagement


ScenarioScenario

Your special ops unit is deployed to a town to Your special ops unit is deployed to a town to assist with postwar recoveryassist with postwar recovery

The objectives:The objectives: Establish rapport with locals, describe missionEstablish rapport with locals, describe mission Ask for directions to local leaderAsk for directions to local leader Meet leader and discuss recovery needsMeet leader and discuss recovery needs Initiate mission (organize reconstruction)Initiate mission (organize reconstruction) Overcome barriers to mission (factions, sabotage, etc.)Overcome barriers to mission (factions, sabotage, etc.)

Impact of communication failure:Impact of communication failure: Failure of missionFailure of mission Hostile encounters with localsHostile encounters with locals


DemoDemo


Overall ArchitectureOverall Architecture

Language Model

Mission Skill Builder

Pedagogical Agent

LearnerModel

Mission Practice Environment

MEDINA Authoring Tool

CurriculumMaterial


Interaction with Mission Skill Interaction with Mission Skill BuilderBuilder

Curriculum

ErrorModel SpeechRecognizer

Speech Data withDysfluencies

LearnerModelLearner Progress,

Feedback

Mission SkillBuilder Curriculum

Authoring Tool(Medina)

Mission Skill Builder Start, Stop

Recognized Text

Error Type

RecognizedText

SpeechData

NLP Parser(Contex)

Feedback

CurriculumData

ResultsVocabulary Check

LanguageData

Parse Trees,Vocabulary Check

Recognized Text,Learner Activities

Speech Data withDysfluencies

Learner Level

Instructor Level


Authoring IssuesAuthoring Issues

Story-oriented authoring poses Story-oriented authoring poses particular challenges:particular challenges:

Authoring is a multidisciplinary team activityAuthoring is a multidisciplinary team activity A mixture of top-down and bottom-up designA mixture of top-down and bottom-up design

Curriculum content is introduced incrementally and Curriculum content is introduced incrementally and opportunisticallyopportunistically

Needed:Needed: Tools to facilitate incremental construction of Tools to facilitate incremental construction of

materialsmaterials Tools to track curriculum coverageTools to track curriculum coverage Tools to check for validity and completenessTools to check for validity and completeness

Scene script - HTML Scene script - XML


Speech ProcessingSpeech Processing

Dual goals of robust recognition and Dual goals of robust recognition and spoken language skill evaluationspoken language skill evaluation

Recognition subsystem goals:Recognition subsystem goals: Robustness to non native speechRobustness to non native speech Lightweight operationLightweight operation

Pronunciation evaluation system goals:Pronunciation evaluation system goals: Multilevel evaluation ranging from utterance level Multilevel evaluation ranging from utterance level

prosody to segmental cuesprosody to segmental cues Feedback requirements differ for learner and game AIFeedback requirements differ for learner and game AI


Speech Recognition- Speech Recognition- System DesignSystem Design

HTK based system HTK based system Bootstrapped from modern standard Arabic and Bootstrapped from modern standard Arabic and

adapted to Levantine speech and lexiconadapted to Levantine speech and lexicon

Recognition networks loaded dynamically Recognition networks loaded dynamically Networks optimized for different stages in schoolhouse Networks optimized for different stages in schoolhouse

and mission training and mission training

Mission training modeMission training mode Large vocabulary (LVCSR) like operationLarge vocabulary (LVCSR) like operation

Schoolhouse mode for skill acquisitionSchoolhouse mode for skill acquisition Limited vocabulary with pronunciation variants and Limited vocabulary with pronunciation variants and

hypothesis rejectionhypothesis rejection


Natural Language ProcessingNatural Language Processing

Functions of NLP:Functions of NLP: Validate authored learning materialsValidate authored learning materials Link dialog examples to authored explanations, Link dialog examples to authored explanations,

translationstranslations Predict & recognize common learner errorsPredict & recognize common learner errors


Managing Pedagogical Drama Managing Pedagogical Drama

Want an engaging pedagogical drama Want an engaging pedagogical drama Unfold based on trainee's language & cultural skills Unfold based on trainee's language & cultural skills Give trainee a sense of control and responsibility Give trainee a sense of control and responsibility Provides pedagogically appropriate consequencesProvides pedagogically appropriate consequences Respond robustly to learner communicative actionsRespond robustly to learner communicative actions Avoid computationally expensive approachesAvoid computationally expensive approaches


MPE ArchitectureMPE Architecture

MissionEngine

UnrealWorld

InputManager

Learner Interface Event

SpeechRecognizer

ActorAgent CharacterAgent

SocialPuppet SocialPuppet

DirectorAgent

ActionScheduler

Script

Action

SocialPuppet UnrealPuppet

LearnerModel

Learner Skill

Learner Speech Learner Game Interaction

Multimodal Learner Action

World Event

Curriculum Material


Technical Details Technical Details

Director uses POMDPs to model trainee and Director uses POMDPs to model trainee and characters characters

Student model informs trainee capabilities Student model informs trainee capabilities

POMDPs used to predict possible story POMDPs used to predict possible story paths paths

Fit parameters of character POMDPs to Fit parameters of character POMDPs to increase likelihood of desirable story path increase likelihood of desirable story path

Example: Adjust importance of being hospitableExample: Adjust importance of being hospitable

Fitting sensitive to cultural & situational Fitting sensitive to cultural & situational factors factors

Example: hospitality for strangers, support for US Example: hospitality for strangers, support for US involvement important goals to adjust involvement important goals to adjust


Evaluation Sessions, May 2004Evaluation Sessions, May 2004

7 learners7 learners 6 some prior 6 some prior

knowledge of TLTSknowledge of TLTS All had prior L2 All had prior L2

language language experienceexperience

All experienced All experienced gamersgamers

Practice w/ SB ~ 1 Practice w/ SB ~ 1 hourhour

MPE sessionMPE session Post session Post session

interviewinterview


General commentsGeneral comments

TLTS viewed as TLTS viewed as potentially potentially muchmuch better than class better than class instructioninstruction

learners readily learners readily identify best features identify best features (one-on-one, ME), can (one-on-one, ME), can anticipate benefitsanticipate benefits

no major interface or no major interface or navigation concernsnavigation concerns

many problems identified many problems identified are already being solvedare already being solved MSBMSB

speech recognitionspeech recognition learner feedbacklearner feedback

MPEMPE avoidanceavoidance


Skill Builder ConclusionsSkill Builder Conclusions

Pronunciation accuracy threshold too Pronunciation accuracy threshold too high for beginnershigh for beginners

Since has been loweredSince has been lowered

Tutor feedback sometimes repetitiveTutor feedback sometimes repetitive Feedback selection algorithm adjustedFeedback selection algorithm adjusted

Lack of transfer from MSB to MPELack of transfer from MSB to MPE New exercises added to MSB to emulate MPE New exercises added to MSB to emulate MPE

dialogdialog


Practice Environment Practice Environment ConclusionsConclusions

Learners reluctant to enter MPELearners reluctant to enter MPE Learners now explicitly encouraged at multiple Learners now explicitly encouraged at multiple

pointspoints

Learners not sure what to doLearners not sure what to do Provide detailed mission briefingsProvide detailed mission briefings Aide helps provide guidanceAide helps provide guidance


Design for Next EvaluationDesign for Next Evaluation

~25 subjects, non-Arabic speakers~25 subjects, non-Arabic speakers Four experimental conditions:Four experimental conditions:

Complete systemComplete system MPE onlyMPE only MSB onlyMSB only MSB without pronunciation feedbackMSB without pronunciation feedback


Backup SlidesBackup Slides


Role of Gesture and CultureRole of Gesture and Culture

Animate gestures in virtual locals:Animate gestures in virtual locals: To promote learner understandingTo promote learner understanding To expose learner to unfamiliar gesturesTo expose learner to unfamiliar gestures

Allow learner to select gestures:Allow learner to select gestures: To learn appropriate gestures for different social To learn appropriate gestures for different social

situationssituations E.g., to build rapport, manage crowdsE.g., to build rapport, manage crowds

Culture training involves:Culture training involves: Training in face threats, face threat mitigation Training in face threats, face threat mitigation

strategiesstrategies Expectations (etiquette) in different social Expectations (etiquette) in different social

situationssituations

Documents

Center for Advanced Research in Technology for Education, USC / ISI The Tactical Language Training Project W. Lewis Johnson, Sunhee Choi, Stacy Marsella,