Upload
isabella-jocelin-flowers
View
214
Download
0
Tags:
Embed Size (px)
Citation preview
Uncertainty, Action, Uncertainty, Action,
and Interactionand Interaction
Eric HorvitzEric HorvitzMicrosoft ResearchMicrosoft Research
May 2002May 2002
Toward Mixed-Initiative User InterfacesToward Mixed-Initiative User Interfaces
Designs that assume Designs that assume from the ground upfrom the ground up that that user may guide, collaborate with automated user may guide, collaborate with automated service to achieve desired resultsservice to achieve desired results
User
Automation
Principles of Mixed-Initiative InteractionPrinciples of Mixed-Initiative Interaction Endow system with ability to infer the likelihood Endow system with ability to infer the likelihood
of a user’s goals, intentionsof a user’s goals, intentions
Attempt to scope precision of action to match Attempt to scope precision of action to match goals and uncertainties goals and uncertainties
Determine the expected value of action given Determine the expected value of action given costs and benefits of actioncosts and benefits of action
Consider status of a user’s attention in timing of Consider status of a user’s attention in timing of actionaction
Allow for dialog at appropriate times to resolve Allow for dialog at appropriate times to resolve key ambiguitieskey ambiguities
Provide efficient means for agent–user Provide efficient means for agent–user collaboration to refine guessescollaboration to refine guesses
Allow efficient Allow efficient directdirect invocation and invocation and terminationtermination
Seek innovative designs that maximize benefit Seek innovative designs that maximize benefit of service, minimize the cost of poor guessesof service, minimize the cost of poor guesses
Allow for natural assumptions of shared Allow for natural assumptions of shared memory of recent interactionsmemory of recent interactions
Continue to learn by observingContinue to learn by observing
Principles of Mixed-Initiative InteractionPrinciples of Mixed-Initiative Interaction
Key goal: Provide the user with clear Key goal: Provide the user with clear advance toward goals advance toward goals
Automated, flexible scoping of Automated, flexible scoping of automated service to precision automated service to precision matching task uncertainty, context matching task uncertainty, context
Automated Scoping and Precision of Automated Scoping and Precision of ServiceService
Prefer automation to do less, but do it correctly
Automated Reasoning about the Automated Reasoning about the Uncertainty of a User’s GoalsUncertainty of a User’s Goals
Automated reasoners must guess about a Automated reasoners must guess about a user’s user’s goalsgoals and and desiredesire for services for services
Good guesses can be quite valuableGood guesses can be quite valuable
……but guessing wrong can be costlybut guessing wrong can be costly
Even valuable automation can be distracting Even valuable automation can be distracting and steal user’s scarce attentional resourcesand steal user’s scarce attentional resources
Minimizing Cost of Guessing WrongMinimizing Cost of Guessing Wrong
Seek design innovation: Advice / assistance Seek design innovation: Advice / assistance valuable when right, but errors with minimal valuable when right, but errors with minimal low costlow cost Natural gestures for declining service Natural gestures for declining service Avoid grabbing focus Avoid grabbing focus Alternate channel overlay: NASA Vista display managerAlternate channel overlay: NASA Vista display manager Nondistracting, simple guessing: Vellum gridpoint guessesNondistracting, simple guessing: Vellum gridpoint guesses
More graceful interaction with potentially More graceful interaction with potentially focused userfocused user
Better timing of services in sync with Better timing of services in sync with availability of attentionavailability of attention
Probability, Utility, & Probability, Utility, & Mixed Initiative InteractionMixed Initiative Interaction
Perspective for designPerspective for designSpecific functions, layering of componentrySpecific functions, layering of componentry Foundations of intelligenceFoundations of intelligence
??*&(#))(@%+%%$#*%$#*&%*&(^*^
Infrastructure, fabric for UI innovationInfrastructure, fabric for UI innovation
ProbabilitiesProbabilities
Infer likelihoods of key uncertainties, take ideal actions
Uncertainty and HCIUncertainty and HCI
• User queryUser query
• User activity User activity
• Content at focusContent at focus
• Data structures Data structures
• User locationUser location
• User profileUser profile
• Vision, speech, soundVision, speech, sound
*Utility-directed actionUtility-directed action
Meshing learning & reasoning with UI designMeshing learning & reasoning with UI design
Beliefs & IntentionsBeliefs & Intentions What does a user believe? What are the user’s goals?What does a user believe? What are the user’s goals?
AttentionAttention What is the user’s workload? What is a user attending to? What is the user’s workload? What is a user attending to?
What What will will a user attend to? What a user attend to? What shouldshould a user attend to? a user attend to? PreferencesPreferences
What does the user like and dislike—and how much?What does the user like and dislike—and how much?
InitiativeInitiative What is the cost and benefit of interaction, interruption, What is the cost and benefit of interaction, interruption,
intervention?intervention? What is the right mix of user / system initiatives?What is the right mix of user / system initiatives?
Critical UncertaintiesCritical Uncertainties
LumièreLumière Project Project
User’s ProfileUser’s ProfileUser’s GoalsUser’s Goals
User’s NeedsUser’s Needs
User ActivityUser Activity
Actions + Words Actions + Words Goals Goals
Joint work with J. Breese, D. Heckerman, K. Rommelse, D. Hovel, et al.
Studies with Human Subjects Studies with Human Subjects
ChallengesChallenges
Architectures for intelligent user Architectures for intelligent user interactioninteraction
Reasoning over timeReasoning over time Sensing activity from systems and Sensing activity from systems and
applicationsapplications Integration of probabilistic information Integration of probabilistic information
retrievalretrieval Models of a user’s competencies over Models of a user’s competencies over
timetime
Big PictureBig Picture
ControlControlNewNew
Perceptions, Perceptions, InteractionsInteractions
EventsEvents UncertainUncertainInference aboutInference about
User, WorldUser, World
ComputationComputationof Ideal UI Actionof Ideal UI Action
Ideal Ideal ActionsActions
EventsEventsSynthesisSynthesis
LearningLearningModelsModels
Inference about a User’s Time-Inference about a User’s Time-Dependent GoalsDependent Goals
TimeTime
Goalt-n
Ej,t-n Ei,t-n
Goalto
Ej,to Ei,to
Goalt-1
Ej,t-1 Ei,t-1
Profile
Profile
Profile
Representing and Updating a Representing and Updating a Persistent “Competency Terrain”Persistent “Competency Terrain”
Skill Catogories
Co
mp
eten
cy
Representing and Updating a Representing and Updating a Persistent “Competency Terrain”Persistent “Competency Terrain”
User’s Skills
Co
mp
eten
cy
TimeTime
Toward a “peripheral nervous system” for Toward a “peripheral nervous system” for sensing user activity sensing user activity SDK with event abstraction languageSDK with event abstraction language Compiler for defining filters for user activityCompiler for defining filters for user activity
Sensing Context and ContentSensing Context and Content
Atomic EventsAtomic Events Modeled EventsModeled Events
TimeTime
EventSource 1
EventSource 2
EventSource n
Eve Event-Specification
Language
Abstraction of Events
Overall LumiOverall Lumièère Architecturere Architecture
BayesianBayesianInferenceInference
EventsEvents
TimeTime
• QueryQuery
• ActionsActionsEvent SynthesisEvent Synthesis
Control SystemControl System
• Probability user Probability user
desires assistancedesires assistance
Lumiere Inference and ActionLumiere Inference and Action
InitiativeInitiative
User vs. system initiativeUser vs. system initiative Allowing fluid collaboration via a mix of Allowing fluid collaboration via a mix of
initiativesinitiatives Toward Toward principles of mixed-initiative principles of mixed-initiative
interactioninteraction Projects: Projects: Lookout, DeepListener, QuartetLookout, DeepListener, Quartet
Reasoning about initiative is a Reasoning about initiative is a
high-payoff opportunity area high-payoff opportunity area
for HCI, Ubicomp, IUIfor HCI, Ubicomp, IUI
Learning by watchingLearning by watching Costs-benefit analysis of initiativeCosts-benefit analysis of initiative Minimize disruption: Prefer Minimize disruption: Prefer doing less,doing less,
but doing it correctlybut doing it correctly
Initiative & Interaction: LookoutInitiative & Interaction: Lookout
?? Critical decision: Critical decision: Do nothing.Do nothing. Ask? Ask? Just do it?Just do it?
Joint work with Andy Jacobs
Real-TimeProbabilistic Inference
Cost--Benefit Analysis
User Actions / Context
UI / Service
• Watch user’s behavior• Store cases, timing info• Learn model from data
Learning and Real-Time BehaviorLearning and Real-Time Behavior in Lookout in Lookout
Lookout in Handsfree ModeLookout in Handsfree Mode
Preferences and InitiativePreferences and Initiative
Expected utility as fundamental in Expected utility as fundamental in decisions about servicesdecisions about services
A: Computer takes action i
A: No action i
D: User desires action iD: User does not desire action i
Service
User’s Desire
u(A,D)
u(A,D)
u(A,D)
u(A,D)
Act
No act
Desired Undesired
Action
No Action
P*
1.0
1.0
0.0
u(A,D)
u(A,D)
u(A,D)
u(A,D)
p(D|E)
eueu((AA) = ) = jj uu((AAii,,DDjj) ) pp((DDjj||EE))eueu((AA) = ) = jj uu((AAii,,DDjj) ) pp((DDjj||EE))eueu((AA) = ) = pp((DD||EE) ) uu((AA,,DD) + ) + pp((DD||EE) ) uu((AA,,DD) ) eueu((AA) = ) = pp((DD||EE) ) uu((AA,,DD) + ) + pp((DD||EE) ) uu((AA,,DD) ) eueu((AA) = ) = pp((DD||EE) ) uu((AA,,DD) + [1 - ) + [1 - pp((DD||EE)] )] uu((AA,,DD) ) eueu((AA) = ) = pp((DD||EE) ) uu((AA,,DD) + [1 - ) + [1 - pp((DD||EE)] )] uu((AA,,DD) )
eueu((AA) = ) = pp((DD||EE) ) uu((AA,,DD) + [1 - ) + [1 - pp((DD||EE)] )] uu((AA,,DD) ) eueu((AA) = ) = pp((DD||EE) ) uu((AA,,DD) + [1 - ) + [1 - pp((DD||EE)] )] uu((AA,,DD) )
Preferences and InitiativePreferences and Initiative
1.0
1.0
0.0
u(A,D)
u(A,D)
u(A,D)
u(A,D)
p(D|E)
Action
No Action
P*
User rushed
Initiative and ContextInitiative and Context
Utility of outcomes as function of context,Utility of outcomes as function of context, uu((AA,,D,D,))
1.0
1.0
0.0
u(A,D)
u(A,D)
u(A,D)
p(D|E)
Action
No Action
P*
No Action
u(A,D)
User rushed
Increase in Amountof Screen Real Estate
u(A,D)
Initiative and ContextInitiative and Context
Utility of outcomes as function of context,Utility of outcomes as function of context, uu((AA,,D,D,))
1.0
1.0
0.0
u(A,D)
u(A,D)
u(A,D)
p(D|E)
Action
No Action
P*
No Action
u(A,D)
User rushed
Increase in Amountof Screen Real Estate
u(A,D)
Initiative and ContextInitiative and Context
Utility of outcomes as function of context,Utility of outcomes as function of context, uu((AA,,D,D,))
1.0
1.0
0.0
u(A,D)
u(A,D)
u(A,D)
p(D|E)
Action
No Action
P*
No Action
u(A,D)
User rushed
Increase in Amountof Screen Real Estate
u(A,D)
Initiative and ContextInitiative and Context
Utility of outcomes as function of context,Utility of outcomes as function of context, uu((AA,,D,D,))
Initiative and ContextInitiative and Context
1.0
1.0
0.0
u(A,D)
u(A,D)
u(A,D)
u(A,D)
p(D|E)
Action
No Action
P*
No Action
u(A,D)
u(A,D)
Increase in Amountof Screen Real Estate
u(A,D)
User rushed
Utility of outcomes as function of context,Utility of outcomes as function of context, uu((AA,,D,D,))
1.0
1.0
0.0
u(A,D)
u(A,D)
u(A,D)
u(A,D)
p(D|E)
Action
P*
Ask
No Action
Expected value of engaging the user in dialogueExpected value of engaging the user in dialogue
Engaging in Dialog about InitiativeEngaging in Dialog about Initiative
Varying Precision of ServiceVarying Precision of Service
Consider contributions across Consider contributions across a spectrum of precision a spectrum of precision
Assume user will refine partial resultsAssume user will refine partial results Under uncertainty, trade off reduced precision for Under uncertainty, trade off reduced precision for
higher accuracyhigher accuracy
ApptDay
Week
Timing of InitiativeTiming of Initiative
Timing is critical: consider patterns of attentionTiming is critical: consider patterns of attention Record length of message and dwell time before Record length of message and dwell time before
calendar invokedcalendar invoked Perform regressionPerform regression
0
2
4
6
8
10
0 500 1000 1500 2000 2500
Length of original message (bytes)Ob
serv
ed d
wel
l bef
ore
act
ion
(sec
)
Conversational ArchitecturesConversational Architectures ProjectProject
DeepListenerDeepListener Bayesian ReceptionistBayesian Receptionist QuartetQuartet
Why do people find it more difficult and Why do people find it more difficult and frustrating to converse with a spoken dialog frustrating to converse with a spoken dialog
system than with a person?system than with a person?
QuestionQuestion
Several answersSeveral answers
• Poor recognition of wordsPoor recognition of words
• Meaning too difficult to captureMeaning too difficult to capture
• Lack of precise user modelsLack of precise user models
• Different social and personality Different social and personality dynamicsdynamics
Interpreting spoken language abounds with Interpreting spoken language abounds with uncertaintyuncertainty
IntuitionsIntuitions Despite uncertainty in humanDespite uncertainty in human––human conversation human conversation
people manage to understand each other quite people manage to understand each other quite well.well.
People consider the source of their uncertainties People consider the source of their uncertainties and pursue actions to resolve confusions.and pursue actions to resolve confusions.
RecognitionRecognition LanguageLanguage Context, topic, meaningContext, topic, meaning Frank troubleshootingFrank troubleshooting
GoalGoal: Models and inference methods that seek : Models and inference methods that seek mutual understanding under uncertainty given mutual understanding under uncertainty given inescapably unreliableinescapably unreliable components. components.
GroundingGrounding
People resolve uncertainties through a People resolve uncertainties through a process of process of groundinggrounding
Process by which participants establish Process by which participants establish and maintain the mutual belief that their and maintain the mutual belief that their utterances have been understood well utterances have been understood well enough for current purposesenough for current purposes
-Clark & Schaefer, 1987 -Clark & Schaefer, 1987
DeepListenerDeepListener
Utility-directed clarification dialogUtility-directed clarification dialog Formal model of “understood well enough”Formal model of “understood well enough” Development environmentDevelopment environment Assessment toolsAssessment tools Focus: Spoken command and control Focus: Spoken command and control
systemssystems
Stakes, Likelihoods, and Stakes, Likelihoods, and Clarification ActionsClarification Actions
Consider stakes of real-world action being Consider stakes of real-world action being consideredconsidered
Should I format your hard drive? Should I format your hard drive?
Should I try to schedule that?Should I try to schedule that?
Should I demolish the King Dome Should I demolish the King Dome nownow??
Consider uncertaintiesConsider uncertainties Consider expected utility of alternative “repair” Consider expected utility of alternative “repair”
actionsactions Costs and benefits of real-world action vs. alternative Costs and benefits of real-world action vs. alternative
dialog repair actionsdialog repair actions
Infer likelihoods of alternative Infer likelihoods of alternative spoken spoken intentionsintentions Likelihoods of different Likelihoods of different spoken intentionsspoken intentions given acoustics given acoustics Optionally condition on goals inferred by user model Optionally condition on goals inferred by user model
external to the speech systemexternal to the speech system
Compute Compute clarificationclarification or real-world or real-world actions with highest expected utilityactions with highest expected utility
Fuse multiple attempts with Bayesian Fuse multiple attempts with Bayesian model that considers confidencesmodel that considers confidences Consider history of utterances within a sessionConsider history of utterances within a session No reason to start over at each turn! ..Leverage what was No reason to start over at each turn! ..Leverage what was
heard beforeheard before
ApproachApproach
Decision ModelDecision Model
Speaker’s Goal(t-1)
User’s SpokenIntention(t-1)
Dialog or Domain-Level
Action(t-1)
ASR Candidate 1Confidence(t-1)
Utility(t-1)
User Actions(t-1)Content at Focus (t-1)
ASR Candidate nConfidence(t-1)
Context
. . .
ASR ReliabilityIndicator(t-1)
External
User Model
Speaker’s Goal(t-1)
User’s SpokenIntention(t-1)
Dialog or Domain-Level
Action(t-1)
ASR Candidate 1Confidence(t-1)
Utility(t-1)
User Actions(t-1)Content at Focus (t-1)
ASR Candidate nConfidence(t-1)
Context
...
Speaker’s Goal(t)
User’s SpokenIntention(t)
Dialog or Domain-Level
Action(t)
ASR Candidate 1Confidence(t)
Utility(t)
User Actions(t)Content at
Focus (t)
ASR Candidate nConfidence(t)
Context
...
ASR ReliabilityIndicator(t-1)
ASR ReliabilityIndicator(t-1)
Dynamic Model for Reasoning Over Dynamic Model for Reasoning Over Multiple TurnsMultiple Turns
Dialog Actions under ConsiderationDialog Actions under Consideration
Perform Perform real-world actionreal-world action (e.g., (e.g., implode the King implode the King Dome nowDome now))
Ask for repetitionAsk for repetition to clarify to clarify
Note hesitationNote hesitation or reflection and try again or reflection and try again
Note potential Note potential overhearingoverhearing of noise and inquireof noise and inquire
Note inattentionNote inattention of user and try to acquire user’s of user and try to acquire user’s attentionattention
Don’t perform action and just Don’t perform action and just go awaygo away
Note problem with conversational interaction and Note problem with conversational interaction and attempt to attempt to troubleshoottroubleshoot
Example: DeepListener for handling Example: DeepListener for handling
confirmation, negationconfirmation, negation
DeepListener: SDK and Real-Time DeepListener: SDK and Real-Time Clarification Dialog SystemClarification Dialog System
Dynamic Bayesian network modeling and Dynamic Bayesian network modeling and inference inference
MS command and control speech systemMS command and control speech system Backchannel animations: MS AgentBackchannel animations: MS Agent
DeepListener: SDK and Real-Time DeepListener: SDK and Real-Time Clarification Dialog SystemClarification Dialog System
Accruing Evidence Over Repeated UtterancesAccruing Evidence Over Repeated Utterances
Beliefs and Actions for Clarification DialogBeliefs and Actions for Clarification Dialog
Clarification
Beliefs about Spoken IntentionBeliefs about Spoken Intention
yes
no overheard
yes
no overheard
Inferred beliefs
Expected Utility of Alternate ActionsExpected Utility of Alternate Actions
Repeat
Again
EngageNoise
Tshoot Disengage
Attention
Repeat
Again
EngageNoise
Tshoot Disengage
Attention
Expected utility of actions
Preference elicitationPreference elicitation For developers (assessment “at factory”)For developers (assessment “at factory”) For users! Prototypical patterns, assessment For users! Prototypical patterns, assessment
wizard, direct detailed assessmentwizard, direct detailed assessment
Toward a more general SDK for command Toward a more general SDK for command and control dialog (e.g., telephony systemsand control dialog (e.g., telephony systems
Preference Assessment and EncodingPreference Assessment and Encoding
Assessing Preferences on OutcomesAssessing Preferences on Outcomes
Speaker’s Goal(t-1)
User’s SpokenIntention(t-1)
Dialog or Domain-Level
Action(t-1)
ASR Candidate 1Confidence(t-1)
Utility(t-1)
ASR Candidate nConfidence(t-1)
External UserModel(t-1)
.. .
Speaker’s Goal(t)
User’s SpokenIntention(t)
ASR Candidate 1Confidence(t)
ASR Candidate nConfidence(t)
ExternalUser Model(t)
.. .
Dialog or Domain-Level
Action(t)
Utility(t)
ASR ReliabilityIndicator(t-1)
ASR ReliabilityIndicator(t)
Troubleshooting Conversation Failures Troubleshooting Conversation Failures Over Multiple StepsOver Multiple Steps
Expected utility of taking action to repair listening situationas function of details of multi-turn dialog “history”
Assessing Frustration with Assessing Frustration with Number of StepsNumber of Steps
yesTshoot
reflect
Repeat
overheard
Engage
no
When Troubleshooting is the When Troubleshooting is the Best ActionBest Action
yes
Tshoot
reflect
Repeat
When Troubleshooting is the When Troubleshooting is the Best ActionBest Action
Considering Attentional IssuesConsidering Attentional Issues
Toward Continuous Listening SystemsToward Continuous Listening Systems
Beyond cumbersome “push-to-talk” Beyond cumbersome “push-to-talk” systemssystems
Discriminating target of speechDiscriminating target of speech Understanding conversation Understanding conversation
maintenance statusmaintenance status Allowing user barge-inAllowing user barge-in
DeepListener StatusDeepListener Status
Application development environmentApplication development environment Mobile telephonyMobile telephony Desktop applicationsDesktop applications Managing “subdialog”Managing “subdialog”
Evaluation: Tradeoffs—steps vs. stakesEvaluation: Tradeoffs—steps vs. stakes Learning by watchingLearning by watching
44 S is proposing activity to L L is considering S’s proposal of
ConversationConversation
33 S is signaling that p for L L is recognizing that p from S
IntentionIntention
22 S is presenting signal to L L is identifying signal from S
SignalSignal
11 S is executing behavior for L L is attending to behavior from S
ChannelChannel
QuartetQuartet:: Multilevel Grounding & Extended Sensing Multilevel Grounding & Extended Sensing
LevelLevel
Maintenance ModuleMaintenance Module
Maintenance ModuleMaintenance Module
User’s ResponseLatency
Used Nameof System
Stressed Nameof System
Repeat SimilarLogical Form
Eye Gaze onSystem
Time Since LastUser Speech
MaintenanceStatus (t)
MaintenanceStatus (t-1)
SignalAccuracy (t-1)
Time Since LastSystem Action
Others Presentin Room
Calendar ShowsMeeting
Telephone in Use
Overall Parse Fit
UtteranceComplexity
Syntactic SketchScore
NumNon-Terminals
Num PhrasalHeads
Final UtteranceConfidence
SignalAccuracy (t)
InterferenceEvent
Num HypothesesPer Word
Energy Floor
Signal Identified
User’s Focus ofAttention (t-1) User’s Focus of
Attention (t)
General ASRQuality
Is Trained forUser
Microphone Type
ThresholdSetting
Custom GrammarFor Domain
Channel Level
Signal Level
Linguistic EvidenceLinguistic Evidence
Nature, correctness of NL parse Nature, correctness of NL parse as evidence sourceas evidence source
Visual EvidenceVisual Evidence
Evidence for user attentionEvidence for user attention
Visual EvidenceVisual Evidence
Evidence for user attentionEvidence for user attention
Acoustical EvidenceAcoustical Evidence
Vision and GroundingVision and Grounding
Vision and GroundingVision and Grounding
Considering Visual and Considering Visual and Linguistic CluesLinguistic Clues
“If you could take me to the next slide that would be great.”
“I can look away and talk about the computer.”
Considering Visual and Considering Visual and Linguistic CluesLinguistic Clues
… and even talk about going to the next page.”
Considering Visual and Considering Visual and Linguistic CluesLinguistic Clues
Bayesian ReceptionistBayesian Receptionist
Continue to Continue to gather gather information information
User’s Goal
Goal 1 Goal n
Subgoal 11
Subgoal 1x1 Subgoal 1xy
Subgoal 1x
Level 0
Level 1
Level 2
Return to previous level Return to previous level of analysisof analysis
Progress to next level of Progress to next level of precision without precision without confirmationconfirmation
Progress to next level Progress to next level of precision after of precision after confirmationconfirmation
EVI
EVI
EVI
Take action in worldTake action in world
Initial utterance, observations
World action
Open request for information
Richer Models of InitiativeRicher Models of InitiativeConversational Architectures ProjectConversational Architectures Project
Joint work with Tim Paek
Goal 1 Goal nLevel 0
User’s Goal
Subgoal 11
Subgoal 1x1 Subgoal 1xy
Subgoal 1x
Level 1
Level 3
VOI
VOI
VOI
Bayesian Models and DialogBayesian Models and Dialog
VOI
Goal 1 Goal nLevel 0
User’s Goal
Subgoal 11
Subgoal 1x1 Subgoal 1xy
Subgoal 1x
Level 1
Level 3
VOI
VOI
VOI
Bayesian Models and DialogBayesian Models and Dialog
VOI
User’s Goal
Goal 1 Goal n
Subgoal 11
Subgoal 1x1 Subgoal 1xy
Subgoal 1x
Level 1
Level 3
VOI
VOI
VOI
Bayesian Models and DialogBayesian Models and Dialog
Level 0
DARPA ISAT Study:DARPA ISAT Study: 2000-2001 2000-2001
Foundations of Augmented CognitionFoundations of Augmented Cognition
(E. Horvitz and M. Pavel, co-chairs)(E. Horvitz and M. Pavel, co-chairs)
Multiple meetings culminating at NAS, Wash Multiple meetings culminating at NAS, Wash DC (August 2001)DC (August 2001)
Augmented Cognition programAugmented Cognition program
Toward Augmented CognitionToward Augmented Cognition
Divided attentionDivided attention Visual search Visual search Concept attainmentConcept attainment MemoryMemory VisualizationVisualization Judgment & decision makingJudgment & decision making Action under limited time and informationAction under limited time and information ComprehensionComprehension Training & educationTraining & education
Apply computation to support / augment human cognition
Results in Cognitive PsychologyResults in Cognitive Psychology
Characterization of limitations in human cognitionCharacterization of limitations in human cognition
Human capabilities static, computational capabilities Human capabilities static, computational capabilities have grown rapidlyhave grown rapidly
Human abilitiesHuman abilities
Computational Computational resources, prowessresources, prowess
Augmented Augmented CognitionCognition
TodayToday
Toward Augmented CognitionToward Augmented Cognition
ExistingExisting
Psychological Psychological
Results on Results on
Cognitive LimitationsCognitive Limitations
TargetTarget
Cognitive TasksCognitive Tasks
MemoryMemory ConceptConcept
AttainmentAttainmentDivided Divided
AttentionAttention
Ab
ilities & efficien
ciesA
bilities &
efficiencies
* * New Cog. Psych. New Cog. Psych. Research Research
* HCI, Aug. Cognition * HCI, Aug. Cognition ResearchResearch
??
HCI and Augmented CognitionHCI and Augmented Cognition
Designing services with knowledge of Designing services with knowledge of cognitive limitationscognitive limitations
Value of probabilistic reasoning for Value of probabilistic reasoning for assessing goals, states, contextassessing goals, states, context
Value of probabilistic reasoning for Value of probabilistic reasoning for assessing goals, states, contextassessing goals, states, context
Key ArenasKey Arenas
AttentionAttention AttentionAttention MemoryMemoryMemoryMemory
Judgment & Decision makingJudgment & Decision makingJudgment & Decision makingJudgment & Decision making
Visualization Visualization & Display& Display
Visualization Visualization & Display& Display
Learning Learning & Training& Training
Learning Learning & Training& Training
Language Language & Interaction& InteractionLanguage Language
& Interaction& Interaction
Neurobiological issuesNeurobiological issuesNeurobiological issuesNeurobiological issues
Meeting FociMeeting Foci
AttentionAttention AttentionAttention MemoryMemoryMemoryMemory
Judgment & Decision makingJudgment & Decision makingJudgment & Decision makingJudgment & Decision making
Visualization Visualization & Display& Display
Visualization Visualization & Display& Display
Learning Learning & Training& Training
Learning Learning & Training& Training
Language Language & Interaction& InteractionLanguage Language
& Interaction& Interaction
Neurobiological issuesNeurobiological issuesNeurobiological issuesNeurobiological issues
Advances in Computation:Advances in Computation:Sensing, Learning, Sensing, Learning,
Reasoning & ApplicationsReasoning & Applications
Advances in Computation:Advances in Computation:Sensing, Learning, Sensing, Learning,
Reasoning & ApplicationsReasoning & Applications
Perspective onPerspective on
ApplicationsApplications
Perspective onPerspective on
ApplicationsApplications
Recent Sample Results from the LabRecent Sample Results from the Lab
UVA/CMUUVA/CMU: Memory, : Memory, spatialization, and contextspatialization, and context
MSRMSR: Alerting, performance, and : Alerting, performance, and memorymemory
UVA/CMU: UVA/CMU: Memory Efficiency and SpatializationMemory Efficiency and Spatialization
(D. Proffitt, R. Pausch, et al.)(D. Proffitt, R. Pausch, et al.)
Significant memory boosts for Significant memory boosts for word-pair trials for single vs. word-pair trials for single vs. multiple screen learningmultiple screen learning
fMRI analysis of differential fMRI analysis of differential activityactivity
InfocockpitInfocockpit
Experiment: Word-Pair RecallExperiment: Word-Pair Recall
TaskTask 10 cue words and 10 target words. 3 lists. Cue words 10 cue words and 10 target words. 3 lists. Cue words
same for all lists. Target words varied from list to list. same for all lists. Target words varied from list to list. Next day: Recall pairs (not told ahead of time)Next day: Recall pairs (not told ahead of time)
Two conditionsTwo conditions Infocockpit condition Infocockpit condition 3 monitors, 3 large projection screens w/ contextual 3 monitors, 3 large projection screens w/ contextual
imageimage & surround sound. Word lists displayed on & surround sound. Word lists displayed on different monitors.different monitors.
Control Control Standard desktop computer.Standard desktop computer.
Recall EnhancementRecall Enhancement
Difference fMRIDifference fMRI
MSR: Studies of Divided Attention, MSR: Studies of Divided Attention, Alerting, NotificationAlerting, Notification
(M. Czerwinski, E. Cutrell, E. Horvitz)(M. Czerwinski, E. Cutrell, E. Horvitz)
Psychological Studies of AttentionPsychological Studies of Attentione.g., Interruption & recoverye.g., Interruption & recovery
(With Mary Czerwinski & Ed Cutrell)(With Mary Czerwinski & Ed Cutrell)
Differing Costs of DisruptionDiffering Costs of Disruption
High-level characterization of taskHigh-level characterization of task
Planning Execution Evaluation0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
Interrupted Phase
Relevant vs. Irrelevant AlertsRelevant vs. Irrelevant Alerts
Total Resume0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
Task Timing
Relevant
Irrelevant
Probing Deeper Structure of DisruptionProbing Deeper Structure of Disruption
Probing “reorientation cost”Probing “reorientation cost” Visual reorientation (e.g., re-acquire position in list Visual reorientation (e.g., re-acquire position in list
being searched)being searched) Conceptual reorientation (e.g., re-acquire context, Conceptual reorientation (e.g., re-acquire context,
goal)goal)
??
Probing Deeper Structure of DisruptionProbing Deeper Structure of Disruption
ChallengeChallenge: Identify target book title in a 3 : Identify target book title in a 3 page list of titles with and without cursor page list of titles with and without cursor markermarker
Two targetsTwo targets .5 cases: Verbatim title.5 cases: Verbatim title .5 cases: Gist: .5 cases: Gist: ee.g., .g., ““A book about Ramses II and A book about Ramses II and
the Nilethe Nile.”.” Remind meRemind me button button DesignDesign
2 (title v. gist search) 2 (title v. gist search) x 2 (marker v. no marker) x 2 (marker v. no marker) x 2 (notification vs. no notification trial) x 2 (notification vs. no notification trial) x 8 (replications per condition) x 8 (replications per condition) [[16 participants, 16 participants, 64 trials per session]64 trials per session]
Probing Deeper Structure of DisruptionProbing Deeper Structure of Disruption
Notifications Notifications Mimicked sound, onset of MSN Messenger 2.0Mimicked sound, onset of MSN Messenger 2.0 Simple multiplication, division problemsSimple multiplication, division problems
Dependent variables Dependent variables Total task timeTotal task time Time to switch to a notificationTime to switch to a notification Number of reminders requested Number of reminders requested Time spent on a notificationTime spent on a notification
Probing Deeper Structure of DisruptionProbing Deeper Structure of Disruption
Cognitive reorientation dominates spatial Cognitive reorientation dominates spatial reorientation.reorientation.
Reminders requested more for notifications Reminders requested more for notifications under higher cognitive loadunder higher cognitive load
Significant relationship between need for Significant relationship between need for reminder and reminder and timingtiming of the interruption of the interruption Participants more likely to request a reminder when Participants more likely to request a reminder when
disrupted early in taskdisrupted early in task
Sample Results on Sample Results on Disruption and ReorientationDisruption and Reorientation
Augmented Cognition: OpportunitiesAugmented Cognition: Opportunities
AttentionAttention Integrate consideration of results on Integrate consideration of results on
divided attention and disruption in divided attention and disruption in monitoring systems.monitoring systems.
e.g.,e.g., Control if, when, and how information about Control if, when, and how information about a monitored system or situation is presented to a monitored system or situation is presented to operators focusing on another, more central task.operators focusing on another, more central task.
Employ results from visual search and Employ results from visual search and
attention in rendering policiesattention in rendering policies
e.g., Develop automated display layout and e.g., Develop automated display layout and optimization considering time-criticality, content.optimization considering time-criticality, content.
Memory and LearningMemory and Learning Harness results on memory and contextual Harness results on memory and contextual
cuescuese.g.,e.g., Automated sensing / generation of cues Automated sensing / generation of cues during learning, coupled with re-generation of during learning, coupled with re-generation of multiple contextual clues during remembering.multiple contextual clues during remembering.
Harness results on ideal spacing for enhanced Harness results on ideal spacing for enhanced learninglearning
e.g.,e.g., Infer ideal time for reinforcement with Infer ideal time for reinforcement with repetition in a training system.repetition in a training system.
Augmented Cognition: OpportunitiesAugmented Cognition: Opportunities
Judgment and Decision MakingJudgment and Decision Making Consider errors of framing and well-Consider errors of framing and well-
characterized biases of judgment, action characterized biases of judgment, action under uncertainty.under uncertainty.
e.g., Guide visual representation of actions, e.g., Guide visual representation of actions, alternatives, and outcomes in a manner that alternatives, and outcomes in a manner that debiases stereotypical errors of judgment in debiases stereotypical errors of judgment in decision making under uncertainty.decision making under uncertainty.
Augmented Cognition: OpportunitiesAugmented Cognition: Opportunities
Potential ApplicationsPotential Applications
Information filtering, triage, and alertingInformation filtering, triage, and alerting Mixed-initiative agent-operator interactionMixed-initiative agent-operator interaction Context-sensitive computingContext-sensitive computing Intelligent remindingIntelligent reminding Managing attention and disruptionManaging attention and disruption Automated visualization in control systemsAutomated visualization in control systems Learning and training systemsLearning and training systems Human-errorHuman-error——aware systemsaware systems Automated sensor fusion and decision Automated sensor fusion and decision
supportsupport Judgment de-biasing systemsJudgment de-biasing systems
Presentations on Key ArenasPresentations on Key Arenas
AttentionAttention (Sperling) (Sperling)
Memory Memory (Landauer)(Landauer)
Judgment & Decision makingJudgment & Decision making (Fischhoff) (Fischhoff)
Visualization & DisplayVisualization & Display (Ellis) (Ellis)
Language & InteractionLanguage & Interaction (Oviatt) (Oviatt)
Learning & TrainingLearning & Training (Carroll) (Carroll)
Neurobiological issuesNeurobiological issues (Levy) (Levy)
Breakout GroupsBreakout Groups Key models & findings on limitations from Key models & findings on limitations from
psychologypsychology
Opportunities for real-world applications Opportunities for real-world applications
Challenges per psychological understandingChallenges per psychological understanding
Challenges per computation, display, HCIChallenges per computation, display, HCI
Take an optimistic perspectiveTake an optimistic perspective
Take a pessimistic perspectiveTake a pessimistic perspective
Evaluation metricsEvaluation metrics
Attention Attention Attention Attention MemoryMemoryMemoryMemory
Judgment & Decision makingJudgment & Decision makingJudgment & Decision makingJudgment & Decision making
Visualization Visualization & Display& Display
Visualization Visualization & Display& Display
Learning Learning & Training & Training
Learning Learning & Training & Training
Language Language & Interaction& InteractionLanguage Language
& Interaction& Interaction
Neurobiological issuesNeurobiological issuesNeurobiological issuesNeurobiological issues
AttentionAttention AttentionAttention
Research challenges Most results (e.g., AOC) focus on short durations (< 500 msec)
Need to understand attention over longer time scales
Links among low-level components, conceptual issues with scaling up to perception, attention, action for larger time scale strategies
Attention and disruption in realistic environments
Opportunities Handling divided attention, attention-guided access, attention-centric rendering and visualization, context-sensitive “attentional security” for learning & performance
Potential Applications Context-sensitive alerting systems, automated monitoring & guiding of attention, attending to anomalies, ideal mixed-initiative interaction
Memory & LearningMemory & LearningMemory & LearningMemory & Learning
Research challenges Basic research, e.g., understanding complex interactions among component processes of memory, links to attention, etc.
Understanding state of knowledge, best strategy per this state
User interfaces, user interaction in real-world tasks and systems
Opportunities Significant results have not seen application!
Specific policies that promote forgetting and impair performance during training actually enhance long-term retention!
e.g., spacing, variation, contextual interference, intermittent feedback
Can make training qualitatively a more difficult experience
Potential Applications Context-sensitive reminder systems
Enhanced situation monitoring, training
Judgment & Decision makingJudgment & Decision makingJudgment & Decision makingJudgment & Decision making
Research challenges Understand fixation on representations, premature closure vs. paralysis of analysis, propagating implications of key changes in assumptions, considering reliability of sources, influence of stress and fatigue; cognitive processes of situation assessment
Dynamics of belief about processes, uncertainty about spatial relations given streams of information
Decisions, judgment in gaming setting with opponent
Opportunities Harness well-characterized results on biases of judgment
Use of representations for communicating about, understanding situations; sensitivity of variables; value of different kinds of modeling effort
Potential Applications Decision making companions that expand considerations, pose reformulatations, debias, assist in real-time, training
Rendering for ideal fusion of information, judgment, action
Visualization & DisplayVisualization & DisplayVisualization & DisplayVisualization & Display
Research challenges Understanding how people integrate information from multiple components of scene into usable concepts
Basic research needed on how to map distinct findings about visual attention, concept attainment to control of display, layout for tasks
Opportunities Map cognitively deliberate serial analyses into fast perceptual recognition Harness knowledge about emergent features, gestalt principles, redundant coding, popout effects (e.g., use of line orientation, length, width, size, number, terminators, intersection, closure, color, intensity, flicker, direction of motion, depth cues, lighting
Exploit findings on timing, animation, scene complexity
Harness knowledge about biases in representation
Potential Applications Displays that adapt to task at hand, employing perceptual and cognitive principles to highlight most important information
Cognitively efficient representations of spatial relationships, dynamics, uncertainty, reliability
Language & InteractionLanguage & InteractionLanguage & InteractionLanguage & Interaction
Research challenges Handling disfluency speech, variations such as hyperarticulation, speech under stress, in mobile settings
Handling signal variability
Handling ambiguity
New languages, principles for mixed-initiative interaction
Opportunities Ideal designs for multimodal input, based on stereotypic patterns of gesture and language
Error reduction, speech interaction with minimal cognitive load
Potential Applications Natural interfaces that allow variable, noisy gesture and language
Automated detection of misunderstandings in communications
Enhanced multimodal interfaces for efficient interaction
Some Findings and Some Findings and RecommendationsRecommendations
Perspective, focus of Augmented Cognition as Perspective, focus of Augmented Cognition as distinguished from much of psychologically-oriented HCI distinguished from much of psychologically-oriented HCI
Interest, enthusiasm among participantsInterest, enthusiasm among participants
Mathematical psychologists traditionally have been Mathematical psychologists traditionally have been focused mainly on basic mechanisms as disjoint from real-focused mainly on basic mechanisms as disjoint from real-world applications, usage.world applications, usage.
Many current psychological models and research results Many current psychological models and research results are not necessarily well-adapted, matched for augmented are not necessarily well-adapted, matched for augmented cognition work.cognition work.
Need to map some basic results at msec time scale, Need to map some basic results at msec time scale, micromechanisms to broader strategic, planning level of micromechanisms to broader strategic, planning level of perception, analysis, decision making, learningperception, analysis, decision making, learning
New research in psychology will be valuable in support of New research in psychology will be valuable in support of Augmented Cognition efforts—some focused in the Augmented Cognition efforts—some focused in the context of real-world taskscontext of real-world tasks
Criticality of interdisciplinary workCriticality of interdisciplinary work––but potential difficulties but potential difficulties with building interdisciplinary teams for augmented with building interdisciplinary teams for augmented cognitioncognition
Some low hanging fruit, but also some very difficult Some low hanging fruit, but also some very difficult challenges in science and technology in this arenachallenges in science and technology in this arena
Even applications of some simple well-characterized Even applications of some simple well-characterized limitations could go a long way in enhancing performancelimitations could go a long way in enhancing performance
Potential value of composing interdisciplinary academic Potential value of composing interdisciplinary academic advisory group in support of potential DARPA program advisory group in support of potential DARPA program
Some Findings and Some Findings and RecommendationsRecommendations
The timing is right for building practical The timing is right for building practical systems via pursuit of a deeper science of systems via pursuit of a deeper science of
humanhuman––computer symbiosiscomputer symbiosis
But…But…
Research and innovation will face Research and innovation will face significant challenges and impediments, significant challenges and impediments,
both technically and professionally.both technically and professionally.
Assuming current predictions Assuming current predictions of future importance as same of future importance as same
as the actual future importanceas the actual future importance
All events
All events
Events assessed as important
All events
Events assessed as important
Events forgotten
All events
Events assessed as important
Events forgotten
Sensed
SummarySummary
Key uncertainties and human-computer Key uncertainties and human-computer interaction…interaction…
Beliefs & intentionsBeliefs & intentions PreferencesPreferences InitiativeInitiative AttentionAttention
Representations & machinery for Representations & machinery for reasoning under uncertainty provide a reasoning under uncertainty provide a
rich fabric for developing valuable rich fabric for developing valuable software services & experiences.software services & experiences.