Explanation and Trust for Adaptive Agents

QuickTime™ and aTIFF (LZW) decompressor

are needed to see this picture.

Explanation and Trust for Adaptive Agents

Alyssa [email protected]

Joint work with Deborah McGuinness (Stanford), Michael Wolverton (SRI), and Paulo Pinheiro da Silva (UTEP)



Outline

Explanation in Adaptive Agents Explanation & Trust: The User

Perspective Explaining Learned Information Discussion & Future Work



Why do we need explanations?

Systems are getting more complex Hybrid and distributed processing Multiple learning components Multiple heterogeneous, distributed information

sources Highly variable reliability of information sources Less transparency of system computation and

reasoning Systems are taking more autonomous control

Guide/assist user actions Perform autonomous actions on behalf of user “reason, learn from experience, be told what to do,

explain what they are doing, reflect on their experience, and respond robustly to surprise” *

* DARPA PAL program: http://www.darpa.mil/ipto/programs/pal/



One Complex System: CALO Cognitive Assistant that Learns and

Organizes Personal office assistant, tasked with:

Noticing things in the cyber and physical environments Aggregating what it notices, thinks, and does Executing, adding/deleting, suspending/resuming

tasks Planning to achieve abstract objectives Anticipating things it may be called upon to do or

respond to Interacting with the user Adapting its behavior in response to past experience,

user guidance Contributed to by 22 different organizations



Working with a Cognitive Assistant

CALO users need to Understand system behavior and responses Trust system reasoning and actions

To believe and act on recommendations from CALO, users need ways of exploring how and why the system acted, responded, recommended, and reasoned the way it did.

Additional wrinkle: CALO knowledge, behavior, and assumptions are constantly changing through several forms of machine learning.

A unified framework for explaining behavior and reasoning is essential for users to trust and adopt cognitive assistants.



Thank you!



The Integrated Cognitive Explanation Environment (ICEE)

Unified framework for explaining logical and task reasoning.

Applicable to multiple task execution systems.

Leverage existing InferenceWeb work for generating formal justifications.

Underlying task reasoning useful beyond explanation.

Provide sample implementation of end-to-end system.



Getting an Explanation

<user>: Why are you doing <subtask>?<system>: I am trying to do <high-level-task>

and <subtask> is one subgoal in the process.

<user>: Why are you doing <high-level-task>?<user>: Why haven’t you completed

<subtask> yet?<user>: Why is <subtask> a subgoal of <high-

level-task>?<user>: When will you finish <subtask>?<user>: What sources did you use to do

<subtask>?

Initial request and answer strategy

Follow-up questions for mixed initiative dialogue



Explanation ExampleSample question type: task motivation

Why are you doing <subtask>?

Strategy: reveal task hierarchyI am trying to do <high-level-task> and <subtask> is one subgoal

in the process.

Alternate strategies: Provide task abstraction Expose preconditions Expose termination conditions Reveal meta-information about task dependencies Explain provenance related to task preconditions or other

knowledge

Possible follow-up suggestions: Request additional detail Request clarification of of the given explanation Request an alternate strategy to the original query

McGuinness, D.L.; Glass, A.; Wolverton, M.; Pinheiro da Silva, P. A Categorization of Explanation Questions for Task Processing Systems. AAAI Workshop on Explanation-Aware Computing (ExaCt-2007), Vancouver, Canada, 2007.



Sample Interface Linked to ICEE



Outline

Explanation in Cognitive Agents Explanation & Trust: The User




Study Procedure 14 participants, in 2 groups

12 men, 2 women Wide range of ages, education,

previous CALO experience Assigned tasks to accomplish

with CALO (many scripted) Told about trust study in

advance Structured interview format Identified 8 themes, in 3 major

categories



Usability

Theme 1: Basic usability is important even in prototype-level systems.

“I can’t tell you how much I would love to have [the system]. But I also can’t tell you how much I can’t stand it.”



Usability

Theme 2: Learning algorithms can give the impression that the user is being ignored.

“You specify something, and [the system] comes up with something completely different. And you’re like, it’s ignoring what I want!”



ExplanationsTheme 3: Users are consistent in wanting to ask specific questions, particularly when they are surprised by responses or failures.

Commonly requested questions: Documentation questions: How do I do this? Why can’t I do this other

thing? What do you mean by this? Explanation questions: What can I do now? What is happening now

and why? When will you finish?

“If there had been an option to ask a question, I would have loved to ask a question.”

“I asked [‘Why?’] all the time, but I wasn’t getting answers!”



Explanations

Theme 4: The granularity of feedback is important.

“I don’t just want an idiot light.”



Trust

Theme 5: Users don’t trust opaque systems; they want transparency.

“The ability to check up on the system, ask it questions, get transparency to verify what it is doing, is the number one thing that would make me want to use it.”



Trust

Theme 6: Access to information provenance can improve trust in both the information and the automated reasoning.

“[The system] needs a better way to have a meta-conversation.”



Trust

Theme 7: Like in politics and the economy, gaining user trust relies on properly managing expectations.

“I was paralyzed with fear about what it would understand and what it would not.”



Trust

Theme 8: Most users have a “trust but verify” attitude that makes system autonomy difficult without explainable verification.

“I trust [the system’s] accuracy, but not its judgment.”



How Explanation Can Help

T1: Basic Usability T2: Being Ignored T3: Questions T4: Granularity of Feedback T5: Transparency T6: Provenance T7: Managing Expectations T8: Autonomy & Verification











Outline

Explanation in Cognitive Agents Explanation & Trust: The User




The Use-Ask-Understand-Update Cycle

Use Ask

Understand

Update



Learning by Instruction

Relatively straight-forward to explain

Store instruction, resulting modification

Strategies present instruction and related meta-information

Demonstrated in CALO with Tailor task learning system



Learning by Demonstration Generalizes user’s demonstration to learn a procedure One data point --> generalization will sometimes be

wrong Specifically, it will occasionally over generalize

Generalize the wrong variables, or too many variables Produce too general a procedure because of a coarse-

grained type hierarchy Explain the relevant aspects of the generalization

process To help the user identify and correct over

generalizations To help the user understand and trust the learned

procedures Working with LAPDOG task learning system in CALO



Support-Vector Machines Augment SVM to gather additional meta-information about the SVM itself:

Support vectors identified by SVM Support vectors nearest to the query point Margin to the query point Average margin over all data points Non-support vectors nearest to the query point Kernel transformation used, if any

Represent SVM learning and meta-information as justification in PML, using added SVM rules

Design abstraction strategies for presenting justification to user as a similarity-based explanation

Demonstrated in CALO with PLIANT preference learning system PLIANT uses user-elicited preferences and past choices to learn user scheduling

preferences Inconsistent user preferences, over-constrained schedules, and necessity of

exploring the preference space result in user confusion about why a schedule is being presented.

Lack of user understanding of PLIANT’s updates creates confusion, mistrust, and the appearance that preferences are being ignored.

Provide justifications of schedule suggestions, without requiring user to understand SVM learning.



Future Work

Using conflicts to drive the learn-explain cycle

Using explanations to identify high-reward learning opportunities

Expand prototype links to CALO learning systems

Support more advanced dialogues and interfaces

User study using ICEE explanations



Resources Explanation questions & strategies:

McGuinness, D.L., Glass, A., Wolverton, M., and Pinheiro da Silva, P. A Categorization of Explanation Questions for Task Processing Systems. AAAI 2007 Workshop on Explanation-Aware Computing (ExaCt-2007), Vancouver, Canada, 2007.

CALO trust study: Glass, A., McGuinness, D.L., and Wolverton, M. Establishing Trust in Adaptive Agents:

A Study of Explanation and Trust in Agents that Learn. Technical Report, KSL-07-04, Knowledge Systems, Artificial Intelligence Laboratory, Stanford University, 2007 (to appear).

Explanation interfaces: McGuinness, D.L., Ding, L., Glass, A., Chang, C., Zeng, H., and Furtado, V. Explanation

Interfaces for the Semantic Web: Issues and Models. 3rd International Semantic Web User Interaction Workshop (SWUI’06), Athens, Georgia, 2006.

Overview of ICEE: McGuinness, D.L., Glass, A., Wolverton, M., and Pinheiro da Silva, P. Explaining Task

Processing in Cognitive Assistants that Learn. Proceedings of the 20th International FLAIRS Conference (FLAIRS-20), Key West, Florida, 2007.

Video demonstration of ICEE: http://iw.stanford.edu/2006/10/ICEE.640.mov



Thank you!(really)

Documents

Explanation and Trust for Adaptive Agents