March 5, 2009Copyright 2009 Vulcan Inc. All Rights Reserved. Overview of AURA: Automated User-centered Reasoning and Acquisition System Presentation by

March 5, 2009 Copyright 2009 Vulcan Inc. All Rights Reserved.

Overview of AURA: Automated User-centered Reasoning and Acquisition System

Presentation by

Mark Greaves

Vulcan Inc.

http://www.vulcan.com

SRI’s AURA project website has papers:

http://www.ai.sri.com/project/aura


Acknowledgment

The project is funded by Vulcan Inc. – a Paul Allen Company

Vulcan Program Management Mark Greaves Dave Gunning Benjamin Grosof


AURA Team

SRI Brian Adair (Chemistry) Sunil Mishra John Pacheco Aaron Spaulding Jing Tien

The Boeing Company Peter Clark John Thompson Phil Harrison

University of Texas at Austin Ken Barker Jason Chaw Bruce Porter Dan Tecuci

Consultants Richard Fikes Art Fortgang (Physics) Karen Hurst (Biology) Bonnie John Shirin Sohrabi Shahin Zarafshar (Biology)


Motivation

``Read a Chapter of a text and answer questions at the back of the chapter’’ Raj Reddy on Three Open Problems in AI, JACM’03

``Build a Knowledge Base by Reading a Textbook’’ Ed Feigenbaum on Some Challenges for

Computational Intelligence, JACM’03


Motivation

Inspired by Dickson’s Final Encyclopedia, the HAL-9000, and the broad SF vision of computing The “Big AI” Vision of computers that work with

people The volume of scientific knowledge has

outpaced our ability to manage it This volume is too great for researchers in a

given domain to keep abreast of all the developments

Research results may have cross-domain implications that are not apparent due to terminology and knowledge volume

“Shallow” information retrieval and keyword indexing systems are not well suited to scientific knowledge management because they cannot reason about the subject matter Example: “What are the reaction products if

metallic copper is heated strongly with concentrated sulfuric acid?” (Answer: Cu2+, SO2(g), and H2O

Response to a query should supply the answer (possibly coupled with conceptual navigation) rather than simply list 1000s of possibly relevant documents


Outline

Research Framework Requirement Analysis Implementation

Knowledge Formulation Question Formulation Question Answering

Evaluation Future Work


Research Framework

Focus on fundamental hard sciences where knowledge is explicitly written down Physics, Chemistry, and Biology

Choose a widely accepted test for competence Advanced Placement Test

The AP test is merely a metric. The system capability should be general enough to answer a broader set of questions

Scope the problem to a manageable size 50 pages of syllabus in each of the three domains


Research Framework

Automatic reading was kept out of (initial) scope Automatic techniques will not produce a fidelity of

representation needed for AP question answering Focus on the basic system and provide ways to

incorporate automatically extracted information


AURA Concept of Operations

Domain Experts Enter Knowledge

Domain Experts add knowledge to theAURA Knowledge Base and imoprtKnowledge using the mapping tool

Users ask questions and get answers and explanations

AURA

AURA Answers Questions


AURA

Automated User-Centered Reasoning and Acquisition System

Aura is a tool to help users formalize knowledge Aura can then reason with that knowledge So users can ask questions and understand the answers.


Outline



Evaluation Potential Application Future Work


Requirement Analysis

Knowledge Representation and Reasoning (KR&R) Requirements

Question Asking Requirements


KR&R Requirements

We undertook a survey of the textbook knowledge and a sample of AP exams in the three domains Systematically enumerated representation capabilities

using a KR&R Ontology (Fikes 2008) Most frequent KR&R types

Structured Objects Rules Mathematical equations Tables Diagrams Computational Knowledge


KR&R Requirements

Rules

Structured Objects


KR&R Requirements

Mathematical equations


KR&R Requirements

Tables


KR&R Requirements

Diagrams

Mitochondrion

Vesicle

Nuclear Envelope

Lysosome


Knowledge Acquisition Design Approach

Structured Objects and Rules Well-known type of knowledge Prior experience with SHAKEN system

Equations Indispensable in Physics, and Chemistry

Tables Could be stated using a conceptual knowledge

interface, but sometimes very tedious Diagrams

Very common, yet a very hard problem Most often the same knowledge can be stated using

text


Question Asking Requirements

The training requirement needs to be kept low The question asker should not have to know

about how the knowledge is represented The questions may contain scenarios


Outline





Knowledge Formulation

The textbook is embedded in the software Provides context, and starting point for formulation

The users never begin from an empty KB Knowledge Engineers provide a library of pre-built

representations The Component Library (CLIB) contains classes representing

physical actions, e.g., Move, Attach, Penetrate, and semantic relations, e.g., agent, object, has-part (Barker, Clark, Porter, KCAP’01)

Some domain-specific knowledge is pump primed

User-centered Design for UI abstractions Concept Maps

Based on extensive research in education Present a collection of rules instead of one rule at a time (Clark, et.

al., KCAP 2001, Chaudhri, EKAW 2003, KCAP 2007)


Example Knowledge












Formulated Knowledge




Mapping Graphs to Axioms

Based on prior work with Shaken & CLIB Clark et. al., KCAP’2001, Chaudhri, et. al. EKAW, 2003

(forall ?c

(=> (instance-of ?c Eucaryotic-Cell)

(exists ?x ?y ?z

(and

(instance-of ?x Nucleus)

(instance-of ?y Chromosome)

(instance-of ?z Plasma-Membrane)

(has-part ?c ?x) (has-part ?c ?y)

(has-part ?c ?z) (is-inside ?y ?x)))))


Outline





Question Formulation

Formallanguage

Unrestrictednatural

languageComputer Processable Language

“A boulder is dropped”“Consider the following possible situation in which a boulder first…”

“xy B(x)R(x,y)C(y)”

Too hard for the user

There lies a “sweet spot” between logic and full NL which is both human-usable and machine-understandable


Question Formulation Cycle

Originaltext

A boulder is dropped.The initial speed of the boulder is 0 m/s.The duration of the drop is 23 seconds.The acceleration of the drop is 7.9 m/s^2.What is the distance of the drop?.

CPL (Controlled english)

Question-Answering

Rewritingadvice

Graph & paraphrase ofsystem’s understanding

A boulder is the object of a dropping.The dropping has a duration of 23 seconds.The dropping has initial speed 23 seconds.The dropping has acceleratio 7.9 m/s^2.The dropping has a distance of unknownWhat is the distance?


Example of Question Formulation

A boulder is dropped.The initial speed of the boulder is 0 m/s.The duration of the drop is 23 seconds.The acceleration of the drop is 7.9 m/s^2.What is the distance of the drop?

An alien measures the height of a cliff by dropping a boulder from rest and measuring the time it takes to hit the ground below. The boulder fell for 23 seconds on a planet with an acceleration of gravity of 7.9 m/s2. Assuming constant acceleration and ignoring air resistance, how high was the cliff?

?


Example Feedback from the System


Outline





Question Answering

Reasoning Control Reasoning Engine

Knowledge Machine (Clark, Porter, 2009) Semantic Matching (Yeh, Porter, AAAI 2006)

Specialized Reasoning Modules Equation solving Chemical compound recognition

Explanation Generation English generation from the knowledge base


Example Answer


Example Answer

A solution that has a pH greater than 7 will result when this substance is dissolved in water.

a. sulfur dioxideb. oxalic acidc. phosphoric acidd. carbonic acide. potassium nitrate

CPLa. there is an aqueous solution of SO2.what is the pH of the solution? AnswerpH = 0

CPL b. there is an aqueous solution of H_2C_2O_4.what is the pH of the solution?AnswerpH is less than 7


Example Answer

What is the relationship between Caveolin and Muscle Cell?(Inspired by Larry Hunter’s work on Biological Discovery)


Example Answer


Outline





Evaluation

How much of the knowledge in the three domains can be captured through a generic knowledge capture and reasoning capability and to what extent does it need to be specialized for each domain? Scaling to a knowledge base of full textbook Applying the same technology to new domains


Basic AURA Performance

Domain experts in each domain constructed a KB for each of the three domains

The KB was tested on a suite of questions The questions were drawn from the 50 pages of

syllabus, and did not cover un-implemented features such as diagrams

Classes Questions Correct Response

Physics 36 137 79% 24 sec

Chemistry 284 235 73% 40 sec

Biology 189 284 71% 8 sec


Generality of Knowledge Formulation

Pump priming is needed for each domain Knowledge in prior chapters Knowledge that cannot be entered using AURA

LOC Classes Relations

CLIB 35693 533 174

Physics 3659 38 91

Chemistry 18804 674 65

Biology 1012 62 3



To what extent was the general knowledge in the CLIB useful across the three domains?

Classes Relations

Total Unique Total Unique

Physics 489 26 1769 55

Chemistry 7078 344 7322 43

Biology 1327 64 1602 56



Chemistry required the use of a reaction editor that was not used in any other domain


Generality of Question Formulation

The overall design of the question formulation dialog was the same across all three domains

Some customization was needed in the CPL across the three domains Semantic role labeling rules: If a Living-Entity Moves an Object, then the Living-

Entity is the agent of the Move Metonymy rules: ``mixed with H2O should be interpreted as mixed with a

substance whose basic unit is an H2O molecule The overall design of the question formulation dialog was the same across all

three domains Some customization was needed in the CPL across the three domains

Semantic role labeling rules: If a Living-Entity Moves an Object, then the Living-Entity is the agent of the Move

Metonymy rules: ``mixed with H2O should be interpreted as mixed with a substance whose basic unit is an H2O molecule

The overall design of the question formulation dialog was the same across all three domains

Some customization was needed in the CPL across the three domains Semantic role labeling rules: If a Living-Entity Moves an Object, then the Living-

Entity is the agent of the Move Metonymy rules: ``mixed with H2O should be interpreted as mixed with a

substance whose basic unit is an H2O molecule

Role Labeling Metonymy

General 102 15

Physics 11 3

Chemistry 9 6

Biology 3 0


Generality of Question Answering

There are multiple question types implemented in the system Computing the values of a slot is the most frequently used question type

The computation of the value of slot can vary greatly in complexity

Physics Chemistry Biology

Compute Slot Value 116 111 33

True/False 13 87 165

Superclass 1 1 38

Comparison 2 1 8

Count of Slot values 0 0 4

Definition 5 29 27

Example of a Class 0 6 0

Relate individuals 0 0 9


Generality Outside the Current Scope

Even though we built the KB only for 50 pages of syllabus, our requirement analysis covered the whole syllabus With the features currently implemented, we expect to be able to

answer 50% of the questions on an AP Exam in the three Science domains

A preliminary exercise for encoding the whole Physics textbook required 36 specific extensions to CLIB

We have analyzed three new domains Micro Economics

Similar to Physics, but also more qualitative US Government and Politics

Some similarities to Biology in the need for approximate matching of descriptions

Environmental Sciences Similar to Physics and Biology, but requires qualitative reasoning

The 50% coverage generalizes to the new domains


Independent Evaluation by BBN

IdealReference

Near-Term Target

Diagnostic(Interaction Effects)

UltimateTarget

Upper bound

Initial Indicator

of KB Quality

• The evaluation employs a 2 x 2 design to test the effects of user experience on KF and QF

• In each cell users will author KBs for a section of an AP syllabus and other users will query the KBs to answer a set of AP questions in that domain


Outline



Evaluation Future Work


Backdrop for Future Work

Direct Knowledge Entry by Domain

ExpertsUpper

Ontology

Mid-LevelTheories

Domain-SpecificTheories

KnowledgeRepresentation

(KR) Expert

DomainExperts

Parallel Development by

Distributed Teams

Knowledgeable

Usable

Embeddable

HPKB, Halo I

RKF, Halo II


Future Work

SILK KR for defaults, processes, higher-order Knowledge Factory in India Expand the expressiveness of the question

formulation interface (Boeing) Database of paraphrases

KR language specification for concept maps and composition operations (with Richard Fikes)

Computational knowledge, qualitative knowledge, and diagrams

Knowledge Debugging Applications, e.g., Bio-med


Summary

AURA is aimed to be a generic computational tool aimed at modeling knowledge in hard sciences We have good results based on 50 pages of textbook

syllabus and questions suite drawn from an advanced placement exam

For more info, e.g., papers: SRI’s AURA project website

http://www.ai.sri.com/project/aura Also: SILK project website

http://silk.projects.semwebcentral.org


Disclaimer: The preceding slides represent the views of the author only. All brands, logos and products are trademarks or registered trademarks of their respective companies.

Thank You

Documents

March 5, 2009Copyright 2009 Vulcan Inc. All Rights Reserved. Overview of AURA: Automated User-centered Reasoning and Acquisition System Presentation by