1 USC INFORMATION SCIENCES INSTITUTE EXPECT TEMPLE: TEMPLate Extension Through Knowledge Acquisition Yolanda Gil Jim Blythe Information Sciences Institute

1USC INFORMATION SCIENCES INSTITUTE EXPECT

TEMPLE:TEMPLate Extension Through

Knowledge Acquisition

Yolanda GilJim Blythe

Information Sciences InstituteUniversity of Southern California

http://www.isi.edu/expect

{gil, blythe}@isi.edu


Acquiring Planning Knowledge Problem: SOF users need to add knowledge to these

planning tools ROEs, commander’s guidance Plan evaluation/critiquing criteria Highlight the information that is important to them Add/extend templates

Approach: provide knowledge acquisition tools to adapt and extend pre-existing planning knowledge Exploit ontologies and background knowledge so users don’t

have to start from scratch KA Scripts guide the user through multiple steps Users manipulate English paraphrases of internal representations

Benefits: Users can extend the tool’s baseline knowledge for the operation


Prototype for adding plan critiques: Expect’s PSM Tool

Questions formulatedbased on background

knowledge

User adds detailed knowledge through English paraphrases


The next 100 days

Allow users to specify and customise “sentinels” that check for new information and alert planners when needed. Our tools generate Java.

Extend ontologies and background knowledge to handle SOF domain.

Integrate with one of the jumpstart applications, probably the travel planning tool, using InterAcT.


Backup slides Description of approach, tools and experiment

from HPKB project.


Key Technologies Guiding users through knowledge acquisition scripts

[Tallis and Gil 99] that capture typical dialogues that users follow to enter new knowledge step by step

Exploiting domain-independent background knowledge about plan evaluation and critiquing [Blythe & Gil 99] that use background knowledge about plan evaluation and critiquing to guide the dialog

An English-based editor [Blythe & Ramachandran 99] that lets the user add or modify internal knowledge by manipulating its English paraphrase, without having to see or understand the internal formal representation


Architecture of TEMPLETemple

(Server)ConstraintAcquisition UI(Client)

Constraintwizard

Constraintviewer

AcquisitionScripts

Background knowledge

Ontologies¤ constraint types¤ actions and plans¤ proactive

suggestions

Method base

Compiler

Executableconstraints

Templatelibrary

Active Templates Toolkit

SOF

Englisheditor

NaturalLanguageGenerator

Domainknowledge

DomainconstraintsDomainmethods

Domainmodelsandtemplates


Evaluation and Critiquing KnowledgePlan ontology(PLANET)

Ontology ofcritiques

Submethods for checking plan resources

Submethods for checkingplan structure

Reused knowledge(ontologies

and methods)

Domain-specificcritiques

Domain-specific submethods

Domain-specific plan critiquing and evaluation system

Domain-specific knowledge

Ontology ofresources


An Ontology of Plan Evaluation Criteria

Captures general knowledge of how to evaluate plans with respect to standard norms of plan development

Ill-formeddescription

Statementcritique

Linkcritique

Structurecritique

Complete statement

Correct statement

Clear statementako

ako

ako

Correct linkako Does the purpose of supporting effort X

support the main effort?isa

Does <unit> have sufficient combatpower to accomplish its mission?

isa

Complete planako

...

...isa Does the COA include a statement

of the reserve forces?


KA Scripts Helps the user add new critiques by using a background

theory of plan evaluation and critiquing. KA Scripts guide the dialog with the user about the new

critique (wizard-type interaction). The tool creates some of the needed methods for the

critiques, helps the user to create new ones (by suggesting initial templates), and ensures consistency with existing knowledge.


English-Based EditorGenerates automatically English paraphrases of problem-solving fragments, and presents alternative text to replace parts ofthe paraphrase based on the ontologies and background knowledge

NL description of method

Alternatives forselected text fragment


First experiment: An ablation test on the PSM-Based KA Scripts Hypothesis: the PSM-Based KA Scripts significantly

reduce the expertise and the effort required to add a new critique to the knowledge base.

KA tasks: add two new critiques to the EXPECT COA critiquer (a completeness check and a resource check)

Knowledge (and tool) ablation experiment: Two tasks done using PSM-Based KA Scripts, two tasks done without

Subjects: four Army officers, previously trained on EXPECT’s language for a day


Sample Tasks Given to Subjects Simple critique: Add a critique to check if the COA

has a security statement. Complex critique: Add a critique to check if each

task in the COA has sufficient force ratio. To compute force ratio, divide the sum of combat

powers of the Blue units assigned to the task by the sum of combat powers of the Red units acted on by the task.

(Two other comparable tasks were also used)


Quantitative results: what users could do Users could complete more tasks using the PSM-

based KA scripts

0

5

10

15

20

25

30

35

easiertask 1

easiertask 2

morecomplextask 1

morecomplex

task 2

All

with PSMToolablated version

LEGEND: indicates total tasks


Quantitative results: speed improvements Time reduction using the PSM-based KA Scripts

02468

101214161820

Completenesscritique

(simpler)

Resourcecheck critique

(more complex)

ablatedversionwith scripts

Time inminutes


Axiom acquisition rates:Experiment with PSM-Based KA Scripts

Adding small amounts of new knowledge

Adding larger amountsof new knowledge

with PSM-BasedKA Scripts

2.12 ax/min

1.26 ax/min

with ablated version

1.1 ax/min

N/A (users were notable to do tasks)


Summary Using the PSM-Based KA Scripts significantly

reduced the time taken to add a critique Using the PSM-Based KA Scripts, all four subjects

successfully added simple critiques to the EXPECT critiquer; three of them successfully added more complex critiques.

Without the PSM-Based KA Scripts, three out of four subjects successfully added simple critiques and two added more complex critiques.

Comments on the tool usability were positive in all cases.


Second experiment with PSM-Based KA Scripts and English-Based Editor Hypothesis: the combination of the PSM-Based KA

Scripts and English-based editor allows a user with very little training to add new critiques.

Single subject usability test: A subject was briefed in COA critiquer and the KA interface (but not about EXPECT) for 20min and asked to add two critiques using the tool

KA tasks: add two new critiques to the EXPECT COA critiquer (a completeness check and a resource check), same used in the previous experiment

Subject: an Army officer with no EXPECT training


Results The subject was able to add two new critiques of both

low and medium complexity. The time taken was comparable to that for the other

four subjects that had previous training in Expect:

02468

101214161820

Completenesscritique

(simpler)

Resourcecheck critique

(more complex)

new user

Average ofother usersTime in

minutes


EXPECT: A User-Centered Framework for Developing KBSs

Method instantiator

Knowledge BaseDomain

ontologiesand factual knowledge

Problemsolving methodsDomain

dependentKBS

KBS compiler

Knowledge-BasedSystem

InterdependencyModel (IM)

EXPECT Ontologies and Method libraries

Plans(PLANET)

Evaluations and Critiques

Evaluation PSMs

Resources(OZONE)

COAontologiesCYC/Sensus

Upper

Instrumentation

KA Strategies

KA toolsEMeD

KA ScriptsPSMTool


EXPECT: A User Centered Approach for Knowledge-Based Planning ToolsKnowledge acquisition technology that can guide users to

specify planning knowledge and develop planning tools Expressive representations

– Loom/Powerloom KR&R– EXPECT’s language to represent problem solving knowledge

Powerful reasoners– Loom/Powerloom pattern classifier & reasoners– abstract problem solving through partial evaluation

ex: how to move <a set of units> from a <location> to another <location> Explicit models of planning knowledge and plan reasoners:

– PLANET ontology of plans, OZONE resource ontology– plan evaluation and planning methods

Expectation-based knowledge acquisition tools– Derive interdependencies between individual knowledge fragments– KA Scripts to guide users in completing complex modifications

Documents

1 USC INFORMATION SCIENCES INSTITUTE EXPECT TEMPLE: TEMPLate Extension Through Knowledge Acquisition Yolanda Gil Jim Blythe Information Sciences Institute