Symbolic Supercomputerfor

Artificial Intelligenceand

Cognitive ScienceResearch

Kenneth D. Forbus

Dedre Gentner

Northwestern University

Overview

• Why symbolic supercomputing?• Off-line experiments

– Work in progress: Large-scale corpus analysis

– Distributed experiments harness

• Interactive Cognitive Architecture experiments– Companion Cognitive Systems (DARPA)

– Explanation Agent

Off-line experiments

• Sensitivity Analysis– Every cognitive simulation has parameters

• Analyzing how performance depends on parameters important for understanding models

– Sensitivity analyses can be expensive• 1994 MAC/FAC simulations took weeks of CPU time

• 2000: 4.8 million SME runs in SEQL sensitivity analyses took 23 days (400 mhz PII), should be 4 days today.

• Corpora Analyses– Text

– Sketches

– Problems

Larger-scale simulations

• Goal: Increased use of automatically generated inputs– Reduce tailorability

– Increase # of stimuli generated and used.

• Processes – Analogical Encoding

– Conceptual problem solving

Symbolic models and parallelism

• Our approach is based on Gentner’s (1983) structure-mapping theory– Assumes parallel processing both within modules and

between modules– Currently emulate on serial processors

• Coarse-grained parallelism could provide important benefits– Continue to simulate within-module parallelism on

single CPUs– Exploit parallel processing between modules

• Incrementally update retrievals during reasoning• Incrementally construct generalizations during reasoning• Reason about domain, interactions, and self in parallel

Traditional Supercomputers ineffective for symbolic processing

• Optimized for– Floating-point processing

– Pipelined, with vector or grid model okay CPUs, low RAM, fast floating point

• Symbolic processing– Involves many pointer operations

– Some floating-point, but over irregular structures (graphs, sparse-vectors)

fast CPUs, high RAM, okay floating point

Optimizing a cluster for symbolic processing

1. Use the fastest CPU available.

2. Distribute the processing in large, functionally-organized units. – Avoid communication overhead– Data-parallel programming style poor fit for

clusters– Replicate knowledge base as needed

3. Organize memory to be as fast as possible.– Maximize RAM, cache– Avoid virtual memory

Why large memories are crucial

• If a program is going to know a lot, it has to put it somewhere

• Example: Subset of Cyc KB contents we use– 35,070 concepts, 8939 relations, and 3,917 functions

– 1,283,835 axioms, divided into 3,537 microtheories

– Added knowledge (DARPA HPKB, CPOF, RKF)• Military tasks, units, equipment

• Countries, international relationships, terrorist incidents

• Qualitative models, terrain, trafficability, visual representation conventions, developed by our group

– Takes roughly 495 MB of storage, due to indexing overhead

• May double in size as we learn by accumulating experiences

Mk2

• Hardware: Linux Networx– 5 year maintenance

contract

• 67 nodes– Dual 3.2Ghz Xeon CPUs

– 3GB RAM/node

– 80GB disk/node

• Allegro Common Lisp for Linux– Provides flexible

development environment

Mk2 Cluster Network

Node 3

Node 2

Node 1

Node 67

Master Host

Cisco PIXFirewall/Router …

Backplane S

ubnet

Frontplane S

ubnet

PublicInternet

Gigabit switched EthernetPacket

filtering, trusted

whitelist of hosts

One-command provisioning, P2P data distribution

system

Qualitative reasoning for

intelligent agents

(ONR AI Program)

ObjectiveCreate science base for intelligent

software agents that can

• Reason about the physical phenomena and systems in a human-like way

• Extend their knowledge incrementally, by communicating with human collaborators in natural language.

Technical Approach• Develop qualitative reasoning techniques for

solving problems under time pressure with partial, incomplete knowledge (“back of the envelope” reasoning)

• Explore the use of qualitative representations as part of the semantics for a natural language system.

• Develop techniques to assimilate controlled-language reports to extend an agent’s models of the physical world.

Knowledge Base(general knowledge + libraries of cases)

ExplanationAgent

New examples

Queries

Situationupdates

Model of ongoingsituation/system

Estimates,warnings

QP Theory in Natural Language Semantics• Idea: Qualitative Process theory can be

used as a framework for understanding NL descriptions of physical phenomena.

– Right level of abstraction– Consistent with human mental models – Support for compositionality

• Approach– Identify syntactic patterns corresponding

to QP theory concepts via corpus analysis

– Recast QP theory in terms of frames– Use controlled subset of English to

simplify parsing, focus on semantics

• Current status– NL system translates paragraph sized

texts about physical processes into formal representations

– Tested on a dozen examples

• Next steps– Expand range of texts handled– Develop knowledge assimilation

techniques to construct knowledge bases by reading multiple texts

(1) A pipe connects cylinder c1 to cylinder c2.

(2) Cylinder C1 contains 5 liters of water.

(3) Cylinder C2 contains 2 liters of water.

(4) Water flows from cylinder C1 to cylinder C2, because the pressure in

cylinder C1 is greater than the pressure in cylinder C2.

(5) The higher the pressure in cylinder C1, the higher the flowrate of the water.

(6) When the pressure in cylinder C2 increases, the flowrate of the water decreases.

Type:

(isa flow3606 Translation-Flow)

Participants:

(isa c1 Container) [QuantityFrame q3609] (isa c2 Container) [QuantityFrame q3603]

Conditions:

(> (pressure c1) (pressure c2))

Quantities: 

[QuantityFrames q3608 and q3605]

Consequences:

(qprop (flowrate flow3606) (pressure c1)) (qprop- (flowrate flow3606) (pressure c2))

(I- (water c1) (flowrate flow3606)) (I+ (water c2) (flowrate flow3606))

C1 C2

The EA natural language system

QP FramesQP Frames

QP Theory

constraints

QP Theory

constraints

Patterns for QP-specific

constituents

Patterns for QP-specific

constituents

Only 15 out of ~100grammar rules are

QP-specific

ProcessRules

ProcessRules

Process Frame Construction

Process Frame Construction

KBKB

QRG-CEgrammarQRG-CEgrammar LexiconLexicon

FactsFacts

WSDDataWSDData

ParserParser Word-SenseDisambiguation

Word-SenseDisambiguation

FrameRulesFrameRules

MergeRulesMergeRules

Frame Construction

Frame Construction

Input text

Input text

Retrieval of semantic

information

Retrieval of semantic

information

1.2 million

fact subset of

Cyc

SvenKuehne’s

Ph.Dthesis

Corpus Analysis (in progress)

• Kuehne and Forbus (2002) used by-hand corpus analysis to identify syntactic patterns

– Four chapters of an introductory science book, 216 sentences total

– 43% of the material in physical explanatory text could be captured via QP theory.

• Do the syntactic patterns that we found for explanatory physical texts apply to everyday texts?

– If they do, what is their coverage?

– How many more patterns are there?

Looking for quantities

• 1999 volume of the New York Times, consisting of 6.4 million sentences • First stage used 30 word list for filtering (7.5 hours)

– ~172,000 sentences output

• Second stage used regular expressions (12 hours)– Derived from vocabulary and syntactic patterns from previous corpus analysis. – Result: ~19,000 sentences worth examining more closely

• Third stage uses modified version of our Explanation Agent NLU system (less than 2 days, 17 hours, on 3 nodes)– Previously, used Quantity and PhysicalQuantity– Generalized to the Cyc concept ScalarInterval,

• Subsumes temperament, monetary values, feeling attributes, formality/politeness of speech, plus others.

• 14,000+ quantities found.– 0.2% of the sentences mention a recognizable quantity– Lexicon limitations may have a strong effect here

• Expanding it via hand-labor (Cycorp) plus co-training is probably necessary• e.g., “intensification of the war effort”

Qualitative changes in the New York Times

• Starting point: Corpus of 6.4 million sentences • Filter using word list of 89 synonyms for

increases, 66 for decreases (~10 hours each)– 62,117 candidate sentences mentioning decreases – 195,452 candidate sentences mentioning increases – Around 4% of corpus– Contrast: 43% of the material in physical explanatory

text could be captured via QP theory.• Larger analysis only concerns qualitative proportionalities • Qualitative representations may play a smaller role in

understanding political texts versus physical texts.• Genre differences: newspapers versus explanatory material

– E.g., “(X i.e., Y)” common on web, not in newspapers

Dexp: Distributed Experiment Tool

• Provides support for running distributed experiments– Written in Common Lisp

– Uses sandbox to avoid configuration issues

• Experimenter divides computation into work units– Example: For N queries, find all of the solutions to

them

– Provides list of work units to dexp as a file, along with a startup file and code tree to use

– Gets back a set of files containing the results.

dexp Architecture

• Experiment Coordinator– Manage distribution and

execution of work units– Collect results

• Experiment pool nodes– Executes a work unit, returns

results.– Execution uses sandbox for

configuration control

• Load Balancer– Dynamically allocate nodes

for work units– Will balances demands from

multiple simultaneous experiments

Loadbalancer (*)

distributed experiment pool

n31n33

n34n15

n65n66

Coordinator

How dexp simplifies experiments: Example

• A experiment analyzing semantic translations in ResearchCyc KB consisted of ~1200 work units– Each consisted of a query to see how many examples in

the KB satisfied the semantic patterns given for verbs

• With 24 nodes, most of the experiment was completed in 34 minutes– Estimate: 11 hours on a single CPU, if no failures

• Five work units churned for 12 hours, failed to finish due to heap blow-out– Most of the results were available quickly– Much easier to diagnose what was going wrong, instead

of waiting for hours to hit a failure.

Companion Cognitive Systems

A new cognitive systems architecture

• Robust reasoning and learning– Companions will learn about

their domains, their users, and themselves.

• Longevity– Companions will operate

continuously over weeks and months at a time.

• Interactivity – Companions will be capable of

high bandwidth interaction with their human partners. This includes taking advice.

– Sketching is a majorinteraction modality

Central hypotheses• Analogical processing will

enable us to create systems with human-like learning and reasoning abilities– Able to handle relational

information– Able to incrementally adapt

and extend their knowledge– Able to apply what they learn

in one domain to other domains

• Using a cluster can make an analogical processing architecture fast enough to be used in interactive systems– Changes the kinds of

experiments that become feasible as well.

Colossus(DARPA, 5 nodes)

Mk2(ONR, 67 nodes)

Companions as Structure-Mapping Architecture

Psychological Bets

• Ubiquitous use of structure-mapping for reasoning and learning– SME for matching

– MAC/FAC for similarity-based retrieval

– SEQL for generalization

• Qualitative representations play central role– Part of visual structure in

spatial reasoning

– Representation of causal knowledge and arguments

Engineering Choices

• Distributed agent architecture using KQML

• Logic-based TMS for working memory

• No hardwired working-memory capacity limits

Companion Architecture Year One

FacilitatorSessionReasoner

MAC/FACDomainTickler

nuSketch System(sKEA or nuSketch

Battlespace)

RelationalConcept

Map

SessionManager

Cluster

User’s Windows box

Master nodeNode Node

w/Thomas Hinrichs, Jeff Usher, Matt Klenk, Greg

Dunham, Emmett

Tomai, Tom Ouyang,

Hyeonkyeong Kim, and

Brian Kyckelhahn

Bennett Mechanical Comprehension Test• Widely used standardized exam

for technicians • Used in cognitive psychology

as indicator of spatial ability • Difficulty lies in breadth of

situations, not narrow technical knowledge

• Best score to date: 10 correct out of a subset of 13 BMCT problems (77%). [P < 0.001]

Q: Which crane is more stable?

Example describes how

physical principles apply to a real-world situation

Analogies with

example provides causal models

needed for solution

Analogical inferences

are surmises,

not certainties

Suggesting visual/conceptual relations by analogy

MAC/FAC

Knowledge Base(including case libraries of examples)

CandidateInferenceExtraction

SuggestionsFiltering

109 candidat

es

184 candidat

es

189 candidat

es

109 candidat

es

Visual/Conceptual Relations: Experimental Results

• Ex1: Focused Tasking– 54 sketches (18 situations

drawn by three KEs) as case library for BMCT experiment

• Round Robin method: For each sketch, remove from library, remove its VCR answers, generate suggestions via analogy– Yielded “exam” of 181

VCR questions– Score = 74.25 (P << 10-5)– Coverage = 54%– Accuracy = 87%

• Ex2: Open tasking– 10 situations selected from

BMCT problems, covering larger range of phenomena (e.g., “a boat moving in water”, “a bicycle”)

– Each situation sketched by two graduate students, told to illustrate the principle(s) you think are important.

• Round Robin method– Yielded “exam” of 138

questions– Score = 21.75 (P < 10-7)– Coverage = 46%– Accuracy = 57%

Facilitator

Executive

nuSketch GUI RelationalConcept

Map

SessionManager

Cluster

User’sWindows box

MAC/FACDomain Model

Tickler

SessionReasoner

DialogueManager

HeadlessnuSketch

SEQLDomain

Generalizer

MAC/FACSelf Model

Tickler

SEQLSelf ModelGeneralizer

MAC/FACUser Model

Tickler

SEQLUser ModelGeneralizer

InteractiveExplanation

Interface

OfflineLearningOffline

LearningOfflineLearning

CompanionsArchitecture as of 9/05

Explanation Agent Prototype

• Use Companions Architecture as infrastructure• Incorporate other ONR advances

– EA NLU system (Sven Kuehne)

– Back of the envelope reasoning (Praveen Paritosh)

– Spatial prepositions model to link language and sketches (Kate Lockwood)

– Analogical Problem Solver (Tom Ouyang)

• Use for cognitive simulations – Natural language, sketching for stimulus input

Back of the Envelope Reasoning (Paritosh)

• Qualitative representations essential for framing the problems, supporting comparisons

• Analogical reasoning used to find similar situations for estimation models, construct qualitative representations via generalization over experience

How much

oxygen is left?

How longto repair

it?

Is anyone still alive in there?

Goal: Develop theories that enable software to reason quantitativelyin real-world situations

Estimate parameter directly

Create estimation model

Find modeling strategy

Find values forparameters

in model

Use known valueif available

Estimate basedon similar situation

Feel for numbers

Problem solving

Back of the Envelope Reasoning Progress

• Implemented BoTE-Solver– Solves 13 problems to date

• Examples• How many K-8 school

teachers are in the USA?

• How much money is spent on newspapers in USA per year?

• What is the total annual gasoline consumption by cars in US?

• What is the annual cost of healthcare in USA?

• How much power can an adult human generate?

• Claim: There is a core collection of strategic knowledge, specifically, seven strategies that capture most of back of the envelope reasoning.

• Source: – Strategies in Bote-Solver

– Analysis of all problems (n=44) from Force and Pressure, Rotation and Mechanics, Heat and Astronomy from Clifford Swartz’s Back-of-the-Envelope Physics.

C1

CARVE: Using analogy to generate qualitative representations

Dimensional partitioning for each quantity(k-means clustering)

(isa Algeria (HighValueContextualizedFn Area AfricanCountries)..

Add these facts to original cases

Structural clustering using SEQL

C1

Input cases

Cj

Quantity 1

L2L1

S3S1

S2

Cases + structural limit points and distributional partitions

Analogical Estimation

• Analogical estimator: makes guesses for a numeric parameter based on analogy.

(GrossDomesticProduct Brazil ?x)– The value is known.

– Find an analogous case for which value is known.

– Find anything in the KB which might be a basis for an estimate.

• Hypothesis: Representations augmented with symbolic representation will lead to more accurate estimates.

Basketball Stats Domain

• Quantities (e.g., points per game, rebounds per game, assists per game, etc.)

• Causal relationships– Being taller helps being able to rebound and block– Power forwards are taller and are expected to shoot,

rebound and block– Being good at getting 3 point field goals means one is a

good shooter, so their free throw success rates will be higher.

• Case library– 15 players from different positions on field– 11 facts per player

(seasonThreePointsPercent JasonKidd 0.404) (qprop seasonThreePointPercent seasonFreeThrowPercent BasketballPlayers)

Results: Errors

0

10

20

30

40

50

60

70

80

Height

Assist

s

Free

thro

ws

Point

s

Reboun

ds

Three

point % All

Enriched mean % error

Raw mean % error

SpaceCase: Motivation

• Recent research points to role of non-geometric properties in spatial preposition use – Coventry 1994; Coventry &

Prat-Sala, 1999; Herskovitz, 1986; Feist & Gentner, 2003; Garrod et al., 1999; Coventry & Garrod, 2004; Carlson & van der Zee, 2005

• Spatial language can affect retrieval of pictures – Feist and Gentner, 2001

• Multimodal interfaces potentially useful for military needs– Language plus diagrams,

other spatial displays

• Software’s notion of similarity needs to be like their human partners– Including visual properties

– Including retrieval, for shared history

– Including shared language

Lockwood, K., Forbus, K., and Usher, J. SpaceCase: a Model of Spatial Preposition Use

Proceedings of CogSci-05, to appear

sKEA Sketching Interface

sKEA Sketching Interface

medium_curvatureground_supports_

figure

dish

firefly

firefly -> insect -> animate

functions as weak container

Sketch corpus crucial for model development

• Building a corpus of sketches– Gathering library of examples from literature

– Use sKEA to capture them in machine-understandable form

– Estimate: ~ 200 sketches will be needed to cover the set of prepositions and phenomena to be modeled

• Cluster will be used for – Regression testing

– Sensitivity analyses: How does performance depend on parameter values?

Problem-solving experiments

• Starting point: Pisan’s (1998) Thermodynamics Problem Solver– Solved 80% of the problems typically found in first four chapters

in engineering thermodynamics textbooks

– Used graphs and property tables

– Produced human-like solutions

• Generalize: Analogical Problem Solver– Focus on conceptual comprehension questions

– Declarative strategies now include analogical processing• when/what to retrieve, what candidate inferences to use, level of

effort in testing

– Experiment in progress: Can strategy variations explain novice/expert differences?

• Pilot results promising, should have full data by end of summer.

Questions?

Technology Transfer

The Whodunit Problem

• Goal: Generate plausible hypotheses about who performed an event.

• Formal version: Given some event E whose perpetrator is unknown, construct a small set of hypotheses {Hp} about the identity of the perpetrator of E. – Include explanations as to

why these are the likely ones

– Able to explain on demand why others are less likely.

Assumptions & Limitations

• Formal inputs. Structured descriptions, including relational information, expressed in CycL.

• Accurate inputs. • One-shot operation. No

incremental updates.• Passive operation.

Doesn’t generate differential diagnosis information

Method 1: Closest Exemplar

1. Use MAC/FAC to retrieve events similar to E.

2. For each similar event, remove it if it doesn't include a candidate inference about the perpetrator.

3. Iterate until enough hypotheses are generated.

4. (Optional) Generate explanations and expectations by analyzing the similarities and differences between each Hp and E.

Probe

SME

SME

SME

CVmatch

CVmatch

CVmatch

CVmatch

Memory pool

Output =memory

item+ SME results

Cheap, fast, non-structural

MAC/FAC models similarity based retrieval

• Scales to large memories• Accounts for psychological phenomena

• Memory pool = All cases concerning the 98 perpetrators, minus the test set.

Method 2: Closest Generalization

• Preprocessing:1. Partition case library

according to perpetrator.2. Use SEQL to construct

generalizations for each perpetrator.

• Generating hypotheses:

1. Given an incident E, pick the n closest generalizations, as determined by SME's structural evaluation score.

Exemplars…

GeneralizationsSME

NewExample

SEQL

SEQL models generalization• Assimilate new exemplars into a

generalization when close enough.• Models psychological data, used to

made successful predictions of human behavior.

• Recent extension: use probability to improve noise immunity

Whodunit Experiment

• Used 3,379 terrorist incidents from Cycorp’s Terrorist knowledge base– Between 6 and 158

propositions per case, 20 on average

• 98 perpetrators involved in at least 3 incidents in the TKB– Pick one incident at

random for test set, remove perpetrator

• Elaborate via inference– Add attributes (e.g., (CityInCountryFn Italy)) using genls hierarchy

• Three performance levels:– Best bet– Top 3: Best plus

plausible alternatives– Top Ten list: Foci for

additional collection, analysis

Whodunit Example

Whodunit Results

Correctness

0%

10%

20%

30%

40%

50%

60%

MAC/FAC SEQL SEQL+P

Top-10Top-3Correct

Adding probability yielded 5% improveme

nt

Pure retrieval

surprisingly good

Symbolic generalization adds valve for weaker criteria

Background Material

Basketball Stats Estimation by Analogy

Given: An estimation problem (seasonThreePointsPercent JasonKidd ?x) and a case library

Find the most similar player to JasonKidd in the case library for whom we know the value for seasonThreePointsPercent.

Use that as an estimate for the given problem.

Compare accuracy over the initial case library, and the case library enriched with representations from CARVE.

SpaceCase

KB

sKEAinput stimulus

inkprocessing

routines

Evidence Rules

Bayesianupdatingalgorithm

Spatial Preposition Label

Performance

• Labeling task (Feist & Gentner, 2003)– <figure> is in/on the

<ground>

• 36 total stimuli– {firefly, coin}

– {bowl, dish, plate, slab, rock, hand}

– {low, medium, high}

• Consistent on all 36 trials for values of parameters given

Modeling a spatial language/memory interaction

• Feist and Gentner (2001)• Use spatial preposition when

showing someone a situation

• Given novel stimulus, they are more likely to claim they have seen it before

• Use SpaceCase to confirm unsuitability of original stimuli for ON

• Retrieval via MAC/FAC– Initial sketch plus variants

stored as memory– Initial as probe retrieves

itself– Initial plus relation for

spatial preposition retrieves plus variant

“On”

initial sketch 0.363

plus variant 0.859

minus variant 0.2428

SpaceCase next steps

• Expand model – more prepositions

– more complex input

• Cross-linguistic modeling

auf

an