The SNePS Approach to Cognitive Robotics

S.C. Shapiro

cse@buff

alo

The SNePS Approach to Cognitive Robotics

Stuart C. Shapiro

Department of Computer Science and Engineering

and Center for Cognitive Science

University at Buffalo

[email protected]

S.C. Shapiro

cse@buff

alo

Outline• Introduction• Intensional Representation & Propositions• SNePS Connectives and Quantifiers• SNeRE Acting Constructs• Example Plans • Representation and Use of Indexicals• A Personal Sense of Time• Summary

S.C. Shapiro

cse@buff

alo

Goal

• A computational cognitive agent that can:– Understand and communicate in English; – Discuss specific, generic, and “rule-like” information;– Reason;– Discuss acts and plans;– Sense;– Act;– Remember and report what it has sensed and done.

S.C. Shapiro

cse@buff

alo

Embodied Cassie

• A computational cognitive agent– Embodied in hardware– or Software-Simulated– Based on SNePS and GLAIR.

S.C. Shapiro

cse@buff

alo

SNePS• Knowledge Representation and Reasoning

– Propositions as Terms

• SNIP: SNePS Inference Package– Specialized connectives and quantifiers

• SNeBR: SNePS Belief Revision

• SNeRE: SNePS Rational Engine

• Interface Languages– SNePSUL: Lisp-Like– SNePSLOG: Logic-Like– GATN for Fragments of English.

S.C. Shapiro

cse@buff

alo

GLAIR Architecture

Knowledge Level

Perceptuo-Motor Level

Sensory-Actuator Level

NL

ActionPercepts

Grounded Layered Architecture with Integrated Reasoning

SNePS

S.C. Shapiro

cse@buff

alo

Interaction with CassieEnglish

(Statement, Question, Command)

(Current) Set of Beliefs[SNePS]

(Updated) Setof Beliefs[SNePS]

Actions[SNeRE]

(New Belief)[SNePS]

English sentence expressingnew belief answering question reporting actions

Answer[SNIP]

GATN Parser

GATN Generator

ReasoningClarification DialogueLooking in World

Reasoning

S.C. Shapiro

cse@buff

alo

Cassie, the BlocksWorld Robot

S.C. Shapiro

cse@buff

alo

Cassie, the FEVAHR

S.C. Shapiro

cse@buff

alo

FEVAHR/Cassie in the Lab

S.C. Shapiro

cse@buff

alo

FEVAHRWorld Simulation

S.C. Shapiro

cse@buff

alo

UXO Remediation Cassie

CassieCorner flag

NonUXO object

Corner flag

UXO

Batterymeter

Corner flag

Drop-off zone

Field

Safe zone

RechargingStation

S.C. Shapiro

cse@buff

alo

Crystal Space Environment

S.C. Shapiro

cse@buff

alo


S.C. Shapiro

cse@buff

alo

Entities, Terms, Symbols, Objects

• Cassie’s mental entity: a person named Bill

• SNePS term: B5• Object in world:

S.C. Shapiro

cse@buff

alo

Intensional Representation

Intensional entities are distincteven if coreferential.

“The morning star is the evening star.”

“George IV wondered if Scott was the author of Waverly.”

S.C. Shapiro

cse@buff

alo

McCarthy’s Telephone Number Problem

Mary's telephone number is Mike's telephone number.

I understand that Mike's telephone number is Mary's telephone number.

Pat knew Mike's telephone number.

I understand that Pat knew Mike's telephone number.

Pat dialed Mike's telephone number.

I understand that Pat dialed Mike's telephone number.

S.C. Shapiro

cse@buff

alo

Answering the Telephone Number Problem

Did Pat dial Mary's telephone number?

Yes, Pat dialed Mary's telephone number.

Did Pat know Mary's telephone number?

I don't know.

S.C. Shapiro

cse@buff

alo

Representing Propositions

Propositions must befirst-class entities of the domain

Represented by terms.

S.C. Shapiro

cse@buff

alo

Discussing Propositions

That Bill is sweet is Mary's favorite proposition.

I understand that Mary's favorite proposition is that Bill is sweet.

Mike believes Mary's favorite proposition.

I understand that Mike believes that Bill is sweet.

S.C. Shapiro

cse@buff

alo


S.C. Shapiro

cse@buff

alo

Logic for NLU &Commonsense Reasoning

Either Pat is a man or Pat is a woman or Pat is a robot.

I understand that Pat is a robot or Pat is a woman or Pat is a man.

Pat is a woman.

I understand that Pat is a woman.

What is Pat?

Pat is a woman and Pat is not a robotand Pat is not a man.

S.C. Shapiro

cse@buff

alo

Representation in FOPL?

Man(Pat) Woman(Pat) Robot(Pat)

S.C. Shapiro

cse@buff

alo



but don’t want inclusive or

S.C. Shapiro

cse@buff

alo



but don’t want inclusive or

Man(Pat) Woman(Pat) Robot(Pat)+ +

T T T

F

TSo don’t want exclusive or either

S.C. Shapiro

cse@buff

alo

andor

andor(i, j){P1, ..., Pn}

True iff at least i, and at most j of the Pi are True

S.C. Shapiro

cse@buff

alo

thresh

thresh(i, j){P1, ..., Pn}

True iff either fewer than i,

or more than j

of the Pi are True

Note: thresh(i, j) ~andor(i, j)

S.C. Shapiro

cse@buff

alo

or-entailment

{P1, ..., Pn} v=> {Q1, ..., Qn}

True iff for all i, j Pi Qj

S.C. Shapiro

cse@buff

alo

and-entailment

{P1, ..., Pn} &=> {Q1, ..., Qn}

True iff for all j

P1 &…& Pn Qj

S.C. Shapiro

cse@buff

alo

Numerical entailment

{P1, ..., Pn} i=> {Q1, ..., Qn}

True iff for all j

andor(i, n){P1, …, Pn } Qj

S.C. Shapiro

cse@buff

alo

Universal Quantifier

all(ū)({R1(ū),..., Rn(ū)} &=> {C1(ū),..., Cm(ū)})

Every ā that satisfies

R1(ū)&…& Rn(ū)

also satisfies

C1(ū),..., Cm(ū)})

S.C. Shapiro

cse@buff

alo

Numerical Quantifiers

nexists(i,j,k)(x)

({P1(x),..., Pn(x)}: {Q(x)})}

There are k individuals that satisfy

P1(x) ... Pn(x)

and, of them, at least i and at most j also satisfy

Q(x)

S.C. Shapiro

cse@buff

alo


S.C. Shapiro

cse@buff

alo

MENTAL ACTS

• Believe(proposition)• Disbelieve(proposition)

S.C. Shapiro

cse@buff

alo

Act Selection

• Do-One({act1 ... actn})

• Snif(if(condition, act),

...

if(condition, act)

[else(act)])

S.C. Shapiro

cse@buff

alo

Act Iteration• Do-All({act1 ... actn})• Sniterate(if(condition, act),

...

if(condition, act),

[else(act)])• Snsequence(act1, ..., actn)• Cascade(act1, ..., actn)• P-Do-All({act1, ..., act2})

S.C. Shapiro

cse@buff

alo

Entity Iteration

WithSome(var, suchthat,

do, [else])

WithAll(var, suchthat,

do, [else])

WithSome+(var, suchthat,

do, [else])

WithNew(vars, thatare, suchthat,

do, [else])

S.C. Shapiro

cse@buff

alo

Proposition/Act Transformers

• Achieve(proposition)• ActPlan(act, plan)• GoalPlan(proposition, act)• Precondition(act, proposition)• Effect(act, proposition)• WhenDo(proposition, act)• WheneverDo(proposition, act)• IfDo(proposition, act)

S.C. Shapiro

cse@buff

alo


S.C. Shapiro

cse@buff

alo

Conditional Plans If a block is on a support then a plan to achieve

that the support is clear is to pick up the block and then put the block on the table.

all(x, y) ({Block(x), Support(y), On(x, y)} &=> {GoalPlan(Clear(y), Snsequence(Pickup(x), Put(x, Table)))})(STRIPS-like representation: No times)

S.C. Shapiro

cse@buff

alo

Use of Conditional Plan

GoalPlan(Clear(B),

Snsequence(Pickup(A),

Put(A, Table)))Remember (cache) derived propositions.

S.C. Shapiro

cse@buff

alo

Use of Conditional Plan

GoalPlan(Clear(B),

Snsequence(Pickup(A),

Put(A, Table)))???SNeBR to the rescue!

S.C. Shapiro

cse@buff

alo

A FEVAHR Acting Ruleall(a, o) ({Agent(a), Thing(o)}

&=> {Precondition(Follow(a, o), Near(a, o)),

GoalPlan(Near(a, o), Goto(a, o)),

Precondition(Goto(a, o), Lookat(a, o)),

ActPlan(Lookat(a, o), Find(a, o))})

Uses a temporal model.

S.C. Shapiro

cse@buff

alo

Acting According to the Rule

S.C. Shapiro

cse@buff

alo


I found a red robot.I am looking at a red robot.

Follow a red robot.

S.C. Shapiro

cse@buff

alo


I went to a red robot.I am near a red robot.I am following a red robot.

I found a red robot.I am looking at a red robot.

Follow a red robot.

S.C. Shapiro

cse@buff

alo

A Plan for Blowing up UXOsall(a)(Agent(a) =>ActPlan(Blowup(a, UXOs), Act(a,Cascade(SearchforUxo(a),

WithSome+(obj, Near(a, obj), WithNew({ch ex},

{Charge(ch), Explosion(ex)}, Possess(a, ch), Cascade(Place(a, ch, obj), Hide(a),

Waitfor(a, ex), SearchforUxo(a))),

goto(a, SafeZone))))))

S.C. Shapiro

cse@buff

alo


S.C. Shapiro

cse@buff

alo

Representation and Use of Indexicals

• Words whose meanings are determined by occasion of use

• E.g. I, you, now, then, here, there

• Deictic Center <*I, *YOU, *NOW>

• *I: SNePS term representing Cassie

• *YOU: person Cassie is talking with

• *NOW: current time.

S.C. Shapiro

cse@buff

alo

Analysis of Indexicals(in input)

• First person pronouns: *YOU• Second person pronouns: *I• “here”: location of *YOU• Present/Past relative to *NOW.

S.C. Shapiro

cse@buff

alo

Generation of Indexicals

• *I: First person pronouns

• *YOU: Second person pronouns

• *NOW: used to determine tense and aspect.

S.C. Shapiro

cse@buff

alo

Come here.

Use of Indexicals 1

S.C. Shapiro

cse@buff

alo

Come here.I came to you, Stu.I am near you.

Use of Indexicals 2

S.C. Shapiro

cse@buff

alo

Who am I?Your name is ‘Stu’and you are a person.

Who have you talked to?

I am talking to you.Talk to Bill.

I am talking to you, Bill.Come here.

Use of Indexicals 3

S.C. Shapiro

cse@buff

alo

Come here.

I found you.I am looking at you.

Use of Indexicals 4

S.C. Shapiro

cse@buff

alo

Come here.

I came to you.I am near you.

I found you.I am looking at you.

Use of Indexicals 5

S.C. Shapiro

cse@buff

alo

Who am I?

I talked to Stuand I am talking to you.

Your name is ‘Bill’and you are a person.

Who are you?I am the FEVAHRand my name is ‘Cassie’.

Who have you talked to?

Use of Indexicals 6

S.C. Shapiro

cse@buff

alo


S.C. Shapiro

cse@buff

alo

A Personal Sense of Time

• *NOW contains SNePS term representing current time.

• *NOW moves when Cassie acts or perceives a change of state.

S.C. Shapiro

cse@buff

alo

B6

Representation of Time

find

lex

action object

B1

!

agentact

eventtime

NOW

!!before after before after

?????????????

I

S.C. Shapiro

cse@buff

alo

Movement of Timet1 t2!before after t3!before after

NOW NOW NOW

S.C. Shapiro

cse@buff

alo

Performing a Punctual Actt1 t3!before after

NOW NOW

t2!before after

!

time

event

S.C. Shapiro

cse@buff

alo

Performing a Durative Actt1

NOW

!before after t2

!

time

event

NOW

t3!

supintsubint

S.C. Shapiro

cse@buff

alo

The Pacemaker

• PML process periodically increments variable COUNT.

• *COUNT = some PML integer.

• Reset to 0 when NOW moves.

• Provides bodily “feel” of passing time.

S.C. Shapiro

cse@buff

alo

Quantizing Time

Cannot conceptualize fine distinctions in time intervals.

So quantize, e.g. into half orders of magnitude (Hobbs, 2000).

S.C. Shapiro

cse@buff

alo

Movement of Time with Pacemaker

NOW COUNT n

hom

0

KL

PML

t1 t2q

!before after

time duration!

S.C. Shapiro

cse@buff

alo

The Problem of the Fleeting Now

How can you reason about “now”

if it never stands still?

S.C. Shapiro

cse@buff

alo

Fleeting Now Example 1

12:15:00: “Is John having lunch now?”

12:15:02: Agent walks to John’s office.

12:17:00: Agent sees John at his desk, eating.

12:19:00: Agent reports “yes”.

Appropriate granularity.

S.C. Shapiro

cse@buff

alo

Fleeting Now Example 2

12:15:00: “Is John having lunch now?”

Agent knows John is at home without a phone.

Agent contemplates driving to John’s home.

Don’t bother---inappropriate granularity.

S.C. Shapiro

cse@buff

alo

The Vagueness of “now”

I’m now giving a talk.I’m now supervising PhD students.I’m now visiting Paris.I’m now living in Buffalo.The agent is now walking to John’s office.The agent is now seeing if John is eating lunch.

Multiple now’s at different granularities.

S.C. Shapiro

cse@buff

alo

NOW-MTF

NOW

Semi-lattice of times, all of which contain *NOW,any of which could be meant by “now”Finite---only conceptualized times of conceptualized states

Maximal Temporal Frame based on *NOW

S.C. Shapiro

cse@buff

alo

Moving NOW with MTF

NOW

S.C. Shapiro

cse@buff

alo

Typical Durations

“If the walk light is on now, cross the street.”Relevant duration is typical duration of walk lights.

“Is John having lunch now?”Relevant duration is typical duration of lunch.

Use quantized typical durations when updating NOW-MTFs.

S.C. Shapiro

cse@buff

alo

Using Appropriate Granularity

NOW

Lunch time

Lunch? Lunch!

Yes!

S.C. Shapiro

cse@buff

alo


S.C. Shapiro

cse@buff

alo

Goal

• A computational cognitive agent/robot

• That can communicate in natural language.

S.C. Shapiro

cse@buff

alo

Intensional Representation& Propositions

• SNePS terms represent mental entities.

• May assert that two entities are coreferential.

• Relations/acts may be declared transparent.

• Propositions are first-class entities.

S.C. Shapiro

cse@buff

alo

SNePS Connectives and Quantifiers

• Designed logical connectivesand rules of inference

More appropriate for NLU and Commonsense reasoningthan in standard FOPC.

S.C. Shapiro

cse@buff

alo

SNeRE Acting Constructs

• Separate, but Coordinated

Syntax and Semantics

For Acting and for Reasoning

S.C. Shapiro

cse@buff

alo

Representation and Useof Indexicals

• Use of Deictic Center for parser to interpret indexicals as current referents

• And for generator to generate indexicals from current referents.

S.C. Shapiro

cse@buff

alo

A Personal Sense of Time

• *NOW is current time.

• Updated when Cassie acts

or perceives a change of state.

• Points into MTF to support vagueness of “now”.

S.C. Shapiro

cse@buff

alo

For More Information

• Personnel

• Manual

• Tutorial

• Bibliography

• ftp’able SNePS source code

• etc.

• http://www.cse.buffalo.edu/sneps/

Documents

The SNePS Approach to Cognitive Robotics