Upload
anaya
View
40
Download
0
Tags:
Embed Size (px)
DESCRIPTION
Bailey Model. Showed how simple hand action verbs may be acquired based on motor control schemas and parameterization. Used Model Merging which allowed for One-shot learning (Maps to recruitment learning) Could label and perform actions (given a command, world state pair) - PowerPoint PPT Presentation
Citation preview
Bailey Model• Showed how simple hand action verbs may be acquired
based on motor control schemas and parameterization.• Used Model Merging which allowed for
– One-shot learning (Maps to recruitment learning)• Could label and perform actions (given a command, world state
pair)• Uses parameters over motor-control schemas as inductive bias
• Limitations– Inference
• Connections between events – Abstract uses
• Event Structure• Frames• Metaphor
Active representations
• Many inferences about actions derive from what we know about executing them
• Representation based on stochastic Petri nets captures dynamic, parameterized nature of actions
Walking:bound to a specific walker with a
direction or goalconsumes resources (e.g., energy)may have termination condition
(e.g., walker at goal) ongoing, iterative action
walker=Harry
goal=home
energy
walker at goal
X-Schema Extensions to Petri Nets
• Parameterization– x-schemas take parameter values (speed, force)
• Walk(speed = slow, dest = store1)
• Dynamic Binding– X-schemas allow run-time binding to different
objects/entities• Grasp(cup1), push(cart1)
• Hierarchical control and durative transitions– Walk is composed of steps which are composed of
stance and swing phases• Stochasticity and Inhibition
– Uncertainties in world evolution and in action selection
Event Structure in Language• Commonplace discourse fragments/blurbs
– Low inflation is starting to pull France out of recession. – E3 continue to push Iran to uphold IAEA obligations.– US Economy on the verge of falling back into recession
after moving forward on an anemic recovery.– Indian Government stumbling in implementing
Liberalization plan.– Moving forward on all fronts, we are going to be ongoing
and relentless as we tighten the net of justice.– The Government is taking bold new steps. We are
loosening the stranglehold on business, slashing tariffs and removing obstacles to international trade.
Event Structure in Language
• Fine-grained • Rich Notion of Contingency Relationships.
– Phenomena: Aspect, Tense, Force-dynamics, Modals, Counterfactuals
• Event Structure Metaphor:– Phenomena: Abstract Actions are
conceptualized in Motion and Manipulation terms.
– Schematic Inferences are preserved.
Aspect• Aspect is the name given to the ways
languages describe the structure of events using a variety of lexical and grammatical devices.– Viewpoints
• is walking, walk– Phases of events
• Starting to walk, walking, finish walking– Inherent Aspect
• run vs cough vs. rub– Composition with
• Temporal modifiers, tense..• Noun Phrases (count vs. mass) etc..
Grammatical Aspect
Languages have grammatical constructions that indicate the type of situation described.
• Progressive: She was running home.• Perfect: I’ve had a wonderful evening.• Inceptive: She started knitting. • Prospective: She’s about to leave.• Resumptive: Peace talks resume.• Iterative: They ran twice around the track.
Phases, Viewpoints, and Aspects
• John is walking to the store.• John is about to walk to the store.• John walked to the store.• John started walking to the store.• John is starting to walk to the store.• John has walked to the store.• John has started to walk to the store.• John is about to start walking to the store.• John resumed walking to the store.• John has been walking to the store.• John has finished walking to the store.• John almost walked to the store.
A Walk X-schema
A Climb X-schema
Common Patterns
Posture = UpEnergy AvailableReadyDest = top(obj)
LoopBEGIN Execute (subschema)END
At DestDone
Posture = UpGround okReady
LoopBEGIN Execute(subschema)END
At DestDone
START FINISH
Pre-motor Versus Motor Cortex
Whenever we perform a complex motor movement, such as picking up a glass and taking a drink, at least two distinct parts of the brain are activated:
The motor cortex, where there are neural ensembles that control “motor synergies” — relatively simple actions like opening or closing the hand, flexing or extending the elbow, turning the wrist, and so on.
Complex motor schemas, however, are carried out by neural circuitry in the pre-motor cortex, circuitry connected via neural bindings to the appropriate synergies in the motor cortex.
In picking up a glass and taking a drink, both pre-motor cortex and motor cortex are activated, as are binding connections between them.
The Controller X-Schema
In modeling complex premotor action schemas, we make the following hypothesis
All complex premotor schemas are compositions of a single type of structure.
The same neural computational structure, when disengaged from the motor cortex, can characterize aspect (that is, event structure) in the world’s languages. When dynamically active, this structure can compute the logic of aspect.
We call this structure the “Controller X-schema.”
The Structure of the Controller X-Schema
•Initial State•Starting Phase Transition•Precentral State•Central Phase Transition (either instantaneous,
prolonged, or ongoing)•Postcentral State*•Ending Phase Transition•Final State
Postcentral Options: *A check to see if a goal state has been achieved *An option to stop/resume *An option to iterate or continue the main process
-Narayanan, 1997
A Schema Controller
• An active controller that sends signals to the embedded schema and transitions based on signals from the embedded schema.• Useful for higher level monitoring and coordination of actions.
Ready DoneStart Process Finish
SuspendCancel
interrupt resume
iterate
A Generic Process Schema
• Part of Conceptual Structure. • Generalizes over actions and events. Has internal state and models evolution of processes.
Ready DoneStart Process Finish
SuspendCancel
interrupt resume
iterate
Aspects of (Climb)
Ready DoneStart Process Finish
SuspendCancel
interruptresume
Iterate
EnergyReady
StandingOn top
HoldFind hold
Pull(self)Stabilize
BINDINGS
About to + (Climb) (Prospective)
Ready DoneStart Process Finish
SuspendCancel
interruptresume
Iterate
EnergyReady
StandingOn top
HoldFind hold
Pull(self)Stabilize
BINDINGS
Cancel + (Climb)
Ready DoneStart Process Finish
SuspendCancel
interruptresume
Iterate
EnergyReady
StandingOn top
HoldFind hold
Pull(self)Stabilize
BINDINGS
Start + (Climb)-ING
Ready DoneStart Process Finish
SuspendCancel
interruptresume
Iterate
EnergyReady
StandingOn top
HoldFind hold
Pull(self)Stabilize
BINDINGS
Be + (Climb)-ING (Progressive)
Ready DoneStart Process Finish
SuspendCancel
interruptresume
Iterate
EnergyReady
StandingOn top
HoldFind hold
Pull(self)Stabilize
BINDINGS
Suspend (Climb)-ING
Ready DoneStart Process Finish
SuspendCancel
interruptresume
Iterate
EnergyReady
StandingOn top
HoldFind hold
Pull(self)Stabilize
BINDINGS
Resumed + (Climb)-ING (Resumptive)
Ready DoneStart Process Finish
SuspendCancel
interruptresume
Iterate
EnergyReady
StandingOn top
HoldFind hold
Pull(self)Stabilize
BINDINGS
Finish (End) + (Climb)-ING
Ready DoneStart Process Finish
SuspendCancel
interruptresume
Iterate
EnergyReady
StandingOn top
HoldFind hold
Pull(self)Stabilize
BINDINGS
Have + (Climb)-ed (Perfect)
EnergyReady
StandingOn top
HoldFind hold
Pull(self)Stabilize
Ready DoneStart Process Finish
SuspendCancel
interruptresume
Iterate
BINDINGS
Embedding: Has Started (to X)
Ready DoneStart Process Finish
Suspendinterruptresume
R DS P F
SC ir
X-Schema for X with bindings
Phasal Aspect Maps to the Controller
Ready DoneStart Process Finish
SuspendCancel
interrupt resume
IterateInceptive (start, begin) Iterative (repeat)
Completive (finish, end)Resumptive(resume)
Embedding: About to start (X)
Ready DoneStart Process Finish
Suspendinterruptresume
R DS P F
SC ir
X-Schema for X with bindings
Embedding: Has Started (to X)
Ready DoneStart Process Finish
Suspendinterruptresume
R DS P F
SC ir
X-Schema for X with bindings
Begins and Ends
• “This is not the end. It is not even the beginning of the end. But it is, perhaps, the end of the beginning." – Speech given at the Lord Mayor's Luncheon,
Mansion House, London, November 10, 1942. Winston Churchill
Embedding: It’s not this (the end)
Ongoing Finish Done
X-Schema for X with bindings
Embedding: It’s not this (beginning of the end)
Ongoing Finish Done
R DP F
SC ir
X-Schema for X with bindings
S
Embedding: But this (The end of the beginning)
Ready DoneStart Process Finish
Suspendinterruptresume
R DS P F
SC ir
X-Schema for X with bindings
Inherent Aspect (Aksionsart)
• Vendler-Dowty-Taylor (VDT) classification– Events and States– Events can be
• Punctual or Durative• Atelic or Telic
– States satisfy the downward entailment property
• If a state holds in some interval, it holds in all sub-intervals of that interval.
Inherent Aspect• Much richer than traditional Linguistic
Characterizations (VDT (durative/atomic, telic/atelic))
• Action patterns– one-shot, repeated, periodic, punctual– decomposition: concurrent, alternatives, sequential
• Goal based schema enabling/disabling• Generic control features;
– interruption, suspension, resumption• Resource usage
Basic Event X-schemas
• State• Event Transition• Simple Event• Simple Action• Complex Event/Process• Complex State
Aspectual Types
Other Transitions in the Controller may be coded
• Lexical items may code interrupts– Stumble is an interrupt to an ongoing walk
• A combination of grammatical and aktionsart may code of the controller phases– Ready to walk : Prospective– Resuming his run: Resumptive– Has been running: Embedded progressive– About to Finish the painting: Embedded Completive.– Canceling the meeting vs. Aborting the meeting.
Interaction of Aspect with Tense
• Reichenbach’s system uses three pointers– Speech Time (S)– Reference Time (R)– Event Time (E)
• Tense is a partial ordering relation between the pointers– Simple Past E < R, E < S– Perfect E < R < S
Viewpoint Aspect (Perfective/Imperfective)
Perfective/ImperfectivePerfective
Imperfective
Simulation and Reference Interval Perfective
Imperfective
Levels of Granularity
• Events can be construed at different levels of granularity based on various contextual factors.– In 1991, McEnroe injured his knee while
playing tennis.– This morning, I injured my knee while playing
tennis.– He is coughing (normal time scale vs. slow-
motion film time scale).
Summary of Aspect Results• Controller mediates between linguistic markings and individual event/verbal x-
schemas (Cogsci99, Coling2002)• Captures regular event structure; inspired by biological control theory• Flexible: specific events may require only a subset of controller; interaction of underlying x-
schemas, linguistic markers and hierarchical abstraction/ decomposition of controller accounts for wide range of aspectual phenomena.
• Important aspectual distinctions, both traditional and novel, can be precisely specified in terms of the interaction of x-schemas with the controller (CogSci97, CogSci98, AAAI99, IJCAI99, CogSci04, CogLing2005):
• stative/dynamic, durative/punctual: natural in x-schemas• telic processes: depletion of resources• continuous processes: consumption of resources• temporary/effortful states; habituals• dynamic interactions with tense, nominals, temporal modifiers• incorporation of world knowledge, pragmatics
• Ongoing Work: Simulation Semantics and Tense-Aspect (with Laura Michaelis)
Simulation hypothesis
We understand utterances by mentally simulating their content.
– Simulation exploits some of the same neural structures activated during performance, perception, imagining, memory…
– Linguistic structure parameterizes the simulation.• Language gives us enough information to simulate
Simulation Semantics
• BASIC ASSUMPTION: SAME REPRESENTATION FOR PLANNING AND SIMULATIVE INFERENCE– Evidence for common mechanisms for recognition and
action (mirror neurons) in the F5 area (Rizzolatti et al (1996), Gallese 96, Buccino 2002, Tettamanti 2004) and from motor imagery (Jeannerod 1996)
• IMPLEMENTATION: – x-schemas affect each other by enabling, disabling or
modifying execution trajectories. Whenever the CONTROLLER schema makes a transition it may set, get, or modify state leading to triggering or modification of other x-schemas. State is completely distributed (a graph marking) over the network.
• RESULT: INTERPRETATION IS IMAGINATIVE SIMULATION!
A Precise Notion of Contingency Relations
Activation:Executing one schema causes the enabling, start or continued execution of another schema. Concurrent and sequential activation.
Inhibition:Inhibitory links prevent execution of the inhibited x-schema by activating an inhibitory arc. The model distinguishes between concurrent and sequential inhibition, mutual inhibition and aperiodicity.
Modification:The modifying x-schema results in control transition of the modified xschema. The execution of the modifying x-schema could result in theinterruption, termination, resumption of the modified x-schema.
General and Domain Knowledge
• Conceptual Knowledge and Inference– Embodied– Language and Domain Independent– Powerful General Inferences– Ubiquitous in Language
• Domain Specific Frames and Ontologies– FrameNet, OWL ontologies
• Metaphor links domain specific to general– E.g., France slipped into recession.
Frames• Frames are conceptual structures that may be culture
specific• Words evoke frames
– The word “talk” evokes the Communication frame– The word buy (sell, pay) evoke the Commercial
Transaction (CT) frame.– The words journey, set out, schedule, reach etc. evoke the
Journey frame.• Frames have roles and constraints like schemas.
– CT has roles vendor, goods, money, customer.• Words bind to frames by specifying binding patterns
– Buyer binds to Customer, Vendor binds to Seller.
Event Frames
Event frames have temporal structure that comprises of the controller event structure and generally have constraints on what precedes them, what happens during them, and what state the world is in once the event has been completed.
Sample Event Frame:Commercial Transaction
Initial state:Vendor has Goods, wants MoneyCustomer wants Goods, has Money
Transition:Vendor transmits Goods to CustomerCustomer transmits Money to Vendor
Final state:Vendor has MoneyCustomer has Goods
Sample Event Frame:Commercial Transaction
Initial state:Vendor has Goods, wants MoneyCustomer wants Goods, has Money
Transition:Vendor transmits Goods to CustomerCustomer transmits Money to Vendor
Final state:Vendor has MoneyCustomer has Goods
(It’s a bit more complicated than that.)
Partial Wordlist for Commercial Transactions
Verbs: pay, spend, cost, buy, sell, charge
Nouns: cost, price, payment
Adjectives: expensive, cheap
Meaning and Syntax
The various words that evoke this frame introduce the elements of the frame in different ways. The identities of the buyer, seller, goods and
money Information expressed in sentences
containing these words occurs in different places in the sentence depending on the word.
Language understanding: analysis & simulation
“Harry walked into the cafe.”
Analysis Process
SemanticSpecification
Utterance
ConstructionsLexicon
General Knowledge
Belief State
CAFE Simulation
construction WALKEDform
selff.phon [wakt]meaning : Walk-Action constraints
selfm.time before Context.speech-time selfm..aspect encapsulated