Embodied Construction Grammar in language (acquisition and) use Jerome Feldman ([email protected]) Computer Science Division, University of California,

Embodied Construction Grammar in language (acquisition and) use

Jerome Feldman([email protected])

Computer Science Division, University of California, Berkeley, andInternational Computer Science Institute

State of the Art

• Limited Commercial Speech Applications transcription, simple response systems • Statistical NLP for Restricted Tasks tagging, parsing, information retrieval• Template-based Understanding programs expensive, brittle, inflexible, unnatural• Essentially no NLU in HCI, QA Systems

What does language do?

“Harry walked to the cafe.” “Harry walked into the cafe.”

A sentence can evoke an imagined scene and resulting inferences:

CAFE CAFE

– Goal of action = at cafe– Source = away from cafe– cafe = point-like location

– Goal of action = inside cafe– Source = outside cafe– cafe = containing location

Language understanding

Interpretation

(Utterance, Situation)

Linguistic knowledge

Conceptual knowledge

Analysis

Language understanding: analysis & simulation

“Harry walked to the cafe.”

Schema Trajector Goalwalk Harry cafe

Cafe

Lexicon

Constructicon

General Knowledge

Belief State

Analysis Process

SemanticSpecification

Utterance

Simulation

Interpretation: x-schema simulation

Constructions can• specify which schemas

and entities are involved in an event, and how they are related

• profile particular stages of an event

• set parameters of an event

energy

walker at goal

walker=Harry goal=home

Harry is walking home.

Phonetics

Semantics

Pragmatics

Morphology

Syntax

Traditional Levels of Analysis

Phonetics

Semantics

Pragmatics

Morphology

Syntax

“Harry walked into the cafe.”

Utterance

Construction Grammar

to

block

walk

Form Meaning

A construction is a form-meaning pair whose properties may not be strictly predictable from other constructions.

(Construction Grammar, Goldberg 1995)

Source

Path

GoalTrajector

Form-meaning mappings for language

Formphonological cuesword orderintonationinflection

Meaningevent structuresensorimotor controlattention/perspectivesocial goals...

Linguistic knowledge consists of form-meaning mappings:

Cafe

Constructions as maps between relations

Mover + Motion + Directionbefore(Motion, Direction)before(Mover, Motion)

“is” + Action + “ing”before(“is”, Action)suffix(Action, “ing”)

Mover + Motionbefore(Mover, Motion)

Form Meaning

ProgressiveActionaspect(Action, ongoing)

MotionEventmover(Motion, Mover)

DirectedMotionEventdirection(Motion, Direction)mover(Motion, Mover)

Complex constructions are mappings between relations in form and relations in meaning.

Embodied Construction Grammar(Bergen and Chang 2002)

• Embodied representations– active perceptual and motor schemas– situational and discourse context

• Construction Grammar– Linguistic units relate form and meaning/function.– Both constituency and (lexical) dependencies allowed.

• Constraint-based (Unification)– based on feature structures (as in HPSG)– Diverse factors can flexibly interact.

schema Containerroles

interiorexteriorportalboundary

Representing image schemas

Interior

Exterior

Boundary

PortalSource

Path

GoalTrajector

These are abstractions over sensorimotor experiences.

schema Source-Path-Goalroles

sourcepathgoaltrajector

schema name

role name

Inference and Conceptual Schemas

• Hypothesis: – Linguistic input is converted into a mental simulation based on bodily-grounded structures.

• Components:– Semantic schemas

• image schemas and executing schemas are abstractions over neurally grounded perceptual and motor representations

– Linguistic units • lexical and phrasal construction representations invoke schemas, in part through metaphor

• Inference links these structures and provides parameters for a simulation engine

Early ExampleUnderstanding News Stories

France fell into recession. Pulled out by Germany

In1991, India set out on a path of liberalization. The Government started to loosen its stranglehold on business and removed obstacles to international trade. Now the Government is stumbling in implementing the liberalization plan.

Task

• Interpret simple discourse fragments/blurbs– France fell into recession. Pulled out by Germany– Economy moving at the pace of a Clinton jog.– US Economy on the verge of falling back into recession after moving

forward on an anemic recovery.– Indian Government stumbling in implementing Liberalization plan.– Moving forward on all fronts, we are going to be ongoing and relentless as

we tighten the net of justice.– The Government is taking bold new steps. We are loosening the

stranglehold on business, slashing tariffs and removing obstacles to international trade.

I/O as Feature Structures

• Indian Government stumbling in implementing liberalization plan



Analysis Process


Utterance

Constructions

General Knowledge

Belief State

CAFE Simulation

construction WALKEDform

selff.phon [wakt]meaning : Walk-Action constraints

selfm.time before Context.speech-time selfm..aspect encapsulated

Embodied Construction Grammar providesformal tools for linguistic description and analysis

motivated largely by cognitive/functional concerns.

• Allows precise specifications of structures/processes involved in acquisition of early constructions–Embodied constructions (structured maps between form and meaning); lexically specific and more general–Usage-based processes of learning new constructions to account for co-occurring utterance-situation pairs

• Bridge to detailed psycholinguistic and neural imaging experiments

Formal Cognitive Linguistics

• Schemas and frames– Image schemas, force dynamics, executing schemas…

• Constructions– Lexical, grammatical, morphological, gestural…

• Maps– Metaphor, metonymy, mental space maps…

• Mental spaces– Discourse, hypothetical, counterfactual…

Embodied constructions

construction HARRYform : [hEriy]meaning : Harry

construction CAFEform : [khaefej]meaning : Cafe

Harry

CAFEcafe

NotationForm Meaning

Constructions have form and meaning poles that are subject to type constraints.

Schema Formalism

SCHEMA <name>

SUBCASE OF <schema>

EVOKES <schema> AS <local name>

ROLES < self role name>: <role restriction>

< self role name> <-> <role name>

CONSTRAINTS <role name> <- <value>

<role name> <-> <role name>

<setting name> :: <role name> <-> <role name>

<setting name> :: <predicate> | <predicate>

A Simple Example

SCHEMA hypotenuse

SUBCASE OF line-segment

EVOKES right-triangle AS rt

ROLES Comment inherited from line-segment

CONSTRAINTS

SELF <-> rt.long-side

Source-Path-Goal

SCHEMA: spg

ROLES:

source: Place

path: Directed Curve

goal: Place

trajector: Entity

Translational Motion

SCHEMA translational motion

SUBCASE OF motion

EVOKES spg AS s

ROLES

mover <-> s.trajector

source <-> s.source

goal <-> s.goal

CONSTRAINTS

before:: mover.location <-> source

after:: mover.location <-> goal

Construction Formalism

CONSTRUCTION<name>

SUBCASE OF <construction>

CONSTRUCTIONAL

EVOKES <construction> AS <local name>

CONSTITUENTS < local name> : <construction>

CONSTRAINTS // as in SCHEMAs

FORM

ELEMENTS

CONSTRAINTS // as in SCHEMAs

MEANING // as in SCHEMAs

The meaning pole may evoke schemas (e.g., image schemas) with a local alias. The meaning pole may include constraints on the schemas (e.g., identification constraints ).

construction TOform

selff.phon [thuw]meaning

evokesTrajector-Landmark as tl

Source-Path-Goal as spg constraints:

tl.trajectorspg.trajectortl.landmarkspg.goal

construction TOform

selff.phon [thuw]meaning

evokesTrajector-Landmark as tl

Source-Path-Goal as spg constraints:

tl.trajectorspg.trajectortl.landmarkspg.goal

Representing constructions: TO

local alias

identification constraint

TO vs. INTO:INTO adds a Container schema and appropriate bindings.

The INTO construction construction INTO

form selff.phon [Inthuw]

meaning evokes

Trajector-Landmark as tl Source-Path-Goal as spg

Container as cont constraints:

tl.trajectorspg.trajectortl.landmarkcontcont.interiorspg.goalcont.exteriorspg.source

Grammatical Construction Example

CONSTRUCTION Spatial-PP

SUBCASE OF Phrase

CONSTRUCTIONAL CONSTITUENTS

rel: Spatial-Preposition

lm: Referring-Exp

CONSTRAINTS

rel.case <-> lm.case

FORM rel < lm

MEANING CONSTRAINTS

rel.landmark <-> lm

The DIRECTED-MOTION construction

construction DIRECTED-MOTIONconstructional

constituentsmover : Thingmotion : Motion-Process direction : Source-Path-Goal

form moverf before motionf

motionf before directionf

meaningevokes Motion-Event as mm.mover moverm

m.motion motionm

m.path directionm

directionm.trajector moverm

motionm.mover moverm

Semantic specification

The analysis process produces a semantic specification that

•includes image-schematic, motor control and conceptual structures

•provides parameters for a mental simulation

Language Understanding Process

Constructional analysis

Semantic Specification



Analysis Process


Utterance

Constructions

General Knowledge

Belief State

CAFE Simulation

construction WALKEDform

selff.phon [wakt]meaning : Walk-Action constraints

selfm.time before Context.speech-time selfm..aspect encapsulated

Simulation-based sense disambiguation

• The scientist walkedinto the laboratory.

• The scientist walkedinto the wall.

Ease of construing nominal as a CONTAINER determines what sense of into is appropriate:

CONTAINER sense CONTACT sense

LAB WALL

Bonk!!

Simulation-based inference

• The teacher drifted into the house.

• The smoke drifted into the house.

Detailed inferences can result from simulation.

Image-schematic content of prepositions must fit with properties of other elements of sentence.

– Final location of Trajector = inside cafe

– Portal = door

– Final location of Trajector = inside (possibly throughout) cafe

– Portal = door/window

World knowledge informs simulation

Physical knowledge of how people and gases interact with houses determines:

–Relation between Trajector and Interior The smoke drifted into the house and filled it.?The teacher drifted into the house and filled it.

–Portal for motion across Boundary The smoke drifted into the house

because the window had been left open.?The teacher drifted into the house

because the window had been left open.

Getting From the Utterance to the SemSpec

Johno Bryant

• Need a grammar formalism– Embodied Construction Grammar (Bergen & Chang 2002)

• Need new models for language analysis – Traditional methods too limited– Traditional methods also don’t get enough leverage out of the

semantics.

Embodied Construction Grammar

• Semantic Freedom– Designed to be symbiotic with cognitive approaches to

meaning – More expressive semantic operators than traditional grammar

formalisms

• Form Freedom– Free word order, over-lapping constituency

• Precise enough to be implemented

Traditional Parsing Methods Fall Short

• PSG parsers too strict– Constructions not allowed to leave constituent order

unspecified

• Traditional way of dealing with incomplete analyses is ad-hoc– Making sense of incomplete analyses is important when

an application must deal with “ill-formed” input(For example, modeling language learning)

• Traditional unification grammar can’t handle ECG’s deep semantic operators.

Our Analyzer

• Replaces the FSMs used in traditional chunking (Abney 96) with much more powerful machines capable of backtracking called construction recognizers

• Arranges these recognizers into levels just like in Abney’s work

• But uses a chart to deal with ambiguity

Our Analyzer (cont’d)

• Uses specialized feature structures to deal with ECG’s novel semantic operators

• Supports a heuristic evaluation metric for finding the “right” analysis

• Puts partial analyses together when no complete analyses are available– The analyzer was designed under the assumption that the grammar

won’t cover every meaningful utterance encountered by the system.

System Architecture

Learner

Semantic Chunker

Semantic Integration

Grammar/UtteranceC

hunk

C

hart

Ranked Analyses

The Levels

• The analyzer puts the recognizer on the level assigned by the grammar writer.– Assigned level should be greater than or equal to the levels of

the construction’s constituents.

• The analyzer runs all the recognizers on level 1, then level 2, etc. until no more levels.

• Recognizers on the same level can be mutually recursive.

Recognizers

• Each Construction is turned into a recognizer

• Recognizer = active representation – seeks form elements/constituents when initiated– Unites grammar and process - grammar isn’t just a static piece of knowledge in

this model.

• Checks both form and semantic constraints– Contains an internal representation of both the semantics and the form– A graph data structure used to represent the form and a feature structure

representation for the meaning.

Recognizer Example

Path

Patient

ActionAgent

Mary kicked the ball into the net.

This is the initial Constituent Graph for caused-motion.

Recognizer Example

Construct:Caused-Motion

Constituent:Agent

Constituent:Action

Constituent:Patient

Constituent:Path

The initial constructional tree for the instance of Caused-Motion that we are trying to create.

Recognizer Example

:

:

action.m

2,

}1{:

:

:

:

path.m

}7{,

}7{:

62:

}1}{3{:

51:

motion.cmcaused

4

:

:

patient.m

3,

:

:

agent.m

1,

6:

4:

5:

motion.mcaused

schemax

tense

cmtrajector

goal

path

source

path

action

cmpatient

agent

refresolved

category

refresolved

category

action

scene

agent

Recognizer Example

processed


Path

Patient

ActionAgent

A node filled with gray is removed.

Recognizer Example


Constituent:Action

Constituent:Patient

Constituent:Path

RefExp:Mary


Recognizer Example

:

:

action.m

2,

}1{:

:

:

:

path.m

}7{,

}7{:

62:

}1}{3{:

51:

motion.cmcaused

4

:

:

patient.m

3,

:

:

agent.m

1,

6:

4:

5:

motion.mcaused

schemax

tense

cmtrajector

goal

path

source

path

action

cmpatient

agent

refresolved

category

Maryrefresolved

Personcategory

action

scene

agent

Recognizer Example

processed


Path

Patient

ActionAgent

Recognizer Example


Verb:kicked

Constituent:Patient

Constituent:Path

RefExp:Mary


Recognizer Example

kickschemax

simpPasttense

cmtrajector

goal

path

source

path

action

cmpatient

agent

refresolved

category

Maryrefresolved

Personcategory

action

scene

agent

:

:

action.m

2,

}1{:

:

:

:

path.m

}7{,

}7{:

62:

}1}{3{:

51:

motion.cmcaused

4

:

:

patient.m

3,

:

:

agent.m

1,

6:

4:

5:

motion.mcaused

Recognizer Example

processed


Path

Patient

ActionAgent

According to the Constituent Graph, The next constituent can either be thePatient or the Path.

Recognizer Example

processed


Path

Patient

ActionAgent

Recognizer Example


Verb:kicked

RefExp:Det Noun

Constituent:Path

RefExp:Mary


NounDet

Recognizer Example

kickschemax

simpPasttense

cmtrajector

goal

path

source

path

action

cmpatient

agent

refresolved

ballcategory

Maryrefresolved

Personcategory

action

scene

agent

:

:

action.m

2,

}1{:

:

:

:

path.m

}7{,

}7{:

62:

}1}{3{:

51:

motion.cmcaused

4

:

:

patient.m

3,

:

:

agent.m

1,

6:

4:

5:

motion.mcaused

Recognizer Example

processed


Path

Patient

ActionAgent

Recognizer Example


Verb:kicked

RefExp:Det Noun

Spatial-Pred:Prep RefExp

RefExp:Mary


NounDet NounDetPrep

RefExp

Recognizer Example

kickschemax

simpPasttense

cmtrajector

netgoal

path

source

path

action

cmpatient

agent

refresolved

ballcategory

Maryrefresolved

Personcategory

action

scene

agent

:

:

action.m

2,

}1{:

:

:

:

path.m

}7{,

}7{:

62:

}1}{3{:

51:

motion.cmcaused

4

:

:

patient.m

3,

:

:

agent.m

1,

6:

4:

5:

motion.mcaused

Scene = Caused-MotionAgent = MaryAction = KickPatient = Path.Trajector = The BallPath = Into the netPath.Goal = The net

After analyzing the sentence, the following identities are asserted in the resulting SemSpec:

Resulting SemSpec

Chunking

0 1 2 3 4 5 6 7 8 9the woman in the lab coat thought you were sleeping

L0 D N P D N N V-tns Pron Aux V-ing

L1 ____NP P_______NP VP NP ______VP

L2 ____NP _________PP VP NP ______VP

L3 ________________________S_____________S

Cite/description

Construction Recognizers

You want to put a cloth on your hand ?

NP NP NP NP NP

Form Meaning“you”<->[Addressee]

Form MeaningD,N <-> [Cloth num:sg]

Form MeaningPP$,N <-> [Hand num:sg

poss:addr]

Like Abney: Unlike Abney:

One recognizer per rule

Bottom up and level-based

Check form and semantics

More powerful/slower than FSMs

Chunk Chart

• Interface between chunking and structure merging• Each edge is linked to its corresponding semantics.

You want to put a cloth on your hand ?

Combining Partial Parses

• Prefer an analysis that spans the input utterance with the minimum number of chunks.

• When no spanning analysis exists, however, we still have a chart full of semantic chunks.

• The system tries to build a coherent analysis out of these semantics chunks.

• This is where structure merging comes in.

Structure Merging

• Closely related to abductive inferential mechanisms like abduction (Hobbs)

• Unify compatible structures (find fillers for frame roles)• Intuition: Unify structures that would have been co-

indexed had the missing construction been defined.• There are many possible ways to merge structures.• In fact, there are an exponential number of ways to

merge structures (NP Hard). But using heuristics cuts down the search space.

Structure Merging Example

Utterance:You used to hate to have the bib put on .

[Addressee < Animate]

Bib < Clothingnum:sggivenness:def

Caused-Motion-ActionAgent: [Animate]Patient: [Entity]Path:On

Before Merging: After Merging:

Caused-Motion-ActionAgent: [Addressee]Patient:

Path:On

Bib < Clothingnum:sggivenness:def

Semantic Density

• Semantic density is a simple heuristic to choose between competing analyses.

• Density of an analysis = (filled roles) / (total roles)• The system prefers higher density analyses because a

higher density suggests that more frame roles are filled in than in competing analyses.

• Extremely simple / useful? but it certainly can be improved upon.

Summary: ECG• Linguistic constructions are tied to a model of simulated

action and perception• Embedded in a theory of language processing

– Constrains theory to be usable– Frees structures to be just structures, used in processing

• Precise, computationally usable formalism– Practical computational applications, like MT and NLU– Testing of functionality, e.g. language learning

• A shared theory and formalism for different cognitive mechanisms– Constructions, metaphor, mental spaces, etc.

Issues in Scaling up to Language

• Knowledge – Lexicon (FrameNet)– Constructicon (ECG)– Maps (Metaphors, Metonymies) (MetaNet)– Conceptual Relations (Image Schemas, X-schemas)

• Computation– Representation (ECG)

• expressiveness, modularity, compositionality– Inference (Simulation Semantics)

• tractable, distributed, probabilistic concurrent, context-sensitive

The Buy schema

schema Buysubcase of Actionevokes Commercial-Transaction as ctroles

selfct.nucleusbuyer actor ct.customer ct.agent goods undergoer ct.goods

The Sell schema

schema Sellsubcase of Actionevokes Commercial-Transaction as ctroles

selfct.nucleusseller actor ct.vendor ct.agent goods undergoer ct.goods

Extending Inferential Capabilities

• Given the formalization of the conceptual schemas– How to use them for inferencing?

• Earlier pilot systems– Used metaphor and Bayesian belief networks– Successfully construed certain inferences– But don’t scale

• New approach– Probabilistic relational models– Support an open ontology

Semantic Web

• The World Wide Web (WWW) contains a large and expanding information base.

• HTML is accessible to humans but does not formally describe data in a machine interpretable form.

• XML remedies this by allowing for the use of tags to describe data (ex. disambiguating crawl)

• Ontologies are useful to describe objects and their inter-relationships.

• DAML+OIL (http://www.daml.org) is an markup language based on XML and RDF that is grounded in description logic and is designed to allow for ontology development, transfer, and use on the web.

http://www.daml.org/

The ICSI/BerkeleyNeural Theory of Language Project

Acquisition of early constructions

ECG

Probabilistic Relation Inference

• Scalable Representation of – States, domain knowledge, ontologies

• (Avi Pfeffer 2000, Koller et al. 2001)• Merges relational database technolgy with Probabilistic reasoning

based on Graphical Models.– Domain entities and relational entities– Inter-entity relations are probabilistic functions– Can capture complex dependencies with both simple and composite slot

(chains).• Inference exploits structure of the domain

Status of PRMs

• Summer Project– Build the basic PRM codebase/infrastructure

• Fall Project– Design Coordinated PRM (CPRM)– Build Interface for testing

• Spring/Summer 03– Implement CPRM to replace Pilot System DBN– Test CPRM for QA

• Related Work– Probabilistic OWL (PrOWL)– Probabilistic FrameNet

Articulating Projects

• FrameNet – NSF (with Colorado, USD)

• SmartKom – International Consortium

• EDU – European Media Lab

• Acquaint – ARDA (with SIMS, Stanford)

Conclusion• NLU is essential to large, open domain QA.

– Much of the web in unstructured data• Substantial Progress in Enabling Technologies

– Knowledge Representation/Inference Techniques• Active Knowledge – X-schemas, Simulation Semantics• Dealing With Uncertainty – PRM’s• Combining Statistics and Structure.• Conceptual Relations – Schemas, Metaphor, ECG

– Scaling Up• CYC, Wordnet, Term-bases• FrameNet, Semantic Web, MetaNet• Open Source

• The goal of NLU can be realized, perhaps!– Anyway, it’s time to try again.

Documents

Embodied Construction Grammar in language (acquisition and) use Jerome Feldman ([email protected]) Computer Science Division, University of California,