Weakly Supervised Learning of Semantic Parsers for Mapping Instructions … · 2017. 10. 17. · Weakly Supervised Learning of Semantic Parsers for Mapping Instructions to Actions

Weakly Supervised Learning of Semantic Parsers for Mapping Instructions to ActionsYoav Artzi and Luke ZettlemoyerUniversity of Washington

TACL VOL. 1(2013):49−62 / NAACL 2013

1

Semantic Parsing

Less Supervision

More Domains

DatabasesLarge knowledge-basesInstructionsReferring expressionsRegular expressions

AnswersDemonstrationsSituated examples

Show me all papers about semantic parsing

�x.paper(x) ^ topic(x, SEMPAR)

Later thi

s

session

2

Semantic Parsing

Less SupervisionMore

Domains

Modeling

Situated Parsing

Show me all papers about semantic parsing

�x.paper(x) ^ topic(x, SEMPAR)

3

Executing Navigation Instructions

place your back against the wall of the t intersection

turn left

go forward along the pink flowered carpet hall two segments to the intersection with the brick hall

4

go to the next sofa

Resolve Referents

5

go to the next sofa

Resolve Referents

6

Disambiguate Word Sense

go to the chair7

Disambiguate Word Sense

go to the chair8

Identify Executable Actions

walk forward twice9

Identify Executable Actions

walk forward twice

move, move? I can do that!

10

Understand Implicit Requests

at the chair, turn right11

Understand Implicit Requests

at the chair, turn right

need to move first

12

Grounded Learning

move forward?turn?

turn to face the chair13

Grounded Learning

move forward?turn?

turn to face the chair14

Learning Signal

at the chair, move forward three steps past the sofa

Instruction:

Demonstration:

15

Learning Signal

Instruction:

Demonstration:

During learning: validate executions of different interpretations against demonstrations

at the chair, move forward three steps past the sofa

16

LearningDemonstrations

ParsingLogical Form

ExecutionJoint

InferenceWeighted

CCG

Situated Signals

Validation

17


Parsing1Logical Form2

Execution3

Lexical Generation4

Joint Inference

Weighted CCG

Situated Signals

Validation

18

Combinatory Categorial Grammarsmove to the chair

S AP/NP NP/N N

�a.move(a) �x.�a.to(a, x) �f.◆x.f(x) �x.chair(x)

>NP

◆x.chair(x)

>AP

�a.to(a, ◆x.chair(x))

S\S�f.�a.f(a) ^ to(a, ◆x.chair(x))

>S

�a.move(a) ^ to(a, ◆x.chair(x))

19


S AP/NP NP/N N


>NP

◆x.chair(x)

>AP



>S


Lexicon Combinators20


S AP/NP NP/N N


>NP

◆x.chair(x)

>AP



>S


Lexicon Combinators

x

z

y

21

Weighted Linear CCGs

• Given a weighted linear model:- CCG lexicon Λ- Feature function - Weights • The best parse is:

• We consider all possible parses y for sentence x given the lexicon Λ

y

⇤= argmax

yw · f(x, y)

f : X ⇥ Y ! Rm

w 2 Rm

22


Parsing✓Logical Form2

Execution3

Lexical Generation4

Joint Inference

Weighted CCG

Situated Signals

Validation

23

Constrain sets

Specific entities

Relations between entities

Sets of entities

Modeling for Semantic Parsing

Nouns

PPs and adjectives

Noun phrases

Verbs

24


Works well for natural language interfaces for DBs

Constrain sets

Specific entities


Sets of entitiesNouns

PPs and adjectives

Noun phrases

Verbs

25


Previous work on instructional language adopted procedural representation

[Matuszek et al. 2010; 2012; Chen, Mooney 2011; Chen 2012; Kim, Mooney 2012]

Constrain sets

Specific entities



PPs and adjectives

Noun phrases

Verbs

26


How can we use this approach for instructions?

Previous work on instructional language adopted procedural representation

Constrain sets

Specific entities



PPs and adjectives

Noun phrases

Verbs

27

Relations between entitiesDavidsonian Events


Name objects

Instructions to execute

Constrain sets

Specific entities


PPs and adjectives

Noun phrases

Verbs

Sets of eventsImperatives

28

Spatial Environment Modeling1 2 3 4 5

1

2

3

4

5

• Maps are graphs of connected positions

• Agent can move forward, turn right and turn left

• Agent perceives clusters of positions

• Clusters capture objects

Agent

29

1 2 3 4 5

1

2

3

4

5

Spatial Environment Modeling





30

Spatial Environment Modeling1 2 3 4 5

1

2

3

4

5





31

Spatial Language1 2 3 4 5

1

2

3

4

5

• Nouns• Noun phrases• Adjectives• Prepositional phrases• Spatial relations

32


1

2

3

4

5

chair

�x.chair(x)

{ },

Nouns denote sets of objects

33


1

2

3

4

5

Noun phrases name specific entities

34


1

2

3

4

5

the chair


◆x.chair(x)

35


1

2

3

4

5

the chair


◆x.chair(x)

Definite determiner depends on agent state

36


1

2

3

4

5

the chair


◆x.chair(x)

Definite determiner depends on agent state

37


1

2

3

4

5

• Nouns• Noun phrases• Adjectives• Prepositional phrases• Spatial relations

38

Modeling Instructions

• Sequences of identical actions are events• Use Neo-Davidsonian event semantics• Represent imperatives as sets of events

39

Modeling Instructions1 2 3 4 5

1

2

3

4

5

move

Imperatives are sets of events

�a.move(a)

{ }, ,

40


1

2

3

4

5

move

{ },�a.move(a)

,Disambiguate by preferring

shorter sequences

Imperatives are sets of events

41


1

2

3

4

5

go to the chair

{ }

Events can be modified by adverbials

�a.move(a)^to(a, ◆x.chair(x))

42

Executing Logical Forms1 2 3 4 5

1

2

3

4

5

go to the chair�a.move(a) ^ to(a, ◆x.chair(x))

Consider all variable assignments to find

satisfying ones

43


1

2

3

4

5


44


1

2

3

4

5


45

Dynamic Models1 2 3 4 5

1

2

3

4

5

move until you reach the chair

�a.move(a)^post(a, intersect(◆x.chair(x), you))

World models change during execution

46


1

2

3

4

5



4

3


47


1

2

3

4

5



4

3Never

inters

ects


48


1

2

3

4

5



Update model to reflect state change


49


1

2

3

4

5



2

3

Update model to reflect state change

Update


50

Implicit Actions1 2 3 4 5

1

2

3

4

5

at the chair, turn left

Consider actions with prefixed implicit actions

�a.turn(a) ^ dir(a, left)^pre(a, intersect(◆x.chair(x), you))

51


1

2

3

4

5




52


1

2

3

4

5



Implicit actions


53

Joint Inference

Language Understanding

World State

Actions

54

• Execution is integrated into parsing• During parsing, execute logical forms and

observe the result

Demonstrations

Parsing✓Logical Form✓

Execution✓

Lexical Generation4

Joint Inference

Weighted CCG

Situated Signals

ValidationLearning

55



Execution✓

Lexical Generation4

Joint Inference

Weighted CCG

Situated Signals

Validation

56

Demonstrations


Execution✓

Lexical Generation4

Joint Inference

Weighted CCG

Situated Signals

Validation

Learning

• Based on a small set of seed templates

• New coarse-to-fine parsing algorithm to gradually prune the potential lexical entries

• Conservative approach to introduce new entries to the model

57



Execution✓

Lexical Generation✓

Joint Inference

Weighted CCG

Situated Signals

Validation

58

Validation-Driven Learning

• Online• 2 steps:- Lexical generation-Parameter update• Driven by a weak validation signal

59


For T iterations, for each training sample:

• Step 1: Lexical generation- Generate a large number of potential lexical entries- Parse with the generated lexicon using the model- Select the best valid parses from the k-best parses- Add their lexical items to the lexicon• Step 2: Update parameters

60


For T iterations, for each training sample:

• Step 1: Lexical generation• Step 2: Update parameters- Parse using the model- Split all parses into two sets:

max scoring valid and invalid

- Find margin violating pairs between the sets- Do a perceptron-style update using these violations

61

Related Work

Supervised seman.c parsing [Kate, Mooney 2006; Wong, Mooney 2007; Muresan 2011 ]

with CCGs [ZeElemoyer, Collins 2005; 2007; Kwiatkowski et al. 2010; 2011]

Weakly supervised seman.c parsing

[Clarke et al. 2010; Goldwasser, Roth 2011; Liang et al. 2011; Kirshnamurthy, Mitchell 2012; Goldwasser

et al. 2011]

Grounded Seman.c Analysis [Liang et al. 2009; Chen et al. 2010; Matuszek et al. 2012]

Execu.ng Instruc.ons with Shallow Representa.on

[Branavan et al. 2009; 2010; Vogel, Jurafsky 2010; Wei et al. 2009; Kollar et al. 2010; Tellex et al. 2011]

Non-‐joint Execu.on of Instruc.ons [Matuszek et al. 2010; 2012; Chen, Mooney 2011; Chen 2012; Kim, Mooney 2012]

62

Experimental Setup

• Seed lexicon from an annotated randomly selected 12 instruction sequences

• Features: lexical, type-raising usage and repetitions in logical coordinations

• Consider only executable parses as complete

63

ResultsSAIL Corpus - Cross Validation

0

14

28

42

56

70

Single Sentence Sequence

Chen and Mooney 2011Chen 2012Kim and Mooney 2012Final State ValidationTrace ValidationKim and Mooney 2013

64


0

14

28

42

56

70


31.93

65.28

30.9

64.25


65


0

14

28

42

56

70


31.93

65.28

30.9

64.25


66

ResultsOracle Corpus - Held-out Set

0

16

32

48

64

80


58.05

78.63

54.63

77.6

Final State ValidationTrace Validation

67

Contributions

• Linguistically-driven modeling of instructional language

• Joint inference for interpretation and execution of grounded language

• General weakly-supervised learning approach for semantic parsers

68

UW SPF

Open source semantic parsing framework

http://yoavartzi.com/spf

Semantic Parser

Flexible High-Order Logic Representation

Learning Algorithms

69

UW SPF

Open source semantic parsing framework

http://yoavartzi.com/spf

Semantic Parser

Flexible High-Order Logic Representation

Learning Algorithms

Navigation code and data available online

http://yoavartzi.com/navi Comin

g up:

ACL tuto

rial

70

[fin]

71

Documents

Weakly Supervised Learning of Semantic Parsers for Mapping Instructions … · 2017. 10. 17. · Weakly Supervised Learning of Semantic Parsers for Mapping Instructions to Actions