Probabilistic Lexical Models for Textual Inference Eyal Shnarch, Ido Dagan, Jacob Goldberger Probabilistic Lexical Models for Textual Inference Eyal Shnarch,

Bar Ilan University @ IBM July 2012

Probabilistic Lexical Models for Textual Inference

Eyal Shnarch, Ido Dagan, Jacob Goldberger

1/34


The entire talk in a single sentencewe address

with a

which

lexical textual

inference

principled probabilistic

model

improves state-of-the

art

2/34


Outlinewe address

with a

which

lexical textual

inference


model


art

1 2 3

3/34


we address

with a

which

lexical textual

inference


model


art

1 2 3

4/34


Textual inference – useful in many NLP apps

improves state-of-the-artprincipled probabilistic model

in Belgium Napoleon was defeated

In the Battle of

Waterloo, 18 Jun 1815,

the French army, led by

Napoleon, was

crushed.

Napoleon was

Emperor of the

French from 1804 to

1815.

lexical textual inference

Napoleon was not tall enough to win

the Battle of Waterloo

At Waterloo Napoleon did surrender...Waterloo - finally facing my Waterloo

Napoleon engaged in a series of wars, and won many

5/34


BIU NLP lab

Chaya Liebeskind

6/34


Lexical textual inference

• Complex systems use parser

• Lexical inference rules link terms from T to H• Lexical rules come from lexical resources• H is inferred from T iff all its terms are inferred

Improves state-of-the-artprincipled probabilistic model lexical textual inference

In the Battle of

Waterloo, 18 Jun

1815, the French

army, led by

Napoleon, was

crushed.

in Belgium Napoleon was defeated

Text Hypothesis

7/34

1st or 2nd order co-occurrence


Textual inference for ranking


In which battle was Napoleon defeated?

In the Battle of



Napoleon, was

crushed.

Napoleon was

Emperor of the

French from 1804 to

1815.





12

3

4

5

a

bc

d

e

8/34


Ranking textual inference – prior work


• Transform T’s parsed tree into H’s parsed tree• Based on principled ML model(Wang et al. 07, Heilman and Smith 10, Wang and Manning 10)

Syntactic-based

methods

• Fast, easy to implement, highly competitive• Practical across genres and languages(MacKinlay and Baldwin 09, Clark and Harrison 10,

Majumdar and Bhattacharyya 10)

Heuristic lexical

methods

9/34


Lexical entailment scores – current practice

• Count covered/uncovered• (Majumdar and Bhattacharyya, 2010; Clark and Harrison, 2010)

• Similarity estimation• (Corley and Mihalcea, 2005; Zanzotto and Moschitti, 2006)

• Vector space• (MacKinlay and Baldwin, 2009)

Mostly heuristic

10/34


model


we address

with a

which

lexical textual

inference


model


art

1 2 3

11/34


Probabilistic model – overview


T

H which battle was Napoleon defeated

Battle of Waterloo French army led by Napoleon was crushed

)( HTP

knowledge integration

term-level

sentence-level

)( 3hTP )( 1hTP

t1 t2 t3 t4 t5 t6

h1 h2 h3

)( 2hTP

annotations are available at

sentence-level only

x1 x2 x3

12/34


Knowledge integration

• Distinguish resources reliability levels• WordNet >> similarity-based thesauri (Lin, 1998; Pantel and Lin, 2002)

• Consider transitive chains length• The longer a chain is the lower its probability

• Consider multiple pieces of evidence• More evidence means higher probability

which battle was Napoleon defeated


t

rule1

rule2

transitive chainr

is a rulemultiple evidence

13/34


Probabilistic model – term level


T


)(rR

chainr

rR

chain

htP )()(


ORt'

ris a rule

is the reliability level of the

resource which suggested r

1)( hTP

ACL 11 short paper this level parameters: one per input lexical resource

t1 t2 t3 t4 t5 t6

h1 h2 h3

multiple evidence

)(

)](1[hchainsc

c

htP

14/34

𝜃𝑊𝑁

𝜃𝑊𝑖𝑘𝑖


Probabilistic model – overview


T



)( HTP

knowledge integration

term-level

sentence-level

)( 3hTP )( 2hTP )( 1hTP

15/34


Probabilistic model – sentence level


we define hidden binary random variables:

xt = 1 iff ht is inferred from T (zero otherwise)


h1 h2 h3

x1 x2 x3

)( 3hTP )( 2hTP )( 1hTP

final sentence-level decision

AND

y

Modeling with AND gate:• Most intuitively• However

• Too strict• Does not model terms dependency

16/34


Probabilistic model – sentence level


),|()( 1 jxiykyPkq tttij

this level parameters

}1,0{,, kji


h1 h2 h3

x1 x2 x3

y1 y2 y3

)( 3hTP )( 2hTP )( 1hTP


we define another binary random variable:

yt – inference decision for the prefix h1… ht

P(yt = 1) is dependent on yt-1 and xt

M-PLM

xt = 1 iff ht is inferred by T (zero otherwise)

17/34


M-PLM – inference


)1(),|()()()1(2

11

n

tttttn xyyPxPxPyP

1,,,

,,

12

1

nn

n

yyy

xx


h1 h2 h3

x1 x2 x3

y1 y2 y3

)( 3hTP )( 2hTP )( 1hTP


qij(k)

)2()()()()()(}1,0{,

1

ji

ijtttt kqjxPikyPk

)3()()( 11 kxPk

can be computed efficiently with a forward algorithm

)4()1()1( nnyP

18/34


M-PLM – summary

Annotation final sentence-level decision


Parameters

resource Observed

Lexical rules which link terms

Learning we developed EM

scheme to jointly learn all parameters

Hidden

19/34


so how our model does?


In which battle was Napoleon defeated?

In the Battle of



Napoleon, was

crushed.

Napoleon was

Emperor of the

French from 1804 to

1815.





12

3

4

5

20/34


we address

with a

which

lexical textual

inference


model


art

1 2 3

21/34


Evaluations – data sets

improves sate-of-the-artprincipled probabilistic model lexical textual inference

Ranking in passage retrieval for QA

(Wang et al. 07)

5700/1500 question-candidate answer pairs from TREC 8-13

Manually annotated

Notable line of work from recent years: Punyakanok et al. 04, Cui et al. 05, Wang et al. 07, Heilman and Smith 10, Wang and Manning 10

Recognizing textual entailment

within a corpus

20,000 text-hypothesis pairs in each RTE-5, RTE-6

Originally constructed for classification

22/34


Evaluations – baselines

Syntactic generative models• Require parsing• Apply sophisticated machine learning methods(Punyakanok et al. 04, Cui et al. 05, Wang et al. 07, Heilman and Smith 10, Wang and Manning 10)

Lexical model – Heuristically Normalized-PLM• AND-gate for the sentence-level• Add heuristic normalizations to addresses its disadvantages (TextInfer

workshop 11)

• Performance in line with best RTE systems


HN-PLM

23/34


QA results – syntactic baselines


MAP MRR40

45

50

55

60

65

70

60.91

69.51

Punyakanok et al.Cui et al. 05Wang and ManningWang et al. 07Heilman and Smith

24/34


QA results – syntactic baselines + HN-PLM


MAP MRR40

45

50

55

60

65

70

60.91

69.51

Punyakanok et al.Cui et al. 05Wang and ManningWang et al. 07Heilman and SmithHN-PLM

+0.7%

+1%

25/34


QA results – baselines + M-PLM


MAP MRR40

45

50

55

60

65

70

60.91

64.38

72.69

69.51

Punyakanok et al.Cui et al. 05Wang and ManningWang et al. 07Heilman and SmithHN-PLMM-PLM

+3.2%

+3.5%

M-PLM

26/34


RTE results – M-PLM vs. HN-PLM


RTE-5 MAP RTE-5 MRR RTE-6 MAP RTE-6 MRR40

45

50

55

60

65

70

75

80

85

HN-PLMM-PLM

+7.3%

+1.9%

+6.0%+3.6%

27/34


First approach - summary

Clean probabilistic lexical model• As a lexical component or as a stand alone inference system• Superiority of principled methods over heuristic ones • Attractive passage retrieval ranking method• Code available - BIU NLP downloads

M-PLM limits• Processing is term order dependent• Lower performance on classification vs. HN-PLM

does not normalize well across hypotheses length

improves state-of-the-artprincipled probabilistic model lexical textual inference

28/34


we

addr

ess

with

a

whi

ch

lexical textual

inference


model


art

1 2 3

A (v

ery)

new

4

second approach:

resources as observers

29/34


each resource is a witness


Battle of Waterloo French army led by Napoleon was crushedt1 t2 t3 t4 t5 t6

h1 h2 h3

t'

30/34


Bottom-up witnesses model


Battle of Waterloo French army led by Napoleon was crushedt1 t2 t3 t4 t5 t6

h1 h2 h3

x1 x2 x3

AND

y

𝑃 (𝑊 (h𝑖 )|𝑥 𝑖=1)= ∏𝑤∈𝑊 (h𝑖)

𝜃𝑤 ⋅ ∏𝑤∉𝑊 (h𝑖)

(1−𝜃𝑤)

𝜂0≝𝑃 (𝑥 𝑖=1∨𝑦=0)𝜂1≝𝑃 (𝑥𝑖=1∨𝑦=1)

𝜃𝑤=𝑃 (𝑤 (𝑥 𝑖 )=1∨𝑥 𝑖=1)

𝜏𝑤=𝑃 (𝑤 (𝑥 𝑖 )=1∨𝑥𝑖=0) 𝑃 (𝑊 (h𝑖 )|𝑥 𝑖=0 )= ∏𝑤∈𝑊 (h𝑖)

𝜏𝑤 ⋅ ∏𝑤∉𝑊 (h𝑖)

(1−𝜏𝑤)

Likelihood

31/34


Advantages of the second approach

Inference:

• Hypothesis length is not an issue• Learn from non-entailing resources• and provide a recall and precision estimation for a resource

¿𝑃 (𝑊 (𝐻 )|𝑦=1 ) ⋅ 𝑃 (𝑦=1)

𝑃 (𝑊 (𝐻 ))

¿𝑃 (𝑊 (𝐻 )|𝑦=1 ) ⋅ 𝑃 (𝑦=1)

𝑃 (𝑊 (𝐻 )|𝑦=0 ) ⋅ 𝑃 ( 𝑦=0 )+𝑃 (𝑊 (𝐻 )|𝑦=1 ) ⋅ 𝑃 (𝑦=1)

32/34


(near) future plans

• Context model• There are other languages than English

• Deploy the new version of a Wikipedia-base lexical resource with the Italian dump

• Test the probabilistic lexical models for other languages• Cross language textual entailment

33/34


Cross Language Textual Entailment

quale battaglia fu sconfitto Napoleone


Italian monolingual

English-Italian phrase table

English monolingual

Thank

You

34/34

Bar Ilan University @ IBM July 2012 35/34


Demo examples:

[Bap,WN] no transitivityJack and Jill go_up the hill to fetch a pail of water Jack and Jill climbed a mountain to get a bucket of fluid

[WN,Wiki] <show graph>Barak Obama's Buick got stuck in Dublin in a large Irish crowdUnited_States_President's car got stuck in Ireland, surrounded by many people

Barak Obama - WN is out of date, need a new version of Wikipedia

Bill_Clinton's Buick got stuck in Dublin in a large Irish crowdUnited_States_President's car got stuck in Ireland, surrounded by many people

------------------------------------------------------------------------------[Bap,WN] this time with <transitivity & multiple evidence> Jack and Jill go_up the hill to fetch a pail of water Jack and Jill climbed a mountain to get a bucket of fluid

[VO,WN,Wiki]in the Battle_of_Waterloo the French army led by Napoleon was crushedin which battle Napoleon was defeated?

------------------------------------------------------------------------------[all]1. in the Battle_of_Waterloo the French army led by Napoleon was crushed 72%

2. Napoleon was not tall enough to win the Battle_of_Waterloo 47%

3. at Waterloo Napoleon did surrender... Waterloo - finally facing my Waterloo 34%

4. Napoleon engaged in a series of wars, and won many 47%

5. Napoleon was Emperor of the French from 1804 to 1815 9% [a bit long run]

36/34

Documents

Probabilistic Lexical Models for Textual Inference Eyal Shnarch, Ido Dagan, Jacob Goldberger Probabilistic Lexical Models for Textual Inference Eyal Shnarch,