FBK August 2012

PLIS : a Probabilistic Lexical Inference System

Eyal Shnarch, Ido Dagan, Jacob Goldberger

PLIS - Probabilistic Lexical Inference System 1/34

FBK August 2012

The entire talk in a single sentence

we

addr

ess

with

a

whi

ch

lexical textual

inference

principled probabilistic

model

improves state-of-the

art

PLIS - Probabilistic Lexical Inference System 2/34

FBK August 2012

Outline

we

addr

ess

with

a

whi

ch

lexical textual

inference

principled probabilistic

model

improves state-of-the

art

1 2 3

PLIS - Probabilistic Lexical Inference System 3/34

FBK August 2012

we

addr

ess

with

a

whi

ch

lexical textual

inference

principled probabilistic

model

improves state-of-the

art

1 2 3

PLIS - Probabilistic Lexical Inference System 4/34

FBK August 2012

Textual entailment

improves state-of-the-artprincipled probabilistic model lexical textual inference

PLIS - Probabilistic Lexical Inference System

in Belgium Napoleon was defeated

At Waterloo Napoleon did

surrender...Waterloo - finally facing my

Waterloo

T entails H if typically a human reading T would infer that H is most likely true

Text Hypothesis

5/34

FBK August 2012

Textual inference – useful in many NLP apps

improves state-of-the-artprincipled probabilistic model

in Belgium Napoleon was defeated

In the Battle of Waterloo, 18 Jun 1815,

the French army, led by

Napoleon, was crushed.

Napoleon was

Emperor of the

French from 1804 to

1815.

lexical textual inference

Napoleon was not tall enough to win

the Battle of Waterloo

At Waterloo Napoleon did surrender...Waterloo - finally facing my Waterloo

Napoleon engaged in a series of wars, and won many

PLIS - Probabilistic Lexical Inference System 6/34

FBK August 2012

BIU NLP lab

PLIS - Probabilistic Lexical Inference System

Chaya Liebeskind

7/34

FBK August 2012

Lexical textual inference

• Complex systems use parser

• Lexical inference rules link terms from T to H• Lexical rules come from lexical resources• H is inferred from T iff all its terms are inferred

Improves state-of-the-artprincipled probabilistic model lexical textual inference

In the Battle of Waterloo, 18 Jun 1815, the French army, led by Napoleon, was crushed.

in Belgium Napoleon was defeated

PLIS - Probabilistic Lexical Inference System

1st or 2nd order co-occurrence

Text Hypothesis

8/34

FBK August 2012 PLIS - Probabilistic Lexical Inference System

Textual inference for ranking

Improves state-of-the-artprincipled probabilistic model lexical textual inference

In which battle was Napoleon defeated?

In the Battle of Waterloo, 18 Jun 1815,

the French army, led by

Napoleon, was crushed.

Napoleon was

Emperor of the

French from 1804 to

1815.

Napoleon was not tall enough to win

the Battle of Waterloo

At Waterloo Napoleon did surrender...Waterloo - finally facing my Waterloo

Napoleon engaged in a series of wars, and won many

12

3

4

5

9/34

FBK August 2012

Ranking textual inference – prior work

Improves state-of-the-artprincipled probabilistic model lexical textual inference

• Transform T’s parsed tree into H’s parsed tree• Based on principled ML model(Wang et al. 07, Heilman and Smith 10, Wang and Manning 10)

Syntactic-based

methods

• Fast, easy to implement, highly competitive• Practical across genres and languages(MacKinlay and Baldwin 09, Clark and Harrison 10,

Majumdar and Bhattacharyya 10)

Heuristic lexical

methods

PLIS - Probabilistic Lexical Inference System 10/34

FBK August 2012

Lexical entailment scores – current practice

• Count covered/uncovered• (Majumdar and Bhattacharyya, 2010; Clark and Harrison, 2010)

• Similarity estimation• (Corley and Mihalcea, 2005; Zanzotto and Moschitti, 2006)

• Vector space• (MacKinlay and Baldwin, 2009)

Mostly heuristic

PLIS - Probabilistic Lexical Inference System

principled probabilistic

model

11/34

FBK August 2012

we

addr

ess

with

a

whi

ch

lexical textual

inference

principled probabilistic

model

improves state-of-the

art

1 2 3

PLIS - Probabilistic Lexical Inference System 12/34

FBK August 2012

Probabilistic model – overview

Improves state-of-the-artprincipled probabilistic model lexical textual inference

T

H which battle was Napoleon defeated

Battle of Waterloo French army led by Napoleon was crushed

)( HTP

knowledge integration

term-level

sentence-level

)( 3hTP )( 1hTP

t1 t2 t3 t4 t5 t6

h1 h2 h3

)( 2hTP

annotations are available at

sentence-level only

PLIS - Probabilistic Lexical Inference System

x1 x2 x3

13/34

FBK August 2012

Knowledge integration

• Distinguish resources reliability levels• WordNet >> similarity-based thesauri (Lin, 1998; Pantel and Lin, 2002)

• Consider transitive chains length• The longer a chain is the lower its probability

• Consider multiple pieces of evidence• More evidence means higher probability

PLIS - Probabilistic Lexical Inference System

which battle was Napoleon defeated

Battle of Waterloo French army led by Napoleon was crushed

t

rule1

rule2

transitive chainr

is a rulemultiple evidence

14/34

FBK August 2012

we

addr

ess

with

a

whi

ch

lexical textual

inference

1st approachForward

improves state-of-the

art

1 2 3

PLIS - Probabilistic Lexical Inference System 15/34

FBK August 2012

Probabilistic model – term level

Improves state-of-the-artprincipled probabilistic model lexical textual inference

T

H which battle was Napoleon defeated

)(rR

chainr

rR

chainhtP )()(

Battle of Waterloo French army led by Napoleon was crushed

ORt'

ris a rule

is the reliability level of the

resource which suggested r

1)( hTP

ACL 11 short paper this level parameters: one per input lexical resource

t1 t2 t3 t4 t5 t6

h1 h2 h3

multiple evidence

)(

)](1[hchainsc

chtP

PLIS - Probabilistic Lexical Inference System

𝜃𝑊𝑁

𝜃𝑊𝑖𝑘𝑖

16/34

FBK August 2012

Probabilistic model – overview

Improves state-of-the-artprincipled probabilistic model lexical textual inference

T

H which battle was Napoleon defeated

Battle of Waterloo French army led by Napoleon was crushed

)( HTP

knowledge integration

term-level

sentence-level

)( 3hTP )( 2hTP )( 1hTP

PLIS - Probabilistic Lexical Inference System 17/34

FBK August 2012

Probabilistic model – sentence level

Improves state-of-the-artprincipled probabilistic model lexical textual inference

we define hidden binary random variables:

xt = 1 iff ht is inferred from T (zero otherwise)

H which battle was Napoleon defeated

h1 h2 h3

x1 x2 x3

)( 3hTP )( 2hTP )( 1hTP

final sentence-level decision

AND

y

Modeling with AND gate:• Most intuitively• However

• Too strict• Does not model terms dependency

PLIS - Probabilistic Lexical Inference System 18/34

HN-PLM

FBK August 2012

Probabilistic model – sentence level

Improves state-of-the-artprincipled probabilistic model lexical textual inference

),|()( 1 jxiykyPkq tttij

this level parameters

}1,0{,, kji

H which battle was Napoleon defeated

h1 h2 h3

x1 x2 x3

y1 y2 y3

)( 3hTP )( 2hTP )( 1hTP

final sentence-level decision

we define another binary random variable:

yt – inference decision for the prefix h1… ht

P(yt = 1) is dependent on yt-1 and xt

M-PLM

xt = 1 iff ht is inferred by T (zero otherwise)

PLIS - Probabilistic Lexical Inference System 19/34

FBK August 2012

M-PLM – inference

Improves state-of-the-artprincipled probabilistic model lexical textual inference

)1(),|()()()1(2

11

n

tttttn xyyPxPxPyP

1,,,,,

12

1

nn

n

yyyxx

H which battle was Napoleon defeated

h1 h2 h3

x1 x2 x3

y1 y2 y3

)( 3hTP )( 2hTP )( 1hTP

final sentence-level decision

qij(k)

)2()()()()()(}1,0{,

1

ji

ijtttt kqjxPikyPk

)3()()( 11 kxPk

can be computed efficiently with a forward algorithm

)4()1()1( nnyP

PLIS - Probabilistic Lexical Inference System 20/34

FBK August 2012

M-PLM – summary

Annotation final sentence-level decision

Improves state-of-the-artprincipled probabilistic model lexical textual inference

Parameters

resource Observed

Lexical rules which link terms

Learning we developed EM

scheme to jointly learn all parameters

Hidden

PLIS - Probabilistic Lexical Inference System 21/34

M-PLM limits• Processing is term order dependent• does not normalize well across hypotheses length

FBK August 2012

we

addr

ess

with

a

whi

ch

lexical textual

inference

2nd approachBackward

improves state-of-the

art

1 2 3

PLIS - Probabilistic Lexical Inference System 22/34

FBK August 2012

The main idea: each resource is a witness

PLIS - Probabilistic Lexical Inference System

which battle was Napoleon defeated

Battle of Waterloo French army led by Napoleon was crushedt1 t2 t3 t4 t5 t6

h1 h2 h3

t'

Improves state-of-the-artprincipled probabilistic model lexical textual inference

23/34

FBK August 2012

Backward witnesses model

PLIS - Probabilistic Lexical Inference System

which battle was Napoleon defeated

Battle of Waterloo French army led by Napoleon was crushedt1 t2 t3 t4 t5 t6

h1 h2 h3

x1 x2 x3

AND

y

𝑃 (𝑊 (h𝑖 )|𝑥 𝑖=1 )= ∏𝑤∈𝑊 (h𝑖)

𝜃𝑤 ⋅ ∏𝑤∉𝑊 (h𝑖)

(1−𝜃𝑤)

𝜂0≝𝑃 (𝑥 𝑖=1∨𝑦=0)𝜂1≝𝑃 (𝑥𝑖=1∨𝑦=1)

𝜃𝑤=𝑃 (𝑤 (𝑥 𝑖 )=1∨𝑥 𝑖=1)𝜏𝑤=𝑃 (𝑤 (𝑥 𝑖 )=1∨𝑥𝑖=0) 𝑃 (𝑊 (h𝑖 )|𝑥 𝑖=0 )= ∏

𝑤∈𝑊 (h𝑖)𝜏𝑤 ⋅ ∏

𝑤∉𝑊 (h𝑖)(1−𝜏𝑤)

Likelihood

Improves state-of-the-artprincipled probabilistic model lexical textual inference

24/34

FBK August 2012

Advantages of the second approach

Inference:

• Hypothesis length is not an issue• Learn from non-entailing resources• and provide a recall and precision estimation for a resource

PLIS - Probabilistic Lexical Inference System

¿𝑃 (𝑊 (𝐻 )|𝑦=1 ) ⋅ 𝑃 (𝑦=1)

𝑃 (𝑊 (𝐻 ))

¿𝑃 (𝑊 (𝐻 )|𝑦=1 ) ⋅ 𝑃 (𝑦=1)

𝑃 (𝑊 (𝐻 )|𝑦=0 ) ⋅ 𝑃 ( 𝑦=0 )+𝑃 (𝑊 (𝐻 )|𝑦=1 ) ⋅ 𝑃 (𝑦=1)

Improves state-of-the-artprincipled probabilistic model lexical textual inference

25/34

FBK August 2012

we

addr

ess

with

a

whi

ch

lexical textual

inference

principled probabilistic

model

improves state-of-the

art

1 2 3

PLIS - Probabilistic Lexical Inference System 26/34

FBK August 2012

Evaluations – data sets

improves sate-of-the-artprincipled probabilistic model lexical textual inference

Ranking in passage retrieval for QA

(Wang et al. 07)

5700/1500 question-candidate answer pairs from TREC 8-13

Manually annotated

Notable line of work from recent years: Punyakanok et al. 04, Cui et al. 05, Wang et al. 07, Heilman and Smith 10, Wang and Manning 10

Recognizing textual entailment

within a corpus

20,000 text-hypothesis pairs in each RTE-5, RTE-6

Originally constructed for classification

PLIS - Probabilistic Lexical Inference System 27/34

FBK August 2012

Evaluations – baselines

Syntactic generative models• Require parsing• Apply sophisticated machine learning methods(Punyakanok et al. 04, Cui et al. 05, Wang et al. 07, Heilman and Smith 10, Wang and Manning 10)

Lexical model – Heuristically Normalized-PLM• AND-gate for the sentence-level• Add heuristic normalizations to addresses its disadvantages (TextInfer

workshop 11)

• Performance in line with best RTE systems

improves sate-of-the-artprincipled probabilistic model lexical textual inference

HN-PLM

PLIS - Probabilistic Lexical Inference System 28/34

FBK August 2012

QA results – syntactic baselines

improves sate-of-the-artprincipled probabilistic model lexical textual inference

MAP MRR40

45

50

55

60

65

70

60.91

69.51

Punyakanok et al.Cui et al. 05Wang and ManningWang et al. 07Heilman and Smith

PLIS - Probabilistic Lexical Inference System 29/34

FBK August 2012

QA results – syntactic baselines + HN-PLM

improves sate-of-the-artprincipled probabilistic model lexical textual inference

MAP MRR40

45

50

55

60

65

70

60.91

69.51

Punyakanok et al.Cui et al. 05Wang and ManningWang et al. 07Heilman and SmithHN-PLM

+0.7%

+1%

PLIS - Probabilistic Lexical Inference System 30/34

FBK August 2012

QA results – baselines + M-PLM

improves sate-of-the-artprincipled probabilistic model lexical textual inference

MAP MRR40

45

50

55

60

65

70

60.91

64.38

72.69

69.51

Punyakanok et al.Cui et al. 05Wang and ManningWang et al. 07Heilman and SmithHN-PLMM-PLM

+3.2%

+3.5%

M-PLM

PLIS - Probabilistic Lexical Inference System 31/34

FBK August 2012

RTE results – M-PLM vs. HN-PLM

improves sate-of-the-artprincipled probabilistic model lexical textual inference

RTE-5 MAP RTE-5 MRR RTE-6 MAP RTE-6 MRR40

45

50

55

60

65

70

75

80

85

HN-PLMM-PLM

+7.3%

+1.9%

+6.0%+3.6%

PLIS - Probabilistic Lexical Inference System 32/34

FBK August 2012

Future work: Cross Lingual Textual Entailment

PLIS - Probabilistic Lexical Inference System

quale battaglia fu sconfitto Napoleone

Battle of Waterloo French army led by Napoleon was crushed

Italian monolingual

English-Italian phrase table

English monolingual

improves state-of-the-artprincipled probabilistic model lexical textual inference

33/34

FBK August 2012

summary

Clean probabilistic lexical model• As a lexical component or as a stand alone inference system• Superiority of principled methods over heuristic ones • Attractive passage retrieval ranking method• Code available – Kateryna has already took it – be our next user!

improves state-of-the-artprincipled probabilistic model lexical textual inference

PLIS - Probabilistic Lexical Inference System

Thank

You

34/34

FBK August 2012 PLIS - Probabilistic Lexical Inference System 35/34

FBK August 2012

Demo examples:

[Bap,WN] no transitivityJack and Jill go_up the hill to fetch a pail of water Jack and Jill climbed a mountain to get a bucket of fluid

[WN,Wiki] <show graph>Barak Obama's Buick got stuck in Dublin in a large Irish crowdUnited_States_President's car got stuck in Ireland, surrounded by many people

Barak Obama - WN is out of date, need a new version of Wikipedia

Bill_Clinton's Buick got stuck in Dublin in a large Irish crowdUnited_States_President's car got stuck in Ireland, surrounded by many people

------------------------------------------------------------------------------[Bap,WN] this time with <transitivity & multiple evidence> Jack and Jill go_up the hill to fetch a pail of water Jack and Jill climbed a mountain to get a bucket of fluid

[VO,WN,Wiki]in the Battle_of_Waterloo the French army led by Napoleon was crushedin which battle Napoleon was defeated?

------------------------------------------------------------------------------[all]1. in the Battle_of_Waterloo the French army led by Napoleon was crushed 72%

2. Napoleon was not tall enough to win the Battle_of_Waterloo 47%

3. at Waterloo Napoleon did surrender... Waterloo - finally facing my Waterloo 34%

4. Napoleon engaged in a series of wars, and won many 47%

5. Napoleon was Emperor of the French from 1804 to 18159% [a bit long run]

PLIS - Probabilistic Lexical Inference System 36/34