1/21

Automatic Discovery of Intentions in Text and its Application to Question

Answering (ACL 2005 Student Research Workshop )

2/17

Abstract Idea:

Semantic relations b/w text concepts denote the core elements of lexical semantics.

Model: Automatic detection of INTENTION semantic relation.

Method: Syntactic + semantic features for a SVM learning

classifier. Results:

90.41% accuracy. Application:

Question Answering.

3/17

Introduction (1/2) Intentions :

Express a human’s goal-oriented private states of mind, including intents, objectives, aims, and purposes (might not be explicitly stated).

Examples: “Mary is going to buy a TV set.” Customer: Where do you have the $1 cups? Salesman: How many do you want?

Problem: Current state-of-the-art NLP systems cannot

extract intentions from open text.

4/17

Introduction (2/2) Intentions :

Expression of a particular action that shall take place in the future, in which the speaker is some sort of agent.

Method: Identify intentions in domain independent

texts. Employ machine learning algorithms to

create models that locate intentions in a given paragraph using a set of 6 syntactic and semantic features.

5/17

Previous Work Purely probabilistic models, empirical

methods, hand-coded constraints…… Decision tree, neural networks, memory-

based learning, support vector machines……

Wiebe et al. (2004) focused on the detection of subjective language such as opinions, evaluations, or emotions in text.

Numerous philosophical studies discuss how intentions relate to other psychological concepts, such as, beliefs, desires, hopes, or expectations.

6/17

Syntax and Semantics of Intention (1/2) Syntax patterns:

SemCor text collection manually classifying the first 2,700 sentences into sentences that contain or not intentions 46 examples were identified.

Focus on detecting intentions encoded by VB1 to VB2.

Example: “John intends to meet Mary today.” (O) “Mary began to play with the dog. (X)

7/17

Syntax and Semantics of Intention (2/2) Semantics of intentions:

Representation for the intention semantic relation: INT(e1, x1, e2)

e1: the event denoting the intention x1: denotes the person that has the intention e2: the intended action or event INT: THEME, LOCATION, MANNER, SOURCE……

Example: Sentence: John intends to meet Mary today. Representation:

Allow the derivation of inference rules.

8/17

Learning Model (1/4) Experimental data:

The first 10,000 sentences of the SemCor2.0 extract 1,873 sentences contain 115 intentions (manually identified)

5-fold-cross-validation: 115 positive + 258 negative examples.

115 258

80%20%

9/17

Learning Model (2/4) 6 Features for intention (VB1 to VB2):

“Mary intends to revise the paper.” 1. The semantic class of the VB1 verb’s

agent or specializations of it. Use an in-house semantic parser to retrieve the

AGENT of the VB1 verb. Value: WordNet semantic class. Specialize the semantic class s of a concept w by

replacing it with its immediate hyponym (h) that subsumes w.

Mary names a person. entity#1 2. The semantic class of the VB1 verb or its

specializations. intend#1 wish#3

10/17

Learning Model (3/4) 6 Features for intention (VB1 to VB2):

“Mary intends to revise the paper.” 3. The semantic class of the VB2 verb’s agent.

Mary names a person. entity#1 4. The semantic class of the VB2 verb or its

specializations. revise#1 act#1

5. A flag indicating if the VB1 verb has an affirmative or a negative form.

“John wants to go for a walk.” (O) “John doesn’t want to go for a walk.” (X)

11/17

Learning Model (4/4) 6 Features for intention (VB1 to VB2):

“Mary intends to revise the paper.” 6. The type of the analyzed sentence.

Primarily concerned with questions. “Where do you plan to go for a walk?”

“Do you plan to go for a walk?” Values: wh-words (begin a question) or n/a (other

types). ? Negative form of the VB2 verb: b/c it does not

effect the objective attribute of the intention. “John intends not to go for a walk.” (O) “John doesn’t intend to go for a walk.” (X)

12/17

Experimental Results (1/3)

5 experiments: 1. Impact of specialization 2. Learning curves 3. Feature impact on the SVM models 4. Impact of word sense

disambiguation 5. C5 results

13/17

Experimental Results (2/3) 1. Impact of specialization:

Using the LIBSVM package and the WordNet semantic classes.

Specialize VB2’s agent semantic class accuracy remained constant specialization of the VB2’s agent does not influence the performance 3rd feature’s value = 2nd specialization level

2. Learning curves How many training examples are needed to reach

90.41% accuracy? All 3 models exhibit a similar behavior with respect

to the change in the training set size.

14/17

Experimental Results (3/3) 3. Feature impact on the SVM models

Perform experiments that use only 5 out of 6 features.

The most influential attribute: VB1 verb’s semantic class or its specializations.

Syntactic features bring an average increase in accuracy of 3.50%.

4. Impact of word sense disambiguation 5. C5 results

B/c of the relatively small numbers of training instances, C5 ignores some of the features and makes wrong decisions.

15/17

Application to QA

Q involving intentions cannot be answered only by keyword-based or simple surface-level matching techniques.

Examples

16/17

Conclusions A method to detect the INTENT

relation encoded by the sentence-level pattern VB1 to VB2 with a 90.41% accuracy.

Future work: Investigate the other INTENTION patterns

as well as other semantic relations such as MOTIVE, IMPLICATION, or MEANING.

Incorporate INTENTION detection module into a QA system and show its impact.

17/17

ThanQ

18/17

Syntactic patterns

Cover 95.65% of 46 examples.

19/17

Semantics _ implications

1. John intends to start his car to go to the park. John intends to go to the park.

2. John intends to buy a car. John intends to pay for it. 4. John intends to go to the park. He’s starting his car

right now. John intends to start his car.

5. John intends to swim in the pool. He knows he’s going to catch a cold in the cold water. John intends to catch a cold.

20/17

1. Impact of specialization

21/17

2. Learning curves

22/17

3. Feature impact on the SVM models

23/17

4. Impact of word sense disambiguation5. C5 results

24/17

Applications to QA