View
216
Download
0
Category
Tags:
Preview:
Citation preview
May 29, 2008 LREC 2008, Marakech, Morocco
Talk Overview
Fine-grained sentiment analysis Definitions Examples
Opinion topic annotation Definitions Issues Approach and Corpus IA agreement
May 29, 2008 LREC 2008, Marakech, Morocco
Background
Sentiment Analysis:
Extraction and representation of attitudes, evaluations, opinions, and sentiment in text.
Fine-grained Sentiment Analysis:
At the level of individual expressions of opinions.
May 29, 2008 LREC 2008, Marakech, Morocco
The Australian press has launched a bitter attack on Italy after seeing their beloved Socceroos eliminated on a controversial late penalty. Italian coach Lippi has been blasted for his favorable comments toward the penalty.
Lippi is preparing his side for the upcoming clash with Ukraine. He hailed 10-man Italy's determination to beat Australia and reiterated that the penalty was rightly given.
Fine-grained vs. Coarse-grained Sentiment Analysis
Coarse-grained Sentiment classification Useful in the product
review domain
Fine-grained Individual expressions of
opinions Multiple opinions per
document (even sentence)
Review 1
Review 2
Positive
Negative
[SThe Australian press] has launched a bitter attack on [TItaly] after seeing their beloved [TSocceroos] eliminated on a controversial late [Tpenalty]. [S+TItalian coach Lippi] has also been blasted for his favorable comments toward [Tthe penalty].
Lippi is preparing his side for the upcoming clash with Ukraine. [SHe] hailed 10-man [TItaly]'s determination to beat Australia and reiterated that the [Tpenalty] was rightly given.
May 29, 2008 LREC 2008, Marakech, Morocco
Fine-grained opinions: ExampleThe Australian press has launched a bitter attack on Italy.
May 29, 2008 LREC 2008, Marakech, Morocco
Fine-grained opinions: Example
Opinion trigger (opinion words)
Source (opinion holder) Polarity – positive/negative Strength Topic (target)
The Australian press has launched a bitter attack on Italy.
Definitions differ, but five main components:
May 29, 2008 LREC 2008, Marakech, Morocco
Fine-grained opinions: Example
Opinion trigger (opinion words)
Source (opinion holder) Polarity – positive/negative Strength Topic (target)
The Australian press has launched a bitter attack on Italy.
Definitions differ, but five main components:
launched a bitter attack
May 29, 2008 LREC 2008, Marakech, Morocco
Fine-grained opinions: Example
Opinion trigger (opinion words)
Source (opinion holder) Polarity – positive/negative Strength Topic (target)
[SThe Australian press] has launched a bitter attack on Italy.
Definitions differ, but five main components:
launched a bitter attack
The Australian press
May 29, 2008 LREC 2008, Marakech, Morocco
Fine-grained opinions: Example
Opinion trigger (opinion words)
Source (opinion holder) Polarity – positive/negative Strength Topic (target)
[SThe Australian press] has launched a bitter attack on Italy.
Definitions differ, but five main components:
launched a bitter attack
The Australian press
negative
May 29, 2008 LREC 2008, Marakech, Morocco
Fine-grained opinions: Example
Opinion trigger (opinion words)
Source (opinion holder) Polarity – positive/negative Strength Topic (target)
[SThe Australian press] has launched a bitter attack on Italy.
Definitions differ, but five main components:
launched a bitter attack
The Australian press
negative
high
May 29, 2008 LREC 2008, Marakech, Morocco
Fine-grained opinions: Example
Opinion trigger (opinion words)
Source (opinion holder) Polarity – positive/negative Strength Topic (target)
[SThe Australian press] has launched a bitter attack on [TItaly]
Definitions differ, but five main components:
launched a bitter attack
The Australian press
negative
high
Italy
May 29, 2008 LREC 2008, Marakech, Morocco
Fine-grained opinions
Five components Source (opinion holder)
e.g. [Bethard et al., 2004] [Choi et al., 2005] [Kim and Hovy, 2006] Opinion trigger (opinion words)
e.g. [Yu and Hatzivassiloglou, 2003] [Riloff and Wiebe, 2003] Polarity – positive/negative
As above Strength
e.g. [Wilson et al. 2004] Topic (target)
????
May 29, 2008 LREC 2008, Marakech, Morocco
Annotating Topics of Fine-grained Opinions
Definitions Issues Approach and Corpus IA agreement
May 29, 2008 LREC 2008, Marakech, Morocco
Examples
(1)[OH John] likes Marseille for its weather and cultural diversity.
(2)[OH Al] thinks that the government should tax gas more in
order to curb CO2 emissions.
May 29, 2008 LREC 2008, Marakech, Morocco
Definitions(1)[OH John] likes Marseille for its weather and cultural diversity.
May 29, 2008 LREC 2008, Marakech, Morocco
Definitions(1)[OH John] likes Marseille for its weather and cultural diversity.
Topic: city of Marseille
Topic - the real-world object, event or abstract entity that is the subject of the opinion as intended by the opinion holder
May 29, 2008 LREC 2008, Marakech, Morocco
Definitions(1)[OH John] likes [TOPIC SPAN Marseille] for its weather and cultural
diversity.
Topic: city of Marseille
Topic - the real-world object, event or abstract entity that is the subject of the opinion as intended by the opinion holder
Topic span - the closest, minimal span of text that mentions the topic
May 29, 2008 LREC 2008, Marakech, Morocco
Definitions(1)[OH John] likes [TARGET+TOPIC SPAN Marseille] for its weather and
cultural diversity.
Topic: city of Marseille
Topic - the real-world object, event or abstract entity that is the subject of the opinion as intended by the opinion holder
Topic span - the closest, minimal span of text that mentions the topic
Target span - the span of text that covers the syntactic surface form comprising the contents of the opinion
May 29, 2008 LREC 2008, Marakech, Morocco
Definitions
(2)[OH Al] thinks that the government should tax gas
more in order to curb CO2 emissions.
May 29, 2008 LREC 2008, Marakech, Morocco
Definitions
(2)[OH Al] thinks that [TARGET SPAN the government should
tax gas more in order to curb CO2 emissions].
May 29, 2008 LREC 2008, Marakech, Morocco
Definitions
(2)[OH Al] thinks that [TARGET SPAN [TOPIC SPAN? the
government] should [TOPIC SPAN? tax gas] more in order
to [TOPIC SPAN? curb [TOPIC SPAN? CO2 emissions]]].
May 29, 2008 LREC 2008, Marakech, Morocco
Definitions
(2)[OH Al] thinks that [TARGET SPAN the government should
tax gas more in order to curb CO2 emissions].
Context:
(3) Although he doesn’t like government imposed taxes, he thinks that a fuel tax is the only effective solution.
May 29, 2008 LREC 2008, Marakech, Morocco
Definitions
(2)[OH Al] thinks that [TARGET SPAN the government should
[TOPIC SPAN tax gas] more in order to curb CO2
emissions].
Context:
(3) Although he doesn’t like government imposed taxes, he thinks that a fuel tax is the only effective solution.
May 29, 2008 LREC 2008, Marakech, Morocco
Related Work
Product reviews E.g. Kobayashi et al. (2004), Yi et al. (2003), Popescu and Etzioni
(2005), Hu and Liu (2004 Limit “topics” to mentions of product names, components, and
their attributes Lexicon look-up Focused on methods for lexicon acquisition
MPQA corpus (Wiebe, Wilson, Cardie, 2004) Fine-grained opinions Topic annotation deemed too difficult Target span annotation is underway
Kim & Hovy (2006) Target span extraction using semantic frames Limited evaluation
May 29, 2008 LREC 2008, Marakech, Morocco
Issues in Opinion Topic Identification Multiple potential topics mentioned within a single target span
(2)[OH Al] thinks that [TARGET SPAN [TOPIC SPAN? the government] should [TOPIC SPAN?
tax gas] more in order to [TOPIC SPAN? curb [TOPIC SPAN? CO2 emissions]]].
Requires context
Topic of an opinion is the entity that comprises the main information goal of the opinion based on the discourse context.
May 29, 2008 LREC 2008, Marakech, Morocco
Issues in Opinion Topic Identification Opinion topics are not always explicitly
mentioned
(4) [OH John] believes the violation of Palestinian
human rights is one of the main factors.
Topic: ISRAELI-PALESTINIAN CONFLICT
(5) [OH I] disagree entirely!
May 29, 2008 LREC 2008, Marakech, Morocco
A Coreference Approach
Hypothesize that the notion of topic coreference will facilitate identification of opinion topics
Easier than specifying the topic of each opinion in isolation
Two opinions are topic-coreferent if they share the same opinion topic.
May 29, 2008 LREC 2008, Marakech, Morocco
Opinion Topic Corpus
Build on the MPQA corpus: 535 Documents manually annotated for fine-
grained opinions No opinion topic annotation Our goal: Add the opinion topic information on
top of the existing opinion annotations Created and used a GUI
(www.cs.pitt.edu/mpqa)
May 29, 2008 LREC 2008, Marakech, Morocco
Annotation Process
List of opinions to be processed
Set of current clusters
Document text
May 29, 2008 LREC 2008, Marakech, Morocco
Interannotator Agreement
Annotator 1 150 of the 535 MPQA documents
Annotator 2 20 of these 150
IAG measures from noun phrase coreference resolution
B3 CEAF
all opinions .64 .55 .69
sentiment-bearing opinions
.72 .73 .80
strong opinions .74 .77 .82
May 29, 2008 LREC 2008, Marakech, Morocco
Interannotator Agreement
Annotator 1 150 of the 535 MPQA documents
Annotator 2 20 of these 150
IAG measures from noun phrase coreference resolution
B3 CEAF
all opinions .64 .55 .69
sentiment-bearing opinions
.72 .73 .80
strong opinions .74 .77 .82
May 29, 2008 LREC 2008, Marakech, Morocco
Baselines all-in-one
assigns all opinions to the same cluster 1 opinion per cluster
assigns each opinion to its own cluster same paragraph
opinions in the same paragraph are assigned to the same cluster
May 29, 2008 LREC 2008, Marakech, Morocco
Results
Baselines
vs. Interannotator agreement
B3 CEAF
all-in-one .37 -.10 .30
1 opinion per cluster .29 .22 .27
same paragraph .55 .31 .50
all opinions .64 .55 .69
sentiment-bearing .72 .73 .80
strong opinions .74 .77 .82
May 29, 2008 LREC 2008, Marakech, Morocco
Thank you
Questions?
Annotation instructions + more information available at:www.cs.cornell.edu/~ves
May 29, 2008 LREC 2008, Marakech, Morocco
Example
The Australian press has launched a bitter attack on Italy after seeing their beloved Socceroos eliminated on a controversial late penalty. Italian coach Lippi has been blasted for his favorable comments toward the penalty.
Lippi is preparing his side for the upcoming clash with Ukraine. He hailed 10-man Italy's determination to beat Australia and reiterated that the penalty was rightly given.
May 29, 2008 LREC 2008, Marakech, Morocco
Example – fine-grained opinions[SThe Australian press] has launched a bitter attack on [TItaly] after seeing [Stheir] beloved [TSocceroos] eliminated on a controversial late [Tpenalty]. [S+TItalian coach Lippi] has also been blasted for his favorable comments toward [Tthe penalty].
Lippi is preparing his side for the upcoming clash with Ukraine. [SHe] hailed 10-man [TItaly]'s determination to beat Australia and reiterated that [Tthe penalty] was rightly given.
May 29, 2008 LREC 2008, Marakech, Morocco
Motivation
Sentiment analysis: Useful as stand-alone application Product reviews Tracking opinions in the press Flame detection, etc.
Opinion information can benefit many NLP applications Multi-Perspective Question Answering
[Stoyanov, Cardie, Litman and Wiebe. AAAI WS 2004] and
[Stoyanov, Cardie and Wiebe. HLT-EMNLP 2005] Opinion-Oriented Information Retrieval Clustering, etc.
Recommended