Jan Wiebe University of Pittsburgh

Claire Cardie Cornell University

Ellen Riloff University of Utah

Opinions in Question Answering

Overview

Techniques and tools to support multi-perspective question answering (MPQA)

Goals: produce high-level summaries of opinions incorporate rich information about

opinions extracted from text

Overview

Opinion-oriented information extraction Extract opinion frames for individual

expressions Combine to create opinion-oriented

“scenario” templates

OpinionSummaryTemplate

MPQA Corpus

Grew out of the 2002 ARDA NRRC Workshop on Multi-Perspective Question Answering

Detailed annotations of opinions Freely available (thanks to David Day):

nrrc.mitre.org/NRRC/publications.htm

Collaborations

Interactions with end-to-end system teams

Integrated corpus annotation Pilot opinion evaluation

Outline

Recent activities Subjective sentence identifier Clause intensity identifier Extended annotation scheme Version 1 Q&A corpus Nested opinions Opinion summaries

What’s next

Subjective Sentence Identifier

Input is unlabeled data Evaluated on manual annotations of

the MPQA corpus Accuracy as good as supervised

systems which classify all sentences

Subjective Sentence Identifier

Bootstraps from a known subjective vocabulary, labeling the sentences it can with confidence

Extraction pattern learner finds clues of subjectivity in that corpus

Incorporated into a statistical model trained on the automatically labeled data

Multiple classification strategies 76% accuracy with 54% baseline 80% subj. precision and 66% subj. recall 80% obj. precision: and 51% obj. recall

Clause-level intensity (strength) identification

Maximum intensity of the opinions in a clause

Neutral, low, medium, high Evaluated on manual annotations of

the MPQA corpus

I am furious that my landlord refused to return my security deposit until I sued them.

Example

return

my

that

am

them

sued

I

to

refused

landlord

furious

I

until

deposit

securitymy

High Strength

Medium Strength

Neutral

Opinionated Sentence

Clause-level intensity (strength) identification

Classification and regression learners Accuracy: how many clauses are assigned

exactly the correct class? Mean Squared Error: how close are the

answers to the right ones? Accuracy: classification > regression

23-79% over baseline MSE: regression > classification

57-64% over baseline

Opinion Frames

direct subjective annotation Span: “strongly criticized and condemned” Source: <writer,many-countries> Strength (intensity): high Attitudes: negative toward the report Target: report

The report has been strongly criticized and condemned by many countries.

Major Attitude Types

Positive Negative Arguing for ones world view Intention

Negative and Positive Example

People are happy because Chavez has fallen, she said.

direct subjective annotation span: are happy source: <writer, she, People> attitude:

attitude annotation span: are happy because Chavez has fallen type: positive and negative positive target: negative target:

target annotation span: Chavez has fallen

target annotation span: Chavez

Arguing for World View Example

Putin remarked that events in Chechnia “could be interpreted only in the context of the struggle against international terrorism.”

direct subjective annotation span: remarked source: <writer, Putin> attitude:

attitude annotation span: could be interpreted only in the context of the struggle against international terrorism type: argue for world view target:

target annotation span: events in Chechnia

Characteristics

Sarcastic "Great, keep on buying dollars so there'll be

more and more poor people in the country," shouted one.

Speculative Leaders probably held their breath…

Characteristics of the linguistic realization

Q&A Corpus

Includes 98 documents from the NRRC corpus, split into four topics:

Kyoto Protocol 2002 elections in Zimbabwe U.S. annual human rights report 2002 coup in Venezuela

Q&A Corpus

Includes 30 questions 15 questions classified as fact

What is the Kyoto Protocol about? What is the Kiko Network? Where did Mugabe vote in the 2002 presidential

election? 15 questions classified as opinion

How do European Union countries feel about the US opposition the Kyoto protocol?

Are the Japanese unanimous in their opinion of Bush’s position on the Kyoto Protocol?

What was the American and British reaction to the reelection of Mugabe?

Q&A Corpus

Answers annotations added by two annotators Minimal spans that constituted or

contributed to an answer Confidence Partial?

Difficulties in Corpus Creation

Annotating answers Difficult to decide what constitutes an answer:

Q: “Did most Venezuelans support the 2002 coup?” A: “Protesters…failed to gain the support of the army.” ???

Not clear what sources to attribute to collective entities

European Union: The EU Parliament? Great Britain? GB government? Tony Blair?

The Japanese: The Japanese government? Emperor Akihito? Empress Michiko? The Kiko Network?

Q&A Corpus

Interannotator agreement 85% on average using Wiebe et. al’s agr(a||b) measure 78% and 93%, respectively for each

annotator

Evaluating MPQA Opinion Annotations

Answer probability: estimate P(opinion answer | opinion question) P(fact answer | fact question)

Low-level opinion information reliable predictor facts: 78% opinions: 93%

Answer rank Sentence-based retrieval Filter based on opinion annotations Examine rank of first sentence w/answer Filtering improves answer rank

Summary Representations of Opinions

Direct subjective annotationSource:Attitude:

OpinionSummaryTemplate

Reporting in text

Clapp sums up the environmental movement’s reaction: “The polluters are unreasonable’’

Charlie was angry at Alice’s claim that Bob was unhappy

Hierarchy of Perspective & Speech Expressions

Charlie was angry at Alice’s claim that Bob was unhappy

angry

claim

implicit speech event

unhappy

sums up

implicit speech event

reaction

Clapp sums up the environmental movement’s reaction: “The polluters are unreasonable’’

Baseline 1: Only filter through writer

66% correct

angry

claim

implicit

unhappy

unhappy

unhappy

Baseline 2: Dependency Tree

72% correct

angry

implicit

claim

unhappy

claim

unhappy

claim

unhappy

ML Approach

Features Parse-based Positional Lexical Genre-specific

IND decision trees (mml criterion)

78% correct

Summary Representations of Opinions

Direct subjective annotationSource:Attitude:

OpinionSummaryTemplate

Opinion Summaries

Summaries based on manual annotations Single-document summaries Opinion annotations grouped by source and target Sources characterized by degree of

subjectivity/objectivity Simple graph-based graphical interface

Overview of entire graph Focus on portion of the graph Drill-down to opinion annotations (highlighted) Grouping/deleting of sources/targets JGRAPH package

The next 6 months

Identify individual expressions of subjectivity

Perform manual annotations Extract Sources Opinion summaries with automatic

annotations