32
Peer Review in Online Forums: Classifying Feedback-Sentiment Greg Harris, Anand Panangadan, and Viktor K. Prasanna 1 University of Southern California

Peer Review in Online Forums: Classifying Feedback-Sentiment

Embed Size (px)

DESCRIPTION

Peer Review in Online Forums: Classifying Feedback-Sentiment. Greg Harris, Anand Panangadan, and Viktor K. Prasanna. University of Southern California. Outline. Introduction to Feedback-Sentiment Slashdot Dataset Interactive Decision Tree Experiment Results. - PowerPoint PPT Presentation

Citation preview

Page 1: Peer Review in Online Forums: Classifying Feedback-Sentiment

1

Peer Review in Online Forums: Classifying Feedback-Sentiment

Greg Harris, Anand Panangadan, and Viktor K. Prasanna

University of Southern California

Page 2: Peer Review in Online Forums: Classifying Feedback-Sentiment

2

Outline

1. Introduction to Feedback-Sentiment2. Slashdot Dataset3. Interactive Decision Tree4. Experiment Results

Page 3: Peer Review in Online Forums: Classifying Feedback-Sentiment

3

Discussion Forums as a Source of Information

• Forums are rich with both information and misinformation.• Clues as to the accuracy of information can be found in replies.

Page 4: Peer Review in Online Forums: Classifying Feedback-Sentiment

4

What is feedback-sentiment?

It is the sentiment in a forum reply directed either toward the parent comment, or toward the author of the parent comment.

Examples include:• (dis)agreeing with a comment/author• showing appreciation• insulting the author• questioning/expressing doubt• listing a counterexample

Page 5: Peer Review in Online Forums: Classifying Feedback-Sentiment

5

What is feedback-sentiment?

Negative example:Citation needed. Just 'cause something has been “known for decades” doesn't make it so.

Positive example:Yeah, they’ve got the worst customer service ever.

Negative example:Um.... Yeah, it is.

Page 6: Peer Review in Online Forums: Classifying Feedback-Sentiment

6

Where can feedback-sentiment be used?

• Fact validation by peer review• Answer selection/validation• Reputation analysis• Expert identification

• Monitoring forum health

Page 7: Peer Review in Online Forums: Classifying Feedback-Sentiment

7

Outline

1. Introduction to Feedback-Sentiment2. Slashdot Dataset3. Interactive Decision Tree4. Experiment Results

Page 8: Peer Review in Online Forums: Classifying Feedback-Sentiment

8

Dataset

Slashdot.org, “News for nerds. Stuff that matters.”• All news summaries (100 thousand) and comments (25

million) spanning June 26, 2002, through August 31, 2013.• Nearly 5 million comments initiate new discussion threads,

the rest are replies.• Dataset available at http://gregharris.info

Page 9: Peer Review in Online Forums: Classifying Feedback-Sentiment

9

First-Sentence Heuristic

• The first sentence in a reply is most likely to contain the sentiment of the author toward the parent comment/author.

Count Sentence Count Sentence33,678 No. 11,184 Nope.26,143 Yes. 10,364 Wrong.23,334 Exactly. 10,128 What?20,718 Really? 9,691 Yeah.19,239 I agree. 8,857 ?18,122 Agreed. 8,734 I disagree.12,070 Huh? 8,481 Not really.11,993 Indeed. 8,383 Right.11,812 Wow. 7,984 Yep.11,577 Why? 7,318 True.

Page 10: Peer Review in Online Forums: Classifying Feedback-Sentiment

10

Challenges• Unsupervised methods did not work for us:

– We used contrast set mining to find phrases that more commonly appear in first-sentences. There were too many to label, and the context was missing.

– The semantic orientation of a phrase is difficult to infer based on statistical association. We started with a seed phrase and calculated the pointwise mutual information (PMI) with other phrases, as in Turney (2002).

– We looked for coherency when replies contained multiple snippets of quoted text. This happened when a reply refuted the parent comment point-by-point.

– We looked for coherency in all replies by the same person.– We tried using principal component analysis (PCA) to see if the first one or

two principal components could identify feedback-sentiment.– We looked for association of feedback-sentiment words and phrases with

profanity.

Page 11: Peer Review in Online Forums: Classifying Feedback-Sentiment

11

Challenges• Response patterns change over time:

Year “citation needed” “This.”2002 0 182003 0 342004 0 392005 0 282006 34 142007 321 412008 1,241 1382009 2,511 4482010 2,433 8252011 2,102 1,4602012 1,925 1,830to 8/31/2013 1,091 1,337

Page 12: Peer Review in Online Forums: Classifying Feedback-Sentiment

12

Challenges

Each forum is different. Slashdot has its own idiosyncrasies:• Poor spelling, grammar, capitalization, and punctuation• Informal• Ad hominem attacks• +1• MOD PARENT UP• RTFA (read the full article)• You must be new here.

Page 13: Peer Review in Online Forums: Classifying Feedback-Sentiment

13

Outline

1. Introduction to Feedback-Sentiment2. Slashdot Dataset3. Interactive Decision Tree4. Experiment Results

Page 14: Peer Review in Online Forums: Classifying Feedback-Sentiment

14

Interactive Decision Tree

• Fast way to explore the data• Focus on most common response patterns

See demo...

Page 15: Peer Review in Online Forums: Classifying Feedback-Sentiment

15

Some Useful Features

• Starts with “no” (No, Nope, Not, Nonsense, Nothing, ...)• Ends in ? or ...• Yelling through all-caps• Profanity• Ends in !

Page 16: Peer Review in Online Forums: Classifying Feedback-Sentiment

16

Outline

1. Introduction to Feedback-Sentiment2. Slashdot Dataset3. Interactive Decision Tree4. Experiment Results

Page 17: Peer Review in Online Forums: Classifying Feedback-Sentiment

17

Baselines

authors-first-sent (authors’ annotation of first sentences)mturk-first-sent (Mechanical Turk annotation of first sentences)mturk-fulltext (Mechanical Turk annotation of full replies)lex-first-sent (first sentence classifier based on word counts from lexicon)lex-fulltext (full text classifier based on word counts from lexicon)opfin-first-sent (OpinionFinder 2.0 run on first sentences)opfin-fulltext (OpinionFinder 2.0 run on full text)RNTN-first-sent (Recursive Neural Tensor Network run on first sentences)dtree-first-sent (first sentence classifier trained on decision tree)dtree-fulltext (dtree-first-sent applied to each sentence in full text)

Page 18: Peer Review in Online Forums: Classifying Feedback-Sentiment

18

Results

Turkers showed agreement of 58%.

Page 19: Peer Review in Online Forums: Classifying Feedback-Sentiment

19

Questions?

back to “unsupervised”

[email protected]://gregharris.info

This work is supported by Chevron U.S.A. Inc. under the joint project, Center for Interactive Smart Oilfield Technologies (CiSoft), at the University of Southern California.

Page 20: Peer Review in Online Forums: Classifying Feedback-Sentiment
Page 21: Peer Review in Online Forums: Classifying Feedback-Sentiment
Page 22: Peer Review in Online Forums: Classifying Feedback-Sentiment
Page 23: Peer Review in Online Forums: Classifying Feedback-Sentiment
Page 24: Peer Review in Online Forums: Classifying Feedback-Sentiment
Page 25: Peer Review in Online Forums: Classifying Feedback-Sentiment

25

return

Page 26: Peer Review in Online Forums: Classifying Feedback-Sentiment

26

Related Work• Hassan et al. (2012), Detecting subgroups in online discussions by modeling positive and

negative relations among participants.• Hassan et al. (2012), What’s with the attitude?: identifying sentences with attitude in

online discussions.• Danescu-Niculescu-Mizil et al. (2013), A computational approach to politeness with

application to social factors.• Sood et al. (2012), Automatic identification of personal insults on social news sites.• Musat et al. (2013), Direct negative opinions in online discussions.• Janin et al. (2003), The ICSI meeting corpus.• Hillard et al. (2003), Detection of agreement vs. disagreement in meetings: Training

with unlabeled data.• Galley et al. (2004), Identifying agreement and disagreement in conversational speech:

Use of bayesian networks to model pragmatic dependencies.• Hahn et al. (2006), Agreement/disagreement classification: Exploiting unlabeled data

using contrast classifiers.• Germesin and Wilson (2009), Agreement detection in multiparty conversation.

Page 27: Peer Review in Online Forums: Classifying Feedback-Sentiment
Page 28: Peer Review in Online Forums: Classifying Feedback-Sentiment
Page 29: Peer Review in Online Forums: Classifying Feedback-Sentiment
Page 30: Peer Review in Online Forums: Classifying Feedback-Sentiment
Page 31: Peer Review in Online Forums: Classifying Feedback-Sentiment
Page 32: Peer Review in Online Forums: Classifying Feedback-Sentiment