Peer Review in Online Forums: Classifying Feedback-Sentiment

1

Peer Review in Online Forums: Classifying Feedback-Sentiment

Greg Harris, Anand Panangadan, and Viktor K. Prasanna

University of Southern California

2

Outline

1. Introduction to Feedback-Sentiment2. Slashdot Dataset3. Interactive Decision Tree4. Experiment Results

3

Discussion Forums as a Source of Information

• Forums are rich with both information and misinformation.• Clues as to the accuracy of information can be found in replies.

4

What is feedback-sentiment?

It is the sentiment in a forum reply directed either toward the parent comment, or toward the author of the parent comment.

Examples include:• (dis)agreeing with a comment/author• showing appreciation• insulting the author• questioning/expressing doubt• listing a counterexample

5

What is feedback-sentiment?

Negative example:Citation needed. Just 'cause something has been “known for decades” doesn't make it so.

Positive example:Yeah, they’ve got the worst customer service ever.

Negative example:Um.... Yeah, it is.

6

Where can feedback-sentiment be used?

• Fact validation by peer review• Answer selection/validation• Reputation analysis• Expert identification

• Monitoring forum health

7

Outline

1. Introduction to Feedback-Sentiment2. Slashdot Dataset3. Interactive Decision Tree4. Experiment Results

8

Dataset

Slashdot.org, “News for nerds. Stuff that matters.”• All news summaries (100 thousand) and comments (25

million) spanning June 26, 2002, through August 31, 2013.• Nearly 5 million comments initiate new discussion threads,

the rest are replies.• Dataset available at http://gregharris.info

9

First-Sentence Heuristic

• The first sentence in a reply is most likely to contain the sentiment of the author toward the parent comment/author.

Count Sentence Count Sentence33,678 No. 11,184 Nope.26,143 Yes. 10,364 Wrong.23,334 Exactly. 10,128 What?20,718 Really? 9,691 Yeah.19,239 I agree. 8,857 ?18,122 Agreed. 8,734 I disagree.12,070 Huh? 8,481 Not really.11,993 Indeed. 8,383 Right.11,812 Wow. 7,984 Yep.11,577 Why? 7,318 True.

10

Challenges• Unsupervised methods did not work for us:

– We used contrast set mining to find phrases that more commonly appear in first-sentences. There were too many to label, and the context was missing.

– The semantic orientation of a phrase is difficult to infer based on statistical association. We started with a seed phrase and calculated the pointwise mutual information (PMI) with other phrases, as in Turney (2002).

– We looked for coherency when replies contained multiple snippets of quoted text. This happened when a reply refuted the parent comment point-by-point.

– We looked for coherency in all replies by the same person.– We tried using principal component analysis (PCA) to see if the first one or

two principal components could identify feedback-sentiment.– We looked for association of feedback-sentiment words and phrases with

profanity.

11

Challenges• Response patterns change over time:

Year “citation needed” “This.”2002 0 182003 0 342004 0 392005 0 282006 34 142007 321 412008 1,241 1382009 2,511 4482010 2,433 8252011 2,102 1,4602012 1,925 1,830to 8/31/2013 1,091 1,337

12

Challenges

Each forum is different. Slashdot has its own idiosyncrasies:• Poor spelling, grammar, capitalization, and punctuation• Informal• Ad hominem attacks• +1• MOD PARENT UP• RTFA (read the full article)• You must be new here.

13

Outline

1. Introduction to Feedback-Sentiment2. Slashdot Dataset3. Interactive Decision Tree4. Experiment Results

14

Interactive Decision Tree

• Fast way to explore the data• Focus on most common response patterns

See demo...

15

Some Useful Features

• Starts with “no” (No, Nope, Not, Nonsense, Nothing, ...)• Ends in ? or ...• Yelling through all-caps• Profanity• Ends in !

16

Outline

1. Introduction to Feedback-Sentiment2. Slashdot Dataset3. Interactive Decision Tree4. Experiment Results

17

Baselines

authors-first-sent (authors’ annotation of first sentences)mturk-first-sent (Mechanical Turk annotation of first sentences)mturk-fulltext (Mechanical Turk annotation of full replies)lex-first-sent (first sentence classifier based on word counts from lexicon)lex-fulltext (full text classifier based on word counts from lexicon)opfin-first-sent (OpinionFinder 2.0 run on first sentences)opfin-fulltext (OpinionFinder 2.0 run on full text)RNTN-first-sent (Recursive Neural Tensor Network run on first sentences)dtree-first-sent (first sentence classifier trained on decision tree)dtree-fulltext (dtree-first-sent applied to each sentence in full text)

18

Results

Turkers showed agreement of 58%.

19

Questions?

back to “unsupervised”

gfharris@usc.eduhttp://gregharris.info

This work is supported by Chevron U.S.A. Inc. under the joint project, Center for Interactive Smart Oilfield Technologies (CiSoft), at the University of Southern California.

25

return

26

Related Work• Hassan et al. (2012), Detecting subgroups in online discussions by modeling positive and

negative relations among participants.• Hassan et al. (2012), What’s with the attitude?: identifying sentences with attitude in

online discussions.• Danescu-Niculescu-Mizil et al. (2013), A computational approach to politeness with

application to social factors.• Sood et al. (2012), Automatic identification of personal insults on social news sites.• Musat et al. (2013), Direct negative opinions in online discussions.• Janin et al. (2003), The ICSI meeting corpus.• Hillard et al. (2003), Detection of agreement vs. disagreement in meetings: Training

with unlabeled data.• Galley et al. (2004), Identifying agreement and disagreement in conversational speech:

Use of bayesian networks to model pragmatic dependencies.• Hahn et al. (2006), Agreement/disagreement classification: Exploiting unlabeled data

using contrast classifiers.• Germesin and Wilson (2009), Agreement detection in multiparty conversation.