How Opinions are Received by Online Communities- Case study on Amazon helpfulness votes

Cristian Danescu-Niculescu-Mizil1, Gueorgi Kossinets2, Jon Kleinberg1, Lillian Lee1

1Dept. of Computer Science, Cornell University, 2Google Inc.WWW 2009

Emin Sadiyev

Cmpe 493

Amazon.com layout

2

Average star rating

Helpfulness ratio

OutlineUsers’ evaluation on online reviews:

Helpfulness votesMake hypothesisProving their validityComing up with a mathematical model that

explains these behaviors

Introduction

OpinionWhat did Y think of X?

4

Introduction

Meta-OpinionWhat did Z think of Y’s opinion of X?

5

The Helpfulness of ReviewsWidely-used web sites include not just reviews, but

also evaluations of the helpfulness of the reviewsThe helpfulness vote

“Was this review helpful to you?”Helpfulness ratio:

“a out of b people found the review itself helpful”

ba

Flow of Presentation

Hypothe-siz-ing

Verify-ing

Model-ing

7

Flow of PresentationHypoth-esizing•Con-formity

•Individ-ual-bias

•Bril-liant-but-cruel

•Quality-only

Verifying Modeling

8

Hypotheses: Social MechanismsWell-studied hypotheses for how social effects

influence group’s reaction to an opinionThe conformity hypothesisThe individual-bias hypothesisThe brilliant-but-cruel hypothesisThe quality-only straw-man hypothesis

9

HypothesesThe conformity hypothesis

Review is evaluated as more helpful when its star rating is closer to the consensus star rating Helpfulness ratio will be the highest of which

reviews have star rating equal to overall averageThe individual-bias hypothesis

When a user considers a review, he or she will rate it more highly if it expresses an opinion that he or she agrees with

10

Hypotheses (contd.)The brilliant-but-cruel hypothesis

Negative reviewers are perceived as more intelligent, competent, and expert than positive reviewers

The Quality-only straw-man hypothesisHelpfulness is being evaluated purely based on

the textual content of reviewsNon-textual factors are simply correlates of

textual quality

11

Flow of Presentation

Hypothe-siz-ing

Verifying•Absolute deviation of helpfulness ratio

•Signed de-viation of helpfulness ratio

•Variance of star rating and help-fulness ra-tio

•Making use of plagia-rism

Model-ing

12

HypothesesConformity•A review is evaluated as more helpful when its star rating is closer to the average star rating

Individual-bias•A review is evaluated as more helpful when its star rating is closer to evaluator’s opinion

Brilliant-but-cruel•A review is evaluated as more helpful when its star rating is below to the average star rating

Quality-only•Only textual infor-mation affects help-fulness evaluation

13

Absolute Deviation from Average Consistent with

conformity hypothesisStrong inverse

correlation between the median helpfulness ratio and the absolute deviation

Reviews with star rating close to the average gets higher helpfulness ratio

14

HypothesesConformity•A review is evaluated as more helpful when its star rating is closer to the average star rating

Individual-bias•A review is evaluated as more helpful when its star rating is closer to evaluator’s opinion

Brilliant-but-cruel•A review is evaluated as more helpful when its star rating is below to the average star rating

Quality-only•Only textual infor-mation affects help-fulness evaluation

15

Signed Deviation from AverageNot consistent with

brilliant-but-cruel hypothesisThere is tendency

towards positivityBlack lines should not be

sloped that way if it is valid hypothesis

16

HypothesesConformity•A review is evaluated as more helpful when its star rating is closer to the average star rating

Individual-bias•A review is evaluated as more helpful when its star rating is closer to evaluator’s opinion

Brilliant-but-cruel•A review is evaluated as more helpful when its star rating is below to the average star rating

Quality-only•Only textual infor-mation affects help-fulness evaluation

17

Addressing Individual-bias EffectsIt is hard to distinguish between the

conformity and the individual-bias hypothesisWe need to examine cases in which individual

people’s opinions do not come from exactly the same distributionCases in which there is high variance in star

ratingsOtherwise conformity and individual-bias are

indistinguishable Everyone has same opinion

18

Variance of Star Rating and Helpfulness Ratio

19

Helpfulness ratio is the highest with reviews of which rating is slightly-above the average

Two-humped camel plots: local minimum around average

Helpfulness ratio is the highest when star ratings of reviews have average value

HypothesesConformity•A review is evaluated as more helpful when its star rating is closer to the average star rating

Individual-bias•A review is evaluated as more helpful when its star rating is closer to evaluator’s opinion

Brilliant-but-cruel•A review is evaluated as more helpful when its star rating is below to the average star rating

Quality-only•Only textual infor-mation affects help-fulness evaluation

20

Quality-only hypothesisPossible other methods

Human annotation Could be subjective

Classification using machine learning methods We cannot guarantee the accuracies of algorithms

Plagiarized reviewsAlmost(not exact) same text

same text could be considered as spam reviewsDifferent non-textual information

If the quality-only straw man hypothesis holds, helpfulness ratios of documents in each pair should be the same

21

PlagiarismMaking use of plagiarism is effective way to

control for the effect of review textDefinition of plagiarized pair(s) of reviews

Two or more reviews of different productsWith near-complete textual overlap

Author takes %70 textual overlap as plagiarism

22

An ExampleSkull-splitting headache guaranteed!•If you enjoy thumping, skull splitting migraine headache, then Sing N Learn is for you.As a longtime language instructor, I agree with the attempt and ef -fort that this series makes, but it is the execution that ultimately weakens Sing N Learn Chinese.To be sure, there are much, much better ways to learn Chinese. In fact, I would recommend this title only as a last resort and after you’ve thoroughly exhausted traditional ways to learn Chinese …

Migraine Headache at No Extra Charge•If you enjoy a thumping, skull splitting migraine headache, then the Sing N Learn series is for you.As a longtime language instructor, I agree with the effort that this series makes, but it is the execution that ultimately weakens Sing N Learn series. To be sure, there are much, much better ways to learn a foreign language. In fact, I would recommend this title only as a last resort and after you’ve thoroughly exhausted traditional ways to learn Korean …

23

Experiments with PlagiarismText quality is not the only explanatory factor

Statistically significant difference between the helpfulness ratios of plagiarized pairs

24

The plagiarized reviews with deviation 1 is significantly more helpful than those with deviation 1.5

HypothesesConformity•A review is evaluated as more helpful when its star rating is closer to the average star rating

Individual-bias•A review is evaluated as more helpful when its star rating is closer to evaluator’s opinion

Brilliant-but-cruel•A review is evaluated as more helpful when its star rating is below to the average star rating

Quality-only•Only textual infor-mation affects help-fulness evaluation

25

Flow of Presentation

Hypothe-siz-ing

Verify-ing

Model-ing•Based on in-dividual bias andmix-tures of distri-butions

26

Authors’ ModelBased on individual bias and mixtures of distributionsTwo distributions: one for positive, one for negative

evaluatorsBalance between positive and negative evaluators: Controversy level:

Density function of helpfulness ratios of positive evaluators Gaussian distribution of which average is -centered

Density function of helpfulness ratios of negative evaluators Gaussian distribution of which average is -centered

27

)( q

)( p

)()()( xqgxpfxh

)(xg

)(xf

p

Validity of the ModelEmpirical observation and model generated

28

ConclusionA review’s perceived helpfulness depends not just on its

content, but also the relation of its score to other scoresThe dependence of the score is consistent with a simple and

natural model of individual-bias in the presence of a mixture of opinion distributions

Directions for further researchVariations in the effect can be used to form hypotheses about

differences in the collective behaviors of the underlying populations

It would be interesting to consider social feedback mechanisms that might be capable of modifying the effects authors observed here

Considering possible outcomes of design problem for systems enabling the expression and dissemination of opinions

29

DiscussionsSo, how can we use this?

In which cases would this information be helpful?

Available information is very limited Star ratings Helpfulness ratios

Conclusion is rather trivialDoes not present new discoveries

30