Embed Size (px)
DESCRIPTION
I used to believe that sentiment analysis was one of the most important tools in a smart marketer's box. The rise of social media offered (I thought) a huge, free, and above all, reliable source of both qualitative and quantitative data about how real people really feel about brands, and how our marketing contributed to those feelings. Over the years, though, I've become increasingly disappointed in both the reality and the promise; to the point at which I'm actively recommending that our clients avoid using social media sentiment in their models or as a performance metric. It's time for us all to stop kidding ourselves, 'fess up, and end the conspiracy. In this short talk, I hope to explain why sentiment analysis doesn't function as a useful social media metric, now or EVER.
Citation preview
Mat Morrison@mediaczar
Tuesday, 5 July 2011
Tuesday, 5 July 2011
Tuesday, 5 July 2011
Elephant in the roomTuesday, 5 July 2011
or Emperor’s ClothesTuesday, 5 July 2011
Tuesday, 5 July 2011
Volume
Tuesday, 5 July 2011
VolumeRelevanceAUTHORITYSentiment
qualified by
Tuesday, 5 July 2011
quick Question:
How Accurate
is Sentiment analysis?
Tuesday, 5 July 2011
The
stinks
Tuesday, 5 July 2011
Robot Human
supervi
sed
Machin
eLearnin
g
Tuesday, 5 July 2011
Robots first...
Tuesday, 5 July 2011
I’m unqualified to
make judgements
about the science...
Tuesday, 5 July 2011
I’m unqualified to
make judgements
about the science...
...and So are you.
Tuesday, 5 July 2011
Because the Science Is hidden in a box
Tuesday, 5 July 2011
Because the Science Is hidden in a box
But scientific Results are Reproducable!!!
Tuesday, 5 July 2011
What’s the buzz around “Green Lantern”?
Tuesday, 5 July 2011
Positive Negative Net
Sysomos (Blogs) 40 17 23
TweetSentiment 29 11 18
WDTIT? 56.8 42 14.8
SocialMention 18 5 13
SM2 20 16 4
Sysomos (Tweets) 6.4 7.6 -1.2
Rotten Tomatoes 27 73 -46
A quick poll of a few sentiment engines’ analysis of the buzz surrounding the film “Green Lantern” demonstrates no correspondence.
Tuesday, 5 July 2011
Tuesday, 5 July 2011
Tuesday, 5 July 2011
RT @BangsandaBun: According to
some, I wasn't deserving/grateful/
cool enough to be there, but I
blogged about my Glastonbury
experience http://t.co/eEInrnu
Turk THis!
Tuesday, 5 July 2011
RT @BangsandaBun: According to
some, I wasn't deserving/grateful/
cool enough to be there, but I
blogged about my Glastonbury
experience http://t.co/eEInrnu
Turk THis!
Robot says Negative
Tuesday, 5 July 2011
I know the bands for Glastonbury
were great this year, but I still
don't think you could have paid me to
be part of that mental crowd! You?
Turk THis!
Tuesday, 5 July 2011
I know the bands for Glastonbury
were great this year, but I still
don't think you could have paid me to
be part of that mental crowd! You?
Turk THis!
Robot says positive
Tuesday, 5 July 2011
SENTIMENT scoring using Amazon Mechanical Turk disagreed with 16/40 (40%) robot judgments.
Only 13/40 (32.5%) of robotic judgments were passed by all three human Judges.
Raw Data available at http://mczar.me/mkue1p
Tuesday, 5 July 2011
But Are Humans Reliable?
Emotional State/Mood
Priming
Cultural bias
Cognitive Bias
Famously not.
???
Tuesday, 5 July 2011
It Gets Worse...
The Same Human Scorer,
when presented with the
same statement on
multiple occasions, may
score it differently each
time.cf. A Guerilla Experiment in Sentiment Analysis http://mczar.me/lrp7Yi
Tuesday, 5 July 2011
Tuesday, 5 July 2011
The
stinks
Tuesday, 5 July 2011
Academic studiesgenerally focus on
Movies & the STOCK Market
Why is this?
Tuesday, 5 July 2011
Social Media data are full of irrelevant...
Tuesday, 5 July 2011
Social Media data are full of irrelevant...
Tuesday, 5 July 2011
Tuesday, 5 July 2011
VolumeRelevanceAuthoritySentiment
Tuesday, 5 July 2011
VolumeRelevanceAuthoritySentiment
All four are needed to make sense of data
But I’m only talking sentiment
Tuesday, 5 July 2011
The
stinks
Tuesday, 5 July 2011
To qualify social media buzz
Why might marketers want to perform Sentiment analysis?
To guide customer service interactions
As a Performance indicator
To identify fans
Tuesday, 5 July 2011
Is sentiment actionable?
Product Query
Product Complaint
Product endorsement
Product review
Product COmparison
...I need to know more
Is it a
...
???
Tuesday, 5 July 2011
!"#$
#$
"#$
%##$
%"#$
&##$
&"#$
'##$
()*+,-+.$ /+0+,-+.$ 12342.5$ 6+-.42.5$ 72.08$ 9:.;<$ 725$
!"#"$%&'#
=)*;>$?)@2<$
A2.-4.@)3>$?)@2<$
B;3C>,;<<$?)@2<$
Massive (and unrepresentative) spike in Net Positive
Sentiment caused by specific campaign activity
THE KPI Problem
Should we be allowed to mark our own homework?
Tuesday, 5 July 2011
Tuesday, 5 July 2011
Tuesday, 5 July 2011
Tuesday, 5 July 2011
![Sapiness{sentiment analyser · Sentiment analysis [10] is a technique used to determine the amount of positive and negative sentiment in a piece of text. Accurate opinion mining software](https://img.pdfslide.us/doc/110x75/5f9a36851637095a481fe409/sapinesssentiment-sentiment-analysis-10-is-a-technique-used-to-determine-the.jpg)
![Does mainstream news influence markets? A semantic ... · Sentiment analysis approach ... Original Sentiment score function •huand liu[2] positive and negative dictionary 5 •I](https://img.pdfslide.us/doc/110x75/5f555e6a992e312769411734/does-mainstream-news-influence-markets-a-semantic-sentiment-analysis-approach.jpg)
