Embed Size (px)
DESCRIPTION
**Update 7/24/2014** Cleaned it up. Likely to be the final version unless I figure out image processing. That may be an entirely different presentation though. ** Presentation given for BarcampNOLA7. Touched on a variety of topics like natural language processing, sentiment analysis, and ethics. Chose the context of Twitter since I'm more familiar with text processing than image processing. Twitter has some unique problems that make it not straightforward to take the data that is covered here. Overall, the presentation went longer than expected (The time frame was 15-30 min). Didn't have much time for discussion, although one spawned about the inaccuracy of sentiment analysis. No questions, though I blame the length of the presentation. I also was expecting the room to be mostly computer programmers, but there were some business people, sales, and marketing. Next time I present, I would give myself more time (45 seems more reasonable) to elaborate on important topics I had to skim through (ethics, programming algorithms, consumer psychology and irrational behavior).
Citation preview
Computer-Assisted Consumer Profiling on Twitter
By Olin Gallethttp://www.olingallet.com
@olingallet
Profiling in General
Takes observable qualities and estimates the person(s) behind those qualities.
Example uses:
Criminal Profiling (Law and Order, Criminal Minds)
Video Games (Bartle Gamer Types)
Marketing (Intelligent Ad Retargeting)
Human Resources
Definitions
Consumer profile – what a person likes and dislikes in terms of products, their spending habits based off their age, gender, marital status, etc.
Sentiment analysis – studying how a person(s) feel about something through scientific measurement; can be positive, negative, or neutral.
Why care?
Better sales
Better targeting
Consumer feedback
Research and Development
Why care?
Better sales
Better targeting
Consumer feedback
Research and Development
Straight From Twitter's Privacy Policy
Advertising:
“To help us deliver ads, measure their performance, and make them more relevant to you based on criteria like your activity on Twitter and visits to our ad partners' websites”
People may object this as an invasion of privacy, but you're stepping into their environment and trying to play by your rules.
Why Twitter?
“Tweets” are limited to 140 characters, providing succinct tidbits of information.
Accessible on mobile devices
Good to use for events such as a concert, convention, etc.
Easily accessible through a JSON API.
Why not Twitter?
“Tweets” are 140 characters, sometimes not enough for a proper sentence.
Can limit descriptiveness
Fake profiles can give false information
Difficulty in traditional natural language processing ->
Natural Language Processing Methods
-Parts-of-speech tagging – utilize a dictionary (ie Wordnet) to identify words as nouns, verbs, etc.
-can't be used with all words, for example:
Fire:
As a verb: I will fire a gun.
As a verb: I will fire that individual.
As a noun: I didn't start the fire.
Sentiment Analysis
Twitter provides sentiment analysis, but it's poor since it only looks for smilies and frownies.
Computers understand the denotation easily, connotation is another story.
-Negation
-Adverb/Adjective Modifiers
-Sarcasm
Basic sentiment analysis searches for emotionally charged words using a dictionary. More advanced versions use machine learning to train the computer.
Brief Word on Machine Learning
-Involves teaching the intelligence what conditions produce a certain result.
-The more data provided, the more confident intelligence becomes.
Importance of Hashtags
Hashtags are often unambiguous
Showcase group thought
Act as keys when searching tweets
What to Consider in Terms of the Consumer
Person giving the message (celebrity status, followers, how often they post)
Date (timestamp, how new or old the product is)
The product
The component of the product (if applicable)
The overall sentiment
A Little More About Data Sources
Consider online reputation (Klout):
-Number of followers
-Number following
-Frequency of tweets
-Variance of tweets
-”Verified” status, Join Date
-Number of retweets, favorites
Online reputation helps filter out spambots, fake profiles,
social honeypots. Best you can do without knowing the relationship between
people.
Good Example:
I dislike my iPhone. The battery life is too short.
(Work this out with group time permitting)
Bad Example:
LMAO dat RiFF RAFF album...iceberg simpson off da chain #neonicon #gettinpaid
-internet acronyms
-non-existant sentence structure
-idiomatic phrase
Analysis Over Time
People are more likely to remember negativity than positivity.
Vengeance breeds vengeance, apologies rebuild trust, counteract vengeance (see Dan Ariely research)
Emotions are contagious
http://www.scientificamerican.com/article/facebook-emotions-are-contagious/
Putting it Together
Use natural language processing to understand what products the consumer cares about.
Use sentiment analysis to understand how they feel about the product.
Address any negativity (if feasible) so negativity about the product.
Help people rationalize their purchases with positivity.
What I want you to take away
- Profiling in general can be wrong.
- Computers can't understand language the same way people can. They won't be able to get it right 100%.
- Consider the ethics.
- Internet is public, hard to keep things private with caching, spiders, hackers, etc.
- Don't let the Internet replace real life. People can be forgiven, online reputation can only be hidden.
Related Topics
Game Theory Probabilities and Statistics Psychology Sociology Natural Language Programming Language Syntax Image Processing, Facial Recognition (for
pictures and Instagram)
Helpful Resources
www.lct-master.org/files/MullenSentimentCourseSlides.pdf
Sentiment Analysis Tutorial
http://wordnet.princeton.edu/
Wordnet, Lexical Database
http://www2.imm.dtu.dk/pubdb/views/publication_details.php?id=6010
AFINN, Sentiment Dictionary
http://gigaom.com/2013/10/03/stanford-researchers-to-open-source-model-they-say-has-nailed-sentiment-analysis/
Stanford Research
http://danariely.com/the-books/an-excerpt-from-chapter-5-of-%E2%80%9Cthe-upside-of-irrationality%E2%80%9D/
Dan Ariely on vengeance
