Upload
others
View
2
Download
0
Embed Size (px)
Citation preview
NATURAL LANGUAGE PROCESSING
Sentiment Analysis on Twitter
Mentor: Prof. Amitabha Mukherjee
ByRohit Kumar JhaSakaar Khurana
Department of Computer Science and Engineering, IIT Kanpur
11/18/2013 1NLP|CS671
Introduction
In the past decade, new forms of communication, such as
microblogging and text messaging have emerged and become
ubiquitous. While there is no limit to the range of information
conveyed by tweets and texts, often these short messages are used to
share opinions and sentiments that people have about what is going
on in the world around them. We plan to work on the following task.
This task was part of SEMEVAL 2013 challenge. The task:
Given a message, classify whether the message is of positive, negative, or neutral sentiment. For messages conveying both a
positive and negative sentiment, whichever is the stronger sentiment should be chosen.
11/18/2013NLP|CS671 2
Motivation
• Tweets and texts are short: a sentence or a headline rather than a document. The
language used is very informal, with creative spelling and punctuation, misspellings,
slang, new words, URLs, and genre-specific terminology and abbreviations, such as,
RT for "re-tweet" and #hashtags, which are a type of tagging for Twitter messages.
• Another aspect of social media data such as Twitter messages is that it includes rich
structured information about the individuals involved in the communication. For
example, Twitter maintains information of who follows whom and re-tweets and tags
inside of tweets provide discourse information.
• Working with these informal text genres presents challenges for natural language
processing beyond those typically encountered when working with more traditional
text genres, such as newswire data.
• Modelling such structured information is important because: (i) it can lead to more
accurate tools for extracting semantic information, and (ii) because it provides means
for empirically studying properties of social interactions (e.g., we can study properties
of persuasive language or what properties are associated with influential users).
11/18/2013NLP|CS671 3
Previous work
11/18/2013NLP|CS671 4
Bag of Words Model• Use a word list where each word has been scored positivity/negativity or sentiment
strength
• Overall polarity determined by the aggregate of polarity of all the words in the text
• In SEMEVAL 2013, Team IITB used Bag of Words model with Discourse Information and able to achieve an accuracy of 39.80%
Naive Bayesian Classifier• Straightforward and frequently used method for supervised learning
• Maximum entropy classifiers are commonly used as alternatives to Naïve
• Bayesian classifier because they do not require statistical independence of the features that serve as predictors
• In SEMEVAM 2013, Team uottawa used Naïve Bayesian Classifier were able to achieve an accuracy of 42.51%
Support Vector Machine• In SEMEVAL 2013, Team NRC-Canada used SVM model with unigram, bigram, POS tags,
negation etc as features and were able to achieve an accuracy of 69.02%
Our Work
Simple and Naive implementation of Bag of
Words Model
Incorporate the use of emoticons
Bag of Words model with Discourse Information
Sentence Weightage assignment in Bag of
Words Model
Incorporate the effect of #hashtags
Incorporate the effect of modifiers like "very",
"too", etc
SVM implementation with unigrams+bigrams, negation, POS tagging,
etc as features
Use Bag of Words Model with SVM
Take care of misspellings, informal use of words
like happpppyyy, abbreviations, etc
11/18/2013NLP|CS671 5
Naive Implementation of Bag of Words
11/18/2013NLP|CS671 6
Important Points
• Use a word list where each word has been scored positivity/negativity or sentiment strength
• Overall polarity determined by the aggregate of polarity of all the words in the text
• Major polarity words have been given polarity between -4 to 4 depending on their polarity, like excellent is +4, but happy is +2, and so on
Results
• Accuracy of ~42% when using Bag of Words Model without any special care
• Team IITB also used Bag of Words model but could achieve only ~40% accuracy. The major difference being probably that they considered only 0, +1, -1 values of polarity of words.
Bag of Words Model with features
11/18/2013 7
Important Features
• Incorporate the use of emoticons. If there are emoticons, clearly showing a particular
emotion, don't proceed further
• Incorporate the use of Discourse Information along with the Bag of Words model
• Assigning double weightage to polarity words in sentences occurring later on in
tweets. But it didn't make an overall difference
• Incorporate the effect of #hashtags. Apply Bag of Words model on the tweets after
doing word boundary segmentation and if some clear sentiment appears, don't
proceed ahead
• Incorporate the effect of modifiers like "very", "too", etc
Results• Achieved accuracy of ~56% after considering these features
NLP|CS671
SVM Implementation
11/18/2013 8
Important Features
• POS: the number of occurrences for each part-of-speech
tags. 'NN','VG','CD','JJ','CC','RB' pos tags are the only ones considered
• emoticons:
- presence/absence of positive and negative emoticons at any position
in the tweet;
- whether the last token is a positive or negative emoticon;
• negation: the number of negated contexts. A negated context also affects
the ngram and lexicon features: each word and associated with it polarity in
a negated context become negated
• frequency: select top 1000 words and their presence
Results
• Achieved accuracy of ~61% after considering these features
NLP|CS671
SVM + Bag of Words Model
11/18/2013NLP|CS671 9
Important Points
• Using a Hybrid Classifier using emoticons, Bag of Words Model and SVM
Classifier
• If there is a clear presence of emoticons denoting the emotion of a tweets,
simply use it
• Otherwise use the Bag of Words model and if there is a certain confidence
level in the polarity reported, simply go with it
• If no clear and confident outcome obtained, use SVM classifier and the
results from the first two steps to decide
Results• Achieved an accuracy of 68.32% with training on only around 8000 tweets,
which is a really small training data
Results
SerialNo.
Methods Used Data Size Accuracy Obtained on SEMEVAL 2013 data
1. Bag of Words Model N.A. 39.80%
2. Naïve Bayes Classifier 10,000 tweets 42.51%
3. Support Vector Machine 1.6 Million tweets
69.02%
4. SVM + Bag of Words Model(Our result)
8,000 tweets 68.32%
11/18/2013NLP|CS671 10
References
Subhabrata Mukherjee, Pushpak Bhattacharyya. Sentiment Analysis in Twitter with
Lightweight Discourse Analysis
http://aclweb.org/anthology//C/C12/C12-1113.pdf
Preslav Nakov, Sara Rosenthal ,Zornitsa Kozareva ,Veselin Stoyanov ,Theresa Wilson. SemEval-
2013 Task 2: Sentiment Analysis in Twitter
http://www.aclweb.org/anthology/S/S13/S13-2052.pdf
Stefano Baccianella, Andrea Esuli, and Fabrizio Sebastiani. 2010. SentiWordNet 3.0: An
enhanced lexical resource for sentiment analysis and opinion mining. In Nicoletta Calzolari (chair),
Khalid Choukri, Bente Maegaard, Joseph Mariani, Jan Odijk, Stelios Piperidis, Mike Rosner, and
Daniel Tapias, editors, Proceedings of the Seventh International Conference on Language
Resources and Evaluation, LREC ’10, pages 2200–2204, Valletta, Malta.
11/18/2013NLP|CS671 11
Summary and Conclusion
Objective
• Given a message, classify whether the message is of positive, negative, or neutral
sentiment. For messages conveying both a positive and negative sentiment,
whichever is the stronger sentiment should be chosen.
Result Obtained
• Achieved an accuracy of 68.32% with training on only around 8000 tweets, which is a
really small training data, compared to what was used by the team that came first
Significance
• Our method achieves good accuracy with relatively small amount of data compared
to that used by others
• Our method is targeted towards applications that cannot afford to use heavy
processing
Dataset Used
• We used the dataset provided by SEMEVAL 2013 for both training and testing
purposes
11/18/2013NLP|CS671 12