21
Semi Automatic Sentiment Analysis Results from a case study in Brazilian Portuguese web 2.0 sites Gleicon Moraes, Marco Aurélio Gerosa [email protected], [email protected]

Semi Automatic Sentiment Analysis

Embed Size (px)

DESCRIPTION

Web 2.0 applications for social networking provide data about users’ mood and opinions in almost real time. Many applications are taking advantage of these data to derive business intelligence. However, the volume of data makes it hard and error-prone to classify sentiments and opinions manually. The combination of data mining techniques and a pipeline to process data from Web 2.0 applications, such as Twitter, Facebook, and Wordpress, makes it possible to apply natural language processing and machine learning techniques to automate partially this task. Therefore, the amount of manual classification is reduced, as the incoming data has already a classification tag that can be easily changed, feeding back the classifier. There is room for improvements and a Brazilian Portuguese Corpus was created to do the initial training of the classifier. The code used for this testing was based on open source libraries and is available as a test bed for different corpora and new algorithms.

Citation preview

Page 1: Semi Automatic Sentiment Analysis

Semi Automatic Sentiment Analysis Results from a case study in Brazilian Portuguese web 2.0 sites

Gleicon Moraes, Marco Aurélio Gerosa

[email protected], [email protected]

Page 2: Semi Automatic Sentiment Analysis

Introduction

•  Popular Web 2.0 applications are based on social networking: Facebook, Twitter, Orkut, Flickr, LinkedIn

•  Status messages, user information, wall posts, like/unline votes, scraps, recommendations are created and exchanged between users.

•  Symetric and Assymetric relationships broadcast these messages between friends (direct connections) and friends of friends.

•  Sentiment and opinions might be objective (up/down votes, recommendations) or subjective (free text)

Gleicon Moraes, Marco Aurélio Gerosa 2/20

Page 3: Semi Automatic Sentiment Analysis

Sentiment Classification

•  Find out what users in a social network think about product, tendency or brand.

•  Compute or help to compute the Return of Investment of a

marketing campaign

•  Create or compose product and services recommendations to other users

•  To measure user satisfaction and experience about a

service.

Gleicon Moraes, Marco Aurélio Gerosa 3/20

Page 4: Semi Automatic Sentiment Analysis

Goals

•  Opinion mining / subjectivity and sentiment analysis review [1]

•  Automate opinion classification (tweet, scrap, message, wall post) using Machine Learning and Information Retrieval techniques.

•  To apply a Bayesian filter (and also try a SVM classifier) to

identify Positive and Negative sentiment on brazilian Portuguese texts.

•  To build a corpus to train and test the classifiers

•  To find out how to measure the filter efficiency. [1] Pang e Lee - Opinion Mining and Sentiment Analysis

Gleicon Moraes, Marco Aurélio Gerosa 4/20

Page 5: Semi Automatic Sentiment Analysis

Related work

•  Thumbs Up? Sentiment Classification using Machine Learning

Techniques: Bayesian filter, Maximum Entropy filter and SVM filter. Training

was made with Movielens dataset, splitting between 70% corpus to training and

30% to test. This corpus is already marked as positive and negative. Conclusion

was that sarcasm on opinions made it difficult to classify the sentiments. There

was no smaller text classification (e.g. A tweet/140 chars) and feedback with

outside text to the classifiers. [1]

[1] Pang, B. Lee L., Cornell University, Vaithyanathan S, IBM: Thumbs Up? Sentiment Classification using Machine Learning Techniques

Gleicon Moraes, Marco Aurélio Gerosa 5/20

Page 6: Semi Automatic Sentiment Analysis

Related work

•  Content-based book Recommendation Using Learning for Text

Categorization and information extracted from the internet to train a classifier,

with a database per user. The combination between collaborative filtering and

content filtering complete each other and help improve the results. [1][2]

[1] Mooney R. J., Roy L., “Content-Based Book Recommendation Using Learning for Text Categorization” (Proceedings of ACM Conference on Digital Libraries, 2000) [2] Dˇzeroski S., Zenko B. “Is Combining Classifiers Better than Selecting the Best One?”

Gleicon Moraes, Marco Aurélio Gerosa 6/20

Page 7: Semi Automatic Sentiment Analysis

Semi-Automatic Sentiment Classification

•  Trained Bayesian Filter on two categories: “positive” and “negative”

•  Feedback feature so false positives and false negatives could be trained back to improve the filter

•  Problem: There is not a brazilian portuguese data matching text to sentiment to do the initial classificator training.

•  Problem: Text composition varies between social networks and groups within these networks. Feeding back data to keep the classificator database updated is fundamental

Gleicon Moraes, Marco Aurélio Gerosa 7/20

Page 8: Semi Automatic Sentiment Analysis

Semi-Automatic Sentiment Classification

•  English language training corpus uses movie reviews in most papers, associated with ratings to tell what that text block express [1]

•  An initial training corpus was made using consumer review data from Brazilian websites like iVox, ReclameAqui, opiniões do MercadoLivre

•  After scrapping each opinion and its rating (stars, rating, or positive/negative indication), stored it on folders ranging from 0.0 to 5.0, each opinion a file inside the proper folder

[1] MovieLens dataset: http://www.grouplens.org/node/73

Gleicon Moraes, Marco Aurélio Gerosa 8/20

Page 9: Semi Automatic Sentiment Analysis

Training composition findings

•  Number of words in negative opinions is bigger than on

positive opinions: 67.575 words in 712 positive opinions versus 81.747 words in 507 negative opinions.

•  Distribution of reviews between minimum and maximum

ratings: more opinions on the extremes (0.0 to 0.5 and 4.5 to 5.0).

Gleicon Moraes, Marco Aurélio Gerosa 9/20

Page 10: Semi Automatic Sentiment Analysis

Composição da base de treinamento - iVox

Gleicon Moraes, Marco Aurélio Gerosa 10/22

Page 11: Semi Automatic Sentiment Analysis

Domain

•  Language domain varies between communities/sites

Gleicon Moraes, Marco Aurélio Gerosa 11/20

Page 12: Semi Automatic Sentiment Analysis

Opinion Sample (Mercadolivre)

positivo (rating 5): "Este alto-falante faz o baile tremer... comprei para montar uma mini-saveiro” negativo (rating 1): "Apesar de custar muito barato recomendo economizar e comprar falantes de marcas conhecidas. Bravox, Selenium. O produto parece recondicionado, e não tem 90Wats nem na china, meu triaxial Pionner de 60Wats aquenta muito mais grave que esse Unlike. Não faça besteira economize mais R$60,00 e compre um Kit 2 vias Selenium ou até Sony ou Bomber que custa quase o mesmo aqui no Mercado Livre"

Gleicon Moraes, Marco Aurélio Gerosa 12/20

Page 13: Semi Automatic Sentiment Analysis

Opinion Sample (iVox)

positivo (rating 5.0): "Economica não tem Adquiri uma web.evo Sundown,à moto é bonita,gostei tanto da Sundown que adquiri mais uma moto Sundow a hunter 90cc. estou com 2 motos e estou muito satisfeito. Quanto ao pessoal da grappa, todos sem exceção sempre bem atenciósos comigo; só tenho a agradecer. " negativo (rating 1): "Contra Todas Não sei o motivo de sua defesa a esta empresa, pois fui enganado a pouco tempo e o engraçado é que liguei para reclamar, bem na hora que o vendedor estava enganando outra pessoa, por um deslize do mesmo o cliente verificou o numero e me ligou dizendo que também havia sido enganado. Entramos com denúncia conjunta na DECON do DF. Razoável Muito Ruim Razoável Muito Ruim"

Gleicon Moraes, Marco Aurélio Gerosa 13/20

Page 14: Semi Automatic Sentiment Analysis

Opinion Sample (Reclame Aqui)

positivo: "Olá, estou passando apenas para parabenizar ao ótimo e sério trabalho da equipe do site reclameaqui.com.br, pois já fui atendido em duas ocasiões reclamadas no site e foi algo bem melhor e mais rápido do que partir para outras atitudes. Parabéns e que cada vez mais possamos ter meios iguais para podermos agilizar o processo de negociação. Obrigado," negativo: "Fiz 2 reclamações contra a MOTOROLA DO BRASIL por propaganda enganosa em seu site www.motorola.com.br sobre o aparelho V3m que no site diz ACOPMPANHA cartão enquanto no meu aparelho nao veio NADA !!! Eles me ligaram e tiram o deles da reta dizendo que a culpa é da VIVO ! MAis perai quem faz o aparelho nao é eles ??? A VIVO so revende !!!! Ah MOTOROLA POR FAVOR NE !!!!! QUERO MEU CARTAO !!!" Gleicon Moraes, Marco Aurélio Gerosa 14/20

Page 15: Semi Automatic Sentiment Analysis

Domain

•  Language Domain [1]: "go read a book” has different meaning related

to each social network. In a book related network might be a positive meaning. In others might mean a negative sentiment.

•  Feeding back data also helps to keep the database updated with new

slangs and combinations that also might cover sarcasm expressions.

•  Events like world cup and television shows might introduce new words and expressions.

[1] Pang e Lee - Opinion Mining and Sentiment Analysis Gleicon Moraes, Marco Aurélio Gerosa 15/20

Page 16: Semi Automatic Sentiment Analysis

Training

•  Split the database between negative (rating: 0.0) and positive (rating 5.0). Later steps added ratings 4.5, 4.0 to positive while negative rating kept the same.

•  Training/Classifying applied on raw data and on data processed a

pipeline of taking out stop words and extracting the stem of remaining words

•  Raw data biased towards negative sentiment, processed data biased towards positive sentiments.

Gleicon Moraes, Marco Aurélio Gerosa 16/20

Page 17: Semi Automatic Sentiment Analysis

Results – raw data

iVox ReclameAqui False results Ratings Negative/Positive Negative Positive Negative Positive No training No messages 1635 268 0 0

0.0 e 5.0 506/720 1634 6 262 1

0.0 e 4.5 + 5.0 506/873 1587 99 169 48 0.0 e 4.0 + 4.5 + 5.0 506/973 1365 165 105 270

Gleicon Moraes, Marco Aurélio Gerosa 17/20

Page 18: Semi Automatic Sentiment Analysis

Results – filtered data

iVox ReclameAqui False results Ratings Negative/Positive Negative Positive Negative Positive No training No messages 1635 268 0 0

0.0 and 5.0 506/720 1635 0 268 0

0.0 and 4.5 + 5.0 506/873 0 261 0 1627 0.0 and 4.0 + 4.5 + 5.0 506/973 0 268 0 1635

Gleicon Moraes, Marco Aurélio Gerosa 18/20

Page 19: Semi Automatic Sentiment Analysis

Measuring efficiency

•  Metrics: Accuracy, Precision Recall

•  Token extraction: words (bag of words) and bigrams.

•  Test between languages and domain: trained and tested the same classifiers and extractors with the Movielens dataset [1]

[1] The MovieLens dataset: http://www.grouplens.org/node/73

Gleicon Moraes, Marco Aurélio Gerosa 19/20

Page 20: Semi Automatic Sentiment Analysis

Efficiency

Movie Review (en) Feature Extractor Accuracy Positive

Precision Negative Precision

Positive Recall

Negative Recall

Bag of Words 0.7280 0.6516 0.9597 0.9800 0.4760

Bigrams 0.8240 0.7613 0.9263 0.9440 0.7040

Consumer Opinion (pt_br) Feature Extractor Accuracy Positive

Precision Negative Precision

Positive Recall

Negative Recall

Bag of Words 0.5984 1.0000 0.5100 0.3099 1.000

Bigrams 0.7049 1.0000 0.5862 0.4930 1.000

Gleicon Moraes, Marco Aurélio Gerosa 20/20

Page 21: Semi Automatic Sentiment Analysis

Conclusion

•  Consumer review database helped on initial training.

•  O keep the messages as is helps makes the database richer with different forms of the same expression

•  Token extraction influences the end result

•  Feeding back helps to keep the database up to date

•  To combine classifiers helps the end results and the precision

•  Contribution: Brazilian portuguese database and scripts used to extract data and to reproduce the experiment at: https://github.com/gleicon/sentiment_analysis

Gleicon Moraes, Marco Aurélio Gerosa 21/20