Embed Size (px)
Citation preview
Suggestion Mining from Opinionated Text
Sapna Negi PhD student Supervisor: Dr. Paul Buitelaar Insight Centre for Data Analytics, National University of Ireland Galway Insight NLP SIG meeting, 24th August, 2016
Opinionated texts Opinion containing text. social media, debates, blogs, feedback, reviews, discussion forums
Suggestion An idea or plan put forward for consideration. Advice, hint, tip, proposal, recommendation etc.
Opinionated texts Opinion containing text. social media, debates, blogs, feedback, reviews, discussion forums
Suggestion An idea or plan put forward for consideration. Advice, hint, tip, proposal, recommendation etc.
Mining
Manually read
and decide?
Car Country Support ...
Swift India 80% of 24000 results
i10 India …
Volkswagen Polo Ireland …
Suggestion
Mining?
Manually answered currently.
Suggestion mining =
automatic answering
Introduction
Asher et al. (2009): Opinion expressions can be categorized in four top-level categories
Asher et al. (2009): Varying proportions of opinion expressions
State of the Art: Opinion Mining = Sentiment Analysis
Hotel Review: Room-service fast and delicious, great selection of food. If you prefer an outside room, ask for one on an upper floor facing towards Cathedral. For a really great breakfast walk a block to the Ameron Cafe.
Camera Review: One of the features that sold me on the canon g3 was
the battery life. I would recommend a larger compact-flash card, at least 128 mb .
Suggestions in Sentiment Analysis Datasets
Hotel Review: Room-service fast and delicious, great selection of food.
If you prefer an outside room, ask for one on an upper floor facing towards Cathedral. For a really great breakfast walk a block to the Ameron Cafe.
Camera Review: One of the features that sold me on the canon g3 was the
battery life. I would recommend a larger compact-flash card, at least 128 mb.
Suggestions from a Sentiment Perspective
Hotel Review: Room-service fast and delicious, great selection of food.
If you prefer an outside room, ask for one on an upper floor facing towards Cathedral. For a really great breakfast walk a block to the Ameron Cafe.
Camera Review: One of the features that sold me on the canon g3 was the
battery life. I would recommend a larger compact-flash card, at least 128 mb.
• Targets other than the central entity
• Special case of conditional sentiments • Sentiments expressed as suggestions, advice
SoTA Sentiment Datasets
Guidelines for SemEval 2015 Aspect Based Sentiment Analysis dataset
Suggestions vs Sentiments in Reviews
Hotel Reviews
Electronics Reviews
Non- Suggestions Suggestions
Non- Suggestions Suggestions
- I recommend doing the upgrade for a trouble free operation - Creative should get some marketing people to work on the names
Research Questions and Related Work
Research Question
Classifier
Suggestion (+ve class)
Non-suggestion (-ve class)
Input sentences
How to automatically
detect suggestions?
Binary text classification task
Research Questions
2. How to automatically
detect suggestions?
Classifier
Suggestion Non-suggestion
Input sentences
1. How to define
suggestions? (for annotation guidelines,
scope, evaluation)
Binary text classification task
Related Work
Related Work Domain-independent approach
Suggestion Definition
Dataset available
Method and Results (F score)
Brun 2013 ✗(product reviews)
✗
✗
Rule based (0.73)
Dong 2013 ✓(tweets)
✗
✓
SVM, FM (0.69)
Wicaksono 2013 ✗ (discussion thread)
✗ ✓ HMM (0.75)
All previous works performed binary text classification
Related Work
Suggestions in Opinionated text
Suggestions in Reviews
Suggestions to brand owners
Suggestions to
fellow customers
Suggestions in Tweets
Suggestions to
brand owners
Advice in discussion
forums
Brun et al. 2013, Ramanand et al. 2013
Dong et al. 2013 Negi and Buitelaar 2015
Wicaksono et al. 2013
Use case specific works, inadequate qualitative analysis for datasets, evaluation, and limited to sentence classification
Qualitative Analysis
Source Example Linguistic properties Receiver
Electronics Reviews
I would recommend doing the upgrade to be sure you have the best chance at trouble free operation.
Subjunctive, Imperative Customer
Electronics Reviews
My one recommendation to Creative is to get some marketing people to work on the names of these things
Imperative Brand owner
Hotel Reviews
Be sure to specify a room at the back of the hotel.
Imperative Customer
Tweets (Windows phone)
Dear Microsoft, release a new zune with your wp7 launch on the 11th. It would be smart
Imperative, subjunctive Brand owner
Travel discussion thread
If you do book your own airfare, be sure you don’t have problems if Insight has to cancel the tour or reschedule it
Conditional, imperative Thread participants
Suggestions across the domains are linguistically similar
Current Work
Suggestions
Suggestions in Reviews
Suggestions to brand owners
Suggestions to fellow
customers
Suggestions on twitter
Suggestions to brand owners
Suggestions on discussion forums
…........
Our Work
Suggestions
Suggestions in Reviews
Suggestions to brand owners
Suggestions to fellow
customers
Suggestions on twitter
Suggestions to brand owners
Suggestions on discussion
forums …........
- Detailed study of suggestion annotation, consistent guidelines, benchmark datasets
- One classifier for all, comparison of performance of multiple classifiers across
datasets - Suggestion representation and summarization
Datasets
Data annotation: crowd sourced annotations
- Using Crowdsourcing - First round of annotations on review datasets: Generic definition of suggestions
Data annotation: crowd sourced annotations
- Low agreement between annotators
- Reasons: Different perception of ‘suggestions’
No. of suggestions Confidence
Electronics (3782) Hotel (8050)
1488
3220 >=0.6
604 1046 >=0.7
562 1024 >=0.8
558 1020 >=0.9
553 1020 1
Data Annotation: Disagreements
Opinion expression Example Confidence > 60%
Instructions/ Imperatives
If you do end up here, be sure to specify a room at the back of the hotel.
✓
Advice I would advise getting an inclusive deal or eating at one of the many local cafes which offered breakfast for a third of the price.
✓
Recommendation for/against
I recommend a trabi safari. ✓
Wish/necessity The furniture is in a serious need of polishing. ✓
Information I got a much better deal at the Marriott Potsdamer Platz on a previous trip.
✗
Praise/criticism It's not that good for the center attractions and not well connected to public transports.
✗
Data Annotation: Disagreements
Opinion expression Example Confidence > 60%
Instructions/ Warnings
Room was big, bath was lovely, but watch out for the tile floor after you shower.
✓
Advice I would advise getting an inclusive deal or eating at one of the many local cafes which offered breakfast for a third of the price.
✓
Recommendation for/against
I recommend a trabi safari. ✓
Wish/necessity The furniture is in a serious need of polishing. ✓
Information I got a much better deal at the Marriott Potsdamer Platz on a previous trip.
✗
Praise/criticism It's not that good for the center attractions and not well connected to public transports.
✗
Explicitly expressed
Implicitly expressed
Final annotations
- Suggestions should explicitly urge the reader to adopt a certain course of action, or recommend a certain entity. - All sentences of less than 4 length were removed from the dataset. Relevant entities should be directly mentioned within the sentence. - Kappa score (2 annotators) of upto 0.81 for explicitly expressed suggestions. 0.72 for tweets.
Datasets: Available from related works
Dataset Sugg / Total Intended receiver
Tweets Microsoft phone (annotations verified) - Dong et al 2013
238 / 3000 Brand owner
Travel discussions (retagged) - Wicaksono et al 2013
1314 / 5183 Thread participants
Datasets: Our datasets
Dataset Sugg / Total Intended receiver
Tweets Microsoft phone 238 / 3000 Brand owner
Travel discussions 1314 / 5183 Thread participants
Hotel reviews 448 / 7534 Customers
Electronics reviews 324 / 3782 Customers
Negi and Buitelaar (2015) Mostly imbalanced datasets
Datasets: Our datasets
Dataset Sugg / total Intended receiver
Tweets Microsoft phone 238 / 3000 Brand owner
Travel discussions 1314 / 5183 Thread participants
Hotel reviews 448 / 7534 Customers
Electronics reviews 324 / 3782 Customers
Suggestion forum (mobile app) 1428 / 5724 Brand owners
Tweets using hash-tags: suggestion, advice, recommendation, warning
1126 / 4099 Variable
Negi et. al (2016) Identification of data sources likely to contain
more number of suggestions
Experiments and Results
Experiments: In-domain training, Cross-fold validation
Data F Rules SVM LSTM CNN
Hotel 0.285
0.543 0.639 0.578
Electronics 0.340 0.640 0.672 0.612
Travel discussion 0.342
0.566 0.617 0.586
Microsoft tweets 0.325
0.616 0.550 0.441
New tweets 0.266
0.632 0.645 0.661
Suggestion forum 0.605
0.712 0.727 0.713
Rules: From related works SVM: Linguistic Features Word embeddings: COMPOSES (Baroni et al. 2014), Twitter Glove (Pennington et al.2014) F scores for positive class
Experiments
SVM features: - Unigram, Bigrams - Imperative mood POS patterns
- Sentiment score summation
- Presence / absence of subject, POS of subject
Rules: - Modal verb (MD) followed by base form of verb (VB)
- Atleast one clause starts with verb present tense
- Presence of suggestion keywords
- Presence of (manually identified) suggestion templates
Comparison with related work
Dataset Related work F1: Related work
LSTM CNN
Travel advice Wicaksono and Myaeng, 2013
0.756 0.762 0.692
Microsoft tweets Dong et al. 2013 0.694 0.550 0.441
Use of non-replicable features (extracted from a private dataset), use of hashtags
Experiments: Cross-domain training
Train/Test F SVM LSTM CNN
Sugg forum / Hotel 0.211 0.452 0.363
Sugg forum / Electronics 0.180 0.516 0.393
Sugg forum / Travel thread 0.273 0.323 0.453
Sugg forum + Travel thread / Hotel 0.306 0.345 0.393
Sugg forum + Travel thread / Electronics
0.259 0.503 0.456
New tweets / Microsoft tweets 0.117 0.246 0.241
Training: datasets with larger no. of suggestions
Experiments: Some Variations
- Features for NNs = Embeddings + POS tag Decreased precision in all the cases, increased Recall - Tweets with preprocessing, reduced the F score - Use of dependency based embeddings (Levy and goldberg, 2014)
Train LSTM CNN
COMP. Deps. COMP. Deps.
Hotel 0.638 0.607 0.578 0.550
Electronics 0.672 0.608 0.611 0.556
Travel discussion
0.617 0.625 0.586 0.564
Sugg forum 0.752 0.732 0.714 0.695
Train/Test LSTM CNN
COMP. Deps. COMP. Deps.
Sugg forum/ hotel
0.450 0.380 0.363 0.367
Sugg forum/ Electronics
0.510 0.470 0.393 0.384
Sugg forum/ Travel advice
0.323 0.340 0.453 0.330
Travel advice/ Hotel
0.316 0.349 0.304 0.292
Experiments: Imperative mood detection
- Features for NNs = Embeddings + POS tag Decreased precision in all the cases, increased Recall - Tweets with preprocessing - Use of dependency based embeddings (Levy and goldberg, 2014)
Train LSTM CNN
COMP. Deps. COMP. Deps.
Hotel 0.638 0.607 0.578 0.550
Electronics 0.672 0.608 0.611 0.556
Travel discussion
0.617 0.625 0.586 0.564
Sugg forum 0.752 0.732 0.714 0.695
Conclusion
§ A dedicated study of suggestions and suggestion mining
§ Benchmark datasets § Yet to discover the one model that fits all. Experimented with straightforward approaches so far. Deep learning based approaches performed better. Challenges: § Not enough datasets for training statistical models. § Sparsely mentioned entities and topics in suggestions. § Varied styles of expressing suggestions: warning, request, advice, instruction etc.
Future Direction
- Domain adaptation, and data augmentation approaches using deep learning - Information extraction from suggestions If you do end up here, be sure to specify a room at the back of the hotel.
Suggestion sentence Sub-type Action/Entity Central phrase
If you do end up here, be sure to specify a room at the back of the hotel.
advice action Specify a room at the back of the hotel
Do not forget to choose a room at the back of the hotel
advice action Choose a room at the back of the hotel
Thank You
Questions / Suggestions?
