ChallengeIntroduction2017Portroz, Slovenia, May, 2017
1. INTRODUCTION 2. CHALLENGE TASKS 3. EVALUATION
[1] comScore/the Kelsey group. Online consumer-generated reviews have significant impact on offline purchase behavior. Press Release, November 2007. http://www.comscore.com/press/ release.asp?press=1928. [2] John A. Horrigan. Online shopping. Pew Internet & American Life Project Report, 2008. [3] Lee Rainie and John Horrigan. Election 2006 online. Pew Internet & American Life Project Report, January 2007 [4] http://www.umiacs.umd.edu/research/LCCD/
Goals
• Bridge research and industry at international level. This year among the organizers: University of Cagliari (Italy), Nanyang Technological University (Singapore), FINSA (Italy)
• Provide a forum for demonstrating the suitability of research approaches in real-world context
• Share own experiences
Challenge Tasks
• Task #1: Polarity Detection • Task #2: Polarity Detection in presence of metaphorical language
• Task #3: Aspect-Based Sentiment Analysis • Task #4: Semantic Sentiment Retrieval • Task #5: Frame Entities Identification • Task #6: Subjectivity and Objectivity detection • Most innovative approach
Challenge Tasks
• Task #1: Polarity Detection • Task #2: Polarity Detection in presence of metaphorical language
• Task #3: Aspect-Based Sentiment Analysis • Task #4: Semantic Sentiment Retrieval • Task #5: Frame Entities Identification • Task #6: Subjectivity and Objectivity detection • Most innovative approach
Task #1: Polarity Detection The main goal of the task is polarity detection. E.g. “Today I went to the mall and bought some desserts and a lot of very nice Christmas gifts”, which should be classified as positive.
Dataset[5] Train: 1M Amazon review, 20 domains, pos/neg balance Test: 33,361 Reviews randomly selected across the same 20 domains
[5] Mauro Dragoni, Andrea Tettamanzi and Célia da Costa Pereira - DRANZIERA: An Evaluation Protocol For Multi-Domain Opinion Mining
Task #2: Polarity Detection in presence of metaphorical language
The main goal of the task is polarity (positive, negative, neutral) detection. E.g. “I just love working for 6.5 hours without a break or anything. Especially when I'm on my period and have awful cramps.”, which should be classified as negative.
Dataset Train: SemEval Test: generated using CrowdFlower
Task #3: Aspect-Based Sentiment Analysis
The output of this task will be a set of aspects of the reviewed product and a binary polarity value associated to each of such aspects.
This task requires a set of aspects (such as ‘speaker’, ‘touchscreen’, ‘camera’, etc.) and a polarity value associated with each of such aspects.
Dataset Train: 5,399 sentences over 2 domains (laptop and restaurant) Test: 1,097 sentence over 2 domains (laptop and hotel)
Task #4: Semantic Sentiment Retrieval
This task focuses on the capability of retrieving relevant documents with respect to opinion-based queries given as input to participant systems.
Example question: “Documents talking about smartphone display.”
This task includes: • Information Retrieval (detect features of given entities) • Named Entity Recognition (detect smartphone models within the
review possibly using some sort of knowledge base) • Sentiment Analysis (aggregate features opinions for the entity
sentiment for either overall or feature based retrieval)
Task #5: Frame Entities Identification
This task will evaluate the capabilities of the proposed systems to identify the objects involved in a typical opinion frame according to their role: • holders, • topics, • opinion concepts (i.e. terms referring to highly polarized concepts).
Example: "The mayor is loved by the people in the city, but he has been criticized by the state government",
an approach should be able to identify that: • “people” and “state government” are the opinion holders, • “is loved” and “has been criticized” represent the opinion concepts, • “mayor” identifies a topic of the opinion and • there are two different opinion polarities mentioned in the sentence.
Task #6: Subjectivity and Objectivity Detection
Given a text, classify it into objective or subjective. Basically, an objective sentence does not contain any opinion within it whereas subjective text does.
The mayor is loved by the people in the city. Subjective
The mayor he has been elected by many voters. Objective
1. INTRODUCTION 2. CHALLENGE TASKS
3. EVALUATION
Evaluation protocol
• Task #1: Precision, Recall, and F-Measure on the inferred polarity
• Task #3: (i) Precision, Recall and F-Measure on the Aspect-Extraction; (ii) Accuracy on the polarities inferred on the correct aspects. Final score: F-Measure* Accuracy
• Most Innovative Approach: taking into account the novelty of the approach in terms of concept-level analysis and exploitationof semantics.
Prizes
• Most Innovative Approach: Springer Voucher of 150 Euros
• As there was only one participant to Task #3 (which was also a participant to Task #1) we removed Task #3 from the competition
Challenge Participants Task #1 • Mattia Atzeni, Amna Dridi and Diego Reforgiato: “Fine-Grained Sentiment Analysis on
Financial Microblogs and News Headlines” • Marco Federici: “A Knowledge-based Approach For Aspect-Based Opinion Mining” • Giulio Petrucci: “The IRMUDOSA System at ESWC-2017 Challenge on Semantic
Sentiment Analysis” • Andi Rexha: “Exploiting Propositions for Opinion Mining” • Walid Iguider and Diego Reforgiato Recupero: Language Independent Sentiment
Analysis of theShukran Social Network using Apache Spark
Task #3 • Marco Federici: “A Knowledge-based Approach For Aspect-Based Opinion Mining”
Challenge Participants Task #1 • Mattia Atzeni, Amna Dridi and Diego Reforgiato: “Fine-Grained Sentiment Analysis on
Financial Microblogs and News Headlines” • Marco Federici: “A Knowledge-based Approach For Aspect-Based Opinion Mining” • Giulio Petrucci: “The IRMUDOSA System at ESWC-2017 Challenge on Semantic
Sentiment Analysis” • Andi Rexha: “Exploiting Propositions for Opinion Mining” • Walid Iguider and Diego Reforgiato Recupero: Language Independent Sentiment
Analysis of theShukran Social Network using Apache Spark
Task #3 • Marco Federici: “A Knowledge-based Approach For Aspect-Based Opinion Mining”
Exploitation Proud to mention that a system that has previously competed in this challenge is being prototyped by BUP Solution s.r.l. (http://bupsolutions.com/), Italian company founded by STLAB of CNR in May 2016
RESEARCH
COMPANIES
SENTIMETRIX INC (USA) http://www.sentimetrix.com/
SentiMetrix tools combine cutting edge NLP technology with advanced network analysis algorithms implemented on top of highly scalable architecture. Expert use of modern machine learning techniques allows the company to achieve highly accurate results in areas ranging from sentiment diffusion and influence measures to bot detection, elections results predictions andmental health evaluations.