NLP & Semantic Computing Group

N L P

SemEval-2017 Task 5: Fine-Grained Sentiment Analysis on Financial Microblogs and News

Keith Cortis, Andre Freitas, Tobias Daudert, Manuela Hurlimann, Manel Zarrouk, Siegfried Handschuh, Brian Davis

Semeval, August 2017

NLP & Semantic Computing Group

Outline

• Motivation & relevance

• Task description

• Challenge results

• Discussions

NLP & Semantic Computing Group

Motivation &

Relevance

NLP & Semantic Computing Group

High societal impact

(Material implication of information and perception)

NLP & Semantic Computing Group

Interpretation of Events and Perception at Scale

NLP & Semantic Computing Group

Fine-grained (in which sense?)

NLP & Semantic Computing Group

Fine-grained (in which sense?)

One sentence main contain multiple target-sentiment attributions

“Sales at the tilmari business went downby 8% to eur 11.8 million, while gallerixstores saw 29% growth to eur 2 million.”

NLP & Semantic Computing Group

Fine-grained (in which sense?)

One sentence main contain multiple target-sentiment attributions

“Sales at the tilmari business went downby 8% to eur 11.8 million, while gallerixstores saw 29% growth to eur 2 million.”

NLP & Semantic Computing Group

Fine-grained (in which sense?)

One sentence main contain multiple target-sentiment attributions

“Sales at the tilmari business went downby 8% to eur 11.8 million, while gallerixstores saw 29% growth to eur 2 million.”

NLP & Semantic Computing Group

Fine-grained (in which sense?)

Continuous polarity scale

“Sales at the tilmari business went downby 8% to eur 11.8 million, while gallerixstores saw 29% growth to eur 2 million.”

- 0.20 + 0.65

NLP & Semantic Computing Group

Other NLP Challenges

• Interpretation requires both domain-specific and commonsense knowledge

• Domain-specific can get really specific!

"$AAPL at pivot area on intradaychart- break here could send this to 50-day SMA, 457.80”

NLP & Semantic Computing Group

Other NLP Challenges

• Need for contextual information

“Sales at the tilmari business went downby 8% to eur 11.8 million, while gallerixstores saw 29% growth to eur 2 million.”

balance sheet

market analysis

NLP & Semantic Computing Group

Task

Description

NLP & Semantic Computing Group

Two Subtasks

Subtask 1: Microblogs

Subtask 2: News Statements &

Headlines

NLP & Semantic Computing Group

Microblog Messages

$EL: 0.95 $NKE: 0.5$SBUX: 0.5 $AAPL : 0.5

“Este Lauder beats on Revenues and EPSand boosts dividend 25% - global growthin the Middle Class trend continues. $EL$NKE $SBUX $AAPL”

NLP & Semantic Computing Group

Data Source Types

• News Statements & Headlines

“First Solar, Vivint Solar Lead Short Interest Trend”

First Solar: -0.7, Vivint Solar: -0.7

NLP & Semantic Computing Group

Test Collection Creation

March 11th and 18th, 2016

October 2011 to June, 2015

August and November, 2015

AP News, Reuters, Handelsblatt, Bloomberg and Forbes

Raw Data Collection

NLP & Semantic Computing Group

Test Collection Creation

1591 messages

1847 messages

1780 newsstatements

Sampling & Filtering

Random sampling

Spam Filtering

NLP & Semantic Computing Group

Test Collection CreationAnnotation

4 paid domain experts120 hours (30 hours per expert)

Random sampling

Spam Filtering

Annotation

NLP & Semantic Computing Group

Annotation

• Identify target entities

• Associated sentiment score:

Continuous scale between:

• -1 (very negative/bearish)

• 1 (very positive/bullish)

• The sentiment is assigned from the point of view of an investment decision

NLP & Semantic Computing Group

Inter-annotator agreement

• Average spearman’s rankcorrelation on sentiment scores wascalculated for each pair of annotators:

0.54 for news headlines

0.69 for microblogs

NLP & Semantic Computing Group

Test Collection Creation

Random sampling

Spam Filtering

Annotation

Annotation

Subtask 1

Subtask 2

1647 Headlines and News Statements

2510 Twitter and StockTwits messages

NLP & Semantic Computing Group

NLP & Semantic Computing Group

Evaluation

• Inspired by Ghosh et al. (2015).

vector space abstraction

rewarding systems which attempt to classify more instances

NLP & Semantic Computing Group

Challenge

Results

NLP & Semantic Computing Group

Participation

• Total: 32 teams

• Track 1: 25 teams

• Track 2: 29 teams

• 22 teams addressed both tracks

• 19 teams submitted a paper

Strong engagement and active mailing list.

NLP & Semantic Computing Group

Results - Subtask 1

(Microblog Messages)

NLP & Semantic Computing Group

Machine Learning Methods

NLP & Semantic Computing Group

Linguistic Resources

NLP & Semantic Computing Group

Common Frameworks

NLP & Semantic Computing Group

Results - Subtask 2

(News Statements and

Headlines)

NLP & Semantic Computing Group

Discussions

NLP & Semantic Computing Group

Alternative Evaluation Metric

NLP & Semantic Computing Group

Linguistic Resources

• Common linguistic resources: Loughran and McDonald Sentiment Word (rank 2) Opinion Lexicon (rank 1, 2) MPQA Subjectivity Lexicon (rank 1, 2).

• New resources were created during the task: Moore and Rayson Seyeditabari et al. Cabanski et al. Schouten et al. Li

NLP & Semantic Computing Group

Approaches

• Most approaches used ML + multiple sentiment lexicon features

• The spectrum of applied ML methods was very broad SVM-Regression (SVR) Ensemble methods Neural Network-based sequence models No conclusive or significant difference between

different ML categories

• The use of domain-specific lexicons impacted results for subtask1

NLP & Semantic Computing Group

Open Questions

• Impact of the use of financial backgroundinformation as background knowledge

• Deeper discussion on the relation between ML and linguistic/semantic phenomena

NLP & Semantic Computing Group

Acknowledgements

Horizon 2020 ICT Program Project SSIX: Social Sentiment analysis financialIndeXes, has received funding from the European Union’s Horizon 2020Research and Innovation Program ICT 2014 – Information and CommunicationsTechnologies under grant agreement No. 645425.