News REcommendation Evaluation Lab (NewsREEL)

Results

Frank Hopfgartner, Benjamin Kille, Andreas Lommatzsch, Martha Larson, Torben Brodt, Jonas Seiler

Registrations

2

Task 1: Online Evaluation

• Provide recommendations for visitors of the news portals of plista’s customers

• Ten portals (local news, sports, business, technology)

• Communication via Open Recommendation Platform (ORP)

Dat

a

• Benchmark own performance with other participants and baseline algorithms during three pre-defined evaluation windows

• Best algorithms determined in final evaluation period

• Standard evaluation metricsEva

luat

ion

Recommend news articles in real-time

Task 2: Offline Evaluation

• Traffic and content updates of nine German-language news content provider websites

• Traffic: Reading article, clicking on recommendations

• Updates: adding and updating news articlesD

ata

• Simulation of data stream using Idomaar framework

• Participants have to predict interactions with data stream

• Quality measured by the ratio of successful predictions by the total number of predictionsE

valu

atio

n

Predict interactions in a simulated data stream

Evaluation Schedule (Task 1)

First evaluation window

Second evaluation window

Third evaluation window

CTR (Task 1)

9

Response rate vs number of requests (Task 1)

10

Switched off throughout the day

Error rate (Task 1)

11

Click-through rate

12

Congratulations to „xyz-2.0“ for achieving highest

CTR with 1.17%

Availability/response rate

13

Congratulations to „flumingsparkteam“ for

achieving highest response rate with

99.64%

Presentations