Real-Time News Recommendation and Lifelogging: An ... · Real-Time News Recommendation and Lifelogging: An Introduction to the Evaluation Campaigns NEWSREEL (CLEF) and Lifelog (NTCIR-12)

Real-Time News Recommendation and Lifelogging: An Introduction to the Evaluation Campaigns NEWSREEL (CLEF) and Lifelog (NTCIR-12)

Frank Hopfgartner @OkapiBM25

A few words about me

Past: Various positions in Berlin (TUB), Dublin (DCU), Berkeley (ICSI), and London (QMUL)

Research on Information Access

Systems

Lecturer in Information Studies (HATII, Glasgow)

PhD in Information Retrieval (University of Glasgow)

Overview

Evaluation Campaigns

Overview NEWSREEL

(CLEF) LifeLog

(NTCIR-12)

Introduction

Career Collaborations Research

affiliated with UC Berkeley

*now: INSIGHT Centre

Smart Information Systems

Collaborations

Recommender Systems

Aggregated Search

Multimedia Analysis

Gamification Large-Scale

Evaluation

Semantic Search

Multimedia Access

Lifelogging

Sensor Analysis Multimedia Retrieval

& Recommendation

User Modelling

User Simulation

Co-Author graph of first 100 publications (2006-2015)

Example: Video Search Engine

[ACM Multimedia’08, ECIR’08, SIVP’08, ACM TOIS’11]

Adapt retrieval results based on •User’s search queries •Relevance feedback

Example: Video Search Engine

[ACM/Springer Multimedia Systems (2010), MTAP (2012)]

Recommend news stories based on user’s long-term interests

Example: Video Browsing

[MMM’12-14, Int. Journal of Multimedia Information Retrieval (2014)]

Video Browser Showdown Competition

Example: Other Video Search Engines

Example: Image Browser

[IICAI’09]

Ostensive Browsing

Example: Visual Lifelog Browser

[published as book chapter, Springer Verlag (2013)]

Providing easy access to visual lifelogs

Example: Enterprise Search Engine

[AAMAS’14]

Aggregated Search

Example: Health Information Portal

[PervasiveHealth’13, MIS’14, Book chapter (2015)]

Semantic Search Multilingual Search

Example: Knowledge Management System

[ECIR’14, Book Chapter (2015)]

Targeting intrinsic motivation using Gamification design elements

Multimedia Analysis

Video segmentation

Voice activity detection

Quality assessment

of user-generated

video

Affective content analysis

[ECIR’09, ACM Multimedia (2014), CMBI (‘12, ‘13), MMM (2010, 2013, 2014) , ICMR (2014)]

Data Analysis

Activity and Energy expenditure estimation using accelerometer data

!

Overview


Overview NEWSREEL

(CLEF) LifeLog

(NCTIR-12)

Introduction


How do we evaluate information access systems?

Document

collection

Topic

set

Relevance

assessments

Test co

llection

Document

collection


TREC CLEF

FIRE

NTCIR

Common dataset Pre-defined tasks Ground truth Evaluation protocol Evaluation metrics

Focus on different domains

Microblogging

Ad-hoc and Web Search

Multimedia

Federated Web Search

XML Retrieval

Information Access in the Legal Domain

Document Similarity

…

Overview


Overview NEWSREEL

(CLEF) LifeLog

(NTCIR-12)

Introduction


CLEF Tracks

Source: http://www.clef-initiative.eu/

eHealth

ImageCLEF

LifeCLEF

Living Labs for IR (LL4IR)

News Recommendation Evaluation Lab (NEWREEL)

Uncovering Plagiarism, Authorship and Social Software Misuse (PAN)

Question Answering (QA)

Social Book Search (SBS)

CL

EF

’15

In CLEF NEWSREEL, participants can develop news recommendation

algorithms and have them tested by millions of users over the period of a few

months in a living lab.

NEWSREEL

Recommender Systems help users to find items that they were not

searching for.

What are recommender systems?

Items?

Example: YouTube

Example: Netflix

Example: News Articles

Source (Image): T. Brodt of plista.com

What are living labs?

Rely on feedback from real users to develop convincing demonstrators that showcase potentials of an idea or a product.

Real-life test and experimentation environment to fill the pre-commercial gap between fundamental research and innovation.

In an IR context…

“A living laboratory on the Web that brings researchers and searchers together is needed to facilitate ISSS (Information-Seeking Support System) evaluation.”

Kelly et al., 2009

A / B testing

Eval

uat

e

submit to SIGIR

CLEF NEWSREEL

20

14

Frank Hopfgartner

Andreas Lommatzsch

Benjamin Kille

Torben Brodt

Tobias Heintz

Co-Organisers

2015

Frank Hopfgartner

Torben Brodt

Benjamin Kille

Jonas Seiler

Balázs Hidasi

Andreas Lommatzsch

Roberto Turrin

Martha Larson

Scenario

Who are the users?

Devices

• Given a dataset, predict news articles a user will click on

Offline Evaluation

• Recommend articles in real-time over several months

Online Evaluation

CLEF NEWSREEL 2014

TA

SK

1

TA

SK

2

@clefnewsreel http://www.clef-newsreel.org/

Predict interactions based on an OFFLINE dataset

Task 1: Offline Evaluation

DA

TASE

T

EVA

LUA

TIO

N

Traffic and content updates of 9 German-language news content provider websites

Traffic: Reading article, clicking on recommendations

Updates: adding and updating news articles

Recorded in June 2013

65 GB, 84 Million records

[Kille et al., 2013]

Dataset split into different time segments

Participants have to predict interactions of these segments

Quality measured by the ratio of successful predictions by the total number of predictions

Recommend news articles in REAL-TIME

Task 2: Online Evaluation

LIV

ING

LA

B

EVA

LUA

TIO

N

Provide recommendations for visitors of the news portals of plista’s customers

Ten portals (local news, sports, business, technology)

Communication via Open Recommender Platform (ORP)

Provide recommendations within <10ms (VM provided if necessary)

Three pre-defined evaluation periods

5-23 February 2014

1-14 April 2014

5-19 May 2014

Evaluation criteria

Number of clicks

Number of requests

Click-through rate

Living Lab Scenario

…

Publisher A

Publisher n

Researcher 1

Researcher n

…

plista ORP

…

Millions of visitors Publishers Teams

Open Recommender Platform

Number of clicks

[CLEF’14]

Number of requests

[CLEF’14]

Click-Through Rate

[CLEF’14]

Overall results

Advertisement: Join the Living Lab - Tutorial at ECIR’15

Overview


Overview NEWSREEL

(CLEF) LifeLog

(NTCIR-12)

Introduction


NTCIR

Sourc

e: H

ideo J

oho

NTCIR-12 Tasks

NT

CIR

-12

Second round:

Search-Intent Mining

Mobile Click

Temporal Information Access

Spoken Query & Spoken Document Retrieval

QA Lab for Entrance Exam

First round:

Medical NLP for Clinical Documents

Personal Lifelog Access & Retrieval

Short Text Conversation

Encourage research advances in organising and retrieving from lifelog data.

LifeLog @ NTCIR-12

Lifelogging Challenges

The challenges are how to sense the person, their actions, their life and make it accessible using appropriate interfaces, search, recommendation engines and visual/aural feedback. Further, exploiting the lifelog to identify context for adaptive information services.

Source (Graphic): DAI-Labor, Berlin

LifeLog @ NTCIR-12

CO

-OR

GA

NIS

ER

S

TIM

EL

INE

Cathal Gurrin, Dublin City University

Hideo Joho, University of Tsukuba

Frank Hopfgartner, University of Glasgow

Brian Moynagh, Dublin City University

Rami Albatal, Dublin City University

27 Feb 2015: Kickoff event

30 Jun 2015: Task Registration

01 Jul 2015: Dataset release

Sep 2015 – Feb 2016: Formal run

01 Feb 2016: Evaluation Results

01 Mar 2016: Paper for the Proceedings

7-10 Jun 2016: NCTIR-12 conference

Multimodal dataset with information needs

Created by 5-10 individuals over

10+ days

TE

ST

CO

LL

EC

TIO

N

1,500 images, location, GSR, heart-rate, others… per lifelogger per day

Accompanying output of 1,000 concepts

Data processed pre-release (removal of personal content; face blurring, translation of concepts)

Detailed user queries and judgments generated by the lifelogging data gatherers

Evaluate different methods of

retrieval and access.

Tasks

T1:

LIFE

LOG

SEM

AN

TIC

AC

CES

S (L

SAT)

T2:

LIFE

LOG

IN

SIG

HT

Models the retrieval need from lifelogs (Known-Item Search)

Retrieve N segments that match information need

Interactive or Automatic participation

Interactive: Time limit for fair and comparative evaluation in an interactive system with users

Automatic: Fully-automatic retrieval system. Automated query processing

Models the need for reflection over lifelog data

Exploratory task, the aim is to:

encourage broad participation

novel methods to visualise and explore lifelogs

Same data as LSAT task

Presented via demo/poster.

Thank you

Come and talk to me

Frank Hopfgartner, PhD [email protected]

@OkapiBM25 www.hopfgartner.co.uk