Placing TaskOrganisers: Adam Rae (Yahoo! Research)

Pascal Kelm (Technische Universität Berlin)

Smile!

?

!

Task Description

• Given a video, how accurately can it be placed on a map and be given latitude and longitude coordinates?

METADATA

Task Overview

• Automatic location annotation of online videos

• 7 teams submitted results (17% up)– 5 veterans– 2 new participants

• First year for code sharing– GitHub (currently)

Data

• Provided– Textual metadata: tags, titles, descriptions– Visual: 9 visual features extracted for key frames

every 4 seconds– Additional media: images with textual and

visual feature data• Available (external)– Up to the participant, but controlled according to

run submission

Data

• Training– 15,563 videos (combination of last year’s training

and test data)– 3,185,258 additional Flickr images

• Test– 4,182 videos

2012 Training

2011 Training

2010 Training

2010 Test2011 Test

2012 Test

Evaluation

• Take the latitude + longitude suggested by participants for each video

• Compute Haversine distance between that and the ‘true’ location

• We group results into buckets of increasing radii, e.g. 1km, 10km, 20km, etc.

Overall Best Results

CEALIST

IRISA

UNICAMP

GENT

TUB

ICSI

TUD

0% 5% 10% 15% 20% 25% 30%

Percentage of correct locations @ 1km

Organiser-connected team

1 10 100 1000 10000 1000000

500

1000

1500

2000

2500

3000

3500

4000

4500

Only Restriction: No new material, gazetteer permitted

ICSI TUD UG-CUUNICAMP CEA_LIST London Baseline

Distance from Ground Truth

Corr

ect T

est V

ideo

s

1 10 100 1000 10000 1000000

500

1000

1500

2000

2500

3000

3500

4000

4500

Restriction: Visual Only

CEA_LIST ICSI IRISA UG-CU UNICAMP TUB

Distance from Ground Truth

Corr

ect T

est V

ideo

s

Detected trends and activity of note

• What classes of approaches were taken (has this change since last year?)– Textual, visual– Graph modelling– User modelling– …combinations of above

• Challenging Assumptions– Spatial locality visual stability?

• Absolute performance lower than last year – but…– Different data set– Less textual metadata in general

Future of the task

• Still room for improvement• Still a valuable task?• Standard of science improving

• Need new organisers! Talk to Pascal and me