CERTH @ MediaEval 2011 Social Event Detection Task

Symeon Papadopoulos, Christos Zigkolis, Yiannis Kompatsiaris, Athena Vakali

Pisa, 1-2 September 2011

2

The problem

• Identify social events in tagged photos collections:– Challenge 1: Soccer matches @ Barcelona, Rome

– Challenge 2: Events @ Paradiso (Amsterdam) and Parc del Forum (Barcelona)

• Alternative formulation:– For each photo of the collection answer the questions:

Q1. Is this photo related to a social event of the given types?

Q2. If yes, to which event is it related?

– Points to classification and clustering as methods to address the problem.

3

Approach

Q1

Q2

Q1 / Q2

4

Photo Filtering (1)

• City classification

– If geo-tagging available (~20%), use it simple nearest-neighbour classifier

– If not, match against city-specific tag models:

• Created from processing independent geo-tagged photo collections

Amsterdam (74) Barcelona (57) London (89) Paris (51) Rome (42)

amsterdamnetherlandshollandnederland….

barcelonacatalunyacataloniaespaña….

londonukunited kingdomgreat britain….

parisfrancefranciaversailles….

romeitalyvaticanoitalia….

TAG MODEL SAMPLES

5

Photo Filtering (2)

• Soccer/Venue classification

– In the case of venue classification, use geo-tagging information if available.

– Match against soccer/venue tag model:

• Parameter (cf. evaluation)

Soccer (53, m1,b)

soccerfootballgoalgoalkeeper…

names of Spanish FCsnames of Italian FCs

+

Paradiso (6, m2,b)

TAG MODEL SAMPLES (baseline)

paradisoconcertfestivalgiglive music

Parc del Forum (8, m2,b)

parc del forumprimavera soundconcertfestival…

names of scheduled bands (last.fm)+

domain knowledge

6

Event Partitioning

• Very simple implementation:

– Find all unique dates of photos that “passed” the first filtering step.

– For each date, find all associated photos and split them into groups based on the city they are classified (same classifier as in Step 1).

– Consider the resulting groups of photos, as the set of events.

7

Event Expansion

• Expand in three ways:

– Photos having the same owner as one of the owners in the event & captured at the same date.

– Photos captured at the same location (<200m) with the event center & at the same date (only for geo-tagged photos)

– Photos belonging to the same cluster (by use of method [1]) & having the same owner as one of the owners in the event (parameter: cluster type)

[1] S. Papadopoulos, C. Zigkolis, Y. Kompatsiaris, A. Vakali. “Cluster-basedLandmark and Event Detection on Tagged Photo Collections”. In IEEE MultimediaMagazine 18(1), pp. 52-63, 2011

8

Evaluation (1)

Challenge 1

NotationParameter 1 (p1): m1,b (baseline tag model), m1,+ (extended soccer tag model)Parameter 2 (p2): tt (use photo title + tags), ttd (use photo description + tt)Parameter 3 (p3): ∅ (no clustering), T (tag-based clustering), V (visual clustering)

9

Evaluation (2)

Challenge 2

NotationParameter 1 (p1): m2,b (baseline tag model), m2,+ (extended venue tag model)Parameter 3 (p3): ∅ (no clustering), T (tag-based clustering), V (visual clustering),

H (hybrid clustering)

m2,+ was created by adding to baseline the names of the bands that played in these venues in the same month (collected from last.fm API)

10

Failure examples (1)C1 - Run1 / False positives

Title: AVUÍ SOM 77.331Tags: …, Campions, Trophy, campnou, soccer, football, caosasuna, barça, fiesta, …

3559542192

Many of the photo tags are related to soccer and even to a soccer event (fiesta, champions).

Title: Sant PereTags: Barcelone, Barcelona, Night Ambiance, Light

3618132279 3580841609

Title: roma 09.Tags: rome, italy coliseum, palatino, chuch, soccer, statues, art

Just one of the tags (soccer) is related to soccer.

Captured at the same date and in the vicinity of the event.

Most of the false positives were due to the expansion step (i.e. same day + close by, or same day + same user)

11

Failure examples (2)C1 - Run2 / False negatives

Title: near Tor di Quinto, Latium, ItalyTags: N/A

Description: s.s. lazio wins the coppa italia

3559542192

Here the event information is only present in the photo description.

Title: Barcelona v. Manchester UnitedTags: Sigma 10-20mm, F4-5.6 EX DC HSM, barcelona, spain, moo2

3571654936 3583033760

Title: DSC_0029Tags: FC Barcelona Fiesta Tri Campions

Event information is encoded in a single tag, but we don’t tokenize tags, so we miss it.

The information could be inferred from title if our tag model contained FC names from different countries.

Most of the false negatives were due to failure in matching the textual metadata of photos to the soccer tag model.

12

Discussion (1)

• Most important factor:– a good tag model to be used for classification

• Marginal contribution of clustering:– expansion by spatio-temporal metadata already captures

most related photos

– tag-based clusters tend to include many of the photos of the same user at the same date

– visual clusters did not yield further improvements as one would hope (at least with employed visual similarity measure: 500 feature vector from clustering SIFT features)

13

Discussion (2)

• Future action: study in detail failure cases and make necessary modifications to approach

• Ways to improve:– better topic/entity classification methods

• better/richer tag models + text matching methods

• more sophisticated methods: e.g. SVMs, relational learning + more discriminative features (text, visual, social)

– more elaborate city classification methods or even precise geo-tagging methods

14

Questions