Discovering User Perceptions of Semantic Similarity in Near-duplicate Multimedia Files

Raynor Vliegendhart (speaker)Martha LarsonJohan Pouwelse

WWW 2012 Workshop on Crowdsourcing Web Search (CrowdSearch 2012),Lyon, France, April 17, 2012.

2

• Introduction

• Crowdsourcing Task

• Results

• Conclusions and Future Work

Outline

3

Question:Are these the same? Why (not)?

Chrono Cross - 'Dream of the Shore Near Another World' Violin/Piano Cover

Chrono Cross Dream of the Shore Near Another World Violin and Piano

sources: YouTube, IQYNEj51EUI (left), Iuh3YrJtK3M (right)

4

Question:Are these the same? Why (not)?

Chrono Cross - 'Dream of the Shore Near Another World' Violin/Piano Cover

Chrono Cross Dream of the Shore Near Another World Violin and Piano

Yes, it’s the same song

sources: YouTube, IQYNEj51EUI (left), Iuh3YrJtK3M (right)

5

Question:Are these the same? Why (not)?

Chrono Cross - 'Dream of the Shore Near Another World' Violin/Piano Cover

Chrono Cross Dream of the Shore Near Another World Violin and Piano

No, these are different

performances by different performers

sources: YouTube, IQYNEj51EUI (left), Iuh3YrJtK3M (right)

6

Functional near-duplicate multimedia items are

items that fulfill the same purpose for the user.

Once the user has one of these items, there is no

additional need for another.

Problem:What constitutes a near duplicate?

7

Problem:What constitutes a near duplicate?

Our work:

• Discovering new notions of user-perceived similarity between multimedia files

• in a file-sharing setting

• through a crowdsourcing task.

8

Motivation:Clustering items in search results

screenshot from Tribler (tribler.org)

9

Motivation:Clustering items in search results

screenshot from Tribler (tribler.org)

10

• Introduction

• Crowdsourcing Task

• Results

• Conclusions and Future Work

Outline

11

• Three multimedia files displayed as search results

• Worker points the odd one out and justifies why

• Challenge: eliciting serious judgments

Crowdsourcing Task:Point the odd one out

12

“Imagine that you downloaded the three items in the list and that you view them.”

Crowdsourcing Task:Eliciting serious judgments (1)

Harry Potter and the Sorcerers Stone Audio Book (478 MB)

Harry Potter and the Sorcerer s Stone (2001)(ENG GER NL) 2Lions- (4.36 GB)

Harry Potter.And.The.Sorcerer.Stone.DVDR.NTSC.SKJACK.Universal.S (4.46 GB)

♫

13

Crowdsourcing Task:Eliciting serious judgments (2)

• Don’t force workers to make a contrast

• Explain the definition of functional similarity

o The items are comparable. They are for all practical purposes the same. Someone would never really need all three of these.

o Each item can be considered unique. I can imagine that someone might really want to download all three of these items.

o One item is not like the other two. (Please mark that item in the list.) The other two items are comparable.

14

Final HIT Design

15

• Introduction

• Crowdsourcing Task

• Results

• Conclusions and Future Work

Outline

16

Dataset

top 100 content 75 queries 75 results lists /32,773 filenames

1000 random triads (test set)28 manually selected triads (validation set)

17

Results

RecruitmentHIT

Main HIT

3 validation triads

1000 test triads+ 28 validation triads mixed in

two HITs run concurrently

(3 workers per test triad)

18

Results

RecruitmentHIT

Main HIT

14 qualified workers

8

3 validation triads

1000 test triads+ 28 validation triads mixed in

free-text judgments for 308

test triads< 36h

19

• Print judgments on small pieces of paper

• Group similar judgments into piles

• Merge piles iteratively

• Label each pile

Card Sort

20

Card Sort

Example: “different language”

• “The third item is a Hindi language version of the movie”

• “This is a Spanish version of the movie represented by

the other two”

• …

21

Different movie vs. TV showNormal cut vs. extended cutCartoon vs. movieMovie vs. bookGame vs. corresponding movieCommentary document vs. movieMovie/TV show vs. unrelated audio albumDifferent episodeEpisodes from different seasonMultiple episodes vs. full seasonDifferent songsSong vs. albumAlbum vs. remixExplicit versionSong vs. collection of songs+videosLanguage of subtitlesMobile vs. normal versionDifferent codec/container (MP4 audio vs. MP3)Crack vs. gameDifferent game, same seriesAddon vs. main applicationList (text document) vs. unrelated item

Different movieMovie vs. trailerComic vs. movieAudiobook vs. movieSequels (movies)Soundtrack vs. corresponding movieMovie vs. wallpaperComplete season vs. individual episodesGraphic novel vs. TV episodeDifferent realization of same legend/storyDifferent albumsCollection vs. albumEvent capture vs. songBonus track includedEvent capture vs. unrelated movieDifferent languageQuality and/or sourceDifferent gameSoftware versionsDifferent applicationDocumentation (pdf) vs. softwareSafe vs. X-Rated

User-perceivedSimilarity Dimensions

22

• Introduction

• Crowdsourcing Task

• Results

• Conclusions and Future Work

Outline

23

• A wealth of user-perceived dimensions of similarity discovered,some we could not have thought of

• Quick results due to interesting crowdsourcing task,with the focus on engagement and encouraging serious workers

Conclusions

24

• Expand experiments, larger worker volume

• Other multimedia search settings

• Crowdsourcing the card sorting process

• Use findings to guide design of clustering algorithmsDone: first version is deployed in Tribler

Future Work

25

Questions?