Logo mining application on Flickr ®By: Ximing Hou

Internet and web-based application are widely used

Enormous data transmit volume

◦ 1.2 ZB in 2010 35 ZB in 2020 (1ZB=1012 GB)

Main contribution: Logo on the Map System (LMS)

Three experiments conducted with LMS

Team project with Zilong Wang

◦ Zilong is responsible for developing the visual matching algorithm

◦ I am in charge of the LMS application development

Algorithm

◦ Integrated based on SIFT- Scale-Invariant Feature Transform

Programming

◦ Matlab

◦ PHP, XML, JavaScript, MySQL

Other web sources

◦ Flickr ® and Google ® API

Why Flickr ®

◦ Billions of web images

◦ Tagged by millions of viewers

◦ Provide API

Why Google ®

◦ Widely used

◦ Ability to label a position on the map

◦ Provide API

Description of LMS

◦ Web-based application

◦ Extract large amount of data from Flickr®

◦ Clean the Image data

◦ Label the location of image data on dynamic map

Basic architecture

Logo MatchingModule

Picture Extraction

Module

logo geography labeling moduleDatabase

Back-End

Front-End

2

31

Flickr ® Website

Google ® Website

Database structure

◦ First to know:

http://farm{farm-id}.static.flickr.com/{server-id}/{id}_{secret}_[mstzb].jpg

Pictures on Flickr® consist of {farm-id}, {server-id}, {secret}, {id}

[mstzb] stands for different picture size.

◦ Database details

MySQL

pic stores standard logo

photo stores web images from Flickr®

Picture extraction module

◦ Input name and amount

◦ Send request to Flickr®

◦ Return XML from Flickr®

◦ Parse XML

◦ Store image information in database

◦ Classify picture by name

Picture Extraction Module

Flickr ® Website

Flickr ® API

Internet

Graphical User Interface

Nam

e

Database

AmountName

XML Reader

Picture extraction module (Cont.)

◦ Sending request:

◦ http://api.flickr.com/services/rest/?method=flickr.photos.search&api_key={api_key}&tags={tags}

&per_page={per_page}&has_geo=1&in_gallery=true&sort=interestingness-desc

◦ Return XML:

<?xml version="1.0" encoding="utf-8" ?> - <rsp stat="ok">- <photos page="1" pages="7449" perpage="2" total="14898"><photo id="5677848399" owner="62409281@N08"

secret="e55f3f02ec" server="5224" farm="6" title="IMG_0934" ispublic="1" isfriend="0" isfamily="0" /> <photo id="5639118139" owner="50831163@N07"

secret="8efce2761d" server="5182" farm="6" title="Australian Open" ispublic="1" isfriend="0" isfamily="0" /> </photos></rsp>

Retrieve picture

information from

XML

Logo matching module

◦ Image data cleaning

Using SIFT algorithm to screen out the picture including the target logo

Searching for Subway

Logo matching module

Picture Extraction Module

Flickr ®

Website

Flickr ® API

Internet

Information of

Raw picture

Standard

logo

Raw

Pictures

SIFT

Algorithm

Updated

information of

picture

Database

XML Reader

。Retrieve web image

from database

。Using improved SIFT to

clean image data

。Update the clean image

data in the database

Logo matching module

Before matching After matching

All the noise data are removed

Logo geography labeling module

◦ If data set are huge after cleaning

◦ New York City, London, Tokyo, Paris, Hong Kong, Chicago, Los

Angeles, Singapore, Sydney, Seoul, Brussels, San

Francisco, Washington, D.C. ,Toronto, Beijing, Berlin, Madrid, New

Castle, Vienna, Boston, Frankfurt, Shanghai, Buenos, Aires, Stockholm, Zurich, M

oscow, Barcelona, Dubai, Rome, Amsterdam, Mexico City…..

◦ City like New Castle

New Castle – UK, Australia?

Logo geography labeling module

。 Retrieve web image

geography information from

database

。Display them on the map

Logo matching module

We can see the

exact position of all

the picture visually

Test the advantage of the visual matching in web mining

with LMS◦ Textual word keywords search for text web source

◦ For web images

Images are different for different people

Lose accuracy

Using visual match search

Example: starbucks

Starbucks

Test the advantage of the visual matching in web mining

with LMS◦ Using textual keyword “starbucks” to search on Flickr®

◦ Download 200 pictures (52.6% are not pictures about Starbucks®)

◦ Using Logo matching module to clean the data

Accuracy = (0.325+0.540)/1=86.5%.

Specificity = 0.540/(0+0.540)=100%

Precision = 0.325/(0.325+0)=100%

Recall = 0.325/(0.325+0.135)=70.6%

◦ Visual matching search is suitable for image searching rather than

textual keywords searching.

Effect of brand name on web searching◦ Some brand names have more meanings

◦ Four brand logos in two groups

◦ Download 50, 100 and 200 pictures of each brand

Starbucks® McDonald Subway® Apple®

Group 1 Group 2

Effect of brand name on web searching◦ 50 pictures P(Starbucks)=23/50=46%

P(mcdonalds)=13/50=26%

P (apple)=2/50=4%

P(subway)=0/50=0%

◦ 100 pictures P(starbucks)=49%

P(mcdonalds)=22%

P(apple)=5%

P(subway)=0%

◦ 200 pictures P(starbucks)=47.5%

P(mcdonalds)=18%

P(apple)=4.5%

P(subway)=0%

P(starbucks)=47.5%

P(mcdonalds)=22%

P(apple)=4.5%

P(subway)=0%

Average

Search by keywords, the brand name

with unique meaning have much

more accurate search result than the

brand name with ambiguities

LMS research on people’s interest on different topic of

pictures in different time ◦ As time goes, people’s interests is changing all the time

2001 - 911 Attack

2008 - Olympics

2010 - Michael Jackson

◦ Two experiments:

Shanghai EXPO (2010)

Vancouver Winter Olympic Games (2010)

200 pictures for each from Flickr® and summarize the time distribution

with LMS

LMS research on people’s interest on different topic of

pictures in different time ◦ Histogram of each result

◦ We can obtain a right result based on the web picture number

statistics with LMS

0

50

100

150

200

2002 2006 2008 2009 2010 2011

Number of pictures for EXPO from 2002 to 2011 in Shanghai

number

0

10

20

30

40

50

60

19

68

19

87

20

00

20

01

20

02

20

03

20

06

20

07

20

08

20

09

20

10

20

11

Number of pictures on Olympics in Vancouver

Number of pictures on Olympics in Vancouver

LMS is an effect tool for web image mining

◦ Data classification

◦ Data cleaning

◦ Visualize the geographical distribution

Future works◦ Install on smart phone (e.g iPhone®)

◦ Data analysis on image authors

Thanks to◦ Uwe R. Zimmer – project meeting, project document

◦ Lexing Xie – project supervising, technical support

◦ Zilong Wang – technical support, cooperation

◦ Everyone in today’s presentation

[1] IDC (2009), Digital Data to Earth: You have run out of memory, retrieved on May 30th, 2011 from TG Daily website:

http://www.tgdaily.com/hardware-features/49611-digital-data-to-earth-you-have-run-out-of-memory

[2] SearchCRM.com (2002), Web mining, retrieved on May 30th, 2011 from SerachCRMwebsite: http://searchcrm.techtarget.com/definition/Web-mining

[3] Modern mind (2009), Brand effectiveness, retrieved on May 30th, 2011 from Modern mind website: http://www.modernmind.com/brand.htm

[4] Lowe, D. G., “Object recognition from local scale-invariant features”, International Conference on Computer Vision, Corfu, Greece, September 1999.

[5] Shapiro, Linda and Stockman, George. "Computer Vision", Prentice-Hall, Inc. 2001

[6] Lowe, D. G., “Distinctive Image Features from Scale-Invariant Keypoints”, International Journal of Computer Vision, 60, 2, pp. 91-110, 2004.

[7] Screenshot of tags on del.icio.us in 2004 and Screenshot of a tag page on del.icio.us, also in 2004, both published by Joshua Schachter on July 9, 2007.

[8] Yan-Tao Zheng, Shi-Yong Neo, Tat-Seng Chua, Qi Tian, “Toward a higher-level visual representation for object-based image retrieval”, November 2008

[9] Branding (2009), brand name development, Retrieved on June, 2nd, 2011 from the Branding website: http://www.brandidentityguru.com/brand-name.htm

[10] Fauna, October 19, 2010, Shanghai World Expo Sees 1+ Million Visitors In A Single Day, Chinasmack