How Spatial Segmentation improves the Multimodal Geo-Tagging

Pascal Kelm

[email protected]

Communication Systems Group

Technische Universität Berlin

Thursday, 04 October 2012

www.nue.tu-berlin.de

Kelm: How Spatial Segmentation improves the Multimodal Geo-

Tagging

What is meant by Spatial Segmentation?

World map is iteratively divided into segments of

different sizes

2


Tagging

Run4: only audio/visual information

Descriptions are pooled for each spatial segment (k-d tree) in

the different hierarchy level

Visual nearest neighbour search in lowest hierarchy

3


Tagging

Visual Region Model 4

Returns the visually most similar areas, which are

represented by a mean feature vector of all training images

and videos of the respective area


Tagging

Run4: Results 5

Th [km] TUB [%]UG-CU

[%]

1 0,1 0,1

10 0,1 0,7

20 0,1 0,9

50 0,1 1,1

100 0,2 2,6

200 0,8 6,9

500 4,1 14,7

1000 14,8 21,2

2000 44,5 28,5

5000 81,0 29,6

10000 98,7 91,4

15000 100,0 95,7

20000 100,0 100,0

0

10

20

30

40

50

60

70

80

90

100

1 10 20 50 100 200 500 1000 2000 5000 100001500020000

Accu

rac

y [

%]

Margin of Error [km]

TUB

UG-CU


Tagging

Run1: No additional data or gazetteers

combines textual and visual features: translation of tags and

extracted words (NLP) from the title and the description.

Porter stemmer and stop-word elimination for each segment

and granularity in the spatial segmentation.

Visual Search for the k-nearest segments in the lowest

hierarchy

6


Tagging

Term-location-distribution:

Term frequency-inverse document frequency:

7


Tagging

Example

Condence scores of the visual approach (right) restricted

to be in the most likely spatial segment determined by

the textual approach (left)

8


Tagging

Run1: Results 9

0

10

20

30

40

50

60

70

80

90

100

1 10 20 50 100 200 500 1000 2000 5000 10000 15000 20000

Accu

rac

y [

%]


TUB

Th [km] TUB [%]1 13,710 32,720 36,550 39,4100 41,8200 44,8500 51,71000 62,42000 76,55000 92,310000 99,415000 100,020000 100,0


Tagging

Run2: No additional data

For the highest hierarchy level the boundaries extraction using

gazetteers (GeoNames, Wikipedia and Google Maps) for the

spell checked words is added.

10


Tagging

Collaborative Systems: Example 11

這是我上次去巴黎。在那裡，我得到了我的城堡在迪斯尼樂園看。

這是我上次去巴黎。在那裡，我得到了我的城堡在迪斯尼樂園看。…


Tagging

Collaborative Systems: Example

這是我上次去巴黎。在那裡，我得到了我的城堡在迪斯尼樂園看。…

Which language is it?

Chinese

This was my last trip to Paris. I visited the castle in Disneyland…

Which words gives us information? Tags?

Trip, Paris, Castle, Disneyland

Which of these nouns have got geographical information?

Paris, Disneyland

12


Tagging

Geographical Ambiguity 13

Paris

France

Canada

Puerto Rico

…

Disneyland

China

USA

France

…

R(ci) = Rank sum

ci = Countries

N = Number of toponym


Tagging

Extracted geo. items

00001: hawaii, kauai, usa

14

hawaii

usa

kauii


Tagging

Results 15

0

10

20

30

40

50

60

70

80

90

100

1 10 20 50 100 200 500 1000 2000 5000 10000 15000 20000

Accu

rac

y [

%]


Run1

Run2

Run4


Tagging

Question

Thanks for your attention!

16


Tagging

Training Set: Weighting 17


Tagging

Training Set: Features 18

Technology

How Spatial Segmentation improves the Multimodal Geo-Tagging