Yuri Rykov Oleg Nagornyy

Olessia Koltsova Herbert Natta

Alexander KremenetsLev Manovich

Damiano CerroneDamon Crockett

Semantic and Geospatial Mapping of Instagram Images in Saint-Petersburg

AINL FRUCT Artificial Intelligence and Natural Language Conference

November 11, 2016

BACKGROUNDDigital urban studies is research field that combines issues and methods of urban sociology, computer science, digital humanities, linguistics and design to retrieve knowledge about everyday life and social organization of cities from diverse data sources.

The availability of large geolocated visual social media data creates new opportunities for studying cities.

The important task is to extract meanings of human experience in different urban areas.

New way to do it is to analyze relations between visual content of shared images and their geographical locations.

DATASET

47,410 Instagram items from Saint-Petersburg during one year period from July 1 2014 to June 30 2015.

Data = image + time stamp + geographical coordinates + user ID + user-generated #hashtags.

Instagram API was used to collect the data.Google vision API was used to recognize entities.

• Can Instagram images be clustered into meaningful categories reflecting human experience?

• Can this experience be related to certain urban areas in a meaningful way?

RQs

Step I: Google Tag Networks and Clusters

GOOGLE TAGS NETWORK AND CLUSTERS (15)

flowers

hair & style

sunrise & sea

dish

drink

clothing

animals

document & mobile device

automobile

portrait

facade & palace

strength training

Art

CLUSTER SAMPLES

Portrait

Cars

Flowers

GOOGLE TAG CLUSTERS

& USER

#HASHTAG CLUSTERS

intersection

Chi^2cluster label

co-occ(corresp.analysis)

TOPICAL SIMILARITY OF IMAGE CLUSTERS

‘PORTRAIT’cluster images on St.Petersburg map

‘ANIMALS’ cluster images on St.Petersburg map

‘DISH’cluster images on St.Petersburg map

Drink‘DRINK’

cluster images on St.Petersburg map

Sunraise‘SUNRISE & SEA’

cluster images on St.Petersburg map

GEOSPATIAL SIMILARITY FOR CLUSTER HEATMAPS

GEOSPATIAL SIMILARITY FOR CLUSTER HEATMAPS

The most similar to many other clusters are “hairstyle” and “animals” clusters, because they are relatively evenly distributed in space: they are geographically independent topics.

The most dissimilar to other clusters are “clothing & fashion”, “power training” and "facade & palace". It indicates that related human experience occurs in the most detached urban places probably reflecting social heterogeneity and exclusiveness.

Such combination of methods has not been applied to digital urban studies before.

Such results can be used to study urban segregation or to rate city areas in terms of their consumer/tourist attractiveness or cultural/entertainment development.

CONCLUSION

Yuri Rykov - National Research University Higher School of Economics, Russia Oleg Nagornyy - Olessia Koltsova - Herbert Natta - University of Rome Tor Vergata, ItalyAlexander Kremenets - makeomatic, RussiaLev Manovich - City University of New York , USADamiano Cerrone - Spatial Intelligence Unit , EstoniaDamon Crockett - Software Studies Initiative, USA

The study was implemented in the framework of the Basic Research Program at the National Research University Higher School of Economics (HSE) in 2016. This research was started at Digital Traces Summer Lab “Meta-Morphologies of St.Petersburg 2016” directed by Lev Manovich and Damiano Cerrone.

ACKNOWLEDGMENTS

WORKING GROUP

THANK YOU FOR ATTENTION!