Synchronizing Multi-User Photo Galleries with MRF

Synchronizing Multi-User Photo Galleries with MRF

Emanuele Sansone, Giulia Boato Minh-Son Dao DISI, University of Trento UIT-HCM Trento, Italy HCMC,

Viet Nam e.sansone@unitn.it, boato@disi.unitn.it sondm@uit.edu.vn

Synchronization and Clustering

Multiple Galleries

Feature Extraction

Offset Estimation

Synchronized Galleries

Stereo Matching I

Stereo camera

Acquired Images

Stereo Matching II

Observed node = pixel intensity

Latent node = possible disparity for a given node

Potentials associated to each link in the network

Inference algorithm

Disparity

Stereo Matching: Model

• Pixels Photos• Pixel Intensitities Image features (Timestamp, GPS, SURF, color

histograms)• States: possible diparities States: possible offsets

Stereo Matching vs. Photo Galleries Synchronization

Synchronization: States

Synchronization: Potentials

Synchronization: Potentials

Synchronization: Potentials

Synchronization: Potentials

Synchronization: Potentials

Synchronization: Potentials

Synchronization: Final Model

Synchronization: Inference

Clustering

• K-means algorithm• 3 runs

Clustering

• K-means algorithm• 3 runs

Color Structure Descriptors(CSD - 64 values) + Local Binary Patterns 3x3 (LBP - 18 values each block)

Clustering

• K-means algorithm• 3 runs

Color Structure Descriptors(CSD - 64 values) + Local Binary Patterns 3x3 (LBP - 18 values each block)

PCA Color Structure Descriptors (6 values)

Clustering

• K-means algorithm• 3 runs

Color Structure Descriptors(CSD - 64 values) + Local Binary Patterns 3x3 (LBP - 18 values each block)

PCA Color Structure Descriptors (6 values)

PCA Color Structure Descriptors (6 values) + TIME

Results: Synchronization

Dataset Precision Accuracy

Vancouver 0.35 0.86

London 0.25 0.89

Results: Synchronization

Dataset Precision Accuracy

Vancouver 0.35 0.86

London 0.25 0.89

• Good performance in terms of accuracy

Results: Synchronization

Dataset Precision Accuracy

Vancouver 0.35 0.86

London 0.25 0.89

• Good performance in terms of accuracy• Only 1/4 galleries correctly synchronized

Results: ClusteringDataset Run Features Jaccard Index

Vancouver

1 CSD-64+LBP3x3 0.1673

2 CSD-6 0.1382

3 CSD-6+TIME 0.1315

London

1 CSD-64+LBP3x3 0.1287

2 CSD-6 0.0742

3 CSD-6+TIME 0.0885

Results: ClusteringDataset Run Features Jaccard Index

Vancouver

1 CSD-64+LBP3x3 0.1673

2 CSD-6 0.1382

3 CSD-6+TIME 0.1315

London

1 CSD-64+LBP3x3 0.1287

2 CSD-6 0.0742

3 CSD-6+TIME 0.0885

• Performance are decreased by using only color descriptors

Results: ClusteringDataset Run Features Jaccard Index

Vancouver

1 CSD-64+LBP3x3 0.1673

2 CSD-6 0.1382

3 CSD-6+TIME 0.1315

London

1 CSD-64+LBP3x3 0.1287

2 CSD-6 0.0742

3 CSD-6+TIME 0.0885

• Performance are decreased by using only color descriptors• In the Vancouver dataset time doesn’t help in clustering

Results: ClusteringDataset Run Features Jaccard Index

Vancouver

1 CSD-64+LBP3x3 0.1673

2 CSD-6 0.1382

3 CSD-6+TIME 0.1315

London

1 CSD-64+LBP3x3 0.1287

2 CSD-6 0.0742

3 CSD-6+TIME 0.0885

• Performance are decreased by using only color descriptors• In the Vancouver dataset time doesn’t help in clustering• In the London dataset time becomes a more reliable feature

Lessons Learned

Synchronization:Only 1/4 galleries correctly synchronized

Lessons Learned

Synchronization:Only 1/4 galleries correctly synchronized• new features (e.g. text tags, social network data)

Lessons Learned

Synchronization:Only 1/4 galleries correctly synchronized• new features (e.g. text tags, social network data)• “no association” between a pair of images

Lessons Learned

Synchronization:Only 1/4 galleries correctly synchronized• new features (e.g. text tags, social network data)• “no association” between a pair of images

Clustering:Time is important!

Thank you