Using Deep Learning to rank and tag millions of hotel...

16,10 0,00 9,80 16,10

5,906,30

0,320,32

Using Deep Learning to rank and tag millions of hotel images15/11/2018 - PyParis 2018

Christopher Lennan (Senior Data Scientist)@chris_lennan

Tanuj Jain (Data Scientist)@tjainn #idealoTech

16,10 0,00 9,80 16,10

5,906,30

0,320,32

Agenda

1. idealo.de

2. Business Motivation

3. Models and Training

4. Image Tagging

5. Image Aesthetics

6. Summary

16,10 0,00 9,80 16,10

5,906,30

0,320,32

Some Key Facts

18 More than 18 years experience

700 “idealos” from 40 nations

Active in 6 different countries (DE, AT, ES, IT, FR, UK)

16 million users/month 1

50.000 shops

Over 330 million offers for 2 million products

Tüv certified comparison portal 2

Germany's 4th largest eCommerce website

16,10 0,00 9,80 16,10

5,906,30

0,320,32

Motivation

16,10 0,00 9,80 16,10

5,906,30

0,320,32

idealo hotel price comparisonhotel.idealo.de

● 2.306.658 accommodations

● 308.519.299 images

● ~ 133 images per

accommodation

16,10 0,00 9,80 16,10

5,906,30

0,320,32

Importance of Photography for Hotels

“.. after price, photography is the most important factor for travelers and prospects scanning OTA sites..”

“.. Photography plays a role of 60% in the decision to book with a particular hotel ..”

“.. study published today by TripAdvisor, it would seem like photos have the greatest impact driving engagement from travelers researching on hotel and B&B pages ..”

16,10 0,00 9,80 16,10

5,906,30

0,320,32

16,10 0,00 9,80 16,10

5,906,30

0,320,32

16,10 0,00 9,80 16,10

5,906,30

0,320,32

1 2 3 4 5

6 7 8 9

10 11 12 13

16,10 0,00 9,80 16,10

5,906,30

0,320,32

Position: 19Position: 1

Current image placement

Image Aesthetics

16,10 0,00 9,80 16,10

5,906,30

0,320,32

Image AestheticsCurrent image placement

Position: 17Position: 3

16,10 0,00 9,80 16,10

5,906,30

0,320,32

Beautiful images should appear earlier in the gallery

1 2 3 4 5

6 7 8 9

10 11 12 13

1 2 3 4 5

6 7 8 9

10 11 12 13

16,10 0,00 9,80 16,10

5,906,30

0,320,32

Ensure different areas get depicted

1 2 3 4

5 6 7 8

Bedroom Bathroom

Restaurant Facade Fitness Studio Kitchen

16,10 0,00 9,80 16,10

5,906,30

0,320,32

Understanding Image Content

1. Tag the image with the hotel property area

2. Predict aesthetic quality

Two part problem

16,10 0,00 9,80 16,10

5,906,30

0,320,32

Models & Training

16,10 0,00 9,80 16,10

5,906,30

0,320,32

Transfer Learning

● Use pre-trained CNN that was trained on millions of images

(e.g. MobileNet or VGG16)

● Replace top layers so that the output fits with classification task

● Train existing and new layer weights

16,10 0,00 9,80 16,10

5,906,30

0,320,32

Transfer LearningCNN architecture (VGG16)

16,10 0,00 9,80 16,10

5,906,30

0,320,32

Training regime

1. Only train the newly added dense layers with high learning rate

2. Then train all layers with low learning rate

Goal: Do not juggle around the pre-trained convolutional weights too much

16,10 0,00 9,80 16,10

5,906,30

0,320,32

Training regime

16,10 0,00 9,80 16,10

5,906,30

0,320,32

● CEL generally used for “one-class” ground truth classifications (e.g. image tagging)

● CEL ignores inter-class relationships between score buckets

Loss functionsCross-entropy loss (CEL)

source: https://ssq.github.io/2017/02/06/Udacity%20MLND%20Notebook/

16,10 0,00 9,80 16,10

5,906,30

0,320,32

Loss functions

● For ordered classes, classification settings can outperform regressions

● Training on datasets with intrinsic ordering can benefit from EMD loss objective

Earth Mover’s Distance (EMD)

16,10 0,00 9,80 16,10

5,906,30

0,320,32

Local AWS

GPU training workflow

ECRpushCustom

AMIdatasets

nvidia-docker

EC2GPU

instance

launchDockerMachine

train script

Docker image

Dockerfile

SSHevaluation script

DockerMachine

EC2GPU

instance

launch

Jupyter notebook

Evaluate launch evaluation container with nvidia-docker

pull image

copy existing model

S3launch training container with nvidia-docker

store train outputs

pull image

copy existing model

16,10 0,00 9,80 16,10

5,906,30

0,320,32

Image Tagging

16,10 0,00 9,80 16,10

5,906,30

0,320,32

Tagging Problem

● Given an image, tag it as belonging to a single class

● Multiclass classification model with classes:○ Bedroom○ Bathroom○ Foyer○ Restaurant○ Swimming Pool○ Kitchen○ View of Exterior (Facade)○ Reception

16,10 0,00 9,80 16,10

5,906,30

0,320,32

Multiple Datasets

Will go over them one-by-one and see:

● Dataset properties● Results● Issues

16,10 0,00 9,80 16,10

5,906,30

0,320,32

Wellness Dataset

● Idealo in-house pre-labelled images

● Mostly pictures of 2 or 3 stars properties

16,10 0,00 9,80 16,10

5,906,30

0,320,32

Wellness Dataset

● Balanced: Equal sample count in all categories for all sets

16,10 0,00 9,80 16,10

5,906,30

0,320,32

Wellness Dataset: Metrics

Top-1- accuracy: 86%

16,10 0,00 9,80 16,10

5,906,30

0,320,32

Wellness Dataset: Wrong PredictionsTrue Class of these images: BATHROOM, Predicted as: RECEPTION

Rectangular structure = Reception with high probability→ BIAS!

16,10 0,00 9,80 16,10

5,906,30

0,320,32

Wellness Dataset: Wrong PredictionsTrue Class of these images: BATHROOM

Wrong true label of images→ NOISE in the dataset!

16,10 0,00 9,80 16,10

5,906,30

0,320,32

Correcting Bias

● Augmentation operations, same for every class:○ Random cropping○ Rotation○ Horizontal flipping

● Data enrichment:○ External data from google images

16,10 0,00 9,80 16,10

5,906,30

0,320,32

Augmented Wellness + Google Dataset: Metrics

16,10 0,00 9,80 16,10

5,906,30

0,320,32

Gotta Clean!

16,10 0,00 9,80 16,10

5,906,30

0,320,32

Cleaning Dataset

● Hand-cleaned each category:

○ Deleted pictures that do not belong in its category

○ Removed duplicates (presence of duplicates can give us wrong

metrics)

○ Added more images from external sources for classes with a small

number of images left after cleaning

16,10 0,00 9,80 16,10

5,906,30

0,320,32

Cleaned Data: Metrics

16,10 0,00 9,80 16,10

5,906,30

0,320,32

Cleaned Dataset: Results

● Bathroom vs. Reception confusion has almost vanished!

● View_of_exterior vs Pool confusion has reduced

● Foyer performance:

○ Most misclassifications of Foyer gets assigned to

Reception

○ This is human problem as well!

16,10 0,00 9,80 16,10

5,906,30

0,320,32

Foyer or Reception?

16,10 0,00 9,80 16,10

5,906,30

0,320,32

Learnings so far

● The model can only be as good as the data (cleaning)

● Foyer is a hard category to predict

16,10 0,00 9,80 16,10

5,906,30

0,320,32

Understanding Model Decisions

16,10 0,00 9,80 16,10

5,906,30

0,320,32

Understanding Decisions: Class Activation Maps

● Use the penultimate Global Average Pooling Layer (GAP) to get class activation map

● Highlights discriminative region that lead to a classification

16,10 0,00 9,80 16,10

5,906,30

0,320,32

Insights With CAMSwimming Pool misclassified as Bathroom

16,10 0,00 9,80 16,10

5,906,30

0,320,32

16,10 0,00 9,80 16,10

5,906,30

0,320,32

16,10 0,00 9,80 16,10

5,906,30

0,320,32

Using rails to misidentify Pool as Bathroom.

16,10 0,00 9,80 16,10

5,906,30

0,320,32

Insights With CAMBathroom correct classification

16,10 0,00 9,80 16,10

5,906,30

0,320,32

16,10 0,00 9,80 16,10

5,906,30

0,320,32

16,10 0,00 9,80 16,10

5,906,30

0,320,32

Using faucets to correctly identify Bathroom.

16,10 0,00 9,80 16,10

5,906,30

0,320,32

Learnings so far

● Attribution techniques like CAM lend interpretability

● CAM can drive data collection in specific directions

16,10 0,00 9,80 16,10

5,906,30

0,320,32

Tagging Next Steps

1. Add still more data

a. Explore manual tagging options for training

(Example: Amazon Mechanical Turk)

2. Add more classes

a. Fitness Studio

b. Conference Room

c. Other

16,10 0,00 9,80 16,10

5,906,30

0,320,32

Image Aesthetics

16,10 0,00 9,80 16,10

5,906,30

0,320,32

Ground Truth Labels

For the NIMA model we need “true” probability distribution over all classes for each image:

● AVA dataset: we have frequencies over all classes for each image

→ normalize frequencies to get “true” probability distribution

(6.151 / 1.334)

16,10 0,00 9,80 16,10

5,906,30

0,320,32

Iterations

We have gone through two iterations of the aesthetic model:

● First iteration - Train on AVA Dataset

● Second iteration - Fine-tune first iteration model on in-house labelled data

16,10 0,00 9,80 16,10

5,906,30

0,320,32

Results - first iteration

Linear correlation coefficient (LCC): 0.5987Spearman's correlation coefficient (SCRR): 0.6072Earth Mover's Distance: 0.2018Accuracy (threshold at 5): 0.74

Aesthetic model - MobileNet

16,10 0,00 9,80 16,10

5,906,30

0,320,32

Examples - first iterationAesthetic model

16,10 0,00 9,80 16,10

5,906,30

0,320,32

16,10 0,00 9,80 16,10

5,906,30

0,320,32

16,10 0,00 9,80 16,10

5,906,30

0,320,32

16,10 0,00 9,80 16,10

5,906,30

0,320,32

16,10 0,00 9,80 16,10

5,906,30

0,320,32

Results - second iteration

● We built a simple labeling application

● http://image-aesthetic-labelling-app-nima.apps.eu.idealo.com/

● ~ 12 people from idealo Reise and Data Science labeled

○ 1000 hotel images for aesthetics

● We fine-tuned the aesthetic model with 800 training images

● Built aesthetic test dataset with 200 images

16,10 0,00 9,80 16,10

5,906,30

0,320,32

Results - second iteration

Linear correlation coefficient (LCC): 0.7986Spearman's correlation coefficient (SCRR): 0.7743Earth Mover's Distance: 0.1236Accuracy (threshold at 5): 0.85

Aesthetic model - MobileNet

16,10 0,00 9,80 16,10

5,906,30

0,320,32

Examples - second iterationAesthetic model

16,10 0,00 9,80 16,10

5,906,30

0,320,32

16,10 0,00 9,80 16,10

5,906,30

0,320,32

16,10 0,00 9,80 16,10

5,906,30

0,320,32

16,10 0,00 9,80 16,10

5,906,30

0,320,32

16,10 0,00 9,80 16,10

5,906,30

0,320,32

ProductionAesthetic model

16,10 0,00 9,80 16,10

5,906,30

0,320,32

ProductionAesthetic model

● To date we have scored ~280 million images

● Distribution of scores (sample of 1 million scores):

16,10 0,00 9,80 16,10

5,906,30

0,320,32

Production - Low Scores Aesthetic model

16,10 0,00 9,80 16,10

5,906,30

0,320,32

Production - Medium Scores Aesthetic model

16,10 0,00 9,80 16,10

5,906,30

0,320,32

Production - High Scores Aesthetic model

16,10 0,00 9,80 16,10

5,906,30

0,320,32

Understanding Model Decisions

16,10 0,00 9,80 16,10

5,906,30

0,320,32

Convolutional Filter Visualisations

Layer 23 MobileNet original

MobileNet Aesthetic

16,10 0,00 9,80 16,10

5,906,30

0,320,32

MobileNet Aesthetic

16,10 0,00 9,80 16,10

5,906,30

0,320,32

MobileNet Aesthetic

16,10 0,00 9,80 16,10

5,906,30

0,320,32

Aesthetic Learnings

● Hotel specific labeled data is key - Aesthetic model improved markedly from 800

additional training samples

● NIMA only requires few samples to achieve good results (EMD loss)

● Labeled hotel images also important for test set (model evaluation)

● Training on GPU significantly improved training time (~30 fold)

16,10 0,00 9,80 16,10

5,906,30

0,320,32

● Continue labeling images for aesthetic classifier

● Introduce new desirable biases in labeling (e.g. low technical quality == low aesthetics)

● Improve prediction speed of models (e.g. lighter CNN architectures)

Aesthetics Next Steps

16,10 0,00 9,80 16,10

5,906,30

0,320,32

Summary

16,10 0,00 9,80 16,10

5,906,30

0,320,32

● Transfer learning allowed us to train image tagging and aesthetic classifiers with a few

thousand domain specific samples

● Showed the importance of having noise-free data for quality predictions

● Use of attribution & visualization techniques helps understand model decisions and

improve them

Summary

16,10 0,00 9,80 16,10

5,906,30

0,320,32

Check us out! #idealoTech

https://github.com/idealo https://medium.com/idealo-tech-blog

16,10 0,00 9,80 16,10

5,906,30

0,320,32

We’re hiring!

Data Engineers, DevOps Engineers across different teamsCheck out our job postings: jobs.idealo.de

16,10 0,00 9,80 16,10

5,906,30

0,320,32

Tanuj Jaintanuj.jain@idealo.de@tjainn

Christopher Lennanchristopher.lennan@idealo.de@chris_lennan

16,10 0,00 9,80 16,10

5,906,30

0,320,32

THE END

Using Deep Learning to rank and tag millions of hotel...

Documents

Applicare a Grant EFSAold.iss.it/binary/efsa/cont/Francesca_Maranghi_Esperienze_di_esperti... · A.2 Travel costs and subsistance allowances 0,00 € 0,00 € 0,00 € 0,00 € A.3

ASSETS · 2020. 9. 14. · 202. 22.476.717,00 ... 1731 731. 0,00 732. 0,00 12. Share of profit or loss of undertakings accounted for under the . equity method 1663. 663 0,00. 664

EL SOLAR AND LA VIVIENDA VERNÁCULA AS EXAMPLES OF ...geobalcanica.org/wp-content/uploads/GBP/2018/GBP.2018.26.pdf · Kaqchikel 103 36 0,00 0,00 Ixil 83 71 0,00 0,01 Teko 53 0 0,00

NAZ. A5* CLASSIFICA 004 - C140 A TEMPO - Equieffe13 87 tzigane de bacon fra 2007 bettella gino [propr. lady farm di barco simone] ita s 2°g 4 63,29 0,00 0,00 0,00 0,00 0,00 09256d

Disclosure of Transfer of Value from Pharmaceutical ... · Paul Ole Bischoff Berlin (All) Luisenstr. (blank) 0,00 0,00 0,00 0,00 825,00 619,74 0,00 1.444,74 Petra Becker Hameln (All)

· 2018-04-20 · Chicaque 1013,45 0,00 Colombia 350,96 0,00 Cusio 360,52 0,00 La Maria 107,21 0,00 La Rambla 649,65 0,00 La Rapida 936,19 0,00 Laguna Grande 495,01 0,00 Las Angustias

# PTS. PART. CAT. CORREDOR CLUB PTA ATQ MONTE ...clubcoma.org/wp-content/uploads/2018/12/2018_Rank-LIGA...2º 46,29 1 F-10 Irene Lopez Martin 2 COMA 46,29 0,00 0,00 0,00 0,00 0,00

Etrea Equestrian Tour Results 001 - COMP. 1 - 6 Y.O. IN 2 ... · 0/4 42,27 0,00 0,00 0,00 0,00 0,00 106SU44 10005471 0,00 15 33 EXPRESSO D'ARGOUGES FRA 2014 AMMENDOLA FIORENZO [TAQUIN

CSI 1* + CSI 3* Results 001 - N.1 6 YO IN TWO PHASES - Etrea4 40 CAPE TOWN 25 GER 2013 CROTTA Sabrina [Owner SARAH CAPPELLIN] [Sponsor: ] SUI S 0/0 40,84 0,00 0,00 90,00 0,00 0,00

Crystal Reports ActiveX Designer - B14PDG...22 Immobilisations reçues en affectation (4) 0,00 0,00 0,00 0,00 23 Immobilisations en cours 1 336 790,87 202 273,17 59 860,97 1 074 656,73

CSIYH-1 + CSI 1* + CSI 2* + CSIV-B Results 001 - N. 1 ... · ita s 0/0 0,00 0,00 0,00 126,00 0,00 0,00 105oz05 10043207 0,00 1 23 desdemona di montefiridolfi ita 2012 checchi matteo

SABADELL RENDIMIENTO, FI...2 2. Datos económicos Periodo actual Periodo anterior 2019 2018 Índice de rotación de la cartera 0,00 0,00 0,00 0,00 Rentabilidad media de la liquidez

NONE 17-19 Maggio 2013 A4* outdoor/indoor cat aggiunte ......30 SILDARCO ITA BAROTTO FRANCESCO ITA 2008 Partito 0,00 0,00 0,0 0,00 0,00 0,00 J Brev.J FISE13515 15304A 31 ELIDE ITA

NAZ. A5* CON STAFFETTA IN NOTTURNA CLASSIFICA 004 ......3 135 TZIGANE DE BACON FRA 2007 BETTELLA GINO [Propr. LADY FARM DI BARCO SIMONE] ITA S 2 G 0 63,45 0,00 0,00 315,00 0,00 0,00

Updates to the Prevalence of Undernourishment2009-11 2010-12 2011-13 2012-14 2013-15 2014-16 2015-17 2016-18 DEC 2917 2949 2979 3010 3049 3071 3089 3093 CV 0,32 0,32 0,32 0,32 0,32

IN COMPLIANCE WITH ARTICLE 1003 CODE OF ......2012/04/17 · $101.00 13456 03/02/2012 EV120065 BARREn DAFFIN ET AL 0.00 0.00 5,00 0,00 0.00 0.00 0,00 150.00 0.00 0,00 0.00 0,00 $155.00

Kамаз-65115 · Проценты по кредиту годовых 403 277,78 310 672,70 0,00 Доход компании 0,00 190 182,69 0,00 Налог на имущество

ADAC Cimbern Rallye 2018minirallysyd.dk/onewebmedia/Løb2018/MRS1...5 0,78 18,58 6 13,93 7 10,59 8 50 10,00 9 9,59 10 70 9,11 0,00 0,00 0,00 ZK / TC 100 0,00 0,00 ZK / TC 101 26 44

Classifica - Weeblyequilazio.weebly.com/uploads/2/...campionati_2012.pdf · adorno paolo ita 1996 8/0 0,00 140,11 0,0 0,00 0,00 0,00 s 2°gr. 000576/m 02688mxx 13 33 gioco del castegno

ESTACIÓN CHICHIMENE · 2013-09-28 · Estacion. 0,00% 80,00% 0,00% 0,00% 20% Pendiente entrega materiales para terminar prefabricados, pintura, prueba hidrostatica, Tie Ins, Dos