Extracting information from images using deep learning and transfer learning. Application to personalized recommendations.
Introduction and Context
Iterative building of a recommender system
Labeling Images Pragmatic deep learning for dummies
Post Processing AKA: Image for BI on steroids
Outline Results More images !
Dataiku
• Founded in 2013 • 90 + employees, 100 + clients • Paris, New-York, London, San Francisco,
Singapore
Data Science Software Editor of Dataiku DSS
DESIGN
Load and prepare your data
PREPARE Build your
models
MODEL Visualize and share
your work
ANALYSE
Re-execute your workflow at
ease
AUTOMATE Follow your production
environment
MONITOR Get predictions
in real time
SCORE PRODUCTIO
N
Client Key Figures
E-business vacation retailer Negotiate the best prize for their clients Discount luxury
Sale Image is paramount Purchase is impulsive
18 Millions of clients. Hundreds of sales opened everyday
Specificities
Highly temporary sales -> Classical recommender system fail -> Time event linked (Christmas, ski, summer)
Expensive Product -> Few recurrent buyers -> Appearance counts a lot
Iterative Building of a Recommender System
Basic Recommendation Engines
Other Factors
One Meta Model to Rule Them All
Recommenders as features
Machine learning to op5mize purchasing
probability
Combine
Recommend
Describe
Recommender system for Home Page Ordering
Cleaning, combining and enrichment of
data
Recommendation Engines Optimization of
home display
the application automatically runs and
compiles heterogeneous data
Generation of recommendations based
on user behaviour
Every customer is shown the 10 sales he is the
most likely to buy
Customer visits Purchases
Sales Images
Metal model combine recommendations to
directly optimize purchasing probability
Meta Model
+7% revenue
Sales information
(A/B testing)
Batch Scoring every night
Why use Image ?
We want do distinguish
« Sun and
Beach »
« Ski »
A picture is worth a thousand words
Integrating Image Information
Sales Images Labelling Model
Pool + Palm Trees Hotel
+ Mountains
Pool + Forest + Hotel + Sea
Sea + Beach +Forest + Hotel
Sales descriptions
vector
CONTENT BASED
Recommender System
Image Labelling For Recommendation Engine Pragma&c Deep learning for “Dummies”
Using Deep Learning models Common Issues
“I don’t have GPUs server” “I don’t have a deep leaning expert”
“I don’t have labelled data” (or too few)
“I don’t have the time to wait for model training ”
I don’t want to pay to pay for private apis” / “I’m afraid their labelling will change over time”
“I don’t have (or few) labelled data” -> Is there similar data ?
Solution 1 : Pre trained models
PLACES DATABASE US SUN DATABASE
205 categories 2.5 M images
307 categories 110 K images
tower: 0.53 skyscraper: 0.26
swimming_pool/outdoor: 0.65 inn/outdoor: 0.06
Solution 1 : Pre trained models If there is open data, there is an open pre trained model ! • Kudos to the community • Check the licensing
Example with Places (Caffe Model Zoo) :
Solution 2 : Transfer Learning
Credit : Fei-‐Fei Li & Andrej Karpathy & Jus5n Johnson hYp://cs231n.stanford.edu/slides/winter1516_lecture11.pdf
Solution 2 : Transfer Learning Use the network as a feature extractor
Fine tune the model Keras makes it super easy !
(Possible to freeze layer to train just the top) hYps://github.com/PGu5/Deep_Learning_tuto
PLACES DATABASE US SUN DATABASE
Training (op5onal)
Pre-‐trained model VGG16
tower: 0.53 skyscraper: 0.26
Re-‐Training
Transferred Data : Last convolu5onal layer features
Re-‐trained model TensorFlow
2 fully connected layers
Caffe Model Zoo
GPU
CPU
GPU
Leverage existing knowledge ! Solution 2 : Transfer Learning
Accuracy: 72%, Top-‐5 Acc: 90 % > state of the art on dataset alone
Transfer
Post Treatment & Results
(Or how we transfer the labelling information)
Using Images informa&on for BI on steroids
Labels post-processing
Complementary information Redondant information
Issue with our approach:
Solution : Matrix Factorization
Topic extraction with Non-Negative Matrix Factorization
• Non Negative Matrix factorization (NMF) X = WH • X : image x tags, non negative • W : image x theme • H : theme x tag (scikit learn implementation)
• Most represented Themes
• Swimming-pool_Apartment_Putting-green • Ocean_Coast_SandBar • Coast_SeaCliff_RockArch • Beach_Coast_BoardWalk • Bridge_Viaduc_River • Palace_BuildingFacade-Mansion • Castle_Mansion_Monastery • HotelRoom_Bedroom_DormRoom
• Dimension Reduction • 200x200 pixels -> 600 tags => 30
themes • Faster content based filtering
• Image often sparse combination of themes
Faster content based filtering
• Each theme has the same explication power
Balanced vector for content based
• Explicability Each theme corresponds to a few labels
Image content detection Topic scores determine the importance of topics in an image
TOPIC TOPIC SCORE (%)
Golf course – Fairway – PuHng green 31
Hotel – Inn – Apartment building outdoor
30
Swimming pool – Lido Deck – Hot tub outdoor
22
Beach – Coast -‐ Harbor 17
TOPIC TOPIC SCORE (%)
Tower – Skyscraper – Office building 62
Bridge – River – Viaduct 38
Note on model performance
• Images labels are used for similarity Calling herb field “putting green”:
• Is not important if all herbs field are called this way. • Would be if we had lot’s of golf trips sales.
• Improving the NN performance ? • Labels are used in NMF and reduced to themes • Themes are used to calculate similarities for CB
recommenders • CB Recommenders are used as a feature in meta model • Meta model give probabilities of purchase = order • Users only check 10 sales…
-> what is the change of online performance for 1% accuracy ?
Results
Results ? 1) Visits : • France and Morocco • Pool displayed
2) First Recommendation • Mostly France &
Mediterranean • Fails to display pools
3) Only Images recommendation • Pool all around the world • Does not respect budget
4) Third column = Right Mix
Results ? 1) Visits : • Spain • Sun & Beach • Pool displayed
2) First Recommendation • Displays nature…
3) Only Images ? • Pool all around the world
4) Third = Right Mix • Get the bungalow
feature !
Learned along the way
What’s next ?
AYrac5veness = % visits with tag / % sales with tag -‐1
For ski sales, indoor pictures performs beYer!
What’s Next ?
Kenya
Prague
Berlin
Cambodia
Conclusion
Do iterative data science ! Start simple and grow Evaluate at each steps Image labelling = BI on steroids
Transfer Learning Kick-start your project Gain time and money Any Data Scientist can do it
Deep Learning Don’t start from scratch ! Is there existing data ? Is there a pre-trained model ?
Thank you for your attention !
Basic Recommendation Engines • Implementation
• Everything in SQL - Vertica - Then Impala - Then Spark
• Collaborative filtering • Score(user, future sale j) = Sum_{i:visited sale} sim(i,j)
• Content based filtering • Sale profile: (sale_id, feature, value) • User profile: (user_id, feature, value) • Sparse coding + join does the trick
Join on user_id
Join on user_id + sales id
One Meta Model to Rule Them All • Negative sampling
• Take all purchases tuples : (user, product, timestamp)-> 1 • Select 5 sales open at the same date the user did not buy -> 0 • The model directly optimize purchasing probability
• Machine learning model • Features : recommender systems. • Logistic Regression
Regularizing effect : we don’t want to overfit leaks.
• Reranking approach. Similar to Google or Yandex (Kaggle challenge)
Classification and not Detection problem… • Only have probabilities of each class • Selecting based on probability threshold fails • Keeping all information is not sparse
Labels post-processing
Deep/Transfer Learning
5-10 tags per images 2s/image with CPU
x20 speed up with GPU Our images
Cafe
-> we keep 5 labels With probabilities per image
Solution 3 : What about APIs ?
Solution 3 : What about APIs ? • Price
• Their cost often rather cheap. Ex: 100 K request for less than 300$
• VS the one of redeveloping (probably not as well)
• Full Database scoring • APIs are often limited query per month. • Make sure to be able to avoid cold start problem
• Stability • Use model versioning • Avoid covariate shift, distribution drift
What about APIs ? Use for generating labels !
How to steal model: • 1) Score part of the database for training • 2) Train a model • 3) Score your entire database !
(Or don’t, it’s illegal)
But I have only 5000 requests ? -> Use Transfer Learning !
Tramèr, Florian, et al. "Stealing Machine Learning Models via Prediction APIs." arXiv preprint arXiv:1609.02943 (2016).
What about APIs ? Use for generating labels !
Experiment: • 5000 requests on API. (4500/500 split) • Transfer learning with MIT Places Pre-trained Model • Scikit learn Multilabel model
(Or don’t, it’s illegal) (demo, not used in any real project)
What about APIs ? Results
Precision 75
Label Probability Label Probability
landscape 1,0000 sunset 0,9998
sky 1,0000 no person 0,9996
outdoors 1,0000 water 0,9990
nature 1,0000 park 0,9849
rock 1,0000 river 0,9678
travel 1,0000 scenic 0,8031
Label Probability Label Probability
beach 1,0000 ocean 1,0000
summer 1,0000 relaxation 1,0000
sand 1,0000 island 1,0000
tropical 1,0000 idyllic 1,0000
travel 1,0000 seashore 0,9998
seascape 1,0000 water 0,9997
(demo, not used in any real project)
Recall 80 Accuracy 95