Deep Learning for Recommender Systems - Budapest RecSys Meetup

Deep Learning forRecommender Systems

Alexandros KaratzoglouSenior Research Scientist @ Telefonica Research

first.lastname@gmail.com@alexk_z

Telefonica Research

Machine Learning

HCI

Network & Systems

Mobile Computing

http://www.tid.es

Why Deep?

ImageNet challenge error rates (red line = human performance)

Why Deep?

Inspiration for Neural Learning

Early aviationattempts aimed atimitating birds, bats

Neural Model

Neuron a.k.a. Unit

Feedforward Multilayered Network

Learning

Stochastic Gradient Descent

Generalization of (Stochastic) Gradient Descent

Stochastic Gradient Descent

Stochastic Gradient Descent

Stochastic Gradient Descent

Feedforward Multilayered Network

Backpropagation

Backpropagation

Does not work well in plain a “normal”multilayer deep network

Vanishing Gradients

Slow Learning

SVM’s easier to train

2nd Neural Winter

Modern Deep Networks

Ingredients:

Rectified Linear Activation function a.k.a. ReLu

Modern Deep Networks

Ingredients:

Dropout:

Modern Deep Networks

Ingredients:

Mini-batches:

Stochastic Gradient Descent

Compute gradient over many (50 -100)

data points (minibatch) and update.

Modern Deep Networks

Ingredients:

Softmax output:

Modern Deep Networks

Ingredients:

Categorical cross entropy loss

Modern Feedforward Networks

Ingredients:

Batch Normalization

Modern Feedforward Networks

Ingredients:

Adagrad a.k.a. adaptive learning rates

Restricted Boltzmann Machines

Restricted Boltzmann Machines

Convolutional Networks

Convolutional Networks

[Krizhevsky 2012]

Convolutional Networks

[Faster R-CNN: Ren, He, Girshick, Sun 2015] [Farabet et al., 2012]

Convolutional Networks

[Faster R-CNN: Ren, He, Girshick, Sun 2015] [Farabet et al., 2012]

Convolutional Networks

Self Driving Cars

Convolutional example slides from Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 6 75

Convolutional Networks

Standford CS231n: Convolutional Neural Networks for Visual Recognition

Convolutional Networks

Convolutional Networks

Convolutional Networks

Convolutional Networks

Convolutional Networks

Convolutional Networks

Convolutional Networks

AlexNet [Krizhevsky et al 2014]

dd

D-tour → Matrix Factorization

2

5

4

5

4

4

1

5

5

4

1

2

5

2

4

1

Convolutional Networks forenhancing Collaborative Filtering

VBPR: Visual Bayesian Personalized Ranking from Implicit Feedback He,etl AAAI 2015

Convolutional Networks for Musicfeature extraction

Deep Learning can be used to learn item profiles e.g. music

Map audio to lower dimensional space where it can be useddirectly for recommendation

Useful in recommending music from the long tail (not popular)

A solution to the cold start problem

Convolutional Networks for Musicfeature extraction

A. van den Oord, S. Dielmann, B. Schrauwen Deep content-based music recommendation NIPS 2014

Convolutional Networks

deepart.io

Recurrent Neural Networks

Recurrent Neural Networks

Long Short Term Memory

Recurrent Neural Networks

Recurrent Neural Networks

PANDARUS:Alas, I think he shall be come approached and the dayWhen little srain would be attain'd into being never fed,And who is but a chain and subjects of his death,I should not sleep.

Second Senator:They are away this miseries, produced upon my soul,Breaking and strongly should be buried, when I perishThe earth and thoughts of many states.

DUKE VINCENTIO:Well, your wit is in the care of side and that.

Second Lord:They would be ruled after this chamber, andmy fair nues begun out of the fact, to be conveyed,Whose noble souls I'll have the heart of the wars.

Clown:Come, sir, I will make did behold your worship.

VIOLA:I'll drink it.

Recurrent Neural Networks

Recurrent Neural Networks

Recurrent Neural Networks

Recurrent Neural Networks

Session-based recommendationwith Recurrent Neural Networks

RNN (GRU) with ranking loss functionICLR 2016 [B. Hidasi, et.al.]

Treat each user session as sequence of clicks

Session-based recommendationwith Recurrent Neural Networks

RNN (GRU) with ranking loss functionICLR 2016 [B. Hidasi, et.al.]

Treat each user session as sequence of clicks

Autoencoders

Autoencoders

Autoencoders

Personalized Autoencoders

Collaborative Denoising Auto-Encoders for Top-N Recommender Systems Wuet.al. WSDM 2016

(Some) Deep Learning Software

Theano: Python Library

TensorFlow: Python Library

Keras: High Level Python Library (Theano &TF)

MXNET: R, Python, Julia

Thanks

● Some slides or parts of slides are taken fromother excellent talks and papers on DeepLearning (e.g. Yan Lecun, Andrej Karpathy andother great deep learning researchers)