Introduction to Deep Learning with Python

From multiplication to convolutional networksHow do ML with Theano

Today’s Talk● A motivating problem● Understanding a model based framework● Theano

○ Linear Regression ○ Logistic Regression○ Net○ Modern Net○ Convolutional Net

Follow alongTutorial code at:https://github.com/Newmu/Theano-TutorialsData at:http://yann.lecun.com/exdb/mnist/Slides at:http://goo.gl/vuBQfe

A motivating problemHow do we program a computer to recognize a picture of a handwritten digit as a 0-9?

What could we do?

A dataset - MNISTWhat if we have 60,000 of these images and their label?

X = images

Y = labels

X = (60000 x 784) #matrix (list of lists)

Y = (60000) #vector (list)

Given X as input, predict Y

An ideaFor each image, find the “most similar” image and guess

that as the label.

An ideaFor each image, find the “most similar” image and guess

that as the label.

KNearestNeighbors ~95% accuracy

Trying thingsMake some functions computing relevant information for

solving the problem

What we can codeMake some functions computing relevant information for

solving the problem

feature engineering

What we can codeHard coded rules are brittle and often aren’t obvious or

apparent for many problems.

A Machine Learning Framework

Inputs Computation Outputs

A … model? - GoogLeNet

from arXiv:1409.4842v1 [cs.CV] 17 Sep 2014

A very simple model

Input Computation Output

3 mult by x 12

Theano intro

imports

Theano intro

imports

theano symbolic variable initialization

Theano intro

imports

theano symbolic variable initializationour model

Theano intro

imports

compiling to a python function

Theano intro

imports

Theano

imports

Theano

imports

training data generation

Theano

imports

symbolic variable initialization

Theano

imports

our model

Theano

imports

our model

model parameter initialization

Theano

imports

our model

metric to be optimized by model

Theano

imports

our model

metric to be optimized by modellearning signal for parameter(s)

Theano

imports

our model

metric to be optimized by modellearning signal for parameter(s)how to change parameter based on learning signal

Theano

imports

our model

Theano

imports

our model

compiling to a python functioniterate through data 100 times and train model on each example of input, output pairs

Theano doing its thing

Logistic Regression

T.dot(X, w)

softmax(X)

0. 0.10. 0.0. 0.0. 0.10.7

Zero One Two Three Four Five Six Seven Eight Nine

Back to Theano

convert to correct dtype

Back to Theano

initialize model parameters

Back to Theano

our model in matrix format

Back to Theano

our model in matrix formatloading data matrices

Back to Theano

now matrix types

Back to Theano

now matrix types

probability outputs and maxima predictions

Back to Theano

now matrix types

probability outputs and maxima predictionsclassification metric to optimize

Back to Theano

now matrix types

compile prediction function

Back to Theano

now matrix types

compile prediction function

train on mini-batches of 128 examples

What it learns

0 1 2 3 4 5 6 7 8 9

What it learns

0 1 2 3 4 5 6 7 8 9

Test Accuracy: 92.5%

An “old” net (circa 2000)

h = T.nnet.sigmoid(T.dot(X, wh))

y = softmax(T.dot(h, wo))

0. 0.10. 0.0. 0.0. 0.0.9

A “old” net in Theano

generalize to compute gradient descent on all model parameters

Understanding SGD

2D moons datasetcourtesy of scikit-learn

2 layers of computationinput -> hidden (sigmoid)hidden -> output (softmax)

Understanding Sigmoid Units

initialize both weight matrices

updated version of updates

What an “old” net learns

A “modern” net - 2012+

h = rectify(T.dot(X, wh))

y = softmax(T.dot(h2, wo))

0. 0.10. 0.0. 0.0. 0.0.9

h2 = rectify(T.dot(h, wh))

Noise(or augmentation)

A “modern” net in Theano

rectifier

Understanding rectifier units

rectifier

numerically stable softmax

rectifier

a running average of the magnitude of the gradient

rectifier

a running average of the magnitude of the gradientscale the gradient based on running average

Understanding RMSprop

2D moons datasetcourtesy of scikit-learn

rectifier

randomly drop values and scale rest

rectifier

randomly drop values and scale rest

Noise injected into modelrectifiers now used2 hidden layers

What a “modern” net learns

Quantifying the difference

What a “modern” net is doing

Convolutional Networks

from deeplearning.net

A convolutional network in Theano

a “block” of computation conv -> activate -> pool -> noise

convert from 4tensor to normal matrix

reshape into conv 4tensor (b, c, 0, 1) format

reshape into conv 4tensor (b, c, 0, 1) formatnow 4tensor for conv instead of matrix

conv weights (n_kernels, n_channels, kernel_w, kerbel_h)

highest conv layer has 128 filters and a 3x3 grid of responses

noise during training

noise during trainingno noise for prediction

What a convolutional network learns

Takeaways● A few tricks are needed to get good results

○ Noise important for regularization○ Rectifiers for faster, better, learning○ Don’t use SGD - lots of cheap simple improvements

● Models need room to compute.● If your data has structure, your model should

respect it.

Resources● More in-depth theano tutorials

○ http://www.deeplearning.net/tutorial/● Theano docs

○ http://www.deeplearning.net/software/theano/library/● Community

○ http://www.reddit.com/r/machinelearning

A plugKeep up to date with indico:https://indico1.typeform.com/to/DgN5SP

Questions?

Introduction to Deep Learning with Python

Data & Analytics

DEEP LEARNING WITH PYTHON - Morris Riedel...2018/04/19 · DEEP LEARNING Programming with TensorFlow & Keras 19th April 2018 Page 7 [1] M. Riedel, Invited YouTube Tutorial on Deep

Deep Learning Type Inference - GitHub Pagesvhellendoorn.github.io/PDF/fse2018-j2t.pdf · Deep Learning Type Inference ... Python 3.x introduced type hints via its typings package,

DEEP LEARNING IN PYTHON - Amazon S3 · Deep Learning in Python Vanishing gradients Occurs when many layers have very small slopes (e.g. due to being on ﬂat part of tanh curve) In

Hands-on Lab: Deep Learning with the Theano Python Library€¦ · Hands-on Lab: Deep Learning with the Theano Python Library Frédéric Bastien ... OO coding language I Numpy: n

INTRODUCTION TO DATA SCIENCE DEEP LEARNING · MACHINE LEARNING, DEEP LEARNING & AI USING PYTHON Introduction ML Fundamentals ML Common Use Cases Understanding Supervised and Unsupervised

Deep Learning with Python (PyData Seattle 2015)

Python for Image Understanding: Deep Learning with Convolutional Neural Nets

Deep Learning with Python

Deep Learning for Computer Vision with Python · Deep Learning for Computer Vision with Python Practitioner Bundle Dr. Adrian Rosebrock 1st Edition (1.2.1)

Hands-on Deep Learning in Python

Deep Learning for Computer Vision with Python · Welcome to the ImageNet Bundle of Deep Learning for Computer Vision with Python, the ﬁnal volume in the series. This volume is meant

Deep Learning for Image Recognition in Python

Deep Learning with SAS and Python: A Comparative …...we conduct an in-depth comparison between SAS and Python on their deep learning modeling with different types of data, including

Deep Learning with Python - Aalto

Hands-on Lab: Deep Learning with the Theano …on-demand.gputechconf.com/gtc/...bastien-theano-python-library.pdfHands-on Lab: Deep Learning with the Theano Python Library Frédéric

DEEP LEARNING IN PYTHON - Amazon S3 · Deep Learning in Python Example as seen by linear regression Predicted Transactions Bank Balance Retired Not Retired Model with no interactions

DEEP LEARNING SPECIALIST...o Analytics Managers and Professionals o Statisticians with an interest in Machine and deep learning • What you will learn o Use Python and SQL to access

Deep Learning with Python - tutorialspoint.com · Deep Learning with Python i About the Tutorial Python is a general-purpose high level programming language that is widely used in

Python (deep learning and machine learning) for EEG signal

An Introduction to Python Machine Learning · Machine Learning? We will understand why Python is a ... TensorFlow - Machine Learning (deep) NLTK - Natural Language processing. Python: