View
10
Download
0
Category
Preview:
Citation preview
Neural NetworksAn Introduction
Warith HARCHAOUI
MAP5, UMR 8145Universite Paris-Descartes
Sorbonne Paris Cite&
Oscaro.comResearch and Development
March 2017
Outline
Supervised Classification and RegressionClassificationRegression
One Neuronfor Regressionfor Classification
Gradient DescentBatch Gradient DescentStochastic Gradient Descent
Several Neurons
Convolutional Neural Networks for Images
Adversarial Networks
Conclusion
Outline
Supervised Classification and RegressionClassificationRegression
One Neuronfor Regressionfor Classification
Gradient DescentBatch Gradient DescentStochastic Gradient Descent
Several Neurons
Convolutional Neural Networks for Images
Adversarial Networks
Conclusion
Supervised ClassificationThe binary case
Given a training set that consists of:
I xi ∈ RD
I yi ∈ {0, 1}for i = 1, . . . , nFind F s.t. F(xi ) ' yiEx:xi is an imageyi = 1 corresponds to “cat”yi = 0 corresponds to “non-cat”
Supervised ClassificationMore than 2 classes
Given a training set that consists of:
I xi ∈ RD
I yi ∈ {0, 1}K one-hot representation
for i = 1, . . . , nFind F s.t. F(xi ) ' yiEx:xi is an imageyi = [1, 0, 0] corresponds to “cat”yi = [0, 1, 0] corresponds to “dog”yi = [0, 0, 1] corresponds to “elephant”
Regression
Given a training set that consists of:
I xi ∈ RD
I yi ∈ RK
for i = 1, . . . , nFind F s.t. F(xi ) ' yiEx:xi is a buildingyi is the rent value of the building
Outline
Supervised Classification and RegressionClassificationRegression
One Neuronfor Regressionfor Classification
Gradient DescentBatch Gradient DescentStochastic Gradient Descent
Several Neurons
Convolutional Neural Networks for Images
Adversarial Networks
Conclusion
One NeuronAn Input-Output Machine
x1
x2
x3
a y
w1
w2
w3
Figure: One Neuron
y = a(w1x1 + w2x2 + w3x3 + b)
One Neuron for RegressionLeast Mean Squares
Prediction:F(xi ) = yi = Wxi + b
Loss:
L(W,b) =1
n
n∑i=1
‖yi − yi‖22
One Neuron for Binary ClassificationLogistic Function
Prediction:
scorei = w>xi + b
P(yi = 1) = pi = Sigmoid(scorei ) =1
1 + exp(−scorei )
Loss:
`(w, b) =∏
i :yi=1
pi∏
i :yi=0
(1− pi ) =n∏
i=1
pyii (1− pi )1−yi
L(w, b) =−1
nlog(`(w, b)) =
−1
n
n∑i=1
yi log(pi )+(1−yi ) log(1−pi )
One neuron for Binary ClassificationLogistic function
Figure: The Sigmoid Function
Sigmoid(a) =1
1 + exp(−a)
One Neuron for Classification of K > 2 classesSoftmax Function
Prediction:
scoreik = wk>xi + bk
pik = SoftMax(scorei ) =exp(scoreik)∑K
k ′=1 exp(scoreik ′)
yi ,k = 1⇔ xi belongs to the kth class
yi ,k = 0⇔ xi does not belong to the kth class
Loss:
`(W,b) =n∏
i=1
K∏k=1
pyi,ki ,k
L(W,b) =−1
nlog(`(W, b)) =
−1
n
n∑i=1
K∑k=1
yi ,k log(pi ,k)
Outline
Supervised Classification and RegressionClassificationRegression
One Neuronfor Regressionfor Classification
Gradient DescentBatch Gradient DescentStochastic Gradient Descent
Several Neurons
Convolutional Neural Networks for Images
Adversarial Networks
Conclusion
Batch Gradient DescentThe common problem
Loss function:
L(W,b) =1
n
n∑i=1
Li (W,b)
Problem:
minW,bL(W,b)
Batch Gradient DescentA Universal Learning Procedure
minw
1
n
n∑i=1
Li (w)
1. Choose a random w and a constant α > 0
2. Iterate:wnew = wold − α∇L(wold)
∇L(wold) =1
n
n∑i=1
∇Li (wold)
Stochastic Gradient DescentA Universal Learning Procedure
minw
1
n
n∑i=1
Li (w)
1. Choose a random w and a constant α > 0
2. Iterate:
2.1 Choose a random subset J ⊂ (1, n) ⊂ N (sometimes reducedto a singleton)
2.2wnew = wold − α
|J|∑j∈J
∇Lj(wold)
Outline
Supervised Classification and RegressionClassificationRegression
One Neuronfor Regressionfor Classification
Gradient DescentBatch Gradient DescentStochastic Gradient Descent
Several Neurons
Convolutional Neural Networks for Images
Adversarial Networks
Conclusion
Several NeuronsThe Power of Back-Propagation
x1i
x2i
x3i
x4i
pi
Hiddenlayer
Inputlayer
Outputlayer
Figure: A Multi-Layer-Perceptron
Several NeuronsThe Power of Back-Propagation
Back-Propagation is just an iterated version of Chain Rule forplenty of functions:
(F ◦ G)′ =(F ′ ◦ G
)× G′
NB: (F ◦ G)(x) = F(G(x))
Three Remarks
1. Non-linearity: Sigmoid, SoftMax, ReLu
ReLu(x) = max(x , 0)
2. Automatic Differentiation thanks to: Theano, Torch, Caffe,Tensorflow, PyTorch
3. GPU Acceleration
Outline
Supervised Classification and RegressionClassificationRegression
One Neuronfor Regressionfor Classification
Gradient DescentBatch Gradient DescentStochastic Gradient Descent
Several Neurons
Convolutional Neural Networks for Images
Adversarial Networks
Conclusion
Convolutional Neural Networks for ImagesConvolutions
x : a PixelI: an Image in gray levelsK: a Kernel = A FilterI ∗ K: Convolution of image I by filter KnonLinearity(I ∗ K): Element-wise non-linearity on the convolutionresult producing a Feature Map
(I ∗ K)(x) =∑
y∈Supp(K)
I(x − y)K(y)
The same neuron of weights K is applied many times (as much asthe number of pixels in I) producing a new image called featuremap.
Convolutional Neural Networks for ImagesConvolutions
Figure: LeNet architecture
Outline
Supervised Classification and RegressionClassificationRegression
One Neuronfor Regressionfor Classification
Gradient DescentBatch Gradient DescentStochastic Gradient Descent
Several Neurons
Convolutional Neural Networks for Images
Adversarial Networks
Conclusion
Adversarial NetworksA Desired Network
yx or z Generator
Figure: Scheme for a Desired Network
Adversarial NetworksBinary Classification Networks
y
y
Discriminator p
Figure: Scheme for Binary Classification Networks
Adversarial NetworksThe full system
yx or z Generator
y
Discriminator p
Figure: Scheme for Adversarial Networks
Adversarial NetworksA New Kind of Loss
G : Generator (e.g. of images) from random noise or a real imageD: Discriminator that distinguished fake examples from realexamples
minwD
maxwG
L
Figure: Adversarial Networks Example
Outline
Supervised Classification and RegressionClassificationRegression
One Neuronfor Regressionfor Classification
Gradient DescentBatch Gradient DescentStochastic Gradient Descent
Several Neurons
Convolutional Neural Networks for Images
Adversarial Networks
Conclusion
ConclusionA Great Book
Figure: The Deep Learning Book
Recommended