Embed Size (px)
Citation preview
1
Tutorial on Deep Learning and Applications
Hai PhanAIM Lab, University of Oregon
Includes slide material sourced from Yoshua Bengio, Geoff Hinton, Yann LeCun, Andrew Ng, Honglak Lee, and Marc’Aurelio Ranzato
2
Outline
• Deep learning– Greedy level-wise training (supervised learning)– Restricted Boltzmann machine (RBM)– Deep belief networks– Stacked autoassociators– Deep Boltzmann machines
• Applications– Human motion modeling– Vision– Language
3
Motivation: why go deep?
• Deep Architectures can be representationally efficient
• Deep representation might allow for a hierarchy of representations– Comprehensibility
• Multiple levels of latent variables allow combinatorial sharing of statistical strength
• Deep architectures work well (vision, audio, NLP, etc.)!
4
Motivation
5
6
7
8
9
10
11
Outline
• Deep learning– Greedy level-wise training (supervised learning)– Restricted Boltzmann machine (RBM)– Deep belief networks– Stacked autoassociators– Deep Boltzmann machines
• Applications– Human motion modeling– Vision– Language
12
Restricted Boltzmann Machine (RBM)
• Binary visible units– Pixels
• Binary hidden units – Feature detectors
13
Training RBM
• Free energy– inspired from physics
•
14
Different Types of Unit
• Gaussian visible units
• Gaussian visible and hidden units
• Softmax units
15
Outline
• Deep learning– Greedy level-wise training (supervised learning)– Restricted Boltzmann machine (RBM)– Deep belief networks– Stacked autoassociators– Deep Boltzmann machines
• Applications– Human motion modeling– Vision– Language
16
17
18
19
20
21
Outline
• Deep learning– Greedy level-wise training (supervised learning)– Restricted Boltzmann machine (RBM)– Deep belief networks– Stacked autoassociators– Deep Boltzmann machines
• Applications– Human motion modeling– Vision– Language
22
Stacked Autoassociators
• Reconstruct x
23
Stacked Autoassociators
24
Benchmarks
25
Benchmarks
• Consistent improvement over Neural Net
26
Outline
• Deep learning– Greedy level-wise training (supervised learning)– Restricted Boltzmann machine (RBM)– Deep belief networks– Stacked autoassociators– Deep Boltzmann machines
• Applications– Human motion modeling– Vision– Language
27
28
29
30
Greedy Layer-wise Retraining of DBM’s
31
Discriminative Fine-tuning of DBM’s
32
33
34
35
Outline
• Deep learning– Greedy level-wise training (supervised learning)– Restricted Boltzmann machine (RBM)– Deep belief networks– Stacked autoassociators– Deep Boltzmann machines
• Applications– Human motion modeling– Vision– Language
36
Conditional Restricted Boltzmann Machines (CRBM)
Auto-regression
37
Conditional Restricted Boltzmann Machines (CRBM)
• http://www.uoguelph.ca/~gwtaylor/thesis/4/
38
Outline
• Deep learning– Greedy level-wise training (supervised learning)– Restricted Boltzmann machine (RBM)– Deep belief networks– Stacked autoassociators– Deep Boltzmann machines
• Applications– Human motion modeling– Vision– Language
39
40
Convolutional Deep Belief Networks
41
42
Outline
• Deep learning– Greedy level-wise training (supervised learning)– Restricted Boltzmann machine (RBM)– Deep belief networks– Stacked autoassociators– Deep Boltzmann machines
• Applications– Human motion modeling– Vision– Language
43
44
45
46
47
48
Thank you!
