• Home

  • Documents

  • Learning neural networks with hidden layers · 2018-05-31 · 4 STAT/CSE 416: Intro to Machine...
10
STAT/CSE 416: Intro to Machine Learning Learning neural networks with hidden layers ©2017 Emily Fox OPTIONAL

Learning neural networks with hidden layers · 2018-05-31 · 4 STAT/CSE 416: Intro to Machine Learning Multilayer neural networks ©2017 Emily Fox Inference and Learning •Forward

  • Upload
    others

  • View
    0

  • Download
    0

Embed Size (px)

Citation preview

Page 1: Learning neural networks with hidden layers · 2018-05-31 · 4 STAT/CSE 416: Intro to Machine Learning Multilayer neural networks ©2017 Emily Fox Inference and Learning •Forward

STAT/CSE 416: Intro to Machine Learning

Learning neural networks with hidden layers

©2017 Emily Fox

OPTIONAL

Page 2: Learning neural networks with hidden layers · 2018-05-31 · 4 STAT/CSE 416: Intro to Machine Learning Multilayer neural networks ©2017 Emily Fox Inference and Learning •Forward

STAT/CSE 416: Intro to Machine Learning2

Optimizing a single-layer neuron

We train to minimize sum of squared errors:

Taking gradients:

`(w) =1

2

X

i

[yi � g(w0 +X

j

wjxi[j])]2

@`

@wj= �

X

i

[yi � g(w0 +X

j

wjxi[j])]@

@wjg(w0 +

X

j

wjxi[j])@`

@wj= �

X

i

[yi � g(w0 +X

j

wjxi[j])]xi[j]g0(w0 +

X

j

wjxi[j])

Solution just depends on g’: derivative of activation function!

©2017 Emily Fox

x[1]

x[2]

x[d]

Σ

…

1

Page 3: Learning neural networks with hidden layers · 2018-05-31 · 4 STAT/CSE 416: Intro to Machine Learning Multilayer neural networks ©2017 Emily Fox Inference and Learning •Forward

STAT/CSE 416: Intro to Machine Learning3

Forward propagation1-hidden layer:

Compute values left to right 1. Inputs: x[1],…,x[d]2. Hidden: v[1],…,v[d]3. Output: y

©2017 Emily Fox

out(x) = g(w0 +X

k

wkg(wk0 +

X

j

wkj x[j]))

For fixed weights, forming predictions is easy!

v[1]

v[2]

x[1]

x[2]

1

y

1

Page 4: Learning neural networks with hidden layers · 2018-05-31 · 4 STAT/CSE 416: Intro to Machine Learning Multilayer neural networks ©2017 Emily Fox Inference and Learning •Forward

STAT/CSE 416: Intro to Machine Learning4

Multilayer neural networks

©2017 Emily Fox

Inference and Learning• Forward pass:

left to right, each hidden layer in turn

• Gradient computation:right to left, propagating gradient for each node

Forward

Gradient

Page 5: Learning neural networks with hidden layers · 2018-05-31 · 4 STAT/CSE 416: Intro to Machine Learning Multilayer neural networks ©2017 Emily Fox Inference and Learning •Forward

STAT/CSE 416: Intro to Machine Learning5

Back-propagation – Learning

• Just gradient descent!!! • Recursive algorithm for computing gradient• For each example- Perform forward propagation - Start from output layer

• Compute gradient of node v[k] with parents u[1],u[2],…:• Update weight • Repeat (move to preceding layer)

©2017 Emily Fox

wkj

Page 6: Learning neural networks with hidden layers · 2018-05-31 · 4 STAT/CSE 416: Intro to Machine Learning Multilayer neural networks ©2017 Emily Fox Inference and Learning •Forward

STAT/CSE 416: Intro to Machine Learning6

Convergence of backprop

Multilayer neural nets not convex- Gradient descent gets stuck in local minima

©2017 Emily Fox

Page 7: Learning neural networks with hidden layers · 2018-05-31 · 4 STAT/CSE 416: Intro to Machine Learning Multilayer neural networks ©2017 Emily Fox Inference and Learning •Forward

STAT/CSE 416: Intro to Machine Learning

Deep features:

Deep learning+

Transfer learning

©2018 Emily Fox

Page 8: Learning neural networks with hidden layers · 2018-05-31 · 4 STAT/CSE 416: Intro to Machine Learning Multilayer neural networks ©2017 Emily Fox Inference and Learning •Forward

STAT/CSE 416: Intro to Machine Learning8

Transfer learning: Use data from one task to help learn on another

Lots of data:Learn

neural netGreat accuracy

on cat v. dog

Some data: Neural net as feature extractor

+

Simple classifier

Great accuracy on 101

categories

Old idea, explored for deep learning by Donahue et al. ’14 & others

©2018 Emily Fox

vs.

Page 9: Learning neural networks with hidden layers · 2018-05-31 · 4 STAT/CSE 416: Intro to Machine Learning Multilayer neural networks ©2017 Emily Fox Inference and Learning •Forward

STAT/CSE 416: Intro to Machine Learning9

What’s learned in a neural net

©2018 Emily Fox

Very specific to Task 1

Should be ignored for other tasks

More genericCan be used as feature extractor

vs.

Neural net trained for Task 1: cat vs. dog

Page 10: Learning neural networks with hidden layers · 2018-05-31 · 4 STAT/CSE 416: Intro to Machine Learning Multilayer neural networks ©2017 Emily Fox Inference and Learning •Forward

STAT/CSE 416: Intro to Machine Learning10

Transfer learning in more detail…

©2018 Emily Fox

Very specific to Task 1

Should be ignored for other tasks

More genericCan be used as feature extractor

For Task 2, predicting 101 categories, learn only end part of neural net

Use simple classifiere.g., logistic regression, SVMs, nearest neighbor,…

Class?Keep weights fixed!

Neural net trained for Task 1: cat vs. dog


Learning Neural Networks (NN)

Learning Neural Networks (NN)

Documents
Machine learning 2 - Neural Networks

Machine learning 2 - Neural Networks

Technology
Deep Learning in Neural Networks: An Overviewpeople.idsia.ch/~juergen/DeepLearning2July2014.pdf · Deep Learning in Neural Networks: An Overview ... numerous neural network experts

Deep Learning in Neural Networks: An Overviewpeople.idsia.ch/~juergen/DeepLearning2July2014.pdf · Deep Learning in Neural Networks: An Overview ... numerous neural network experts

Documents
Machine Learning and Neural Networks

Machine Learning and Neural Networks

Documents
Neural Networks Self Organizing Mapscancelli/retineu11_12/SOM.pdf · Neural Networks Self Organizing Maps NN 4 1 Unsupervised Learning Neural networks for unsupervised learning attempt

Neural Networks Self Organizing Mapscancelli/retineu11_12/SOM.pdf · Neural Networks Self Organizing Maps NN 4 1 Unsupervised Learning Neural networks for unsupervised learning attempt

Documents
Métodos Actuales de Machine Learning Neural Networksdc.exa.unrc.edu.ar/rio2016/sites/default/files/Neural Networks... · Métodos Actuales de Machine Learning Neural Networks

Métodos Actuales de Machine Learning Neural Networksdc.exa.unrc.edu.ar/rio2016/sites/default/files/Neural Networks... · Métodos Actuales de Machine Learning Neural Networks

Documents
Neural Networks and Machine Learning

Neural Networks and Machine Learning

Documents
DEEP LEARNING AND NEURAL NETWORKS Deep Learning History of Artificial Neural Networks Keras ... Recurrent Neural Networks (RNNs) Long Term Short Memory (I-ST M) ... Fractal Analytics

DEEP LEARNING AND NEURAL NETWORKS Deep Learning History of Artificial Neural Networks Keras ... Recurrent Neural Networks (RNNs) Long Term Short Memory (I-ST M) ... Fractal Analytics

Documents
Artificial Neural Networks Lect3: Neural Network Learning rules

Artificial Neural Networks Lect3: Neural Network Learning rules

Engineering
Deep Learning in Neural Networks: An Overviejuergen/DeepLearning28May2014.pdf · Deep Learning in Neural Networks: An Overview ... 1 Introduction to Deep Learning ... Neural Fitted

Deep Learning in Neural Networks: An Overviejuergen/DeepLearning28May2014.pdf · Deep Learning in Neural Networks: An Overview ... 1 Introduction to Deep Learning ... Neural Fitted

Documents
Introduction to Deep Neural Networks - Deep Learning · Introduction to Deep Neural Networks 0. Logistics Spring 2020 1. Neural Networks are taking over! •Neural networks have become

Introduction to Deep Neural Networks - Deep Learning · Introduction to Deep Neural Networks 0. Logistics Spring 2020 1. Neural Networks are taking over! •Neural networks have become

Documents
Neural Networks for Machine Learning Lecture 3a Learning

Neural Networks for Machine Learning Lecture 3a Learning

Documents
Convolutional Neural Networks for Deep Learning: An Introduction · 2017-10-05 · Introduction: Convolutional Neural Networks (CNN) Convolutional Neural Networks: A deep learning

Convolutional Neural Networks for Deep Learning: An Introduction · 2017-10-05 · Introduction: Convolutional Neural Networks (CNN) Convolutional Neural Networks: A deep learning

Documents
Lecture 10: Neural Networks and Deep Learningsaravanan-thirumuruganathan.github.io/cse5334... · Neural Networks Deep Learning Convolutional Neural Networks Recurrent Neural Networks

Lecture 10: Neural Networks and Deep Learningsaravanan-thirumuruganathan.github.io/cse5334... · Neural Networks Deep Learning Convolutional Neural Networks Recurrent Neural Networks

Documents
Supervised Learning in Neural Networks

Supervised Learning in Neural Networks

Documents
Arti cal Neural Networks - University of Chicagottic.uchicago.edu/~shubhendu/Slides/NeuralNets1.pdf · Arti cal Neural Networks ... • Deep Learning: Convolutional Neural Networks

Arti cal Neural Networks - University of Chicagottic.uchicago.edu/~shubhendu/Slides/NeuralNets1.pdf · Arti cal Neural Networks ... • Deep Learning: Convolutional Neural Networks

Documents
Neural networks Computer vision Neural Networks and ... · Neural networks Computer vision Neural Networks and Learning Machines (3rd Edition) Author: Simon's H. Haykin Published:

Neural networks Computer vision Neural Networks and ... · Neural networks Computer vision Neural Networks and Learning Machines (3rd Edition) Author: Simon's H. Haykin Published:

Documents
Deep Learning for Neural Networks - viXra · Deep Learning for Neural Networks Today, deep neural networks with different architectures, such as convolutional, recurrent and autoencoder

Deep Learning for Neural Networks - viXra · Deep Learning for Neural Networks Today, deep neural networks with different architectures, such as convolutional, recurrent and autoencoder

Documents
CS536: Machine Learning Artificial Neural Networks Neural Networks

CS536: Machine Learning Artificial Neural Networks Neural Networks

Documents
Neural networks and deep learning

Neural networks and deep learning

Technology
Learning Algorithm and Neural Networks

Learning Algorithm and Neural Networks

Documents
Neural Networks and Learning Machines

Neural Networks and Learning Machines

Documents
Perceptrons and Learning Learning in Neural Networks

Perceptrons and Learning Learning in Neural Networks

Documents
Modal Learning Neural Networks - WSEAS

Modal Learning Neural Networks - WSEAS

Documents
Neural Networks - Lecture 81 Unsupervised competitive learning Particularities of unsupervised learning Data clustering Neural networks for clustering

Neural Networks - Lecture 81 Unsupervised competitive learning Particularities of unsupervised learning Data clustering Neural networks for clustering

Documents
Machine Learning and Neural Networks - UPM€¦ ·  · 2015-05-271 CVG-UPM COMPUTER VISION P. Campoy Neural Networks and Pattern Recognition? by? Machine Learning and Neural Networks

Machine Learning and Neural Networks - UPM€¦ ·  · 2015-05-271 CVG-UPM COMPUTER VISION P. Campoy Neural Networks and Pattern Recognition? by? Machine Learning and Neural Networks

Documents
The Next Generation Neural Networks: Deep Learning and Spiking Neural Networks ·  · 2015-08-10Networks: Deep Learning and Spiking Neural Networks ADVANCED SEMINAR submitted by

The Next Generation Neural Networks: Deep Learning and Spiking Neural Networks ·  · 2015-08-10Networks: Deep Learning and Spiking Neural Networks ADVANCED SEMINAR submitted by

Documents
Reinforcement Learning and Neural networks

Reinforcement Learning and Neural networks

Documents
Neural Networks: Learning Cost function Machine Learning

Neural Networks: Learning Cost function Machine Learning

Documents
Artificial Neural Networks Connectionist Learning Machinesci.louisville.edu/kerem/evraklar/sunum/CECS694_01_seminar.pdfArtificial Neural Networks Connectionist Learning Machines

Artificial Neural Networks Connectionist Learning Machinesci.louisville.edu/kerem/evraklar/sunum/CECS694_01_seminar.pdfArtificial Neural Networks Connectionist Learning Machines

Documents