Upload
others
View
44
Download
0
Embed Size (px)
Citation preview
Neural Networks - Deep Learning
Artificial Intelligence @ Allegheny College
Janyl Jumadinova
March 4-6, 2020
Credit: Google Workshop
Janyl Jumadinova Neural Networks - Deep Learning March 4-6, 2020 1 / 35
Neural Networks
Janyl Jumadinova Neural Networks - Deep Learning March 4-6, 2020 2 / 35
Neural Networks
Janyl Jumadinova Neural Networks - Deep Learning March 4-6, 2020 3 / 35
Neural Networks
Neural computing requires a number of neurons, to be connectedtogether into a neural network.
Neurons are arranged in layers.
Two main hyperparameters that control the architecture or topology ofthe network: 1) the number of layers, and 2) the number of nodes in eachhidden layer.
Janyl Jumadinova Neural Networks - Deep Learning March 4-6, 2020 4 / 35
Neural Networks
Neural computing requires a number of neurons, to be connectedtogether into a neural network.
Neurons are arranged in layers.
Two main hyperparameters that control the architecture or topology ofthe network: 1) the number of layers, and 2) the number of nodes in eachhidden layer.
Janyl Jumadinova Neural Networks - Deep Learning March 4-6, 2020 4 / 35
Activation Functions
The activation function is generally non-linear.
Linear functions are limited because the output is simply proportionalto the input.
Janyl Jumadinova Neural Networks - Deep Learning March 4-6, 2020 5 / 35
Activation Functions
Janyl Jumadinova Neural Networks - Deep Learning March 4-6, 2020 6 / 35
Network structures
Two phases in each iteration:
1 Calculating the predicted output y, known as feed-forward
2 Updating the weights and biases, known as backpropagation
Janyl Jumadinova Neural Networks - Deep Learning March 4-6, 2020 7 / 35
Feed-forward example
Feed-forward networks:
Single-layer perceptronsMulti-layer perceptrons
Janyl Jumadinova Neural Networks - Deep Learning March 4-6, 2020 8 / 35
Feed-forward example
Feed-forward networks:
Single-layer perceptronsMulti-layer perceptrons
Janyl Jumadinova Neural Networks - Deep Learning March 4-6, 2020 8 / 35
Single-layer Perceptrons
Output units all operate separately – no shared weights.
Adjusting weights moves the location, orientation, and steepness of cliff.
Janyl Jumadinova Neural Networks - Deep Learning March 4-6, 2020 9 / 35
Multi-layer Perceptrons
Layers are usually fully connected.
Numbers of hidden units typically chosen by hand.
Janyl Jumadinova Neural Networks - Deep Learning March 4-6, 2020 10 / 35
Neural Networks
Janyl Jumadinova Neural Networks - Deep Learning March 4-6, 2020 11 / 35
Neural Networks
Janyl Jumadinova Neural Networks - Deep Learning March 4-6, 2020 12 / 35
Neural Networks
A fully connected NN layer
Janyl Jumadinova Neural Networks - Deep Learning March 4-6, 2020 13 / 35
Implementation as Matrix Multiplication
Janyl Jumadinova Neural Networks - Deep Learning March 4-6, 2020 14 / 35
Non-Linear Data Distributions
Janyl Jumadinova Neural Networks - Deep Learning March 4-6, 2020 15 / 35
Janyl Jumadinova Neural Networks - Deep Learning March 4-6, 2020 16 / 35
Deep Learning
Most current machine learning works well because of human-designedrepresentations and input features.
Machine learning becomes just optimizing weights to best make afinal prediction.
Deep learning algorithms attempt to learn multiple levels ofrepresentation of increasing complexity/abstraction.
Janyl Jumadinova Neural Networks - Deep Learning March 4-6, 2020 17 / 35
Deep Learning
Most current machine learning works well because of human-designedrepresentations and input features.
Machine learning becomes just optimizing weights to best make afinal prediction.
Deep learning algorithms attempt to learn multiple levels ofrepresentation of increasing complexity/abstraction.
Janyl Jumadinova Neural Networks - Deep Learning March 4-6, 2020 17 / 35
Deep Learning
Each neuron implements a relatively simple mathematical function.
y = g(w · x + b)
The composition of 106 − 109 such functions is powerful.
Janyl Jumadinova Neural Networks - Deep Learning March 4-6, 2020 18 / 35
Deep Learning
Each neuron implements a relatively simple mathematical function.
y = g(w · x + b)
The composition of 106 − 109 such functions is powerful.
Janyl Jumadinova Neural Networks - Deep Learning March 4-6, 2020 18 / 35
Deep Learning
Book: http://www.deeplearningbook.org/
Chapter 5
“A core idea in deep learning is that we assume that the data wasgenerated by the composition of factors or features, potentially at multiplelevels in a hierarchy.”
Janyl Jumadinova Neural Networks - Deep Learning March 4-6, 2020 19 / 35
Results get better (to a degree) with:
more data
bigger models
more computation
Better algorithms, new insights and improved methods help, too!
Janyl Jumadinova Neural Networks - Deep Learning March 4-6, 2020 20 / 35
Results get better (to a degree) with:
more data
bigger models
more computation
Better algorithms, new insights and improved methods help, too!
Janyl Jumadinova Neural Networks - Deep Learning March 4-6, 2020 20 / 35
TensorFlow
Janyl Jumadinova Neural Networks - Deep Learning March 4-6, 2020 21 / 35
Adoption of Deep Learning Tools on GitHub
Janyl Jumadinova Neural Networks - Deep Learning March 4-6, 2020 22 / 35
TensorFlow
Epoch: a training iteration (one pass through the dataset).
Batch: Portion of the dataset (number of samples after dataset hasbeen divided).
Regularization: a set of techniques that helps learning models toconverge(http://www.godeep.ml/regularization-using-tensorflow/).
Janyl Jumadinova Neural Networks - Deep Learning March 4-6, 2020 23 / 35
TensorFlow
Operates over tensors: n-dimensional arrays
Using a flow graph: data flow computation framework
Janyl Jumadinova Neural Networks - Deep Learning March 4-6, 2020 24 / 35
TensorFlow
Operates over tensors: n-dimensional arrays
Using a flow graph: data flow computation framework
Janyl Jumadinova Neural Networks - Deep Learning March 4-6, 2020 24 / 35
TensorFlow
Operates over tensors: n-dimensional arrays
Using a flow graph: data flow computation framework
Janyl Jumadinova Neural Networks - Deep Learning March 4-6, 2020 24 / 35
TensorFlow
5.7 ← Scalar
Number, Float, etc.
Janyl Jumadinova Neural Networks - Deep Learning March 4-6, 2020 25 / 35
TensorFlow
Janyl Jumadinova Neural Networks - Deep Learning March 4-6, 2020 26 / 35
TensorFlow
Janyl Jumadinova Neural Networks - Deep Learning March 4-6, 2020 27 / 35
TensorFlow
Tensors have a Shape that is described with a vector
[1000, 256, 256, 3]
10000 Images
Each Image has 256 Rows
Each Row has 256 Pixels
Each Pixel has 3 values (RGB)
Janyl Jumadinova Neural Networks - Deep Learning March 4-6, 2020 28 / 35
TensorFlow
Tensors have a Shape that is described with a vector
[1000, 256, 256, 3]
10000 Images
Each Image has 256 Rows
Each Row has 256 Pixels
Each Pixel has 3 values (RGB)
Janyl Jumadinova Neural Networks - Deep Learning March 4-6, 2020 28 / 35
TensorFlow
Computation is a dataflow graph
Janyl Jumadinova Neural Networks - Deep Learning March 4-6, 2020 29 / 35
TensorFlow
Computation is a dataflow graph with tensors
Janyl Jumadinova Neural Networks - Deep Learning March 4-6, 2020 30 / 35
TensorFlow
Computation is a dataflow graph with state
Janyl Jumadinova Neural Networks - Deep Learning March 4-6, 2020 31 / 35
Core TensorFlow data structures and concepts
Graph: A TensorFlow computation, represented as a dataflow graph:- collection of ops that may be executed together as a group.
Operation: a graph node that performs computation on tensors
Tensor: a handle to one of the outputs of an Operation:- provides a means of computing the value in a TensorFlow Session.
Janyl Jumadinova Neural Networks - Deep Learning March 4-6, 2020 32 / 35
Core TensorFlow data structures and concepts
Graph: A TensorFlow computation, represented as a dataflow graph:- collection of ops that may be executed together as a group.
Operation: a graph node that performs computation on tensors
Tensor: a handle to one of the outputs of an Operation:- provides a means of computing the value in a TensorFlow Session.
Janyl Jumadinova Neural Networks - Deep Learning March 4-6, 2020 32 / 35
Core TensorFlow data structures and concepts
Graph: A TensorFlow computation, represented as a dataflow graph:- collection of ops that may be executed together as a group.
Operation: a graph node that performs computation on tensors
Tensor: a handle to one of the outputs of an Operation:- provides a means of computing the value in a TensorFlow Session.
Janyl Jumadinova Neural Networks - Deep Learning March 4-6, 2020 32 / 35
TensorFlow
Constants
Placeholders: must be fed with data on execution.
Variables: a modifiable tensor that lives in TensorFlow’s graph ofinteracting operations.
Session: encapsulates the environment in which Operation objectsare executed, and Tensor objects are evaluated.
Janyl Jumadinova Neural Networks - Deep Learning March 4-6, 2020 33 / 35
TensorFlow
Constants
Placeholders: must be fed with data on execution.
Variables: a modifiable tensor that lives in TensorFlow’s graph ofinteracting operations.
Session: encapsulates the environment in which Operation objectsare executed, and Tensor objects are evaluated.
Janyl Jumadinova Neural Networks - Deep Learning March 4-6, 2020 33 / 35
TensorFlow
Constants
Placeholders: must be fed with data on execution.
Variables: a modifiable tensor that lives in TensorFlow’s graph ofinteracting operations.
Session: encapsulates the environment in which Operation objectsare executed, and Tensor objects are evaluated.
Janyl Jumadinova Neural Networks - Deep Learning March 4-6, 2020 33 / 35
TensorFlow
Constants
Placeholders: must be fed with data on execution.
Variables: a modifiable tensor that lives in TensorFlow’s graph ofinteracting operations.
Session: encapsulates the environment in which Operation objectsare executed, and Tensor objects are evaluated.
Janyl Jumadinova Neural Networks - Deep Learning March 4-6, 2020 33 / 35
TensorFlow
Janyl Jumadinova Neural Networks - Deep Learning March 4-6, 2020 34 / 35
TensorFlow
https://playground.tensorflow.org
Janyl Jumadinova Neural Networks - Deep Learning March 4-6, 2020 35 / 35