Upload
norman-oconnor
View
25
Download
4
Embed Size (px)
DESCRIPTION
20.5 Nerual Networks. Thanks: Professors Frank Hoffmann and Jiawei Han, and Russell and Norvig. Biological Neural Systems. Neuron switching time : > 10 -3 secs Number of neurons in the human brain: ~10 10 Connections (synapses) per neuron : ~10 4 –10 5 Face recognition : 0.1 secs - PowerPoint PPT Presentation
Citation preview
20.5 Nerual Networks
Thanks: Professors Frank Hoffmann and Jiawei Han, and
Russell and Norvig
Biological Neural Systems
Neuron switching time : > 10-3 secs Number of neurons in the human brain: ~1010
Connections (synapses) per neuron : ~104–105
Face recognition : 0.1 secs High degree of distributed and parallel
computation Highly fault tolerent Highly efficient Learning is key
Excerpt from Russell and Norvig
A Neuron
Computation: input signals input function(linear)
activation function(nonlinear) output signal
ajoutput links
ak
outputInput links
Wkj
ai = output(inj)
inj
j
kkjj IWin *
Part 1. Perceptrons: Simple NN
x1
x2
xn
.
..
w1
w2
wn
a=i=1n wi xi
Xi’s range: [0, 1]
1 if a y= 0 if a <
y
{
inputsweights
activation output
Decision Surface of a Perceptron
x1
x2
Decision linew1 x1 + w2 x2 = w
1
1 1
0
0
00
0
1
Linear Separability
x1
x2
10
0 0
Logical AND
x1 x2 a y
0 0 0 0
0 1 1 0
1 0 1 0
1 1 2 1
w1=1w2=1=1.5 x1
10
0
w1=?w2=?= ?
1
Logical XOR
x1 x2 y
0 0 0
0 1 1
1 0 1
1 1 0
Threshold as Weight: W0
x1
x2
xn
.
..
w1
w2
wn
w0
x0=-1
a= i=0n wi xi
y
1 if a y= 0 if a <{
=w0
Training the Perceptron p742
Training set S of examples {x,t} x is an input vector and t the desired target vector Example: Logical And S = {(0,0),0}, {(0,1),0}, {(1,0),0}, {(1,1),1}
Iterative process Present a training example x , compute network output y ,
compare output y with target t, adjust weights and thresholds
Learning rule Specifies how to change the weights w and thresholds
of the network as a function of the inputs x, output y and target t.
Perceptron Learning Rule
w’=w + (t-y) x
wi := wi + wi = wi + (t-y) xi (i=1..n) The parameter is called the learning rate.
In Han’s book it is lower case L It determines the magnitude of weight updates wi .
If the output is correct (t=y) the weights are not changed (wi =0).
If the output is incorrect (t y) the weights wi are changed such that the output of the Perceptron for the new weights w’i is closer/further to the input xi.
Perceptron Training Algorithm
Repeatfor each training vector pair (x,t)
evaluate the output y when x is the inputif yt then
form a new weight vector w’ accordingto w’=w + (t-y) x
else do nothing
end if end forUntil y=t for all training vector pairs or # iterations > k
Perceptron Convergence Theorem
The algorithm converges to the correct classification
if the training data is linearly separable and learning rate is sufficiently small
If two classes of vectors X1 and X2 are linearly separable, the application of the perceptron training algorithm will eventually result in a weight vector w0, such that w0 defines a Perceptron whose decision hyper-plane separates X1 and X2 (Rosenblatt 1962).
Solution w0 is not unique, since if w0 x =0 defines a hyper-plane, so does w’0 = k w0.
Experiments
Perceptron Learning from Patterns
x1
x2
xn
.
..
w1
w2
wn
Input pattern
Associationunits
weights (trained)
Summation Thresholdfixed
Association units (A-units) can be assigned arbitrary Booleanfunctions of the input pattern.
Part 2. Multi Layer Networks
Output nodes
Input nodes
Hidden nodes
Output vector
Input vector
Gradient Descent Learning Rule
Consider linear unit without threshold and continuous output o (not just –1,1) Output=oj=-w0 + w1 x1 + … + wn xn
Train the wi’s such that they minimize the squared error Error[w1,…,wn] = ½ jD (Tj-oj)2
where D is the set of training examples
Neuron with Sigmoid-Function
x1
x2
xn
.
..
w1
w2
wn
a=i=1n wi xi
Output=o=(a) =1/(1+e-a)
o
inputsweights
activation output
Sigmoid Unit
x1
x2
xn
.
..
w1
w2
wn
w0
x0=-1
a=i=0n wi xi
o
o=(a)=1/(1+e-a)
(x) is the sigmoid function: 1/(1+e-x)
d(x)/dx= (x) (1- (x))
Derive gradient decent rules to train:• one sigmoid function
E/wi = -j(Tj-O) o (1-o) xij
• derivation: see next page
Explantion: Gradient Descent Learning Rule
wi = Ojp(1-Oj
p) (Tjp-Oj
p) xip
xi
wji
yj
activation ofpre-synaptic neuron
error j ofpost-synaptic neuron
derivative of activation function
learning rate
Gradient Descent: Graphical
D={<(1,1),1>,<(-1,-1),1>, <(1,-1),-1>,<(-1,1),-1>}
(w1,w2)
(w1+w1,w2 +w2)
Perceptron vs. Gradient Descent Rule
Perceptron rule w’i = wi + (t-o) xi
derived from manipulation of decision surface.
Gradient descent rule w’i = wi + (1-y) (t-y) xi
derived from minimization of error function
E[w1,…,wn] = ½ p (t-y)2
by means of gradient descent.