w1
A Review: ArchitectureA Review: Architecture
Basic architecture : Input layer, output layer
1
X1
Xi
Xn
Y
b
wi
wn
Single-layer net for pattern classification
Output UnitInput Unit
Biases & ThresholdsBiases & Thresholds
A bias acts exactly as a weight on a connection from a unit whose activation is always 1
Increasing the bias increases the net input to the unit If a bias is included, the activation function is typically
taken to be
f(net) =
1 if net 0
-1 if net < 0
Where net = b + xiwi
i
Biases & ThresholdsBiases & Thresholds
Some do not use bias weight, but use a fixed threshold for the activation function
f(net) =1 if net
-1 if net <
Where net = xiwii
Biases & ThresholdsBiases & Thresholds
w1
1
X1
X2
Y
b
w2
Single-layer net for pattern classification
Output UnitInput Unit
Linear SeparableLinear Separable For a particular output unit, the desired response is a
‘yes’(output signal 1) if the input pattern is the member of its class and ‘no’(output signal –1)
We want one of two responses Activation function – step function
From the net input formula we can draw a line, a plane or hyperplane
y_iny_in = = bb + + xi wi
bb + + xi wi = 0
Linear SeparableLinear Separable
The problem is said ‘linearly separable’ if there are weights (and a bias) so that all of the training input vectors for which the correct response is +1 lie on one side of the decision boundary and all of the training input vectors for which the correct response is –1 lie on the other side
Draw the decision boundary using this equation
bb + + xi wi = 0
Linear SeparabilityLinear Separability
x10
-
-+
-
Linear SeparabilityLinear Separability
x10
+
++
-
Linear SeparabilityLinear Separability
x10
+
+-
-
In the case of an elementary perceptron, the n-dimensional space is divided by a hyperplane into two decision regions. The hyperplane is defined by the linearly separable function:
01
n
iiiwx
Linear SeparabilityLinear Separability
Linear SeparabilityLinear Separability
x1
x2
Class A2
Class A1
1
2
x1w1 + x2w2 = 0
(a) Two-input perceptron. (b) Three-input perceptron.
x2
x1
x3x1w1 + x2w2 + x3w3 = 0
12
Linear SeparabilityLinear Separability
x1
x2
1
(a) AND (x1 x2)
1
x1
x2
1
1
(b) OR (x1 x2)
x1
x2
1
1
(c) Exclusive-OR(x1 x2)
00 0
A perceptron can learn the operations A perceptron can learn the operations ANDAND and and OROR, but , but not not Exclusive-ORExclusive-OR. .
The HEBB NETThe HEBB NET
The earliest and simplest learning ruleThe earliest and simplest learning rule
Proposed that learning occurs by modification of the synapse Proposed that learning occurs by modification of the synapse strengths (weights).strengths (weights).
If two interconnected neurons are both “on” or “off” at the same If two interconnected neurons are both “on” or “off” at the same time, then the weight between those neurons should be time, then the weight between those neurons should be increased.increased.
Hebb net also used for training other specific netsHebb net also used for training other specific nets
The HEBB NETThe HEBB NET
A single layer netsA single layer nets
Interconnected neurons will be between an input unit and Interconnected neurons will be between an input unit and one output unit.one output unit.
Suit for bipolar form of data (1, -1)Suit for bipolar form of data (1, -1)
Limitation for binary data (see examples 2.5 & 2.6) Limitation for binary data (see examples 2.5 & 2.6)
The HEBB NETThe HEBB NET A single layer netsA single layer nets
w1
1
X1
X2
Y
b
w2
Single-layer net for pattern classification
Output UnitInput Unit
The HEBB NETThe HEBB NET
bipolar form of data (1, -1)bipolar form of data (1, -1)
Weight updateWeight update
wi(new) = wi(old) + xi y
The AlgorithmThe AlgorithmStep 0: Initialize all weights:
wi = 0 (i= 1 to n)
Step 1: For each input training vector and target output pair, s:t, do steps 2-4
Step 2. Set activations for input units:xi = si (i = 1 to n).
Step 3. Set activation for output unit:y = t (t= target)
Step 4. Adjust the weights forwi(new) = wi(old) + wi (i = 1 to n).
wi =xi y Adjust the bias:b(new) = b(old) + y.
w1
1
X1
X2
Y
b
w2
Single-layer net for pattern classification
Output UnitInput Unit
The ApplicationThe Application
1. Hebb net for And function: binary inputs and binary targets
2. Hebb net for And function: binary inputs and bipolar targets
3. Hebb net for And function: bipolar inputs and bipolar targets
Input Target
(x1 x2 1)(1 1 1)(1 0 1)(0 1 1)(0 0 1)
1000
w1 = x1tw2 = x2 tb = t
Input Target Weight Changes Weights
(x1 x2 1) (1 1 1) (1 0 1) (0 1 1) (0 0 1)
1000
( w1 w2 b) (1 1 1) (0 0 0)(0 0 0)(0 0 0)
(w1 w2 b)(0 0 0)(1 1 1) (1 1 1) (1 1 1)(1 1 1)
Hebb net for Hebb net for AndAnd function: function: binarybinary inputs and inputs and binarybinary targetstargets
w1
1
X1
X2
Y
b
w2
Output UnitInput Unit
Hebb net for Hebb net for AndAnd function: function: binarybinary inputs and inputs and binarybinary targetstargets
Input Target Weight Changes Weights
(x1 x2 1)
(1 1 1) (1 0 1) (0 1 1) (0 0 1)
1000
( w1 w2 b)
(1 1 1) (0 0 0)(0 0 0)(0 0 0)
(w1 w2 b)
(0 0 0)(1 1 1) (1 1 1) (1 1 1)(1 1 1)
The response of the net correct for the first input pattern but not forthe 2, 3, 4th pattern because the target values is 0, no learning occurs.
Using binary target values prevents the net from learning any pattern for which the target is “off”
How a net learns
1. By adjusting weights
2. If no weight adjusted :: no learning occurs
w1
1
X1
X2
Y
b
w2
Output UnitInput Unit
Hebb net for Hebb net for AndAnd function: function: binarybinary inputs and inputs and binarybinary targetstargets
Input Target Weight Changes Weights
(x1 x2 1) (1 1 1) (1 0 1) (0 1 1) (0 0 1)
1000
( w1 w2 b) (1 1 1) (0 0 0)(0 0 0)(0 0 0)
(w1 w2 b)(0 0 0)(1 1 1) (1 1 1) (1 1 1)(1 1 1)
No Learning occur
Hebb net for Hebb net for AndAnd function: function: binarybinary inputs and inputs and binarybinary targetstargets
x2
x100
0
-1
-1
+
bb + + xi wi = 1 + x1(1)+ x2(1)=0
Separating lines x2= -x1-1Basic formula to draw separating linebb + + xi wi = 0
w1
1
X1
X2
Y
b
w2
Input Target
(x1 x2 1)
(1 1 1)(1 0 1)(0 1 1)(0 0 1)
1-1-1-1
w1 = x1t
w2 = x2 t
b = t
Input Target Weight Changes Weights
(x1 x2 1)
(1 1 1) (1 0 1) (0 1 1) (0 0 1)
1-1-1-1
( w1 w2 b)
(1 1 1) (-1 0 -1)(0 -1 -1)(0 0 -1)
(w1 w2 b)
(0 0 0)(0 1 0) (0 1 0) (0 0 -1)(0 0 -2)
Hebb net for Hebb net for AndAnd function: function: binarybinary inputs and inputs and bipolarbipolar targetstargets
w1
1
X1
X2
Y
b
w2
Hebb net for Hebb net for AndAnd function: function: binarybinary inputs and inputs and bipolarbipolar targetstargets
The response of the net correct for the first input pattern and forthe 2, 3, 4th patterns shows that learning continues for each of these since the target value is now -1.
However, these weights do not provide the correct response for the first input pattern
(x1 x2 1)
(1 1 1) (1 0 1) (0 1 1) (0 0 1)
1-1-1-1
( w1 w2 b)
(1 1 1) (-1 0 -1)(0 -1 -1)(0 0 -1)
(w1 w2 b)
(0 0 0)(0 1 0) (0 1 0) (0 0 -1)(0 0 -2)
Hebb net for Hebb net for AndAnd function: function: binarybinary inputs and inputs and bipolarbipolar targetstargets
x1
x2
--
-
-1
-1
+
Basic formula to draw separating line
bb + + xi wi = 0
bb + + xi wi = 1 + x1(1)+ x2(1)=0
Separating lines x2= -x1-1
w1
1
X1
X2
Y
b
w2
Input Target
(x1 x2 1)
(1 1 1)(1 -1 1)(-1 1 1)(-1 -1 1)
1-1-1-1
w1 = x1t
w2 = x2 t
b = t
Input Target Weight Changes Weights
(x1 x2 1)
(1 1 1) (1 -1 1) (-1 1 1) (-1 -1 1)
1-1-1-1
( w1 w2 b)
(1 1 1) (-1 1 -1)(1 -1 -1)(1 1 -1)
(w1 w2 b)
(0 0 0)(1 1 1) (0 2 0) (1 1 -1)(2 2 -2)
Hebb net for Hebb net for AndAnd function: function: bipolarbipolar inputs and inputs and bipolarbipolar targetstargets
w1
1
X1
X2
Y
b
w2
Hebb net for Hebb net for AndAnd function: function: bipolarbipolar inputs and inputs and bipolarbipolar targetstargets
The response of the net correct for the first input pattern andthe 2, 3, 4th patterns.
(x1 x2 1)
(1 1 1) (1 -1 1) (-1 1 1) (-1 -1 1)
1-1-1-1
( w1 w2 b)
(1 1 1) (-1 1 -1)(1 -1 -1)(1 1 -1)
(w1 w2 b)
(0 0 0)(1 1 1) (0 2 0) (1 1 -1)(2 2 -2)
w1
1
X1
X2
Y
b
w2
Hebb net for Hebb net for AndAnd function: function: bipolarbipolar inputs and inputs and bipolarbipolar targetstargets
x1
x2
--
-
-1
-1
+-
Basic formula to draw separating line
bb + + xi wi = 0bb + + xi wi = -2 + x1(2)+ x2(2)=0
Separating lines x2= -x1+1
1
1
Application: Character RecognitionApplication: Character Recognition
# . . . #. # . # .. . # . .. # . # .# . . . #
. # # # .# . . . ## . . . ## . . . #. # # # .
Pattern 1 Pattern 2
1 –1 –1 –1 1, -1 1 –1 1 –1,-1 –1 1 –1 –1, -1 1 –1 1 –1,1 –1 –1 –1 1
-1 1 1 1 –1, 1 –1 –1 –1 11 –1 –1 –1 1, 1 –1 –1 –1 1,1 –1 –1 –1 1, -1 1 1 1 -1