W1w1 A Review: Architecture Basic architecture : Input layer, output layer 1 X1X1 XiXi XnXn Y b wiwi wnwn Single-layer net for pattern classification Output

w1

A Review: ArchitectureA Review: Architecture

Basic architecture : Input layer, output layer

1

X1

Xi

Xn

Y

b

wi

wn

Single-layer net for pattern classification

Output UnitInput Unit

Biases & ThresholdsBiases & Thresholds

A bias acts exactly as a weight on a connection from a unit whose activation is always 1

Increasing the bias increases the net input to the unit If a bias is included, the activation function is typically

taken to be

f(net) =

1 if net 0

-1 if net < 0

Where net = b + xiwi

i


Some do not use bias weight, but use a fixed threshold for the activation function

f(net) =1 if net

-1 if net <

Where net = xiwii


w1

1

X1

X2

Y

b

w2



Linear SeparableLinear Separable For a particular output unit, the desired response is a

‘yes’(output signal 1) if the input pattern is the member of its class and ‘no’(output signal –1)

We want one of two responses Activation function – step function

From the net input formula we can draw a line, a plane or hyperplane

y_iny_in = = bb + + xi wi

bb + + xi wi = 0

Linear SeparableLinear Separable

The problem is said ‘linearly separable’ if there are weights (and a bias) so that all of the training input vectors for which the correct response is +1 lie on one side of the decision boundary and all of the training input vectors for which the correct response is –1 lie on the other side

Draw the decision boundary using this equation

bb + + xi wi = 0

Linear SeparabilityLinear Separability

x10

-

-+

-


x10

+

++

-


x10

+

+-

-

In the case of an elementary perceptron, the n-dimensional space is divided by a hyperplane into two decision regions. The hyperplane is defined by the linearly separable function:

01

n

iiiwx



x1

x2

Class A2

Class A1

1

2

x1w1 + x2w2 = 0

(a) Two-input perceptron. (b) Three-input perceptron.

x2

x1

x3x1w1 + x2w2 + x3w3 = 0

12


x1

x2

1

(a) AND (x1 x2)

1

x1

x2

1

1

(b) OR (x1 x2)

x1

x2

1

1

(c) Exclusive-OR(x1 x2)

00 0

A perceptron can learn the operations A perceptron can learn the operations ANDAND and and OROR, but , but not not Exclusive-ORExclusive-OR. .

The HEBB NETThe HEBB NET

The earliest and simplest learning ruleThe earliest and simplest learning rule

Proposed that learning occurs by modification of the synapse Proposed that learning occurs by modification of the synapse strengths (weights).strengths (weights).

If two interconnected neurons are both “on” or “off” at the same If two interconnected neurons are both “on” or “off” at the same time, then the weight between those neurons should be time, then the weight between those neurons should be increased.increased.

Hebb net also used for training other specific netsHebb net also used for training other specific nets


A single layer netsA single layer nets

Interconnected neurons will be between an input unit and Interconnected neurons will be between an input unit and one output unit.one output unit.

Suit for bipolar form of data (1, -1)Suit for bipolar form of data (1, -1)

Limitation for binary data (see examples 2.5 & 2.6) Limitation for binary data (see examples 2.5 & 2.6)

The HEBB NETThe HEBB NET A single layer netsA single layer nets

w1

1

X1

X2

Y

b

w2




bipolar form of data (1, -1)bipolar form of data (1, -1)

Weight updateWeight update

wi(new) = wi(old) + xi y

The AlgorithmThe AlgorithmStep 0: Initialize all weights:

wi = 0 (i= 1 to n)

Step 1: For each input training vector and target output pair, s:t, do steps 2-4

Step 2. Set activations for input units:xi = si (i = 1 to n).

Step 3. Set activation for output unit:y = t (t= target)

Step 4. Adjust the weights forwi(new) = wi(old) + wi (i = 1 to n).

wi =xi y Adjust the bias:b(new) = b(old) + y.

w1

1

X1

X2

Y

b

w2



The ApplicationThe Application

1. Hebb net for And function: binary inputs and binary targets

2. Hebb net for And function: binary inputs and bipolar targets

3. Hebb net for And function: bipolar inputs and bipolar targets

Input Target

(x1 x2 1)(1 1 1)(1 0 1)(0 1 1)(0 0 1)

1000

w1 = x1tw2 = x2 tb = t

Input Target Weight Changes Weights

(x1 x2 1) (1 1 1) (1 0 1) (0 1 1) (0 0 1)

1000

( w1 w2 b) (1 1 1) (0 0 0)(0 0 0)(0 0 0)

(w1 w2 b)(0 0 0)(1 1 1) (1 1 1) (1 1 1)(1 1 1)

Hebb net for Hebb net for AndAnd function: function: binarybinary inputs and inputs and binarybinary targetstargets

w1

1

X1

X2

Y

b

w2




(x1 x2 1)

(1 1 1) (1 0 1) (0 1 1) (0 0 1)

1000

( w1 w2 b)

(1 1 1) (0 0 0)(0 0 0)(0 0 0)

(w1 w2 b)

(0 0 0)(1 1 1) (1 1 1) (1 1 1)(1 1 1)

The response of the net correct for the first input pattern but not forthe 2, 3, 4th pattern because the target values is 0, no learning occurs.

Using binary target values prevents the net from learning any pattern for which the target is “off”

How a net learns

1. By adjusting weights

2. If no weight adjusted :: no learning occurs

w1

1

X1

X2

Y

b

w2




(x1 x2 1) (1 1 1) (1 0 1) (0 1 1) (0 0 1)

1000

( w1 w2 b) (1 1 1) (0 0 0)(0 0 0)(0 0 0)

(w1 w2 b)(0 0 0)(1 1 1) (1 1 1) (1 1 1)(1 1 1)

No Learning occur


x2

x100

0

-1

-1

+

bb + + xi wi = 1 + x1(1)+ x2(1)=0

Separating lines x2= -x1-1Basic formula to draw separating linebb + + xi wi = 0

w1

1

X1

X2

Y

b

w2

Input Target

(x1 x2 1)

(1 1 1)(1 0 1)(0 1 1)(0 0 1)

1-1-1-1

w1 = x1t

w2 = x2 t

b = t


(x1 x2 1)

(1 1 1) (1 0 1) (0 1 1) (0 0 1)

1-1-1-1

( w1 w2 b)

(1 1 1) (-1 0 -1)(0 -1 -1)(0 0 -1)

(w1 w2 b)

(0 0 0)(0 1 0) (0 1 0) (0 0 -1)(0 0 -2)

Hebb net for Hebb net for AndAnd function: function: binarybinary inputs and inputs and bipolarbipolar targetstargets

w1

1

X1

X2

Y

b

w2


The response of the net correct for the first input pattern and forthe 2, 3, 4th patterns shows that learning continues for each of these since the target value is now -1.

However, these weights do not provide the correct response for the first input pattern

(x1 x2 1)

(1 1 1) (1 0 1) (0 1 1) (0 0 1)

1-1-1-1

( w1 w2 b)

(1 1 1) (-1 0 -1)(0 -1 -1)(0 0 -1)

(w1 w2 b)

(0 0 0)(0 1 0) (0 1 0) (0 0 -1)(0 0 -2)


x1

x2

--

-

-1

-1

+

Basic formula to draw separating line

bb + + xi wi = 0

bb + + xi wi = 1 + x1(1)+ x2(1)=0

Separating lines x2= -x1-1

w1

1

X1

X2

Y

b

w2

Input Target

(x1 x2 1)

(1 1 1)(1 -1 1)(-1 1 1)(-1 -1 1)

1-1-1-1

w1 = x1t

w2 = x2 t

b = t


(x1 x2 1)

(1 1 1) (1 -1 1) (-1 1 1) (-1 -1 1)

1-1-1-1

( w1 w2 b)

(1 1 1) (-1 1 -1)(1 -1 -1)(1 1 -1)

(w1 w2 b)

(0 0 0)(1 1 1) (0 2 0) (1 1 -1)(2 2 -2)

Hebb net for Hebb net for AndAnd function: function: bipolarbipolar inputs and inputs and bipolarbipolar targetstargets

w1

1

X1

X2

Y

b

w2


The response of the net correct for the first input pattern andthe 2, 3, 4th patterns.

(x1 x2 1)

(1 1 1) (1 -1 1) (-1 1 1) (-1 -1 1)

1-1-1-1

( w1 w2 b)

(1 1 1) (-1 1 -1)(1 -1 -1)(1 1 -1)

(w1 w2 b)

(0 0 0)(1 1 1) (0 2 0) (1 1 -1)(2 2 -2)

w1

1

X1

X2

Y

b

w2


x1

x2

--

-

-1

-1

+-

Basic formula to draw separating line

bb + + xi wi = 0bb + + xi wi = -2 + x1(2)+ x2(2)=0

Separating lines x2= -x1+1

1

1

Application: Character RecognitionApplication: Character Recognition

# . . . #. # . # .. . # . .. # . # .# . . . #

. # # # .# . . . ## . . . ## . . . #. # # # .

Pattern 1 Pattern 2

1 –1 –1 –1 1, -1 1 –1 1 –1,-1 –1 1 –1 –1, -1 1 –1 1 –1,1 –1 –1 –1 1

-1 1 1 1 –1, 1 –1 –1 –1 11 –1 –1 –1 1, 1 –1 –1 –1 1,1 –1 –1 –1 1, -1 1 1 1 -1

Documents

W1w1 A Review: Architecture Basic architecture : Input layer, output layer 1 X1X1 XiXi XnXn Y b wiwi wnwn Single-layer net for pattern classification Output