Associative Learning. Simple Associative Network

Associative Learning

Simple Associative Network

Banana Associator

Unsupervised Hebb Rule

Banana Recognition Example

Example

Problems with Hebb Rule

• Weights can become arbitrarily large

• There is no mechanism for weights to decrease

Hebb Rule with Decay

Example: Banana Associator

Example

Problem of Hebb with Decay

Instar (Recognition Network)

Instar Operation

Vector Recognition

Instar Rule

Graphical Representation

Example

Training

Further Training

Kohonen Rule

Outstar (Recall Network)

Outstar Operation

Outstar Rule

Example - Pineapple Recall

Definitions

Iteration 1

Convergence

Boltzmann Learning• Stochastic learning process with a recurrent structure• State of a neuron is +1 or –1 and some neurons are free (adaptive state)

and others are clamped (frozen state)• Boltzmann machine is characterized by an energy function

• Free neurons change state with probability:

• The learning rule is given by:

Where kjis the correlation with neurons in clamped states and

kj is the correlation with the neurons in a frozen state

j jk

jkkj xxwE 21

)/exp(1

1)(

TExxP

kkk

kjppw kjkjkj

Hidden

Z-1

Z-1

Z-1

Z-1

Delay

Visible

Clamped