Pattern Recognition: Statistical and Neural

1

Pattern Recognition:Statistical and Neural

Lonnie C. Ludeman

Lecture 19

Oct 26, 2005

Nanjing University of Science & Technology

2

Lecture 19 Topics

1. Structures of Optimal Statistical Classifiers

2. Neural Network History

3. Biological Neural Networks

4. Modified McColloch Pitts Model – Example

5. Artificial Neural Element - Definition

3

Structures from Statistical Decision Theory and Perceptron Algorithm

(a). Two Class Likelihood Ratio test

(b) Structures to Motivate Neural Networks

(c) Two Class General Linear

(d). N Class General minimum P(error)

(e) N Class Gaussian

4

if l(x) > T decide x from C1 < T decide x from C2 = T decide x randomly

Decision Rule

between C1 and C2

5

if f(x) > T<C2

C1Decision Rule

6

if f(x) > T<C2

C1Decision Rule

7

if p(x | Ck) > p(x | Cj ) for all j ≠ k decide Ck

Decision Rule

8(e) M Class Gaussian

9

These Structures will be seen to be similar to the structures used in designs with Neural Networks

10

Important Events in Neural Networks History

1943 McColloch Pitts Model

1958 Perceptron Algorithm- Rosenblatt

1960-62 ADALINE – Widrow and Hoff

1969 Minsky and Papert- Limitations

1980’s Grossberg , Hopfield, and Rumelhart – Backpropagation algorithms, Adaptive Resonance Theory

Period of Revival

1990’s Maturation of Field

11

IMPORTANT BOOKS

1990 Artificial Neural Systems- Jacek M. Zurada

1992 Neural Networks and Fuzzy Systems- Bart Kosco

1994 Neural Networks: A Comprehensive Foundation- Simon Haykin

12

Neural Networks

Biological (Real)

Mathematical (Artificial)

How do you tell them apart ???

Squeeze them !!!

Excite them !!!

13

Biological Neuron

14

Action Potential

15

16

Biological Neural Network

17

End Bulb Connection

18

Integration at axon-dendrite junction

19

Modified McColloch Pitts Neuronal Model

(Threshold Logic Unit (TLU) )

20

What can we do with a Modified McColloch-Pitts Model ???

Question

Answer

Surprisingly we can model all logical expressions !!!

21

T = n - 1/2 T = 1/2

Implementation of Logical AND and OR

22

Since we can model logical AND, OR and

NOT we can model all logical expressions

Implementation of Logical NOT

-1/2

23

Example – Implementation of a given logic expression

Given:

Implement f(x) using Modified McColloch-Pitts Neurons

24

Solution:

-1/2

-1/2

-1/2

-1/2

2.5

3.5

T= 1/2

-1

25

Artificial Neural Element (ANE)

Node

netInput Vector

26

net = w1x1 + w2x2 + …+ wnxn + wn+1

= wTx

y = f(net) = f( wTx)

Artificial Neural Element Mathematical Model

Nonlinear Activation Function

Linear Operation on x

NonlinearOperation on x

27

Artificial Neural Element Nodal Representation

Vector Notation

28

Some Activation Functions

29

30

Hyperbolic Tangent Activation Function

f(net) Equivalent Form

Definition

31

Hyperbolic Activation Function

32

33

Non Monitonic Activation Functions can be very Useful

Examples:

Potential Function Approach

Radial Basis Functions

34

Summary Lecture 19

1. Structures of Optimal Statistical Classifiers

2. Neural Network History

3. Biological Neural Networks

4. Modified McColloch Pitts Model – Example

5. Artificial Neural Element - Definition

35

End of Lecture 19