1 Mehran University of Engineering and Technology, Jamshoro Department of Electronic,...

1

Mehran University of Engineering and Technology, Jamshoro

Department of Electronic, Telecommunication and Bio-Medical Engineering

8th Term

Neural Networks and Fuzzy Logic

By

Dr. Mukhtiar Ali Unar

Email: mukhtiar_unar@yahoo.com

Telephone: 772279

2

Books:

1. Haykin, S., Neural Networks – A Comprehensive Foundation, Second edition or latest, McMillan.

2. Hagan, M.T., Demuth, H.B., and Beale, M., Neural Network Design, PWS Publishing Company.

3

Main Features of Neural Networks:

Artificial Neural Networks (ANNs) learn by experience rather than by modelling or programming.

ANN architectures are distributed, inherently parallel and potentially real time.

They have the ability to generalize. They do not require a prior understanding of the

process or phenomenon being studied. They can form arbitrary continuous non-linear

mappings. They are robust to noisy data. VLSI implementation is easy

4

Some Applications

Pattern recognition including character Pattern recognition including character recognition, speech recognition, face recognition, speech recognition, face recognition, on-line signature recognition recognition, on-line signature recognition colour recognition etc.colour recognition etc.

Weather forecasting, load forecasting.Weather forecasting, load forecasting. Intelligent routers, intelligent traffic Intelligent routers, intelligent traffic

monitoring, intelligent filter design.monitoring, intelligent filter design. Intelligent controller designIntelligent controller design Intelligent modelling.Intelligent modelling.

and many moreand many more

5

limitations Conventional neural networks are black box models. Tools for analysis and model validation are not well

established. An intelligent machine can only solve some specific

problem for which it is trained. Human brain is very complex and cannot be fully simulated

with present computing power. An artificial neural network does not have capability of human brain.

Issue: What is the difference between human brain neurons

and other animal brain neurons?

6

Definition:A universal definition of ANNs is not available,

however, the following definition summarizes the basic features of an ANN:

Artificial Neural Networks, also called Neurocomputing or Parallel Distributed Processes (PDP) or connectionist networks or simply neural networks are interconnected assemblies of simple processing elements, called neurons, units or nodes, whose functionality is loosely based on the biological neuron. The processing ability of the network is stored in the inter-unit connection strength, or weights, obtained by a process of adaptation to, or learning from a set of training patterns.

7

Biological Neuron:A

sim

pli

fied

vie

w o

f a

bio

logi

cal (

real

) n

euro

n

8

Examples of some real neurons

9 Image of the vertical organization of neurons in the primary visual cortex

10

Soma or Cell Body

The central part of a neuron is called the soma or cell body which contains the nucleus and the protein synthesis machinery. The size of soma of a typical neuron is about 10 to 80m. Almost all the logical functions are realized in this part of a neuron.

11

The Dendrites

The dendrites represent a highly branching tree of fibers and are attached to the soma. The word dendrite has been taken from the Greek word dendro which means tree. Dendrites connect the neuron to a set of other neurons. Dendrites either receive inputs from other neurons or connect other dendrites to the synaptic outputs.

12

The Axon

It is a long tubular fiber which divides itself into a number of branches towards its end. Its length can be from 100 m to 1 m. The function of an axon is to transmit the generated neural activity to other neurons or to muscle fibers. In other words, it is output channel of the neuron. The point where the axon is connected to its cell body is called the Hillock zone.

13

SynapsesThe junction at which a signal is passed from one

neuron to the next is called a synapse (from the Greek verb to join). It has a button like shape with diameter around 1 m. Usually a synapse is not a physical connection (the axon and the dendrite do not touch) but there is a gap called the synaptic cleft that is normally between 200Å to 500Å (1Å = 10-10 m). The strength of synaptic connection between neurons can be chemically altered by the brain in response to favorable and unfavorable stimuli in such a way as to adapt the organism to function optimally within its environment.

14

“The nerve fibre is clearly a signalling mechanism of limited scope. It can only transmit a succession of brief explosive waves, and the message can only be varied by changes in the frequency and in the total number of these waves. … But this limitation is really a small matter, for in the body the nervous units do not act in isolation as they do in our experiments. A sensory stimulus will usually affect a number of receptor organs, and its result will depend on the composite message in many nerve fibres.” Lord Adrian, Nobel Acceptance Speech, 1932.

15

We now know it’s not quite that simple

Single neurons are highly complex Single neurons are highly complex electrochemical deviceselectrochemical devices

Synaptically connected networks are only Synaptically connected networks are only part of the storypart of the story

Many forms of interneuron communication Many forms of interneuron communication now known – acting over many different now known – acting over many different spatial and temporal scalesspatial and temporal scales

16

Single neuron activity

• Membrane potential is the voltage difference between a neuron and its surroundings (0 mV)

CellCell

CellCell

0 Mv

Membrane potential

17

Single neuron activity •If you measure the membrane potential of a neuron and print it out on the screen, it looks like:

spike

18

Single neuron activity

•A spike is generated when the membrane potential is greater than its threshold

19

Abstraction•So we can forget all sub-threshold activity and concentrate on spikes (action potentials), which are the signals sent to other neurons

Spikes

20

• Only spikes are important since other neurons receive them (signals)

• Neurons communicate with spikes

• Information is coded by spikes

• So if we can manage to measure the spiking time, we decipher how the brain works ….

21

Again its not quite that simple

• spiking time in the cortex is random

22

With identical input for the identical neuron

spike patterns are similar, but not identical

23

Recording from a real neuron: membrane potential

24

Single spiking time is meaninglessTo extract useful information, we have to average

to obtain the firing rate r

for a group of neurons in a local circuit where neuron codes the same information over a time window

Local circuit

=

Time window = 1 sec

r =

Hz

25

So we can have a network of these local groups

w1: synaptic strength

wn

r1

rn

R f w rj j ( )

Hence we have firing rate of a group of neurons

26

ri is the firing rate of input local circuit

The neurons at output local circuits receive signals in the form

The output firing rate of the output local circuit is then given by R

where f is the activation function, generally a Sigmoidal function of some sort

N

iiirw

1

)(1

N

iiirwfR

wi weight, (synaptic strength) measuring the strength of the interaction between neurons.

27

Artificial Neural networks

Local circuits (average to get firing rates)

Single neuron (send out spikes)

28

Is it possible to simulate human brain?

No, with present computing power is it almost No, with present computing power is it almost impossible to simulate a full behavior of impossible to simulate a full behavior of human brain.human brain.

Reasons:Reasons: It is estimated that the human cerebral It is estimated that the human cerebral

cortex contains 100 billion neurons. Each cortex contains 100 billion neurons. Each neuron has as many as 1000 dendrites and, neuron has as many as 1000 dendrites and, hence within the cerebral cortex there are hence within the cerebral cortex there are approximately 100,000 billion synapses. approximately 100,000 billion synapses.

The behavior of the real nervous system is The behavior of the real nervous system is very complex and not yet fully known.very complex and not yet fully known.

29

Model of a single artificial neuron

w2

w1

wp

f

x1

x2

Activation function

xp

b

y

p

1iii bpwfy

30

Model of a single artificial neuron

w2

w1

wp

f

x1

x2

Activation function

xp

y

w0x0

For simplicity, the threshold contribution b may be treated as an extra input to the neuron, as shown below, where x0 = 1, w0 = b.

p

0iiixwfy

In this case

31

Activation Functions:The activation function of a neuron, also The activation function of a neuron, also

known as transfer function, may be linear or known as transfer function, may be linear or non-linear. A particular activation function non-linear. A particular activation function is chosen to satisfy some specification of is chosen to satisfy some specification of the problem that the neuron is attempting to the problem that the neuron is attempting to solve. solve.

A variety of activation functions have been A variety of activation functions have been proposed. Some of these are discussed proposed. Some of these are discussed below:below:

32

(a) The hard limit activation function:(a) The hard limit activation function: This This activation function sets the output of the activation function sets the output of the neuron to 0 if the function argument is less neuron to 0 if the function argument is less than 0, or 1 if its argument is greater than or than 0, or 1 if its argument is greater than or equal to 0. (see figure below). This function equal to 0. (see figure below). This function is generally used to create neurons that is generally used to create neurons that classify inputs into two distinct classes.classify inputs into two distinct classes.

0

1

33

Example 1:Example 1: The input to a neuron is 3, and its The input to a neuron is 3, and its weight is 1.8. weight is 1.8.

(a) What is the net input to the activation (a) What is the net input to the activation function?function?

(b) What is the output of the neuron if it has (b) What is the output of the neuron if it has the hard limit transfer function?the hard limit transfer function?

Solution:Solution:

(a)(a) Net input = u = 1.8Net input = u = 1.83 = 5.43 = 5.4

(b)(b) Neuron output = f(u) = 1.Neuron output = f(u) = 1.

34

Example 2:Example 2: Repeat example 1 if its bias is Repeat example 1 if its bias is

(i) –2 (ii) –6(i) –2 (ii) –6

Solution:Solution: (a) net input = u = (1.8 (a) net input = u = (1.83) + (-2) 3) + (-2)

= 5.4 – 2 = 3.4= 5.4 – 2 = 3.4

output = f(u) = 1.0output = f(u) = 1.0

(b) net input = u = (1.8(b) net input = u = (1.83) + (-6) 3) + (-6)

= 5.4 – 6 = -0.6= 5.4 – 6 = -0.6

output = 0.output = 0.

35

Example 3:Example 3: Given a two input neuron with Given a two input neuron with the following parameters: b = 1.4, W = the following parameters: b = 1.4, W = [1.2 3.4] and P = [-3 5], calculate the [1.2 3.4] and P = [-3 5], calculate the neuron output for the hard limit transfer neuron output for the hard limit transfer function.function.

Solution: Solution: net input to the activation function net input to the activation function

==

Neuron output = f(u) = 1.0Neuron output = f(u) = 1.0

u

8.144.15

34.32.1u

36

(b) Symmetrical hard limit:(b) Symmetrical hard limit:

This activation function is defined as follows:This activation function is defined as follows:

y = -1 u < 0y = -1 u < 0

y = +1 u y = +1 u 0 0

Where u is the net input and y is the output of Where u is the net input and y is the output of the function (see figure below):the function (see figure below):

-1

0

1

y

u0

37

Example 4:Example 4: Given a two output neuron Given a two output neuron with the following parameters: b = 1.2, W with the following parameters: b = 1.2, W = [3 2], and p = [-5 6]. Calculate the = [3 2], and p = [-5 6]. Calculate the neuron output for the symmetrical hard neuron output for the symmetrical hard limit activation functionlimit activation function..

Solution: Solution:

Output = y = f(u) = -1Output = y = f(u) = -1

8.12.16

523u

38

(c) Linear activation function:(c) Linear activation function:

The output of a linear activation function is The output of a linear activation function is equal to its input:equal to its input:

y = uy = u

as shown in the following figure:as shown in the following figure:

0

0 u

y

39

(d) The log-sigmoid activation function:(d) The log-sigmoid activation function:

This activation function takes the input This activation function takes the input (which may have any value between plus (which may have any value between plus and minus infinity) and squashes the output and minus infinity) and squashes the output into the range 0 to 1, according to the into the range 0 to 1, according to the expression:expression:

ue1

1y

A plot of this function is given below:

0 u0

1

40

Example 5:Example 5: Repeat example 2 for a log- Repeat example 2 for a log-sigmoid activation function.sigmoid activation function.

Solution:Solution:

(a) net input = u = 3.4 (a) net input = u = 3.4

output = f(u) = 1/(1 + eoutput = f(u) = 1/(1 + e-3.4-3.4) = 0.9677) = 0.9677

(b) net input u = -0.6(b) net input u = -0.6

output = f(u) = 1/(1 + eoutput = f(u) = 1/(1 + e0.60.6) = 0.3543) = 0.3543

Example 6:Example 6: Repeat example 3 for a log Repeat example 3 for a log sigmoid activation function.sigmoid activation function.

Solution:Solution: output = 1 output = 1

41

(e) Hyperbolic Tangent Sigmoid:(e) Hyperbolic Tangent Sigmoid:

This activation function is shown in the This activation function is shown in the following figure and is defined following figure and is defined mathematically as follows:mathematically as follows:

uu

uu

ee

eey

0-1

0

1

42

Other choices of activation functions

0

+1

-1

Saturation Limiter Gaussian function Schmitt trigger

+1

-1

and many more

43

Artificial Neural Network (ANN) architectures:

Connecting several neurons in some specific manner yields an artificial neural network.

The architecture of an ANN defines the network structure, that is the number of neurons in the network and their interconnectivity.

In a typical ANN architecture, the artificial neurons are connected in layers and they operate in parallel.

The weights or the strength of connection between the neurons are adapted during use to yield good performance.

Each ANN architecture has its own learning rule.

44

Classes of ANN architectures:(a) Single Layer feedforward networks:

Input layer Output layer

45

(b) Multilayer Feedforward Networks

Input layer Hidden layer Output layer

Ful

ly c

onne

cted

fee

dfor

war

d ne

twor

k

wit

h on

e hi

dden

laye

r

46

(b) Multilayer Feedforward NetworksP

arti

ally

con

nect

ed f

eedf

orw

ard

netw

ork

wit

h on

e hi

dden

laye

r

Input signal

Input layer Hidden layer Output layer

47

Recurrent or Feedback Networks:

Rec

urr

ent

net

wor

k w

ith

no

hid

den

neu

ron

s

48

Recurrent Networks: example 2

49

Lattice Structures: