8/9/2019 07 Neural Networks1
1/73
Negnevitsky, Pearson Education, 2005 1
Lecture 7Lecture 7
Artificial neural networks:Artificial neural networks:Supervised learningSupervised learning
Introduction, or how the brain worksIntroduction, or how the brain works
The neuron as a simple computing elementThe neuron as a simple computing element
The perceptronThe perceptron
Multilayer neural networksMultilayer neural networks
Accelerated learning in multilayer neural networksAccelerated learning in multilayer neural networks The Hopfield networkThe Hopfield network
Bidirectional associative memories (BAMBidirectional associative memories (BAM
!ummary!ummary
8/9/2019 07 Neural Networks1
2/73
Negnevitsky, Pearson Education, 2005 2
Introduction, or how the brain worksIntroduction, or how the brain works
Machine learning involves adaptive mechanismsMachine learning involves adaptive mechanismsthat enable computers to learn from e"perience,that enable computers to learn from e"perience,
learn by e"ample and learn by analogy# $earninglearn by e"ample and learn by analogy# $earning
capabilities can improve the performance of ancapabilities can improve the performance of an
intelligent system over time# The most popularintelligent system over time# The most popular
approaches to machine learning areapproaches to machine learning are artificialartificial
neural networksneural networks andand genetic algorithmsgenetic algorithms# This# This
lecture is dedicated to neural networks#lecture is dedicated to neural networks#
8/9/2019 07 Neural Networks1
3/73
Negnevitsky, Pearson Education, 2005 3
AA neural networkneural network can be defined as a model ofcan be defined as a model of
reasoning based on the human brain# The brainreasoning based on the human brain# The brain
consists of a densely interconnected set of nerveconsists of a densely interconnected set of nerve
cells, or basic information%processing units, calledcells, or basic information%processing units, called
neuronsneurons##
The human brain incorporates nearly &' billionThe human brain incorporates nearly &' billion
neurons and ' trillion connections,neurons and ' trillion connections,synapsessynapses,,
between them# By using multiple neuronsbetween them# By using multiple neurons
simultaneously, the brain can perform its functionssimultaneously, the brain can perform its functionsmuch faster than the fastest computers in e"istencemuch faster than the fastest computers in e"istence
today#today#
8/9/2019 07 Neural Networks1
4/73
Negnevitsky, Pearson Education, 2005 4
)ach neuron has a very simple structure, but an)ach neuron has a very simple structure, but an
army of such elements constitutes a tremendousarmy of such elements constitutes a tremendous
processing power#processing power#
A neuron consists of a cell body,A neuron consists of a cell body, somasoma, a number, a number
of fibers calledof fibers called dendritesdendrites, and a single long fiber, and a single long fiber
called thecalled the axonaxon##
8/9/2019 07 Neural Networks1
5/73
Negnevitsky, Pearson Education, 2005 5
Biological neural networkBiological neural network
Soma Soma
Synapse
Synapse
Dendrites
Axon
Synapse
Dendrites
Axon
8/9/2019 07 Neural Networks1
6/73
Negnevitsky, Pearson Education, 2005 6
*ur brain can be considered as a highly comple",*ur brain can be considered as a highly comple",
non%linear and parallel information%processingnon%linear and parallel information%processing
system#system# Information is stored and processed in a neuralInformation is stored and processed in a neural
network simultaneously throughout the wholenetwork simultaneously throughout the whole
network, rather than at specific locations# In othernetwork, rather than at specific locations# In otherwords, in neural networks, both data and itswords, in neural networks, both data and its
processing areprocessing are globalglobal rather than local#rather than local#
$earning is a fundamental and essential$earning is a fundamental and essentialcharacteristic of biological neural networks# Thecharacteristic of biological neural networks# The
ease with which they can learn led to attempts toease with which they can learn led to attempts to
emulate a biological neural network in a computer#emulate a biological neural network in a computer#
8/9/2019 07 Neural Networks1
7/73 Negnevitsky, Pearson Education, 2005 7
An artificial neural network consists of a number ofAn artificial neural network consists of a number of
very simple processors, also calledvery simple processors, also called neuronsneurons, which, which
are analogous to the biological neurons in the brain#are analogous to the biological neurons in the brain# The neurons are connected by weighted linksThe neurons are connected by weighted links
passing signals from one neuron to another#passing signals from one neuron to another#
The output signal is transmitted through theThe output signal is transmitted through theneuron+s outgoing connection# The outgoingneuron+s outgoing connection# The outgoing
connection splits into a number of branches thatconnection splits into a number of branches that
transmit the same signal# The outgoing branchestransmit the same signal# The outgoing branchesterminate at the incoming connections of otherterminate at the incoming connections of other
neurons in the network#neurons in the network#
8/9/2019 07 Neural Networks1
8/73 Negnevitsky, Pearson Education, 2005 8
Architecture of a tpical artificial neural networkArchitecture of a tpical artificial neural network
InputLayer OutputLayer
MiddleLayer
InputS
ignals
Ou
tputS
ignals
8/9/2019 07 Neural Networks1
9/73 Negnevitsky, Pearson Education, 2005 9
Analog between biological andAnalog between biological and
artificial neural networksartificial neural networks
BiologicalNeuralNetwork Artifcial NeuralNetworkSomaDendriteAxonSynapse
NeuronInputOutputWeight
8/9/2019 07 Neural Networks1
10/73 Negnevitsky, Pearson Education, 2005 10
!he neuron as a simple computing element!he neuron as a simple computing element
"iagram of a"iagram of aneuronneuron
Neuron Y
Input Signals
x1
x2
xn
OutputSignals
Y
Y
Y
w2
w1
wn
Weights
8/9/2019 07 Neural Networks1
11/73 Negnevitsky, Pearson Education, 2005 11
The neuron computes the weighted sum of the inputThe neuron computes the weighted sum of the input
signals and compares the result with asignals and compares the result with a thresholdthreshold
valuevalue
,,
# If the net input is less than the threshold,# If the net input is less than the threshold,
the neuron output is But if the net input isthe neuron output is But if the net input is
greater than or e-ual to the threshold, the neurongreater than or e-ual to the threshold, the neuron
becomes activated and its output attains a value .becomes activated and its output attains a value .
The neuron uses the following transfer orThe neuron uses the following transfer or activationactivation
functionfunction//
This type of activation function is called aThis type of activation function is called a signsign
functionfunction##
==n
iiiwxX
1
8/9/2019 07 Neural Networks1
12/73 Negnevitsky, Pearson Education, 2005 12
Activation functions of a neuronActivation functions of a neuron
Step function Sign function
.&
%&
'
.&
%&
'X
Y
X
Y
& &
%&
' X
Y
Sigmoidfunction
%&
' X
Y
Linear function