Upload
joy
View
240
Download
0
Embed Size (px)
Citation preview
8/7/2019 Lect 1(Intro)
1/19
Artificial Neural Networks
- Introduction -
8/7/2019 Lect 1(Intro)
2/19
Reference Books and Journals
Neural Networks: A Comprehensive
Foundation by Simon Haykin
Neural Networks for Pattern Recognition byChristopher M. Bishop
Some papers
8/7/2019 Lect 1(Intro)
3/19
Overview
Neural Network (NN) or Artificial Neural Networks
(ANN) is a computing paradigm
The key element of this paradigm is
the novel structure of the information processing system
consisting of a large number of highly interconnected
processing elements (neurons) working in unison to solve
specific problems
Development of NNs date back to the early 1940sMinsky and Papert, published a book (in 1969)
summed up a general feeling of frustration (against neural
networks) among researchers
8/7/2019 Lect 1(Intro)
4/19
Overview (Contd.)
Experienced an upsurge in popularity in the
late 1980s
Result of the discovery of new techniques anddevelopments and general advances in computer
hardware technology
Some NNs are models of biological neural
networks and some are not
8/7/2019 Lect 1(Intro)
5/19
Overview (Contd.)
Historically, much of the inspiration for the
field of NNs came from the desire to produce
artificial systems capable ofsophisticated, perhaps intelligent, computations
similar to those that the human brain routinely
performs, and thereby possibly to enhance our
understanding of the human brain.
8/7/2019 Lect 1(Intro)
6/19
Overview (Contd.)
Most NNs have some sort oftraining rule. In other words,
NNs learn from examples
as children learn to recognize dogs from examples of dogs) and
exhibit some capability forgeneralization beyond the training data
Neural computing must not be considered as a competitor to
conventional computing.
Rather should be seen as complementary
Most successful neural solutions have been those which operate in
conjunction with existing, traditional techniques.
8/7/2019 Lect 1(Intro)
7/19
Overview (Contd.)
Digital Computers Deductive Reasoning. Weapply known rules to input data
to produce output
Computation is centralized,
synchronous, and serial. Memory is packetted, literally
stored, location addressable
Not fault tolerant. One transis-
tor goes and it no longer works.
Exact. Static connectivity.
Applicable if well defined rules
with precise input data.
Neural Networks Inductive Reasoning. Given input
and output data (training
examples), we construct rules
Computation is collective,
asynchronous, and parallel.
Memory is distributed,
internalized, short term and
content addressable.
Fault tolerant, redundancy, and
sharing of responsibilities.
Inexact.
Dynamic connectivity.
Applicable if rules are unknown
or complicated, or if data are
noisy or partial.
8/7/2019 Lect 1(Intro)
8/19
Why Neural Networks
Adaptive learning
An ability to learn how to do tasks based on the data given for training or initial
experience.
Self-Organization
An ANN can create its own organization or representation of the information itreceives during learning time.
Real Time Operation
An ANN computations may be carried out in parallel, and special hardware
devices are being designed and manufactured which take advantage of this
capability.Fault Tolerance via Redundant Information Coding:
Partial destruction of a network leads to the corresponding degradation of
performance. However, some network capabilities may be retained even with
major network damage.
8/7/2019 Lect 1(Intro)
9/19
What can you do with an NN
and what not?
In principle, NNs can compute any computable function,
i.e., they can do everything a normal digital computer can
do.
In practice, NNs are especially useful forclassification andfunction approximationproblems.
NNs are, at least today, difficult to apply successfully to
problems that concern manipulation of symbols and
memory.There are no methods for training NNs that can magically
create information that is not contained in the training
data.
8/7/2019 Lect 1(Intro)
10/19
Who is concerned with NNs?
Computer scientists want to find out about the propertiesof non-symbolic information processing with neural netsand about learning systems in general.
Statisticians use neural nets as flexible, nonlinearregression and classification models.
Engineers of many kinds exploit the capabilities of neuralnetworks in many areas, such as signal processing andautomatic control.
Cognitive scientists view neural networks as a possibleapparatus to describe models of thinking andconsciousness (High-level brain function).
Neuro-physiologists use neural networks to describe andexplore medium-level brain function (e.g. memory,sensory system, motorics).
8/7/2019 Lect 1(Intro)
11/19
Who is concerned with NNs?
Physicists use neural networks to model phenomena instatistical mechanics and for a lot of other tasks.
Biologists use Neural Networks to interpret nucleotidesequences.
Philosophers and some other people may also be interestedin Neural Networks for various reasons
8/7/2019 Lect 1(Intro)
12/19
Biological inspiration
Animals are able to react adaptively to changes in their
external and internal environment, and they use their
nervous system to perform these behaviours.
An appropriate model/simulation of the nervous systemshould be able to produce similar responses and
behaviours in artificial systems.
The nervous system is build by relatively simple units, the
neurons, so copying their behavior and functionality
should be the solution.
8/7/2019 Lect 1(Intro)
13/19
Biological inspiration (Contd.)
8/7/2019 Lect 1(Intro)
14/19
Biological inspiration (Contd.)
The brain is a collection of about 10 billion interconnected
neurons
Each neuron is a cell that usesbiochemical reactions to receive,
process and transmit information.
Each terminal button is connected to other neurons across
a small gap called a synapse
A neuron's dendritic tree is connected to a thousand
neighbouring neurons
When one of those neurons fire
a positive or negative charge is received by one of the dendrites.
The strengths of all the received charges are added together
through the processes of spatial and temporal summation.
8/7/2019 Lect 1(Intro)
15/19
Artificial neurons
Neurons work by processing information. They receive and
provide information in form of spikes.
The McCullogh-Pitts model
Inputs
Outputw2
w1
w3
wn
wn-1
.. .
x1
x2
x3
xn-1
xn
y)(;
1
zHyxwzn
i
ii ===
8/7/2019 Lect 1(Intro)
16/19
Artificial neurons
Nonlinear generalization of the McCullogh-Pitts
neuron:
),( wxfy=
y is the neurons output, x is the vector of inputs, and w
is the vector of synaptic weights.
Examples:
2
2
2
||||
11
a
wx
axw
ey
ey
T
=
+
= sigmoidal neuron
Gaussian neuron
8/7/2019 Lect 1(Intro)
17/19
Activation Functions
The activation function is generally non-linear
Linear functions are limited because the output is simply
proportional to the input
8/7/2019 Lect 1(Intro)
18/19
Activation Functions (Contd.)
8/7/2019 Lect 1(Intro)
19/19
Artificial neurons
Nonlinear generalization of the McCullogh-Pitts
neuron:
),( wxfy=
y is the neurons output, x is the vector of inputs, and w
is the vector of synaptic weights.
Examples:
2
2
2
||||
11
a
wx
axw
ey
ey T
=
+
= sigmoidal neuron
Gaussian neuron