Neural Net III

8/13/2019 Neural Net III

1/11

An Introduction to Neural Networks

Prof. Leslie SmithCentre for Cognitive and Computational Neuroscience

Department of Computing and MathematicsUniversity of Stirling.

[email protected] maor update! "# $cto%er &''(! minor update "" )pril &''* and &" Sept "++&! links

updated ,they -ere really out of date &" Sept "++&/ fi0 to math font ,thanks Sietse

1rou-er " )pril "++23his document is a roughly 43ML5ised version of a talk given at the NS6N meeting in

7din%urgh8 Scotland8 on "* 9e%ruary &''(8 then updated a fe- times in response to

comments received. Please email me comments8 %ut remem%er that this -as originallyust the slides from an introductory talk:

Overview:

;hy -ould anyone -ant a

;hat is a neural net-ork>

Some algorithms and architectures.

;here have they %een applied>

;hat ne- applications are likely>

Some useful sources of information.

Some comments added Sept "++&

N7;! ?uestions and ans-ersarising from this tutorial

Why would anyone want a `new' sort of computer?
http://www.cs.stir.ac.uk/~lss/http://nevis.stir.ac.uk/http://nevis.stir.ac.uk/mailto:[email protected]://www.cs.stir.ac.uk/~lss/NNIntro/InvSlides.html#why%23whyhttp://www.cs.stir.ac.uk/~lss/NNIntro/InvSlides.html#what%23whathttp://www.cs.stir.ac.uk/~lss/NNIntro/InvSlides.html#algs%23algshttp://www.cs.stir.ac.uk/~lss/NNIntro/InvSlides.html#algs%23algshttp://www.cs.stir.ac.uk/~lss/NNIntro/InvSlides.html#where%23wherehttp://www.cs.stir.ac.uk/~lss/NNIntro/InvSlides.html#new%23newhttp://www.cs.stir.ac.uk/~lss/NNIntro/InvSlides.html#new%23newhttp://www.cs.stir.ac.uk/~lss/NNIntro/InvSlides.html#info%23infohttp://www.cs.stir.ac.uk/~lss/NNIntro/InvSlides.html#info%23infohttp://www.cs.stir.ac.uk/~lss/NNIntro/InvSlides.html#comments2001%23comments2001http://www.cs.stir.ac.uk/~lss/NNIntro/qanda.htmlhttp://www.cs.stir.ac.uk/~lss/http://nevis.stir.ac.uk/mailto:[email protected]://www.cs.stir.ac.uk/~lss/NNIntro/InvSlides.html#why%23whyhttp://www.cs.stir.ac.uk/~lss/NNIntro/InvSlides.html#what%23whathttp://www.cs.stir.ac.uk/~lss/NNIntro/InvSlides.html#algs%23algshttp://www.cs.stir.ac.uk/~lss/NNIntro/InvSlides.html#where%23wherehttp://www.cs.stir.ac.uk/~lss/NNIntro/InvSlides.html#new%23newhttp://www.cs.stir.ac.uk/~lss/NNIntro/InvSlides.html#info%23infohttp://www.cs.stir.ac.uk/~lss/NNIntro/InvSlides.html#comments2001%23comments2001http://www.cs.stir.ac.uk/~lss/NNIntro/qanda.html


2/11

;hat are ,everyday computer systems good at... .....and not so good at>

Good at Not so ood at

9ast arithmeticnteracting -ith noisy data or data

from the environment

Doing precisely -hat the programmer programsthem to do Massive parallelism

Massive parallelism

9ault tolerance

)dapting to circumstances

;here can neural net-ork systems help>

-here -e can=t formulate an algorithmic solution.

-here -e canget lots of e0amples of the %ehaviour -e re?uire.

-here -e need to pick out the structure from e0isting data.

What is a neural network?

Neural Net-orks are a different paradigm for computing!

von Neumann machines are %ased on the processingAmemory a%straction of

human information processing.

neural net-orks are based on the parallel architecture of animal brains.

Neural net-orks are a form of multiprocessor computer system8 -ith

simple processing elements

a high degree of interconnection

simple scalar messages adaptive interaction %et-een elements

) %iological neuron may have as many as &+8+++ different inputs8 and may send its

output ,the presence or a%sence of a short5duration spike to many other neurons.Neurons are -ired up in a 25dimensional pattern.


3/11

Beal %rains8 ho-ever8 are orders of magnitude more comple0 than any artificial neural

net-ork so far considered.

70ample! ) simple single unit adaptive net-ork!

3he net-ork has " inputs8 and one output. )ll are %inary. 3he output is

& if ;++ ;& & ;%E +

+ if ;++ ;& & ;%FG +

;e -ant it to learn simple $B!

output a & if either +or &is &.

Alorithms and

Architectures!

"he simple #erceptron:

3he net-ork adapts as follo-s!

change the weight by an amountproportional to the difference

between the desired output and the

actual output.

)s an e?uation!

H ;iG I ,D56.i

-here I is the learning rate8 D is the desired output8 and 6 is the actual output.

3his is called thePerceptron Learning Rule8 and goes %ack to the early &'(+=s.

;e e0pose the net to the patterns!

I$ I% &esired output

+ + +

+ & && + &

& & &

;e trainthe net-ork on these e0amples. ;eights after each epoch ,e0posure to completeset of patterns


4/11

)t this point ,* the net-ork has finished

learning. Since ,D56G+ for all patterns8

the -eights cease adapting. Singleperceptrons are limited in -hat they can

learn!

f -e have t-o inputs8 the decision

surface is a line. ... and its e?uation is

&G ,;+A;&.+ ,;%A;&

n general8 they implement a

simple hyperplane decision

surface

3his restricts the possi%le

mappings availa%le.

&evelopments from the simple

perceptron:

1ack5Propagated Delta BuleNet-orks ,1P ,sometimes

kno-n and multi5layer

perceptrons ,MLPs and Badial1asis 9unction Net-orks ,B19

are %oth -ell5kno-n

developments of the Delta rulefor single layer net-orks ,itself adevelopment of the Perceptron Learning Bule. 1oth can learn ar%itrary mappings or

classifications. 9urther8 the inputs ,and outputs can have real values

ack(#ropaated &elta )ule Networks *#+

is a development from the simple Delta rule in -hich e0tra hidden layers,layers

additional to the input and output layers8 not connected e0ternally are added. 3he

net-ork topology is constrained to %efeedforward! i.e. loop5free 5 generally connections

are allo-ed from the input layer to the first ,and possi%ly only hidden layer/ from thefirst hidden layer to the second8...8 and from the last hidden layer to the output layer.

"ypical # network architecture:

3he hidden layer learns to recode,or toprovide a representationfor the inputs. More

than one hidden layer can %e used.

3he architecture is more po-erful than single5layer net-orks! it can %e sho-n that anymapping can %e learned8 given t-o hidden layers ,of units.


5/11

3he units are a little more comple0 than those in

the original perceptron! their inputAoutput graph

is

)s a function!

6 G & A ,& e0p,5k.,J ; in Kin

3he graph sho-s the output for

k=0.5, , and 0,as the activation

varies from 5&+ to &+.

"rainin # Networks

3he -eight change rule is a

development of the perceptronlearning rule. ;eights are

changed %y an amount

proportional to the error at that

unittimes the output of the unitfeeding into the weight.

Bunning the net-ork consists of

9or-ard pass!

the outputs are calculatedand the error at the output

units calculated.

1ack-ard pass!3he output unit error is used to alter -eights on the output units. 3hen the error at

the hidden nodes is calculated ,%y back!propagatingthe error at the output units

through the -eights8 and the -eights on the hidden nodes altered using these

values.9or each data pair to %e learned a for-ard pass and %ack-ards pass is performed. 3his is

repeated over and over again until the error is at a lo- enough level ,or -e give up.

)adial asis function Networks

Badial %asis function net-orks are also feedfor-ard8 %ut have only onehidden layer.

"ypical ), architecture:

Like 1P8 B19 nets can learn ar%itrary mappings! the primary difference is in the hidden

layer.


6/11

B19 hidden layer units have a receptive

field-hich has a centre" that is8 a

particular input value at -hich they have ama0imal output.3heir output tails off as the

input moves a-ay from this point.

enerally8 the hidden unit function is a

aussian!

aussians -ith three different standard deviations.

"rainin ), Networks!

B19 net-orks are trained %y

deciding on ho- many hidden units there

should %e deciding on their centres and the sharpnesses

,standard deviation of their aussians

training up the output layer.

enerally8 the centres and SDs are decided on first %ye0amining the vectors in the training data. 3he output

layer -eights are then trained using the Delta rule. 1P

is the most -idely applied neural net-ork techni?ue.B19s are gaining in popularity.

Nets can %e

trained on classification data ,each output represents one class8 and then used

directly as classifiers of ne- data.

trained on ,08f,0 points of an unkno-n function f8 and then used to interpolate.

B19s have the advantage that one can add e0tra units -ith centres near parts of the input-hich are difficult to classify. 1oth 1P and B19s can also %e used for processing time5

varying data! one can consider a windowon the data!

Net-orks of this form ,finite5impulse response have %een used in many applications.

3here are also net-orks -hose architectures are specialised for processing time5series.

-nsupervised networks:

Simple Perceptrons8 1P8 and B19 net-orks need a teacher to tell the net-ork -hat thedesired output should %e. 3hese are supervised net-orks.


7/11

n an unsupervised net8 the net-ork adapts purely in response to its inputs. Such net-orks

can learn to pick out structure in their input.

Applications for unsupervised nets

clustering data!e0actly one of a small num%er of output units comes on in response to an input.

reducing the dimensionality of data!

data -ith high dimension ,a large num%er of input units is compressed into alo-er dimension ,small num%er of output units.

)lthough learning in these nets can %e

slo-8 running the trained net is veryfast 5 even on a computer simulation

of a neural net.

.ohonen clusterin Alorithm:

5 takes a high5dimensional input8 and

clusters it8 %ut retaining sometopological ordering of the output.

)fter training8 an input -ill cause

somethe output units in some area to

%ecome active.

Such clustering ,and dimensionality reduction is very useful as a preprocessing stage8

-hether for further neural net-ork data processing8 or for more traditional techni?ues.

Where are Neural Networks applica/le?

..... or are they ust a solution in search of a pro%lem>

Neural net-orks cannot do anything that cannot %e done using traditional computing

techni?ues8 -"they can do some things -hich -ould other-ise %e very difficult.

n particular8 they can form a model from their training data ,or possi%ly input dataalone.

3his is particularly useful -ith sensory data8 or -ith data from a comple0 ,e.g. chemical8manufacturing8 or commercial process. 3here may %e an algorithm8 %ut it is not kno-n8

or has too many varia%les. t is easier to let the net-ork learn from e0amples.

Neural networks are /ein used:

in investment analysis!


8/11

to attempt to predict the movement of stocks currencies etc.8 from previous data.

3here8 they are replacing earlier simpler linear models.

in signature analysis!as a mechanism for comparing signatures made ,e.g. in a %ank -ith those stored.

3his is one of the first large5scale applications of neural net-orks in the US)8 and

is also one of the first to use a neural net-ork chip.in process control!

there are clearly applications to %e made here! most processes cannot %e

determined as computa%le algorithms. Ne-castle University Chemical7ngineering Department is -orking -ith industrial partners ,such as eneca and

1P in this area.

in monitoring!

net-orks have %een used to monitor the state of aircraft engines. 1y monitoring vi%ration levels and sound8

early -arning of engine pro%lems can %e given.

1ritish Bail have also %een testing a similar application monitoring diesel

engines.

in marketing!

net-orks have %een used to improve marketing mailshots. $ne techni?ue is to run

a test mailshot8 and look at the pattern of returns from this. 3he idea is to find apredictive mapping from the data kno-n a%out the clients to ho- they have

responded. 3his mapping is then used to direct further mailshots.

"o pro/e further:

) rather longer introduction,-hich is more commercially oriented is hosted %y StatSoft8

nc.

3here is also another introduction ,including some history %yStergiou and Siganos.

3heNatural Computing )pplications 9orumruns meetings ,-ith attendees from industry8commerce and academe on applications of Neural Net-orks. Contact NC)9 through

their -e%site8 %y telephone ,+&22" "('*'8 or %y fa0 ,+&22" "O&"'

nternet addresses!#euro#et-hich -as at ings College8 London8 -as a 7uropean

Net-ork of 70cellence in Neural Net-orks -hich finished in March "++&. 4o--ever8their -e%siteremains a very useful source of information

$%%% &omputational $ntelligence 'ociety ,-as 777 Neural Net-orks Society

http!AA---.ieee5cis.orgApu%lish a num%er of ournals on neural net-orks and relatedareas.

-rote si0 lectures on 1ack Propagationrather a long time ago8 in a no- defunct ;P.

Still8 they are reada%le...
http://www.statsoftinc.com/textbook/stneunet.htmlhttp://www.doc.ic.ac.uk/~nd/surprise_96/journal/vol4/cs11/report.htmlhttp://www.ncaf.org.uk/http://www.kcl.ac.uk/neuronet/http://www.ieee-cis.org/http://www.cs.stir.ac.uk/~lss/31X7/BPlectures/http://www.cs.stir.ac.uk/~lss/31X7/BPlectures/http://www.statsoftinc.com/textbook/stneunet.htmlhttp://www.doc.ic.ac.uk/~nd/surprise_96/journal/vol4/cs11/report.htmlhttp://www.ncaf.org.uk/http://www.kcl.ac.uk/neuronet/http://www.ieee-cis.org/http://www.cs.stir.ac.uk/~lss/31X7/BPlectures/


9/11

#ewscomp.ai.neural5nets has an very useful set of fre?uently asked ?uestions ,9)Q=s8

availa%le as a ;;; document at! ftp!AAftp.sas.comApu%AneuralA9)Q.html

0ourses

Quite a fe- organisations run courses! -e used to run a & year Masters course in NeuralComputation! unfortunately8 this course in in a%eyance. ;e can even run courses to suit

you. ;e are a%out to start up a centre in Computational ntelligence8 called NC37.

1ore 2pecific Information

Some further information a%out applications can %e found from the %ook Stimulationnitiative for 7uropean Neural )pplications ,S7N).

9or more information on Neural Net-orks in the Process ndustries8 try). 1ulsari=s

home page .

3he company 1rainMaker has a nice list ofreferences on applications of its soft-are

packagethat sho-s the %readth of applications areas.

3ournals!

3he %est ournal for application5oriented information is

#eural &omputing and (pplications8 Springer5Rerlag. ,address! S-eetapple 4o8

Catteshall Bd.8 odalming8 UO 2D

ooks!

3here=s a lot of %ooks on Neural Computing. See the 9)Qa%ove for a much longer list.

9or a not5too5mathematical introduction8 try

9ausett L.8)undamentals of #eural #etworks8 Prentice54all8 &''. S1N + &2 +""#+ '

or

urney .8(n $ntroduction to #eural #etworks8 UCL Press8 &''O8 S1N & *#O"* #+2

4aykin S.8#eural #etworks8 "nd 7dition8 Prentice 4all8 &'''8 S1N + &2 "O22#+ & is amore detailed %ook8 -ith e0cellent coverage of the -hole su%ect.

Where are neural networks oin?

) great deal of research is going on in neural net-orks -orld-ide.
ftp://ftp.sas.com/pub/neural/FAQ.htmlhttp://www.cs.stir.ac.uk/courses/mscnc.htmlhttp://www.cs.stir.ac.uk/courses/mscnc.htmlhttp://www.cn.stir.ac.uk/incite/http://www.nici.kun.nl/Publications/1998/11339.htmlhttp://www.nici.kun.nl/Publications/1998/11339.htmlhttp://www.abo.fi/~abulsarihttp://www.abo.fi/~abulsarihttp://www.abo.fi/~abulsarihttp://www.abo.fi/~abulsarihttp://www.calsci.com/Applications.htmlhttp://www.calsci.com/Applications.htmlhttp://www.calsci.com/Applications.htmlhttp://www.calsci.com/Applications.htmlhttp://www.cs.stir.ac.uk/~lss/NNIntro/InvSlides.html#FAQ%23FAQftp://ftp.sas.com/pub/neural/FAQ.htmlhttp://www.cs.stir.ac.uk/courses/mscnc.htmlhttp://www.cs.stir.ac.uk/courses/mscnc.htmlhttp://www.cn.stir.ac.uk/incite/http://www.nici.kun.nl/Publications/1998/11339.htmlhttp://www.nici.kun.nl/Publications/1998/11339.htmlhttp://www.abo.fi/~abulsarihttp://www.abo.fi/~abulsarihttp://www.calsci.com/Applications.htmlhttp://www.calsci.com/Applications.htmlhttp://www.cs.stir.ac.uk/~lss/NNIntro/InvSlides.html#FAQ%23FAQ


10/11

3his ranges from %asic research into ne- and more efficient learning algorithms8 to

net-orks -hich can respond to temporally varying patterns ,%oth ongoing at Stirling8 to

techni?ues for implementing neural net-orks directly in silicon. )lready one chipcommercially availa%le e0ists8 %ut it does not include adaptation. 7din%urgh University

have implemented a neural net-ork chip8 and are -orking on the learning pro%lem.

Production of a learning chip -ould allo- the application of this technology to a -hole

range of pro%lems -here the price of a PC and soft-are cannot %e ustified.

3here is particular interest in sensory and sensing applications! nets -hich learn to

interpret real5-orld sensors and learn a%out their environment.

New Application areas:

Pen PC=s

PC=s -here one can -rite on a ta%let8 and the -riting -ill %e recognised and

translated into ,)SC te0t.Speech and Rision recognition systems

Not ne-8 %ut Neural Net-orks are %ecoming increasingly part of such systems.3hey are used as a system component8 in conunction -ith traditional computers.

;hite goods and toys

)s Neural Net-ork chips %ecome availa%le8 the possi%ility of simple cheapsystems -hich have learned to recognise simple entities ,e.g. -alls looming8 or

simple commands like o8 or Stop8 may lead to their incorporation in toys and

-ashing machines etc. )lready the apanese are using a related technology8 fuTTylogic8 in this -ay. 3here is considera%le interest in the com%ination of fuTTy and

neural technologies.

2ome comments *added 2eptem/er 4$$%+

Beading this through8 it is a %it outdated! not that there=s anything incorrect a%ove8 %ut the

-orld has moved on. Neural Net-orks should %e seen as part of a larger field sometimes

called 'oft &omputingor#atural &omputing. n the last fe- years8 there has %een a realmovement of the discipline in three different directions!

Neural net-orks8 statistics8 generative models8 1ayesian inference

3here is a sense in -hich these fields are coalescing. 3he real pro%lem is makingconclusions from incomplete8 noisy data8 and all of these fields offer something in

this area. Developments in the mathematics underlying these fileds have sho-n

that there are real similarities in the techni?ues used. Chris 1ishop=s%ookNeuralNet-orks for Pattern Becognition8 $0ford University Press is a good start on this

area.

Neuromorphic Systems

70isting neural net-ork ,and indeed other soft computing systems are generallysoft-are models for solving static pro%lems on PCs. 1ut -hy not free the concept

from the -orkstation> 3he area of neuromorphic systems is concerned -ith real5

time implementations of neurally inspired systems8 generally implemented
http://www.ncrg.aston.ac.uk/NNPR/index.htmlhttp://www.ncrg.aston.ac.uk/NNPR/index.html


11/11

directly in silicon8 for sensory and motor tasks. )nother aspect is direct

implementation of detailed aspects of neurons in silicon ,see 1iological Neural

Net-orks %elo-. 3he main centres -orld-ide are at the nstitute forneuroinformaticsat urich8 and at the Center for Neuromorphic Systems

7ngineeringat Caltech. 3here are also some useful links atthis page,from a U

7PSBC Net-ork Proect on Silicon and Neuro%iology1iological Neural Net-orks

3here is real interest in ho- neural net-ork research and neurophysiology can

come together. 3he pattern recognition aspects of )rtificial Neural Net-orksdon=t really e0plain too much a%out ho- real %rains actually -ork. 3he field

called Computational Neurosciencehas taken inspiration from %oth artificial

neural net-orks and neurophysiology8 and attempts to put the t-o together.

1ack to top of page

1ack to Leslie Smith=s homepageLast updated" *onday, +!-ul!+00 0/"+"1 2'3

$f you have any difficulties accessing this page, or you have any 4ueriessuggestions arising from

this page, please email"

Prof Leslie ' 'mith 6lss6nospam7please89cs.stir.ac.uk8
http://www.ini.unizh.ch/index.htmlhttp://www.ini.unizh.ch/index.htmlhttp://www.erc.caltech.edu/http://www.erc.caltech.edu/http://www.cs.stir.ac.uk/SilicoNeural/links.htmhttp://www.cs.stir.ac.uk/SilicoNeural/links.htmhttp://www.cs.stir.ac.uk/SilicoNeural/http://www.hirnforschung.net/cneuro/http://www.hirnforschung.net/cneuro/http://www.cs.stir.ac.uk/~lss/NNIntro/InvSlides.html#top%23tophttp://www.cs.stir.ac.uk/~lss/index.htmlhttp://www.cs.stir.ac.uk/http://www.ini.unizh.ch/index.htmlhttp://www.ini.unizh.ch/index.htmlhttp://www.erc.caltech.edu/http://www.erc.caltech.edu/http://www.cs.stir.ac.uk/SilicoNeural/links.htmhttp://www.cs.stir.ac.uk/SilicoNeural/http://www.hirnforschung.net/cneuro/http://www.cs.stir.ac.uk/~lss/NNIntro/InvSlides.html#top%23tophttp://www.cs.stir.ac.uk/~lss/index.html

Documents

Neural Net III