7/21/2019 Statistical methods in ANN
http://slidepdf.com/reader/full/statistical-methods-in-ann 1/42
MODULE-IV
STATISTICAL METHODS IN ANN
7/21/2019 Statistical methods in ANN
http://slidepdf.com/reader/full/statistical-methods-in-ann 2/42
Module 4
Statistical Methods: Boltzmann's Training - Cauchy
training - Artificial specific heat methods - applications
to general non-linear optimization problems
7/21/2019 Statistical methods in ANN
http://slidepdf.com/reader/full/statistical-methods-in-ann 3/42
Statistical Methods are used for
Training ANN
roducing out!ut fro" trained net#or$
Training Methods
Deter"inistic Methods
Statistical Training Methods
7/21/2019 Statistical methods in ANN
http://slidepdf.com/reader/full/statistical-methods-in-ann 4/42
Deter"inistic Training Method
%ollo#s a ste! &' ste! !rocedure(
)eights are changed &ased on their current
*alues of #eight(
It also &ased on the desired out!ut and the
actual out!ut(
E(g(+-erce!tron Training Algorith"(
,ac$ ro!agation Algorith" etc
7/21/2019 Statistical methods in ANN
http://slidepdf.com/reader/full/statistical-methods-in-ann 5/42
Statistical Training Methods
Ma$e !seudo rando" change in the #eights
.etains onl' those change #hich results in
i"!ro*e"ents(
7/21/2019 Statistical methods in ANN
http://slidepdf.com/reader/full/statistical-methods-in-ann 6/42
GENERAL PROCEDURE
( FOR STTISTICAL TRAINING METHOD) A!!l' a set of in!ut and co"!ute the resulting
out!ut
Co"!are the result #ith target/ find the error(
The o&0ecti*e of the training is to "ini"i1e the error(
Select a #eight in rando" and ad0ust it &' a s"all
rando" a"ount(
7/21/2019 Statistical methods in ANN
http://slidepdf.com/reader/full/statistical-methods-in-ann 7/42
If the ad0ust"ent i"!ro*es our o&0ecti*e retain
the change
Other#ise return the #eight to the !re*ious
*alue
.e!eat the !rocedures until the net#or$ is
trained to the desired le*el
7/21/2019 Statistical methods in ANN
http://slidepdf.com/reader/full/statistical-methods-in-ann 8/42
The local minima problem
The o&0ecti*e function "ini"i1ation !ro&le" can get
tra!!ed in !oor solution(
)eight
AB
O & 0 e c t i * e % u n c t i o n
7/21/2019 Statistical methods in ANN
http://slidepdf.com/reader/full/statistical-methods-in-ann 9/42
If the o&0ecti*e function is at A and if the rando"
#eight changes are s"all then the #eight
ad0ust"ent #ill &e re0ected(
The su!erior #eight setting at !oint , #ill ne*er
found and the s'ste" #ill &e tra!!ed in local
"ini"a instead of glo&al "ini"a at !oint ,(
If the rando" #eight changes are large &oth !oint
A and , are *isited fre2uentl'/ &ut so #ill e*er'
other !oint(The #eight #ill change so drasticall' that it #ill
ne*er settle at desired !oint(
7/21/2019 Statistical methods in ANN
http://slidepdf.com/reader/full/statistical-methods-in-ann 10/42
Solution & Eplanation
Statistical "ethod o*erco"e local "ini"a !ro&le" &'
a #eight ad0ust"ent strateg'(
E3a"!le+
Let the fig( re!resents a &all on a surface in a
&o3(
If the &o3 is sha$en *iolentl' /then the &all #ill
"o*e ra!idl' fro" one side to the other side(
The !ro&a&ilit' to occu!' an' !oint on the
surface is e2ual for all !oints(
7/21/2019 Statistical methods in ANN
http://slidepdf.com/reader/full/statistical-methods-in-ann 11/42
If the *iolence of sha$ing is graduall' reduced the &all
#ill stic$ to &oth !oint A and ,(
If the sha$ing is again reduced it #ill settle to !oint ,(
7/21/2019 Statistical methods in ANN
http://slidepdf.com/reader/full/statistical-methods-in-ann 12/42
The ANN are trained in the sa"e #a' as through
rando" #eight ad0ust"ent(
At first large rando" ad0ust"ent are "ade(
The #eight change that i"!ro*es the o&0ecti*e
function is retained(
The a*erage ste! si1e is hen graduall' reduced to
reach glo&al "ini"u"(
7/21/2019 Statistical methods in ANN
http://slidepdf.com/reader/full/statistical-methods-in-ann 13/42
Annealin! " Bolt#mann La$ %
Annealing+-If a "etal is raised to a te"!erature
a&o*e "elting !oint /the ato"s are in *iolent rando"
"otion( The ato"s al#a's tend to reach a "ini"u"
energ' state( As the "etal is graduall' cooled theato"s enters a "ini"u" !ossi&le energ' state
corres!onds to each te"!erature(
)/(
ep)( kT e
e P
−
α
4e56!ro&a&ilit' that the s'ste" is in a state #ith
energ' e(/$ ,olt1"ann7s constant( T 8te"!erature(
7/21/2019 Statistical methods in ANN
http://slidepdf.com/reader/full/statistical-methods-in-ann 14/42
Simulate Annealin! "Bolt#mann Train!%
Define a *aria&le T that re!resents an artificial
te"!erature( 4Start #ith T at large *alue5(
A!!l' a set of in!ut to the net#or$/ and calculate
the out!uts and o&0ecti*e function(
Ma$e a rando" change #eight and recalculate the
net#or$ out!ut(
Calculate ne# o&0ecti*e function(
7/21/2019 Statistical methods in ANN
http://slidepdf.com/reader/full/statistical-methods-in-ann 15/42
If the o&0ecti*e function is reduced/ retain the
#eight change(
If the #eight change results in an increase in
#eight change /calculate the !ro&a&ilit' of acce!ting
the #eight change(
)/(ep)( kT cc P
−
α
4c56!ro&a&ilit' of a change of c in the o&0ecti*e
function/$ ,olt1"ann7s constant( T 8te"!erature(
7/21/2019 Statistical methods in ANN
http://slidepdf.com/reader/full/statistical-methods-in-ann 16/42
Select a rando" nu"&er r fro" a unifor"
distri&ution &et#een 1ero and one(
If !4c5 is greater than r/ retain the change other#ise
return the #eight to !re*ious *alue(
This allo#s the s'ste" to ta$e a ste! in a
direction that #orsen the o&0ecti*efunction/hence esca!es fro" local
"ini"u"(
.e!eat the #eight change !rocess o*er each of the
#eights in the net#or$/ graduall' reducing the
te"!erature T untill an acce!ta&l' lo# *alue for
o&0ecti*e function is o&tained(
7/21/2019 Statistical methods in ANN
http://slidepdf.com/reader/full/statistical-methods-in-ann 17/42
'o$ to elect $ei!ht)arti*icial Temperature *or trainin!
The si1e of the rando" #eight change is selected &'
*arious "ethods(
Eg+-
4#56ro&a&ilit' of a #eight change of si1e #(
)/ep()( !!T w P −=ω
T6artificial te"!erature
To achie*e glo&al "ini"u" at the earliest thecooling rate is usuall' e3!ressed as follo#s
))"(log()( #
t
T t T
+
=
7/21/2019 Statistical methods in ANN
http://slidepdf.com/reader/full/statistical-methods-in-ann 18/42
9The "ain disad*antage of ,olt1"ann7s training is *er'
lo# cooling rate and hence long co"!utations(
9,olt1"ann7s "achine usuall' ta$es i"!ractical ti"e
for training(
7/21/2019 Statistical methods in ANN
http://slidepdf.com/reader/full/statistical-methods-in-ann 19/42
Cauch+ Trainin!
Cauch' training "ethod is "ore ra!id than
,olt1"ann training(
Cauch' training su&stitutes cauch'7s distri&ution for
,olt1"ann:s distri&ution(
Caush'7s distri&ution has longer ;tails</ hence "ore
!ro&a&ilit' for larger ste! si1e(
The te"!erature reduction rate is changed to
in*erse linear( 4%or ,olt1"ann training it #as in*erse
logarith"ic(5
7/21/2019 Statistical methods in ANN
http://slidepdf.com/reader/full/statistical-methods-in-ann 20/42
7/21/2019 Statistical methods in ANN
http://slidepdf.com/reader/full/statistical-methods-in-ann 21/42
7/21/2019 Statistical methods in ANN
http://slidepdf.com/reader/full/statistical-methods-in-ann 22/42
7/21/2019 Statistical methods in ANN
http://slidepdf.com/reader/full/statistical-methods-in-ann 23/42
7/21/2019 Statistical methods in ANN
http://slidepdf.com/reader/full/statistical-methods-in-ann 24/42
7/21/2019 Statistical methods in ANN
http://slidepdf.com/reader/full/statistical-methods-in-ann 25/42
7/21/2019 Statistical methods in ANN
http://slidepdf.com/reader/full/statistical-methods-in-ann 26/42
7/21/2019 Statistical methods in ANN
http://slidepdf.com/reader/full/statistical-methods-in-ann 27/42
7/21/2019 Statistical methods in ANN
http://slidepdf.com/reader/full/statistical-methods-in-ann 28/42
7/21/2019 Statistical methods in ANN
http://slidepdf.com/reader/full/statistical-methods-in-ann 29/42
7/21/2019 Statistical methods in ANN
http://slidepdf.com/reader/full/statistical-methods-in-ann 30/42
7/21/2019 Statistical methods in ANN
http://slidepdf.com/reader/full/statistical-methods-in-ann 31/42
7/21/2019 Statistical methods in ANN
http://slidepdf.com/reader/full/statistical-methods-in-ann 32/42
7/21/2019 Statistical methods in ANN
http://slidepdf.com/reader/full/statistical-methods-in-ann 33/42
7/21/2019 Statistical methods in ANN
http://slidepdf.com/reader/full/statistical-methods-in-ann 34/42
7/21/2019 Statistical methods in ANN
http://slidepdf.com/reader/full/statistical-methods-in-ann 35/42
7/21/2019 Statistical methods in ANN
http://slidepdf.com/reader/full/statistical-methods-in-ann 36/42
7/21/2019 Statistical methods in ANN
http://slidepdf.com/reader/full/statistical-methods-in-ann 37/42
7/21/2019 Statistical methods in ANN
http://slidepdf.com/reader/full/statistical-methods-in-ann 38/42
7/21/2019 Statistical methods in ANN
http://slidepdf.com/reader/full/statistical-methods-in-ann 39/42
7/21/2019 Statistical methods in ANN
http://slidepdf.com/reader/full/statistical-methods-in-ann 40/42
7/21/2019 Statistical methods in ANN
http://slidepdf.com/reader/full/statistical-methods-in-ann 41/42